Building an AI Research Assistant with Firecrawl and AI SDK

​

What You’ll Build

​

Prerequisites

• Node.js 18 or later installed
• An OpenAI API key from platform.openai.com
• A Firecrawl API key from firecrawl.dev
• Basic knowledge of React and Next.js

1

Create a New Next.js Project

Start by creating a fresh Next.js application and navigating into the project directory:

Copy

Ask AI

npx create-next-app@latest ai-sdk-firecrawl && cd ai-sdk-firecrawl

When prompted, select the following options:

• TypeScript: Yes
• ESLint: Yes
• Tailwind CSS: Yes
• App Router: Yes
• Use src/ directory: No
• Import alias: Yes (@/*)

2

Install Dependencies

​

Install AI SDK Packages

The AI SDK is a TypeScript toolkit that provides a unified API for working with different LLM providers:

Copy

Ask AI

npm i ai @ai-sdk/react zod

These packages provide:

• ai: Core SDK with streaming, tool calling, and response handling
• @ai-sdk/react: React hooks like useChat for building chat interfaces
• zod: Schema validation for tool inputs

Learn more at ai-sdk.dev/docs.

​

Install AI Elements

AI Elements provides pre-built UI components for AI applications. Run the following command to scaffold all the necessary components:

Copy

Ask AI

npx ai-elements@latest

This sets up AI Elements in your project, including conversation components, message displays, prompt inputs, and tool call visualizations.Documentation: ai-sdk.dev/elements/overview.

​

Install OpenAI Provider

Install the OpenAI provider to connect with OpenAI’s models:

Copy

Ask AI

npm install @ai-sdk/openai

3

Build the Frontend Chat Interface

Create the main page at app/page.tsx and copy the code from the Code tab below. This will be the chat interface where users interact with the AI assistant.

• Preview
• Code

​

Understanding the Frontend

The frontend uses AI Elements components to provide a complete chat interface:Key Features:

• Conversation Display: The Conversation component automatically handles message scrolling and display
• Message Rendering: Each message part is rendered based on its type (text, reasoning, tool calls)
• Tool Visualization: Tool calls are displayed with collapsible sections showing inputs and outputs
• Interactive Controls: Users can toggle web search, select models, and attach files
• Message Actions: Copy and retry actions for assistant messages

4

Add Markdown Rendering Support

To ensure the markdown from the LLM is correctly rendered, add the following import to your app/globals.css file:

Copy

Ask AI

@source "../node_modules/streamdown/dist/index.js";

This imports the necessary styles for rendering markdown content in the message responses.

5

Build the Basic API Route

Create the chat API endpoint at app/api/chat/route.ts. This route will handle incoming messages and stream responses from the AI.

Copy

Ask AI

import { streamText, UIMessage, convertToModelMessages } from "ai";
import { createOpenAI } from "@ai-sdk/openai";

const openai = createOpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
});

// Allow streaming responses up to 5 minutes
export const maxDuration = 300;

export async function POST(req: Request) {
  const {
    messages,
    model,
    webSearch,
  }: {
    messages: UIMessage[];
    model: string;
    webSearch: boolean;
  } = await req.json();

  const result = streamText({
    model: openai(model),
    messages: convertToModelMessages(messages),
    system:
      "You are a helpful assistant that can answer questions and help with tasks.",
  });

  // send sources and reasoning back to the client
  return result.toUIMessageStreamResponse({
    sendSources: true,
    sendReasoning: true,
  });
}

This basic route:

• Receives messages from the frontend
• Uses the OpenAI model selected by the user
• Streams responses back to the client
• Doesn’t include tools yet - we’ll add those next

6

Configure Environment Variables

Create a .env.local file in your project root:

Copy

Ask AI

touch .env.local

Add your OpenAI API key:

Copy

Ask AI

OPENAI_API_KEY=sk-your-openai-api-key

The OPENAI_API_KEY is required for the AI model to function.

7

Test the Basic Chat

Now you can test the AI SDK chatbot without Firecrawl integration. Start the development server:

Copy

Ask AI

npm run dev

Open localhost:3000 in your browser and test the basic chat functionality. The assistant should respond to messages, but won’t have web scraping or search capabilities yet.

8

Add Firecrawl Tools

Now let’s enhance the assistant with web scraping and search capabilities using Firecrawl.

​

Install Firecrawl SDK

Firecrawl converts websites into LLM-ready formats with scraping and search capabilities:

Copy

Ask AI

npm i @mendable/firecrawl-js

​

Create the Tools File

Create a lib folder and add a tools.ts file inside it:

Copy

Ask AI

mkdir lib && touch lib/tools.ts

Add the following code to define the web scraping and search tools:

lib/tools.ts

Copy

Ask AI

import FirecrawlApp from "@mendable/firecrawl-js";
import { tool } from "ai";
import { z } from "zod";

const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });

export const scrapeWebsiteTool = tool({
  description: 'Scrape content from any website URL',
  inputSchema: z.object({
    url: z.string().url().describe('The URL to scrape')
  }),
  execute: async ({ url }) => {
    console.log('Scraping:', url);
    const result = await firecrawl.scrape(url, {
      formats: ['markdown'],
      onlyMainContent: true,
      timeout: 30000
    });
    console.log('Scraped content preview:', result.markdown?.slice(0, 200) + '...');
    return { content: result.markdown };
  }
});

export const searchWebTool = tool({
  description: 'Search the web using Firecrawl',
  inputSchema: z.object({
    query: z.string().describe('The search query'),
    limit: z.number().optional().describe('Number of results'),
    location: z.string().optional().describe('Location for localized results'),
    tbs: z.string().optional().describe('Time filter (qdr:h, qdr:d, qdr:w, qdr:m, qdr:y)'),
    sources: z.array(z.enum(['web', 'news', 'images'])).optional().describe('Result types'),
    categories: z.array(z.enum(['github', 'research', 'pdf'])).optional().describe('Filter categories'),
  }),
  execute: async ({ query, limit, location, tbs, sources, categories }) => {
    console.log('Searching:', query);
    const response = await firecrawl.search(query, {
      ...(limit && { limit }),
      ...(location && { location }),
      ...(tbs && { tbs }),
      ...(sources && { sources }),
      ...(categories && { categories }),
    }) as { web?: Array<{ title?: string; url?: string; description?: string }> };

    const results = (response.web || []).map((item) => ({
      title: item.title || item.url || 'Untitled',
      url: item.url || '',
      description: item.description || '',
    }));

    console.log('Search results:', results.length);
    return { results };
  },
});

​

Understanding the Tools

Scrape Website Tool:

• Accepts a URL as input (validated by Zod schema)
• Uses Firecrawl’s scrape method to fetch the page as markdown
• Extracts only the main content to reduce token usage
• Returns the scraped content for the AI to analyze

Search Web Tool:

• Accepts a search query with optional filters
• Uses Firecrawl’s search method to find relevant web pages
• Supports advanced filters like location, time range, and content categories
• Returns structured results with titles, URLs, and descriptions

Learn more about tools: ai-sdk.dev/docs/foundations/tools.

9

Update the API Route with Firecrawl Tools

Now update your app/api/chat/route.ts to include the Firecrawl tools we just created.

View complete app/api/chat/route.ts code

Copy

Ask AI

import { streamText, UIMessage, stepCountIs, convertToModelMessages } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { scrapeWebsiteTool, searchWebTool } from "@/lib/tools";

const openai = createOpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
});

export const maxDuration = 300;

export async function POST(req: Request) {
  const {
    messages,
    model,
    webSearch,
  }: {
    messages: UIMessage[];
    model: string;
    webSearch: boolean;
  } = await req.json();

  const result = streamText({
    model: openai(model),
    messages: convertToModelMessages(messages),
    system:
      "You are a helpful assistant that can answer questions and help with tasks.",
    // Add the Firecrawl tools here
    tools: {
      scrapeWebsite: scrapeWebsiteTool,
      searchWeb: searchWebTool,
    },
    stopWhen: stepCountIs(5),
    toolChoice: webSearch ? "auto" : "none",
  });

  return result.toUIMessageStreamResponse({
    sendSources: true,
    sendReasoning: true,
  });
}

The key changes from the basic route:

• Import stepCountIs from the AI SDK
• Import the Firecrawl tools from @/lib/tools
• Add the tools object with both scrapeWebsite and searchWeb tools
• Add stopWhen: stepCountIs(5) to limit execution steps
• Set toolChoice to “auto” when web search is enabled, “none” otherwise

Learn more about streamText: ai-sdk.dev/docs/reference/ai-sdk-core/stream-text.

10

Add Your Firecrawl API Key

Update your .env.local file to include your Firecrawl API key:

Copy

Ask AI

OPENAI_API_KEY=sk-your-openai-api-key
FIRECRAWL_API_KEY=fc-your-firecrawl-api-key

Get your Firecrawl API key from firecrawl.dev.

11

Test the Complete Application

Restart your development server:

Copy

Ask AI

npm run dev

Open localhost:3000 and test the enhanced assistant:

1. Toggle the “Search” button to enable web search
2. Ask: “What are the latest features from firecrawl.dev?”
3. Watch as the AI calls the searchWeb or scrapeWebsite tool
4. See the tool execution in the UI with inputs and outputs
5. Read the AI’s analysis based on the scraped data

​

How It Works

​

Message Flow

1. User sends a message: The user types a question and clicks submit
2. Frontend sends request: useChat sends the message to /api/chat with the selected model and web search setting
3. Backend processes message: The API route receives the message and calls streamText
4. AI decides on tools: The model analyzes the question and decides whether to use scrapeWebsite or searchWeb (only if web search is enabled)
5. Tools execute: If tools are called, Firecrawl scrapes or searches the web
6. AI generates response: The model analyzes tool results and generates a natural language response
7. Frontend displays results: The UI shows tool calls and the final response in real-time

​

Tool Calling Process

1. The model receives the user’s message and available tool descriptions
2. If the model determines a tool is needed, it generates a tool call with parameters
3. The SDK executes the tool function with those parameters
4. The tool result is sent back to the model
5. The model uses the result to generate its final response

​

Key Features

​

Model Selection

• GPT-5 Mini (Thinking): Recent OpenAI model with advanced reasoning capabilities
• GPT-4o Mini: Fast and cost-effective model

​

Web Search Toggle

• Enabled: AI can call scrapeWebsite and searchWeb tools as needed
• Disabled: AI responds only with its training knowledge

​

Customization Ideas

​

Add More Tools

• Database lookups for internal company data
• CRM integration to fetch customer information
• Email sending capabilities
• Document generation

​

Change the AI Model

import { anthropic } from "@ai-sdk/anthropic";

const result = streamText({
  model: anthropic("claude-4.5-sonnet"),
  // ... rest of config
});

​

Customize the UI

• Modify component styles in the component files
• Add new variants to existing components
• Create custom components that match the design system

​

Best Practices

1. Use appropriate tools: Choose searchWeb to find relevant pages first, scrapeWebsite for single pages, or let the AI decide
2. Monitor API usage: Track your Firecrawl and OpenAI API usage to avoid unexpected costs
3. Handle errors gracefully: The tools include error handling, but consider adding user-facing error messages
4. Optimize performance: Use streaming to provide immediate feedback and consider caching frequently accessed content
5. Set reasonable limits: The stopWhen: stepCountIs(5) prevents excessive tool calls and runaway costs

​

Related Resources