LlamaIndex - Firecrawl Docs

Integrate Firecrawl with LlamaIndex to build AI applications with vector search and embeddings powered by web content.

Setup

npm install llamaindex @llamaindex/openai firecrawl

Create .env file:

FIRECRAWL_API_KEY=your_firecrawl_key
OPENAI_API_KEY=your_openai_key

Note: If using Node < 20, install dotenv and add import 'dotenv/config' to your code.

RAG with Vector Search

This example demonstrates how to use LlamaIndex with Firecrawl to crawl a website, create embeddings, and query the content using RAG.

import { Firecrawl } from 'firecrawl';
import { Document, VectorStoreIndex, Settings } from 'llamaindex';
import { OpenAI, OpenAIEmbedding } from '@llamaindex/openai';

Settings.llm = new OpenAI({ model: "gpt-4o" });
Settings.embedModel = new OpenAIEmbedding({ model: "text-embedding-3-small" });

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const crawlResult = await firecrawl.crawl('https://firecrawl.dev', {
  limit: 10,
  scrapeOptions: { formats: ['markdown'] }
});
console.log(`Crawled ${crawlResult.data.length } pages`);

const documents = crawlResult.data.map((page: any, i: number) =>
  new Document({
    text: page.markdown,
    id_: `page-${i}`,
    metadata: { url: page.metadata?.sourceURL }
  })
);

const index = await VectorStoreIndex.fromDocuments(documents);
console.log('Vector index created with embeddings');

const queryEngine = index.asQueryEngine();
const response = await queryEngine.query({ query: 'What is Firecrawl and how does it work?' });

console.log('\nAnswer:', response.toString());

Python: FireCrawlWebReader

In Python, LlamaIndex’s FireCrawlWebReader loads Firecrawl results as LlamaIndex documents. Set mode to scrape, crawl, map, or search, and pass Firecrawl options in params.

pip install llama-index-readers-web firecrawl-py

import os

from llama_index.readers.web import FireCrawlWebReader

reader = FireCrawlWebReader(
    api_key=os.environ["FIRECRAWL_API_KEY"],
    mode="crawl",
    params={"limit": 10, "scrape_options": {"formats": ["markdown"]}},
)

documents = reader.load_data(url="https://firecrawl.dev")
print(f"Loaded {len(documents)} documents")

Use load_data(url=...) for scrape, crawl, and map, and load_data(query=...) for search. See the FireCrawlWebReader reference for details. For more examples, check the LlamaIndex documentation.