LlamaIndex | Firecrawl

Installation

pip install 'firecrawl-py>=4.3.3' llama-index llama-index-readers-web

Usage

FireCrawlWebReader supports scrape, crawl, map, search, and extract modes. Set FIRECRAWL_API_KEY in your environment. The query engine in these examples uses OpenAI by default, so also set OPENAI_API_KEY (or configure another LLM in LlamaIndex).

Using Firecrawl to Gather an Entire Website

import os

from llama_index.core import SummaryIndex
from llama_index.readers.web import FireCrawlWebReader

# Initialize FireCrawlWebReader to crawl a website
firecrawl_reader = FireCrawlWebReader(
    api_key=os.environ["FIRECRAWL_API_KEY"],  # Get a key at https://www.firecrawl.dev/app/api-keys
    mode="crawl",  # "scrape", "crawl", "map", "search", or "extract"
    params={"limit": 10},  # Optional Firecrawl parameters for the chosen mode
)

# Load documents from the site
documents = firecrawl_reader.load_data(url="https://www.paulgraham.com/")
index = SummaryIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)

Using Firecrawl to Gather a Single Page

import os

from llama_index.core import SummaryIndex
from llama_index.readers.web import FireCrawlWebReader

# Initialize the FireCrawlWebReader with your API key and desired mode
firecrawl_reader = FireCrawlWebReader(
    api_key=os.environ["FIRECRAWL_API_KEY"],  # Get a key at https://www.firecrawl.dev/app/api-keys
    mode="scrape",  # "scrape", "crawl", "map", "search", or "extract"
)

# Load documents from a specified URL
documents = firecrawl_reader.load_data(url="https://www.paulgraham.com/worked.html")
index = SummaryIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)