Quickstart | Firecrawl

Welcome to Firecrawl

Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.

How to use it?

We provide an easy to use API with our hosted version. You can find the playground and documentation here. You can also self host the backend if you’d like.

Check out the following resources to get started:

API: Documentation
SDKs: Python, Node, Go, Rust
LLM Frameworks: Langchain (python), Langchain (js), Llama Index, Crew.ai, Composio, PraisonAI, Superinterface, Vectorize
Low-code Frameworks: Dify, Langflow, Flowise AI, Cargo, Pipedream
Others: Zapier, Pabbly Connect
Want an SDK or Integration? Let us know by opening an issue.

Self-host: To self-host refer to guide here.

API Key

To use the API, you need to sign up on Firecrawl and get an API key.

Features

Scrape: scrapes a URL and get its content in LLM-ready format (markdown, structured data via LLM Extract, screenshot, html)
Crawl: scrapes all the URLs of a web page and return content in LLM-ready format
Map: input a website and get all the website urls - extremely fast
Search: search the web and get full content from results
Extract: get structured data from single page, multiple pages or entire websites with AI.

Powerful Capabilities

LLM-ready formats: markdown, structured data, screenshot, HTML, links, metadata
The hard stuff: proxies, anti-bot mechanisms, dynamic content (js-rendered), output parsing, orchestration
Customizability: exclude tags, crawl behind auth walls with custom headers, max crawl depth, etc…
Media parsing: pdfs, docx, images.
Reliability first: designed to get the data you need - no matter how hard it is.
Actions: click, scroll, input, wait and more before extracting data

You can find all of Firecrawl’s capabilities and how to use them in our documentation

Installing Firecrawl

pip install firecrawl-py

Scraping

To scrape a single URL, use the scrape_url method. It takes the URL as a parameter and returns the scraped data as a dictionary.

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a website:
scrape_result = app.scrape_url('firecrawl.dev', formats=['markdown', 'html'])
print(scrape_result)

Response

SDKs will return the data object directly. cURL will return the payload exactly as shown below.

{
  "success": true,
  "data" : {
    "markdown": "Launch Week I is here! [See our Day 2 Release 🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥 Get 2 months free...",
    "html": "<!DOCTYPE html><html lang=\"en\" class=\"light\" style=\"color-scheme: light;\"><body class=\"__variable_36bd41 __variable_d7dc5d font-inter ...",
    "metadata": {
      "title": "Home - Firecrawl",
      "description": "Firecrawl crawls and converts any website into clean markdown.",
      "language": "en",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "Turn any website into LLM-ready data.",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "https://firecrawl.dev",
      "statusCode": 200
    }
  }
}

Crawling

Used to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.

Usage

from firecrawl import FirecrawlApp, ScrapeOptions

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Crawl a website:
crawl_result = app.crawl_url(
  'https://firecrawl.dev', 
  limit=10, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
)
print(crawl_result)

If you’re using cURL or async crawl functions on SDKs, this will return an ID where you can use to check the status of the crawl.

{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v1/crawl/123-456-789"
}

Check Crawl Job

Used to check the status of a crawl job and get its result.

crawl_status = app.check_crawl_status("<crawl_id>")
print(crawl_status)

Response

The response will be different depending on the status of the crawl. For not completed or large responses exceeding 10MB, a next URL parameter is provided. You must request this URL to retrieve the next 10MB of data. If the next parameter is absent, it indicates the end of the crawl data.

{
  "status": "scraping",
  "total": 36,
  "completed": 10,
  "creditsUsed": 10,
  "expiresAt": "2024-00-00T00:00:00.000Z",
  "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=10",
  "data": [
    {
      "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
      "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
      "metadata": {
        "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl",
        "language": "en",
        "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
        "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.",
        "ogLocaleAlternate": [],
        "statusCode": 200
      }
    },
    ...
  ]
}

Extraction

With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:

v1 is only supported on node, python and cURL at this time.

from firecrawl import JsonConfig, FirecrawlApp
from pydantic import BaseModel
app = FirecrawlApp(api_key="<YOUR_API_KEY>")

class ExtractSchema(BaseModel):
    company_mission: str
    supports_sso: bool
    is_open_source: bool
    is_in_yc: bool

json_config = JsonConfig(
    schema=ExtractSchema
)

llm_extraction_result = app.scrape_url(
    'https://firecrawl.dev',
    formats=["json"],
    json_options=json_config,
    only_main_content=False,
    timeout=120000
)

print(llm_extraction_result.json)

Output:

JSON

{
    "success": true,
    "data": {
      "json": {
        "company_mission": "AI-powered web scraping and data extraction",
        "supports_sso": true,
        "is_open_source": true,
        "is_in_yc": true
      },
      "metadata": {
        "title": "Firecrawl",
        "description": "AI-powered web scraping and data extraction",
        "robots": "follow, index",
        "ogTitle": "Firecrawl",
        "ogDescription": "AI-powered web scraping and data extraction",
        "ogUrl": "https://firecrawl.dev/",
        "ogImage": "https://firecrawl.dev/og.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Firecrawl",
        "sourceURL": "https://firecrawl.dev/"
      },
    }
}

Extracting without schema (New)

You can now extract without a schema by just passing a prompt to the endpoint. The llm chooses the structure of the data.

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "Extract the company mission from the page."
      }
    }'

Output:

JSON

{
    "success": true,
    "data": {
      "json": {
        "company_mission": "AI-powered web scraping and data extraction",
      },
      "metadata": {
        "title": "Firecrawl",
        "description": "AI-powered web scraping and data extraction",
        "robots": "follow, index",
        "ogTitle": "Firecrawl",
        "ogDescription": "AI-powered web scraping and data extraction",
        "ogUrl": "https://firecrawl.dev/",
        "ogImage": "https://firecrawl.dev/og.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Firecrawl",
        "sourceURL": "https://firecrawl.dev/"
      },
    }
}

Interacting with the page with Actions

Firecrawl allows you to perform various actions on a web page before scraping its content. This is particularly useful for interacting with dynamic content, navigating through pages, or accessing content that requires user interaction.

Here is an example of how to use actions to navigate to google.com, search for Firecrawl, click on the first result, and take a screenshot.

It is important to almost always use the wait action before/after executing other actions to give enough time for the page to load.

Example

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a website:
scrape_result = app.scrape_url('firecrawl.dev', 
    formats=['markdown', 'html'], 
    actions=[
        {"type": "wait", "milliseconds": 2000},
        {"type": "click", "selector": "textarea[title=\"Search\"]"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "write", "text": "firecrawl"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "press", "key": "ENTER"},
        {"type": "wait", "milliseconds": 3000},
        {"type": "click", "selector": "h3"},
        {"type": "wait", "milliseconds": 3000},
        {"type": "scrape"},
        {"type": "screenshot"}
    ]
)
print(scrape_result)

Output

{
  "success": true,
  "data": {
    "markdown": "Our first Launch Week is over! [See the recap 🚀](blog/firecrawl-launch-week-1-recap)...",
    "actions": {
      "screenshots": [
        "https://alttmdsdujxrfnakrkyi.supabase.co/storage/v1/object/public/media/screenshot-75ef2d87-31e0-4349-a478-fb432a29e241.png"
      ],
      "scrapes": [
        {
          "url": "https://www.firecrawl.dev/",
          "html": "<html><body><h1>Firecrawl</h1></body></html>"
        }
      ]
    },
    "metadata": {
      "title": "Home - Firecrawl",
      "description": "Firecrawl crawls and converts any website into clean markdown.",
      "language": "en",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "Turn any website into LLM-ready data.",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "http://google.com",
      "statusCode": 200
    }
  }
}

Open Source vs Cloud

Firecrawl is open source available under the AGPL-3.0 license.

To deliver the best possible product, we offer a hosted version of Firecrawl alongside our open-source offering. The cloud solution allows us to continuously innovate and maintain a high-quality, sustainable service for all users.

Firecrawl Cloud is available at firecrawl.dev and offers a range of features that are not available in the open source version:

Contributing

We love contributions! Please read our contributing guide before submitting a pull request.

On this page

Welcome to Firecrawl
How to use it?
API Key
Features
Powerful Capabilities
Installing Firecrawl
Scraping
Response
Crawling
Usage
Check Crawl Job
Response
Extraction
Extracting without schema (New)
Interacting with the page with Actions
Example
Output
Open Source vs Cloud
Contributing

Welcome to Firecrawl

Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.

How to use it?

We provide an easy to use API with our hosted version. You can find the playground and documentation here. You can also self host the backend if you’d like.

Check out the following resources to get started:

API: Documentation
SDKs: Python, Node, Go, Rust
LLM Frameworks: Langchain (python), Langchain (js), Llama Index, Crew.ai, Composio, PraisonAI, Superinterface, Vectorize
Low-code Frameworks: Dify, Langflow, Flowise AI, Cargo, Pipedream
Others: Zapier, Pabbly Connect
Want an SDK or Integration? Let us know by opening an issue.

Self-host: To self-host refer to guide here.

API Key

To use the API, you need to sign up on Firecrawl and get an API key.

Features

Scrape: scrapes a URL and get its content in LLM-ready format (markdown, structured data via LLM Extract, screenshot, html)
Crawl: scrapes all the URLs of a web page and return content in LLM-ready format
Map: input a website and get all the website urls - extremely fast
Search: search the web and get full content from results
Extract: get structured data from single page, multiple pages or entire websites with AI.

Powerful Capabilities

LLM-ready formats: markdown, structured data, screenshot, HTML, links, metadata
The hard stuff: proxies, anti-bot mechanisms, dynamic content (js-rendered), output parsing, orchestration
Customizability: exclude tags, crawl behind auth walls with custom headers, max crawl depth, etc…
Media parsing: pdfs, docx, images.
Reliability first: designed to get the data you need - no matter how hard it is.
Actions: click, scroll, input, wait and more before extracting data

You can find all of Firecrawl’s capabilities and how to use them in our documentation

Installing Firecrawl

pip install firecrawl-py

Scraping

To scrape a single URL, use the scrape_url method. It takes the URL as a parameter and returns the scraped data as a dictionary.

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a website:
scrape_result = app.scrape_url('firecrawl.dev', formats=['markdown', 'html'])
print(scrape_result)

Response

SDKs will return the data object directly. cURL will return the payload exactly as shown below.

{
  "success": true,
  "data" : {
    "markdown": "Launch Week I is here! [See our Day 2 Release 🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥 Get 2 months free...",
    "html": "<!DOCTYPE html><html lang=\"en\" class=\"light\" style=\"color-scheme: light;\"><body class=\"__variable_36bd41 __variable_d7dc5d font-inter ...",
    "metadata": {
      "title": "Home - Firecrawl",
      "description": "Firecrawl crawls and converts any website into clean markdown.",
      "language": "en",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "Turn any website into LLM-ready data.",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "https://firecrawl.dev",
      "statusCode": 200
    }
  }
}

Crawling

Used to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.

Usage

from firecrawl import FirecrawlApp, ScrapeOptions

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Crawl a website:
crawl_result = app.crawl_url(
  'https://firecrawl.dev', 
  limit=10, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
)
print(crawl_result)

If you’re using cURL or async crawl functions on SDKs, this will return an ID where you can use to check the status of the crawl.

{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v1/crawl/123-456-789"
}

Check Crawl Job

Used to check the status of a crawl job and get its result.

crawl_status = app.check_crawl_status("<crawl_id>")
print(crawl_status)

Response

{
  "status": "scraping",
  "total": 36,
  "completed": 10,
  "creditsUsed": 10,
  "expiresAt": "2024-00-00T00:00:00.000Z",
  "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=10",
  "data": [
    {
      "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
      "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
      "metadata": {
        "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl",
        "language": "en",
        "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
        "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.",
        "ogLocaleAlternate": [],
        "statusCode": 200
      }
    },
    ...
  ]
}

Extraction

With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:

v1 is only supported on node, python and cURL at this time.

from firecrawl import JsonConfig, FirecrawlApp
from pydantic import BaseModel
app = FirecrawlApp(api_key="<YOUR_API_KEY>")

class ExtractSchema(BaseModel):
    company_mission: str
    supports_sso: bool
    is_open_source: bool
    is_in_yc: bool

json_config = JsonConfig(
    schema=ExtractSchema
)

llm_extraction_result = app.scrape_url(
    'https://firecrawl.dev',
    formats=["json"],
    json_options=json_config,
    only_main_content=False,
    timeout=120000
)

print(llm_extraction_result.json)

Output:

JSON

{
    "success": true,
    "data": {
      "json": {
        "company_mission": "AI-powered web scraping and data extraction",
        "supports_sso": true,
        "is_open_source": true,
        "is_in_yc": true
      },
      "metadata": {
        "title": "Firecrawl",
        "description": "AI-powered web scraping and data extraction",
        "robots": "follow, index",
        "ogTitle": "Firecrawl",
        "ogDescription": "AI-powered web scraping and data extraction",
        "ogUrl": "https://firecrawl.dev/",
        "ogImage": "https://firecrawl.dev/og.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Firecrawl",
        "sourceURL": "https://firecrawl.dev/"
      },
    }
}

Extracting without schema (New)

You can now extract without a schema by just passing a prompt to the endpoint. The llm chooses the structure of the data.

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "Extract the company mission from the page."
      }
    }'

Output:

JSON

{
    "success": true,
    "data": {
      "json": {
        "company_mission": "AI-powered web scraping and data extraction",
      },
      "metadata": {
        "title": "Firecrawl",
        "description": "AI-powered web scraping and data extraction",
        "robots": "follow, index",
        "ogTitle": "Firecrawl",
        "ogDescription": "AI-powered web scraping and data extraction",
        "ogUrl": "https://firecrawl.dev/",
        "ogImage": "https://firecrawl.dev/og.png",
        "ogLocaleAlternate": [],
        "ogSiteName": "Firecrawl",
        "sourceURL": "https://firecrawl.dev/"
      },
    }
}

Interacting with the page with Actions

Here is an example of how to use actions to navigate to google.com, search for Firecrawl, click on the first result, and take a screenshot.

It is important to almost always use the wait action before/after executing other actions to give enough time for the page to load.

Example

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a website:
scrape_result = app.scrape_url('firecrawl.dev', 
    formats=['markdown', 'html'], 
    actions=[
        {"type": "wait", "milliseconds": 2000},
        {"type": "click", "selector": "textarea[title=\"Search\"]"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "write", "text": "firecrawl"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "press", "key": "ENTER"},
        {"type": "wait", "milliseconds": 3000},
        {"type": "click", "selector": "h3"},
        {"type": "wait", "milliseconds": 3000},
        {"type": "scrape"},
        {"type": "screenshot"}
    ]
)
print(scrape_result)

Output

{
  "success": true,
  "data": {
    "markdown": "Our first Launch Week is over! [See the recap 🚀](blog/firecrawl-launch-week-1-recap)...",
    "actions": {
      "screenshots": [
        "https://alttmdsdujxrfnakrkyi.supabase.co/storage/v1/object/public/media/screenshot-75ef2d87-31e0-4349-a478-fb432a29e241.png"
      ],
      "scrapes": [
        {
          "url": "https://www.firecrawl.dev/",
          "html": "<html><body><h1>Firecrawl</h1></body></html>"
        }
      ]
    },
    "metadata": {
      "title": "Home - Firecrawl",
      "description": "Firecrawl crawls and converts any website into clean markdown.",
      "language": "en",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "Turn any website into LLM-ready data.",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "http://google.com",
      "statusCode": 200
    }
  }
}

Open Source vs Cloud

Firecrawl is open source available under the AGPL-3.0 license.

Firecrawl Cloud is available at firecrawl.dev and offers a range of features that are not available in the open source version:

Contributing

We love contributions! Please read our contributing guide before submitting a pull request.

On this page

Welcome to Firecrawl
How to use it?
API Key
Features
Powerful Capabilities
Installing Firecrawl
Scraping
Response
Crawling
Usage
Check Crawl Job
Response
Extraction
Extracting without schema (New)
Interacting with the page with Actions
Example
Output
Open Source vs Cloud
Contributing

​Welcome to Firecrawl

​How to use it?

​API Key

​Features

​Powerful Capabilities

​Installing Firecrawl

​Scraping

​Response

​Crawling

​Usage

​Check Crawl Job

​Response

​Extraction

​Extracting without schema (New)

​Interacting with the page with Actions

​Example

​Output

​Open Source vs Cloud

​Contributing

Using the API

Scrape Endpoints

Crawl Endpoints

Map Endpoints

Search Endpoints

Extract Endpoints

Account Endpoints

​Welcome to Firecrawl

​How to use it?

​API Key

​Features

​Powerful Capabilities

​Installing Firecrawl

​Scraping

​Response

​Crawling

​Usage

​Check Crawl Job

​Response

​Extraction

​Extracting without schema (New)

​Interacting with the page with Actions

​Example

​Output

​Open Source vs Cloud

​Contributing

Welcome to Firecrawl

How to use it?

API Key

Features

Powerful Capabilities

Installing Firecrawl

Scraping

Response

Crawling

Usage

Check Crawl Job

Response

Extraction

Extracting without schema (New)

Interacting with the page with Actions

Example

Output

Open Source vs Cloud

Contributing

Welcome to Firecrawl

How to use it?

API Key

Features

Powerful Capabilities

Installing Firecrawl

Scraping

Response

Crawling

Usage

Check Crawl Job

Response

Extraction

Extracting without schema (New)

Interacting with the page with Actions

Example

Output

Open Source vs Cloud

Contributing