
Welcome to Firecrawl
Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.How to use it?
We provide an easy to use API with our hosted version. You can find the playground and documentation here. You can also self host the backend if you’d like. Check out the following resources to get started:- API: Documentation
- SDKs: Python, Node
- LLM Frameworks: Langchain (python), Langchain (js), Llama Index, Crew.ai, Composio, PraisonAI, Superinterface, Vectorize
- Low-code Frameworks: Dify, Langflow, Flowise AI, Cargo, Pipedream
- Community SDKs: Go, Rust (v1)
- Others: Zapier, Pabbly Connect
- Want an SDK or Integration? Let us know by opening an issue.
API Key
To use the API, you need to sign up on Firecrawl and get an API key.Features
- Scrape: scrapes a URL and get its content in LLM-ready format (markdown, summary, structured data via json mode, screenshot, html)
- Crawl: scrapes all the URLs of a web page and return content in LLM-ready format
- Map: input a website and get all the website urls - extremely fast
- Search: search the web and get full content from results
- Extract: get structured data from single page, multiple pages or entire websites with AI.
Powerful Capabilities
- LLM-ready formats: markdown, summary, structured data, screenshot, HTML, links, metadata
- The hard stuff: proxies, anti-bot mechanisms, dynamic content (js-rendered), output parsing, orchestration
- Lightning fast: Get results in seconds—built for speed and high-throughput use cases.
- Customizability: exclude tags, crawl behind auth walls with custom headers, max crawl depth, etc…
- Media parsing: pdfs, docx, images.
- Reliability first: designed to get the data you need - no matter how hard it is.
- Actions: click, scroll, input, wait and more before extracting data
Installing Firecrawl
Scraping
To scrape a single URL, use thescrape
method. It takes the URL as a parameter and returns the scraped data as a dictionary.
Response
SDKs will return the data object directly. cURL will return the payload exactly as shown below.Crawling
The crawl feature allows you to automatically discover and extract content from a URL and all of its accessible subpages. With our SDKs, simply call the crawl method—this will submit a crawl job, wait for it to finish, and return the complete results for the entire site.Usage
start crawl
functions on SDKs, this will return an ID
where you can use to check the status of the crawl.
Get Crawl Status
Used to check the status of a crawl job and get its result.Response
The response will be different depending on the status of the crawl. For not completed or large responses exceeding 10MB, anext
URL parameter is provided. You must request this URL to retrieve the next 10MB of data. If the next
parameter is absent, it indicates the end of the crawl data.
JSON mode
With JSON mode, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:JSON
Search
Firecrawl’s search API allows you to perform web searches and optionally scrape the search results in one operation.- Choose specific output formats (markdown, HTML, links, screenshots)
- Choose specific sources (web, news, images)
- Search the web with customizable parameters (location, etc.)
Response
SDKs will return the data object directly. cURL will return the complete payload.JSON
Extracting without schema
You can now extract without a schema by just passing aprompt
to the endpoint. The llm chooses the structure of the data.
JSON
Interacting with the page with Actions
Firecrawl allows you to perform various actions on a web page before scraping its content. This is particularly useful for interacting with dynamic content, navigating through pages, or accessing content that requires user interaction. Here is an example of how to use actions to navigate to google.com, search for Firecrawl, click on the first result, and take a screenshot. It is important to almost always use thewait
action before/after executing other actions to give enough time for the page to load.
Example
Output
Open Source vs Cloud
Firecrawl is open source available under the AGPL-3.0 license. To deliver the best possible product, we offer a hosted version of Firecrawl alongside our open-source offering. The cloud solution allows us to continuously innovate and maintain a high-quality, sustainable service for all users. Firecrawl Cloud is available at firecrawl.dev and offers a range of features that are not available in the open source version: