Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.
We provide an easy to use API with our hosted version. You can find the playground and documentation here. You can also self host the backend if you’d like.
The response will be different depending on the status of the crawl. For not completed or large responses exceeding 10MB, a next URL parameter is provided. You must request this URL to retrieve the next 10MB of data. If the next parameter is absent, it indicates the end of the crawl data.
{ "status": "scraping", "total": 36, "completed": 10, "creditsUsed": 10, "expiresAt": "2024-00-00T00:00:00.000Z", "next": "https://api.firecrawl.dev/v1/crawl/123-456-789?skip=10", "data": [ { "markdown": "[Firecrawl Docs home page!...", "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...", "metadata": { "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl", "language": "en", "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3", "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.", "ogLocaleAlternate": [], "statusCode": 200 } }, ... ]}
With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:
v1 is only supported on node, python and cURL at this time.
from firecrawl import FirecrawlApp, JsonConfigfrom pydantic import BaseModel, Field# Initialize the FirecrawlApp with your API keyapp = FirecrawlApp(api_key='your_api_key')class ExtractSchema(BaseModel): company_mission: str supports_sso: bool is_open_source: bool is_in_yc: booljson_config = JsonConfig( extractionSchema=ExtractSchema.model_json_schema(), mode="llm-extraction", pageOptions={"onlyMainContent": True})llm_extraction_result = app.scrape_url( 'https://firecrawl.dev', formats=["json"], json_options=json_config)print(llm_extraction_result)
Output:
JSON
{ "success": true, "data": { "json": { "company_mission": "AI-powered web scraping and data extraction", "supports_sso": true, "is_open_source": true, "is_in_yc": true }, "metadata": { "title": "Firecrawl", "description": "AI-powered web scraping and data extraction", "robots": "follow, index", "ogTitle": "Firecrawl", "ogDescription": "AI-powered web scraping and data extraction", "ogUrl": "https://firecrawl.dev/", "ogImage": "https://firecrawl.dev/og.png", "ogLocaleAlternate": [], "ogSiteName": "Firecrawl", "sourceURL": "https://firecrawl.dev/" }, }}
Firecrawl allows you to perform various actions on a web page before scraping its content. This is particularly useful for interacting with dynamic content, navigating through pages, or accessing content that requires user interaction.
Here is an example of how to use actions to navigate to google.com, search for Firecrawl, click on the first result, and take a screenshot.
It is important to almost always use the wait action before/after executing other actions to give enough time for the page to load.
To deliver the best possible product, we offer a hosted version of Firecrawl alongside our open-source offering. The cloud solution allows us to continuously innovate and maintain a high-quality, sustainable service for all users.
Firecrawl Cloud is available at firecrawl.dev and offers a range of features that are not available in the open source version: