Note: this is using v0 version of the Firecrawl API which is being deprecated. We recommend switching to v1.
Installation
To install the Firecrawl Python SDK, you can use pip:Usage
- Get an API key from firecrawl.dev
- Set the API key as an environment variable named
FIRECRAWL_API_KEY
or pass it as a parameter to theFirecrawlApp
class.
Scraping a URL
To scrape a single URL, use thescrape_url
method. It takes the URL as a parameter and returns the scraped data as a dictionary.
Extracting structured data from a URL
With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:Crawling a Website
To crawl a website, use thecrawl_url
method. It takes the starting URL and optional parameters as arguments. The params
argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
The wait_until_done
parameter determines whether the method should wait for the crawl job to complete before returning the result. If set to True
, the method will periodically check the status of the crawl job until it is completed or the specified timeout
(in seconds) is reached. If set to False
, the method will return immediately with the job ID, and you can manually check the status of the crawl job using the check_crawl_status
method.
wait_until_done
is set to True
, the crawl_url
method will return the crawl result once the job is completed. If the job fails or is stopped, an exception will be raised.
Checking Crawl Status
To check the status of a crawl job, use thecheck_crawl_status
method. It takes the job ID as a parameter and returns the current status of the crawl job.