Crawl

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

The base URL to start crawling from

Enables the crawler to navigate from a specific URL to previously linked pages.

Allows the crawler to follow links to external websites.

Specifies URL patterns to exclude from the crawl by comparing website paths against the provided regex patterns. For example, if you set "excludePaths": ["blog/*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

Ignore the website sitemap when crawling

Specifies URL patterns to include in the crawl by comparing website paths against the provided regex patterns. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

Maximum number of pages to crawl. Default limit is 10000.

Maximum depth to crawl relative to the entered URL.

The URL to send the webhook to. This will trigger for crawl started (crawl.started) ,every page crawled (crawl.page) and when the crawl is completed (crawl.completed or crawl.failed). The response will be the same as the /scrape endpoint.