Crawl
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Body
The base URL to start crawling from
Specifies URL patterns to exclude from the crawl by comparing website paths against the provided regex patterns. For example, if you set "excludePaths": ["blog/*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.
Specifies URL patterns to include in the crawl by comparing website paths against the provided regex patterns. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.
Maximum depth to crawl relative to the entered URL.
Ignore the website sitemap when crawling
Maximum number of pages to crawl. Default limit is 10000.
Enables the crawler to navigate from a specific URL to previously linked pages.
Allows the crawler to follow links to external websites.
The URL to send the webhook to. This will trigger for crawl started (crawl.started) ,every page crawled (crawl.page) and when the crawl is completed (crawl.completed or crawl.failed). The response will be the same as the /scrape endpoint.