Firecrawl Docs

POST

crawl

curl --request POST \
  --url https://api.firecrawl.dev/v1/crawl \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "<string>",
  "excludePaths": [
    "<string>"
  ],
  "includePaths": [
    "<string>"
  ],
  "maxDepth": 2,
  "ignoreSitemap": false,
  "ignoreQueryParameters": false,
  "limit": 10000,
  "allowBackwardLinks": false,
  "allowExternalLinks": false,
  "webhook": {
    "url": "<string>",
    "headers": {},
    "metadata": {},
    "events": [
      "completed"
    ]
  },
  "scrapeOptions": {
    "formats": [
      "markdown"
    ],
    "onlyMainContent": true,
    "includeTags": [
      "<string>"
    ],
    "excludeTags": [
      "<string>"
    ],
    "headers": {},
    "waitFor": 0,
    "mobile": false,
    "skipTlsVerification": false,
    "timeout": 30000,
    "jsonOptions": {
      "schema": {},
      "systemPrompt": "<string>",
      "prompt": "<string>"
    },
    "actions": [
      {
        "type": "wait",
        "milliseconds": 2,
        "selector": "#my-element"
      }
    ],
    "location": {
      "country": "US",
      "languages": [
        "en-US"
      ]
    },
    "removeBase64Images": true,
    "blockAds": true,
    "proxy": "basic"
  }
}'

{
  "success": true,
  "id": "<string>",
  "url": "<string>"
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

url

string

required

The base URL to start crawling from

excludePaths

string[]

URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

includePaths

string[]

URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

maxDepth

integer

default:

Maximum depth to crawl relative to the entered URL.

ignoreSitemap

boolean

default:

false

Ignore the website sitemap when crawling

ignoreQueryParameters

boolean

default:

false

Do not re-scrape the same path with different (or none) query parameters

limit

integer

default:

10000

Maximum number of pages to crawl. Default limit is 10000.

allowBackwardLinks

boolean

default:

false

Enables the crawler to navigate from a specific URL to previously linked pages.

allowExternalLinks

boolean

default:

false

Allows the crawler to follow links to external websites.

webhook

object

A webhook specification object.

scrapeOptions

object

scrapeOptions.formats

enum<string>[]

Formats to include in the output.

Available options:

markdown,

html,

rawHtml,

links,

screenshot,

screenshot@fullPage,

json

scrapeOptions.onlyMainContent

boolean

default:

true

Only return the main content of the page excluding headers, navs, footers, etc.

scrapeOptions.includeTags

string[]

Tags to include in the output.

scrapeOptions.excludeTags

string[]

Tags to exclude from the output.

scrapeOptions.headers

object

Headers to send with the request. Can be used to send cookies, user-agent, etc.

scrapeOptions.waitFor

integer

default:

Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load.

scrapeOptions.mobile

boolean

default:

false

Set to true if you want to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots.

scrapeOptions.skipTlsVerification

boolean

default:

false

Skip TLS certificate verification when making requests

scrapeOptions.timeout

integer

default:

30000

Timeout in milliseconds for the request

scrapeOptions.jsonOptions

object

Extract object

scrapeOptions.actions

object[]

Actions to perform on the page before grabbing the content

scrapeOptions.location

object

Location settings for the request. When specified, this will use an appropriate proxy if available and emulate the corresponding language and timezone settings. Defaults to 'US' if not specified.

scrapeOptions.removeBase64Images

boolean

Removes all base 64 images from the output, which may be overwhelmingly long. The image's alt text remains in the output, but the URL is replaced with a placeholder.

scrapeOptions.blockAds

boolean

default:

true

Enables ad-blocking and cookie popup blocking.

scrapeOptions.proxy

enum<string>

Specifies the type of proxy to use.

basic: Proxies for scraping sites with none to basic anti-bot solutions. Fast and usually works.
stealth: Stealth proxies for scraping sites with advanced anti-bot solutions. Slower, but more reliable on certain sites.

If you do not specify a proxy, Firecrawl will automatically attempt to determine which one you need based on the target site.

Available options:

basic,

stealth

Response

200

application/json

Successful response

success

boolean

string

url

string

Suggest edits Raise issue

Get Batch Scrape Errors Get Crawl Status

curl --request POST \
  --url https://api.firecrawl.dev/v1/crawl \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "<string>",
  "excludePaths": [
    "<string>"
  ],
  "includePaths": [
    "<string>"
  ],
  "maxDepth": 2,
  "ignoreSitemap": false,
  "ignoreQueryParameters": false,
  "limit": 10000,
  "allowBackwardLinks": false,
  "allowExternalLinks": false,
  "webhook": {
    "url": "<string>",
    "headers": {},
    "metadata": {},
    "events": [
      "completed"
    ]
  },
  "scrapeOptions": {
    "formats": [
      "markdown"
    ],
    "onlyMainContent": true,
    "includeTags": [
      "<string>"
    ],
    "excludeTags": [
      "<string>"
    ],
    "headers": {},
    "waitFor": 0,
    "mobile": false,
    "skipTlsVerification": false,
    "timeout": 30000,
    "jsonOptions": {
      "schema": {},
      "systemPrompt": "<string>",
      "prompt": "<string>"
    },
    "actions": [
      {
        "type": "wait",
        "milliseconds": 2,
        "selector": "#my-element"
      }
    ],
    "location": {
      "country": "US",
      "languages": [
        "en-US"
      ]
    },
    "removeBase64Images": true,
    "blockAds": true,
    "proxy": "basic"
  }
}'

{
  "success": true,
  "id": "<string>",
  "url": "<string>"
}

Using the API

Scrape Endpoints

Crawl Endpoints

Map Endpoints

Extract Endpoints

Search Endpoints

Account Endpoints

Crawl

Authorizations

Body

Response