​

/scrape endpoint

The updated /scrape endpoint has been redesigned for enhanced reliability and ease of use. The structure of the new /scrape request body is as follows:

{
  "url": "<string>",
  "formats": ["markdown", "html", "rawHtml", "links", "screenshot", "json"],
  "includeTags": ["<string>"],
  "excludeTags": ["<string>"],
  "headers": { "<key>": "<value>" },
  "waitFor": 123,
  "timeout": 123
}

​

Formats

You can now choose what formats you want your output in. You can specify multiple output formats. Supported formats are:

• Markdown (markdown)
• HTML (html)
• Raw HTML (rawHtml) (with no modifications)
• Screenshot (screenshot or screenshot@fullPage)
• Links (links)
• JSON (json) By default, the output will be include only the markdown format.

​

Details on the new request body

The table below outlines the changes to the request body parameters for the /scrape endpoint in V1.

The new json format is described in the llm-extract section.

​

/crawl endpoint

We’ve also updated the /crawl endpoint on v1. Check out the improved body request below:

{
  "url": "<string>",
  "excludePaths": ["<string>"],
  "includePaths": ["<string>"],
  "maxDepth": 2,
  "ignoreSitemap": true,
  "limit": 10,
  "allowBackwardLinks": true,
  "allowExternalLinks": true,
  "scrapeOptions": {
    // same options as in /scrape
    "formats": ["markdown", "html", "rawHtml", "screenshot", "links"],
    "headers": { "<key>": "<value>" },
    "includeTags": ["<string>"],
    "excludeTags": ["<string>"],
    "onlyMainContent": true,
    "waitFor": 123
  }
}

​

Details on the new request body

The table below outlines the changes to the request body parameters for the /crawl endpoint in V1.