Scraping behind authentication
Learn how to bypass authentication on a website by forwading cookies.
Note: this example is using v0 version of the Firecrawl API. You can adapt the sdk call by specifying the version in the
FirecrawlApp(version="v0")
initializer. Current version of Firecrawl API is v1.
With Firecrawl, you can send cookies in headers to bypass authentication on a website. This can be useful when you need to scrape content from a website that requires you to be logged in.
Example
To bypass authentication, you need to include the headers
parameter in your request and set the Cookies
header with the appropriate cookie values that you can grab from your browser’s cookies. If you need help grabbing cookies, take a look at this article: Grabbing cookies