Skip to main content
Important: Only use authenticated scraping on systems where you have explicit permission from both parties (yourself and the platform owner), such as internal, self-hosted tools or resources you fully control. Do not use authentication on platforms unless you are certain it abides by the site’s Terms and Conditions and get written permission when in doubt. Using session cookies improperly can violate terms of service or laws; always confirm you are authorized to access protected content in this way.

​
Overview

The recommended approach for authenticated scraping is cookie-based authentication, where you:
  1. Login manually to your application
  2. Extract the session cookie from DevTools
  3. Use the cookie with Firecrawl to access protected pages
Cookie Expiration Times:
  • Internal tools: Often 7-30 days or longer
  • Other tools: Often hours or minutes
Internal tools typically have longer cookie lifespans, making this method ideal for recurring scraping tasks.

​
Setup

1

Get API Key

Get your Firecrawl API key from firecrawl.dev/app
2

Install Dependencies

npm
npm install @mendable/firecrawl-js
Node.js < v20: If you’re using Node.js version 19 or earlier, you’ll also need to install dotenv:
npm install dotenv
And import it with import 'dotenv/config' at the top of your file.
3

Configure Environment

Create a .env file:
.env
FIRECRAWL_API_KEY=your_firecrawl_api_key

​
Step 1: Extract Cookies from DevTools

Demo Application: You can practice with our demo app at https://firecrawl-auth.vercel.app
  • Email: test@example.com
  • Password: password123
1

Login to Your Application

Navigate to https://firecrawl-auth.vercel.app and login with the credentials above
2

Open DevTools

Press F12 or right-click → “Inspect”
3

Navigate to Application Tab

Click the Application tab (Chrome) or Storage tab (Firefox)
4

Find and Copy Cookie

  1. Expand Cookies in the sidebar
  2. Click on your domain
  3. Find the auth-token cookie
  4. Double-click the Value and copy it
DevTools Cookies View
For the demo app, the cookie looks like: 
auth-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJleGFtcGxlLXVzZXItaWQiLCJlbWFpbCI6InRlc3RAZXhhbXBsZS5jb20ifQ.example-signature-hash
Important: Cookies are sensitive credentials. Never share them publicly or commit them to version control. Treat them like passwords.

​
Step 2: Use Cookies with Firecrawl

import FirecrawlApp from "@mendable/firecrawl-js";

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

const result = await app.scrape("https://firecrawl-auth.vercel.app/dashboard", {
  formats: ["markdown", "screenshot"],
  headers: {
    Cookie: 'auth-token=COOKIE_GOES_HERE'
  },
  waitFor: 3000 // Wait 3 seconds for the page to load
});

console.log("=== Markdown ===\n" + result.markdown + "\n\n=== Screenshot URL ===\n" + result.screenshot);

​
Best Practices

Cookie Security

  • Store cookies in environment variables
  • Never commit cookies to git
  • Rotate cookies regularly
  • Use .gitignore for .env files

Cookie Expiration

  • Check expiration times in DevTools
  • Set up alerts before expiration
  • Re-extract cookies when they expire
  • Consider using form-based auth for short-lived cookies

Rate Limiting

  • Respect the application’s rate limits
  • Add delays between requests
  • Monitor for 429 (Too Many Requests) errors
  • Use exponential backoff for retries

Error Handling

  • Check for 401/403 errors (expired cookies)
  • Validate response content
  • Log authentication failures
  • Have fallback authentication methods

​
Troubleshooting

Possible causes:
  • Cookie has expired
  • Cookie was copied incorrectly
  • Application requires additional headers
  • Session was invalidated on the server
Solutions:
  • Re-extract cookies from DevTools after a fresh login
  • Check if you need multiple cookies (session + CSRF token)
  • Verify the cookie domain matches your target URL
For short-lived sessions:
  • Use form-based authentication instead
  • Automate the login process with actions
  • Set up a cron job to refresh cookies
  • Consider requesting longer session times from your internal tool’s admin
Cookie Lifespan for Internal Tools: Many internal tools set cookies with 7-30 day expiration times, making them ideal for recurring scraping tasks. Check your cookie’s Expires field in DevTools to see how long it’s valid.
⌘I