---
name: firecrawl
description: Use when building AI agents that need to search the web, scrape
  pages, extract structured data, interact with dynamic content, crawl entire
  websites, or monitor pages for changes. Agents should reach for Firecrawl when
  they need clean, LLM-ready web data in markdown or JSON format.
metadata:
  mintlify-proj: firecrawl
  version: "1.0"
---

# Firecrawl Skill

## Product Summary

Firecrawl is the context API for AI agents. It provides a unified interface to search the web, scrape any page, extract structured data, interact with dynamic content, crawl entire websites, and monitor pages for changes. All endpoints return clean Markdown or structured JSON optimized for LLM consumption.

**Key files and commands:**
- Base URL: `https://api.firecrawl.dev/v2`
- Authentication: `Authorization: Bearer fc-YOUR-API-KEY` (or keyless for `/scrape`, `/search`, `/parse`)
- SDKs: Python (`firecrawl`), Node.js (`firecrawl`), CLI (`firecrawl-cli`), Go, Ruby, Rust, PHP, Elixir, Java, .NET
- Primary docs: https://docs.firecrawl.dev

**Core endpoints:**
- `POST /scrape` — Extract content from a single URL
- `POST /batch/scrape` — Scrape multiple URLs in parallel
- `POST /crawl` — Recursively crawl an entire website
- `POST /search` — Search the web and scrape results
- `POST /interact` — Click, fill forms, navigate pages
- `POST /monitor` — Schedule recurring checks and detect changes
- `POST /agent` — Autonomous web data gathering with AI
- `POST /parse` — Parse local files (PDFs, DOCX, HTML, etc.)

## When to Use

Reach for Firecrawl when:

- **Scraping a single page**: Extract markdown, HTML, JSON, screenshots, or structured data from a URL
- **Scraping multiple URLs**: Use batch scrape to process a list of URLs concurrently
- **Crawling entire sites**: Recursively discover and scrape all pages on a domain
- **Searching and scraping**: Search the web and get full page content from results
- **Extracting structured data**: Use JSON mode with a schema or prompt to extract specific fields
- **Interacting with dynamic content**: Click buttons, fill forms, navigate pages, then scrape
- **Monitoring for changes**: Schedule recurring checks on pages or entire sites and get alerts
- **Autonomous research**: Use the Agent endpoint when you don't know URLs upfront and need deep web navigation
- **Parsing local files**: Convert PDFs, DOCX, images, HTML to markdown or structured data
- **Handling JavaScript-heavy sites**: Firecrawl renders JavaScript and handles anti-bot measures automatically

## Quick Reference

### Essential Scrape Parameters

| Parameter | Type | Default | Purpose |
|-----------|------|---------|---------|
| `url` | string | required | URL to scrape |
| `formats` | array | `["markdown"]` | Output formats: `markdown`, `html`, `json`, `screenshot`, `links`, `images`, `product`, `branding`, `audio`, `video`, `question`, `highlights` |
| `maxAge` | integer | 172800000 | Cache freshness in ms (2 days); set to 0 for fresh content |
| `proxy` | string | `"auto"` | Proxy type: `basic`, `enhanced`, or `auto` |
| `mobile` | boolean | false | Emulate mobile device |
| `waitFor` | integer | 0 | Delay in ms before fetching (for JS rendering) |
| `headers` | object | — | Custom headers (cookies, user-agent, etc.) |
| `location` | object | `{country: "US"}` | ISO country code and preferred languages |

### JSON Extraction (Structured Data)

```python
# With schema
result = firecrawl.scrape(
    url="https://example.com",
    formats=[{
        "type": "json",
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "number"}
            }
        }
    }]
)

# Without schema (LLM chooses structure)
result = firecrawl.scrape(
    url="https://example.com",
    formats=[{
        "type": "json",
        "prompt": "Extract product name and price"
    }]
)
```

### Batch Scrape Multiple URLs

```python
result = firecrawl.batch_scrape(
    urls=["https://example.com/page1", "https://example.com/page2"],
    formats=["markdown"],
    max_concurrency=5
)
```

### Crawl Entire Website

```python
result = firecrawl.crawl(
    url="https://example.com",
    limit=100,
    scrape_options={
        "formats": ["markdown"],
        "only_main_content": True
    }
)
```

### Search and Scrape

```python
result = firecrawl.search(
    query="AI agents",
    scrape_options={
        "formats": ["markdown"]
    }
)
```

### Interact with Dynamic Content

```python
# Create browser session
session = firecrawl.create_browser_session()

# Execute actions
result = firecrawl.interact(
    session_id=session.id,
    actions=[
        {"type": "click", "selector": "#search-button"},
        {"type": "write", "text": "firecrawl"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "scrape"}
    ]
)
```

### Monitor Page for Changes

```python
monitor = firecrawl.create_monitor(
    name="pricing-monitor",
    targets=[{
        "type": "scrape",
        "urls": ["https://example.com/pricing"]
    }],
    schedule={"text": "every 6 hours"},
    webhook={"url": "https://your-webhook.com"}
)
```

### Agent (Autonomous Research)

```python
result = firecrawl.agent(
    prompt="Find the latest AI research papers on arxiv.org and extract title, authors, and abstract"
)
```

## Decision Guidance

| Scenario | Use This | Why |
|----------|----------|-----|
| Single known URL, need markdown | `/scrape` with `formats: ["markdown"]` | Synchronous, 1 credit, fast |
| Single known URL, need structured data | `/scrape` with `formats: [{type: "json", schema: {...}}]` | Deterministic extraction, 5 credits (1 base + 4 LLM) |
| Multiple URLs, same options | `/batch/scrape` | Parallel processing, efficient |
| Entire website, unknown URLs | `/crawl` | Discovers via sitemap + link traversal |
| Search + scrape results | `/search` with `scrapeOptions` | One call for search + full content |
| Dynamic content (clicks, forms) | `/interact` | Browser session, stateful |
| Don't know URLs, need research | `/agent` | Autonomous navigation, deep search |
| Local file (PDF, DOCX) | `/parse` | For files not accessible by URL |
| Monitor for changes | `/monitor` | Scheduled checks, webhooks, alerts |
| Product pages | `/scrape` with `formats: ["product"]` | Deterministic, no schema needed, cheaper |
| JavaScript-heavy site | `/scrape` with `waitFor` or `/interact` | Firecrawl renders JS automatically |

## Workflow

### Typical Scraping Task

1. **Understand the requirement**: What data do you need? Single page or multiple? Structured or markdown?
2. **Check if URL is accessible**: Verify the URL is public and not behind authentication
3. **Choose the right endpoint**:
   - Single URL → `/scrape`
   - Multiple URLs → `/batch/scrape`
   - Entire site → `/crawl`
   - Search first → `/search`
   - Dynamic content → `/interact`
   - Unknown URLs → `/agent`
4. **Set output format**:
   - Markdown for LLM context: `formats: ["markdown"]`
   - Structured data: `formats: [{type: "json", schema: {...}}]`
   - Product data: `formats: ["product"]`
5. **Configure options**:
   - Cache: `maxAge: 0` for fresh, `maxAge: 600000` for 10-min cache
   - Proxy: `proxy: "auto"` for anti-bot sites
   - Mobile: `mobile: true` for responsive testing
   - Location: `location: {country: "DE"}` for geo-specific content
6. **Handle response**:
   - Check `success: true` and `data.metadata.statusCode` (2xx or 304 = clean load)
   - Extract data from `data.markdown`, `data.json`, `data.product`, etc.
   - Handle errors with retry logic (see Common Gotchas)
7. **Verify results**: Spot-check extracted data matches expectations

### Batch Processing Workflow

1. Prepare list of URLs
2. Call `/batch/scrape` with `urls` array and `scrape_options`
3. Get back job ID
4. Poll `GET /batch/scrape/{id}` until `status: "completed"`
5. Paginate through results if response exceeds 10MB (use `next` parameter)
6. Process each page in `data` array

### Crawl Workflow

1. Start crawl with `POST /crawl` (returns job ID immediately)
2. Poll `GET /crawl/{id}` to check status
3. When `status: "completed"`, retrieve all pages from `data` array
4. For large crawls, paginate using `next` parameter
5. Check `GET /crawl/{id}/errors` for failed pages (network errors, robots.txt blocks)

## Common Gotchas

- **Cache vs. freshness**: By default, Firecrawl returns cached results (2-day window). Set `maxAge: 0` to force fresh scrape, but this is slower and more likely to fail. Use `maxAge: 600000` for a 10-minute cache window as a middle ground.

- **Status code confusion**: The HTTP response status (200) is different from the page's status code (`data.metadata.statusCode`). Always check both: first confirm `success: true`, then check `statusCode` for the actual page outcome.

- **Batch scrape vs. crawl**: Batch scrape is for a known list of URLs; crawl discovers URLs automatically. Don't use batch scrape for entire sites — use crawl instead.

- **JSON extraction costs**: JSON mode adds 4 credits per page. For simple markdown, use `formats: ["markdown"]` (1 credit). For structured data, use `formats: ["json"]` only when needed.

- **Product format is deterministic**: The `product` format extracts from structured data (JSON-LD, schema.org, etc.) without an LLM call. It's cheaper and faster than JSON mode for product pages, but fails silently on non-product pages.

- **Crawl scope by default**: Crawl only follows child paths by default. Use `crawlEntireDomain: true` to include siblings/parents, `allowSubdomains: true` for subdomains, `allowExternalLinks: true` to follow off-domain links.

- **Regex patterns in crawl**: `includePaths` and `excludePaths` match URL pathname (not full URL) by default. Use `regexOnFullURL: true` to match query parameters. Patterns are Rust regex (RE2-style); look-around and backreferences not supported.

- **Prompt injection risk**: When scraping untrusted URLs with LLM-backed formats (json, question, highlights), set `checkPromptInjection: true` to scan for injection attempts before extraction. Adds 4 credits if check runs.

- **PII redaction**: Set `redactPII: true` to redact personally identifiable information from markdown. Adds 4 credits. Choose mode: `accurate` (model-only, precise), `aggressive` (heuristics + model, higher recall), or `fast` (heuristics only, no model call).

- **Interact vs. actions**: The newer `/interact` endpoint is stateful and more powerful than the `actions` parameter on `/scrape`. Use `/interact` for complex multi-step workflows; use `actions` only for simple pre-scrape interactions.

- **Webhook signature verification**: Always verify the `X-Firecrawl-Signature` header (HMAC-SHA256) on incoming webhooks. Never process unverified webhooks.

- **Rate limits**: 429 responses mean you've hit rate limits. Implement exponential backoff. SDKs retry automatically for transient errors (408, 409, 5xx) but not for 429.

- **Result expiration**: Crawl and batch scrape results are available for 24 hours via API. After that, view them in activity logs only.

- **Lockdown mode**: Set `lockdown: true` to serve only from cache (no outbound requests). Useful for compliance/air-gapped environments. Costs 5 credits on hit, 1 on miss.

- **Zero data retention**: Set `zeroDataRetention: true` to prevent caching/logging. Some options force this automatically (e.g., `actions`, `headers`, `profile`).

## Verification Checklist

Before submitting work with Firecrawl:

- [ ] Confirmed `success: true` in response
- [ ] Checked `data.metadata.statusCode` is 2xx or 304 (not 4xx/5xx)
- [ ] Verified extracted data matches expected format (markdown, JSON, product, etc.)
- [ ] Spot-checked 2–3 samples of extracted content for accuracy
- [ ] Confirmed credit usage is within budget (1 base + format surcharges)
- [ ] For batch/crawl jobs: confirmed `status: "completed"` before processing results
- [ ] For webhooks: verified `X-Firecrawl-Signature` header
- [ ] For sensitive data: confirmed `storeInCache: false` or `zeroDataRetention: true` if needed
- [ ] For dynamic content: confirmed page loaded fully (check `waitFor` or `/interact` actions)
- [ ] For crawls: checked `GET /crawl/{id}/errors` for failed pages

## Resources

**Comprehensive navigation:**
- [llms.txt](https://docs.firecrawl.dev/llms.txt) — Full index of all documentation pages for agent navigation

**Critical documentation:**
1. [Introduction & Quick Start](https://docs.firecrawl.dev/introduction) — Overview of Search, Scrape, Interact
2. [Scrape Feature Docs](https://docs.firecrawl.dev/features/scrape) — All scrape formats, extraction, caching, actions
3. [API Reference v2](https://docs.firecrawl.dev/api-reference/v2-introduction) — Complete endpoint reference with examples
4. [Crawl Feature Docs](https://docs.firecrawl.dev/features/crawl) — Crawl configuration, scope, webhooks, error handling
5. [Agent Feature Docs](https://docs.firecrawl.dev/features/agent) — Autonomous research without URLs
6. [Error Reference](https://docs.firecrawl.dev/api-reference/errors) — Error codes, causes, retry guidance
7. [Rate Limits](https://docs.firecrawl.dev/rate-limits) — Plan limits and concurrency
8. [SDKs Overview](https://docs.firecrawl.dev/sdks/overview) — Python, Node, CLI, and community SDKs

---

> For additional documentation and navigation, see: https://docs.firecrawl.dev/llms.txt