# Advanced Scraping Guide
Source: https://docs.firecrawl.dev/advanced-scraping-guide
Configure scrape options, browser actions, crawl, map, and the agent endpoint with Firecrawl's full API surface.
Reference for every option across Firecrawl's scrape, crawl, map, and agent endpoints.
## Basic scraping
To scrape a single page and get clean markdown content, use the `/scrape` endpoint.
```python Python theme={null}
# pip install firecrawl-py
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
doc = firecrawl.scrape("https://firecrawl.dev")
print(doc.markdown)
```
```js Node theme={null}
// npm install @mendable/firecrawl-js
import { Firecrawl } from 'firecrawl-js';
const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' });
const doc = await firecrawl.scrape('https://firecrawl.dev');
console.log(doc.markdown);
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://docs.firecrawl.dev"
}'
```
## Scraping PDFs
Firecrawl supports PDFs. Use the `parsers` option (e.g., `parsers: ["pdf"]`) when you want to ensure PDF parsing. You can control the parsing strategy with the `mode` option:
* **`auto`** (default) — attempts fast text-based extraction first, then falls back to OCR if needed.
* **`fast`** — text-based parsing only (embedded text). Fastest, but skips scanned/image-heavy pages.
* **`ocr`** — forces OCR parsing on every page. Use for scanned documents or when `auto` misclassifies a page.
`{ type: "pdf" }` and `"pdf"` both default to `mode: "auto"`.
```json theme={null}
"parsers": [{ "type": "pdf", "mode": "fast", "maxPages": 50 }]
```
## Scrape options
When using the `/scrape` endpoint, you can customize the request with the following options.
### Formats (`formats`)
The `formats` array controls which output types the scraper returns. Default: `["markdown"]`.
**String formats**: pass the name directly (e.g. `"markdown"`).
| Format | Description |
| ---------- | ---------------------------------------------------------------------------- |
| `markdown` | Page content converted to clean Markdown. |
| `html` | Processed HTML with unnecessary elements removed. |
| `rawHtml` | Original HTML exactly as returned by the server. |
| `links` | All links found on the page. |
| `images` | All images found on the page. |
| `summary` | An LLM-generated summary of the page content. |
| `branding` | Extracts brand identity (colors, fonts, typography, spacing, UI components). |
**Object formats**: pass an object with `type` and additional options.
| Format | Options | Description |
| ---------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `json` | `prompt?: string`, `schema?: object` | Extract structured data using an LLM. Provide a JSON schema and/or a natural-language prompt (max 10,000 characters). |
| `screenshot` | `fullPage?: boolean`, `quality?: number`, `viewport?: { width, height }` | Capture a screenshot. Max one per request. Viewport max resolution is 7680×4320. Screenshot URLs expire after 24 hours. |
| `changeTracking` | `modes?: ("json" \| "git-diff")[]`, `tag?: string`, `schema?: object`, `prompt?: string` | Track changes between scrapes. Requires `"markdown"` to also be in the formats array. |
| `attributes` | `selectors: [{ selector: string, attribute: string }]` | Extract specific HTML attributes from elements matching CSS selectors. |
### Content filtering
These parameters control which parts of the page appear in the output. When `onlyMainContent` is `true` (the default), boilerplate (nav, footer, etc.) is stripped. `includeTags` and `excludeTags` are applied against the original page DOM, not the post-filtered result, so your selectors should target elements as they appear in the source HTML. Set `onlyMainContent: false` to use the full page as the starting point for tag filtering.
| Parameter | Type | Default | Description |
| ----------------- | --------- | ------- | ---------------------------------------------------------------------------- |
| `onlyMainContent` | `boolean` | `true` | Return only the main content. Set `false` for the full page. |
| `includeTags` | `array` | — | HTML tags, classes, or IDs to include (e.g. `["h1", "p", ".main-content"]`). |
| `excludeTags` | `array` | — | HTML tags, classes, or IDs to exclude (e.g. `["#ad", "#footer"]`). |
### Timing and cache
| Parameter | Type | Default | Description |
| --------- | -------------- | ----------- | ------------------------------------------------------------------------------------------------------ |
| `waitFor` | `integer` (ms) | `0` | Extra wait time before scraping, on top of smart-wait. Use sparingly. |
| `maxAge` | `integer` (ms) | `172800000` | Return a cached version if fresher than this value (default is 2 days). Set `0` to always fetch fresh. |
| `timeout` | `integer` (ms) | `60000` | Max request duration before aborting (default is 60 seconds). Minimum is 1000 (1 second). |
### PDF parsing
| Parameter | Type | Default | Description |
| --------- | ------- | --------- | -------------------------------------------------------------------------------- |
| `parsers` | `array` | `["pdf"]` | Controls PDF processing. `[]` to skip parsing and return base64 (1 credit flat). |
```json theme={null}
{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": 10 }
```
| Property | Type | Default | Description |
| ---------- | --------------------------- | ------------ | ------------------------------------------------------------------------------------- |
| `type` | `"pdf"` | *(required)* | Parser type. |
| `mode` | `"fast" \| "auto" \| "ocr"` | `"auto"` | `fast`: text-based extraction only. `auto`: fast with OCR fallback. `ocr`: force OCR. |
| `maxPages` | `integer` | — | Cap the number of pages to parse. |
### Actions
Run browser actions before scraping. This is useful for dynamic content, navigation, or user-gated pages. You can include up to 50 actions per request, and the combined wait time across all `wait` actions and `waitFor` must not exceed 60 seconds.
| Action | Parameters | Description |
| ------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `wait` | `milliseconds?: number`, `selector?: string` | Wait for a fixed duration **or** until an element is visible (provide one, not both). When using `selector`, times out after 30 seconds. |
| `click` | `selector: string`, `all?: boolean` | Click an element matching the CSS selector. Set `all: true` to click every match. |
| `write` | `text: string` | Type text into the currently focused field. You must focus the element with a `click` action first. |
| `press` | `key: string` | Press a keyboard key (e.g. `"Enter"`, `"Tab"`, `"Escape"`). |
| `scroll` | `direction?: "up" \| "down"`, `selector?: string` | Scroll the page or a specific element. Direction defaults to `"down"`. |
| `screenshot` | `fullPage?: boolean`, `quality?: number`, `viewport?: { width, height }` | Capture a screenshot. Max viewport resolution is 7680×4320. |
| `scrape` | *(none)* | Capture the current page HTML at this point in the action sequence. |
| `executeJavascript` | `script: string` | Run JavaScript code in the page. Return values are available in the `actions.javascriptReturns` array of the response. |
| `pdf` | `format?: string`, `landscape?: boolean`, `scale?: number` | Generate a PDF. Supported formats: `"A0"` through `"A6"`, `"Letter"`, `"Legal"`, `"Tabloid"`, `"Ledger"`. Defaults to `"Letter"`. |
```python Python theme={null}
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key='fc-YOUR-API-KEY')
doc = firecrawl.scrape('https://example.com', {
'actions': [
{ 'type': 'wait', 'milliseconds': 1000 },
{ 'type': 'click', 'selector': '#accept' },
{ 'type': 'scroll', 'direction': 'down' },
{ 'type': 'click', 'selector': '#q' },
{ 'type': 'write', 'text': 'firecrawl' },
{ 'type': 'press', 'key': 'Enter' },
{ 'type': 'wait', 'milliseconds': 2000 }
],
'formats': ['markdown']
})
print(doc.markdown)
```
```js Node theme={null}
import { Firecrawl } from 'firecrawl-js';
const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' });
const doc = await firecrawl.scrape('https://example.com', {
actions: [
{ type: 'wait', milliseconds: 1000 },
{ type: 'click', selector: '#accept' },
{ type: 'scroll', direction: 'down' },
{ type: 'click', selector: '#q' },
{ type: 'write', text: 'firecrawl' },
{ type: 'press', key: 'Enter' },
{ type: 'wait', milliseconds: 2000 }
],
formats: ['markdown']
});
console.log(doc.markdown);
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "wait", "milliseconds": 1000 },
{ "type": "click", "selector": "#accept" },
{ "type": "scroll", "direction": "down" },
{ "type": "click", "selector": "#q" },
{ "type": "write", "text": "firecrawl" },
{ "type": "press", "key": "Enter" },
{ "type": "wait", "milliseconds": 2000 }
],
"formats": ["markdown"]
}'
```
#### Action execution notes
* **Write** requires a preceding `click` to focus the target element.
* **Scroll** accepts an optional `selector` to scroll a specific element instead of the page.
* **Wait** accepts either `milliseconds` (fixed delay) or `selector` (wait until visible).
* Actions run **sequentially**: each step completes before the next begins.
* Actions are **not supported for PDFs**. If the URL resolves to a PDF the request will fail.
#### Advanced action examples
**Taking a screenshot:**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "click", "selector": "#load-more" },
{ "type": "wait", "milliseconds": 1000 },
{ "type": "screenshot", "fullPage": true, "quality": 80 }
]
}'
```
**Clicking multiple elements:**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "click", "selector": ".expand-button", "all": true },
{ "type": "wait", "milliseconds": 500 }
],
"formats": ["markdown"]
}'
```
**Generating a PDF:**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "pdf", "format": "A4", "landscape": false }
]
}'
```
**Executing JavaScript (e.g. extracting embedded page data):**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "executeJavascript", "script": "document.querySelector(\"#__NEXT_DATA__\").textContent" }
],
"formats": ["markdown"]
}'
```
The return value of each `executeJavascript` action is captured in the `actions.javascriptReturns` array of the response.
### Full scrape example
The following request combines multiple scrape options:
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://docs.firecrawl.dev",
"formats": [
"markdown",
"links",
"html",
"rawHtml",
{ "type": "screenshot", "fullPage": true, "quality": 80 }
],
"includeTags": ["h1", "p", "a", ".main-content"],
"excludeTags": ["#ad", "#footer"],
"onlyMainContent": false,
"waitFor": 1000,
"timeout": 15000,
"parsers": ["pdf"]
}'
```
This request returns markdown, HTML, raw HTML, links, and a full-page screenshot. It scopes content to `
)}
```
## Scrape a page
Create `src/pages/api/scrape.ts`:
```typescript theme={null}
import type { APIRoute } from "astro";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: import.meta.env.FIRECRAWL_API_KEY,
});
export const POST: APIRoute = async ({ request }) => {
const { url } = await request.json();
const result = await firecrawl.scrape(url);
return new Response(JSON.stringify(result), {
headers: { "Content-Type": "application/json" },
});
};
```
Or scrape at request time in a server-rendered page (`src/pages/scrape.astro`):
```astro theme={null}
---
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: import.meta.env.FIRECRAWL_API_KEY,
});
const target = Astro.url.searchParams.get("url");
let markdown = null;
if (target) {
const result = await firecrawl.scrape(target);
markdown = result.markdown;
}
---
Scraped Content
{markdown ?
{markdown}
:
Pass ?url= to scrape a page
}
```
## Interact with a page
Create `src/pages/api/interact.ts`:
```typescript theme={null}
import type { APIRoute } from "astro";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: import.meta.env.FIRECRAWL_API_KEY,
});
export const POST: APIRoute = async () => {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
await firecrawl.stopInteraction(scrapeId);
return new Response(JSON.stringify({ output: response.output }), {
headers: { "Content-Type": "application/json" },
});
};
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# AWS Lambda
Source: https://docs.firecrawl.dev/quickstarts/aws-lambda
Use Firecrawl with AWS Lambda to search, scrape, and interact with web data in serverless functions.
## Prerequisites
* AWS account with Lambda access
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
mkdir firecrawl-lambda && cd firecrawl-lambda
npm init -y
npm install @mendable/firecrawl-js
```
## Search the web
Create `index.mjs` with a search handler:
```javascript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function handler(event) {
const body = JSON.parse(event.body || "{}");
if (body.action === "search") {
const results = await firecrawl.search(body.query, { limit: 5 });
return {
statusCode: 200,
body: JSON.stringify(results),
};
}
return { statusCode: 400, body: JSON.stringify({ error: "Unknown action" }) };
}
```
## Scrape a page
Add a `scrape` action to the same handler:
```javascript theme={null}
if (body.action === "scrape") {
const result = await firecrawl.scrape(body.url);
return {
statusCode: 200,
body: JSON.stringify(result),
};
}
```
## Interact with a page
Add an `interact` action to control a live browser session — click buttons, fill forms, and extract dynamic content:
```javascript theme={null}
if (body.action === "interact") {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
await firecrawl.stopInteraction(scrapeId);
return {
statusCode: 200,
body: JSON.stringify({ output: response.output }),
};
}
```
## Deploy
Package and deploy with the AWS CLI:
```bash theme={null}
zip -r function.zip index.mjs node_modules/
aws lambda create-function \
--function-name firecrawl-scraper \
--runtime nodejs20.x \
--handler index.handler \
--zip-file fileb://function.zip \
--role arn:aws:iam::YOUR_ACCOUNT:role/lambda-role \
--environment Variables="{FIRECRAWL_API_KEY=fc-YOUR-API-KEY}" \
--timeout 60
```
Set the Lambda timeout to at least 30 seconds. Scraping dynamic pages and interact sessions can take longer than the default 3-second timeout.
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Bun
Source: https://docs.firecrawl.dev/quickstarts/bun
Use Firecrawl with Bun to build fast web scraping and search servers.
## Prerequisites
* Bun 1.0+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
bun add @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
Bun has a built-in HTTP server. Create `index.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
Bun.serve({
port: 3000,
async fetch(req) {
const url = new URL(req.url);
if (req.method === "POST" && url.pathname === "/search") {
const { query } = await req.json();
const results = await firecrawl.search(query, { limit: 5 });
return Response.json(results);
}
return new Response("Not found", { status: 404 });
},
});
console.log("Server running on port 3000");
```
Run it:
```bash theme={null}
bun run index.ts
```
## Scrape a page
Add a `/scrape` route to the same server:
```typescript theme={null}
if (req.method === "POST" && url.pathname === "/scrape") {
const { url: targetUrl } = await req.json();
const result = await firecrawl.scrape(targetUrl);
return Response.json(result);
}
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
```typescript theme={null}
if (req.method === "POST" && url.pathname === "/interact") {
const { url: targetUrl } = await req.json();
const result = await firecrawl.scrape(targetUrl, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await firecrawl.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
await firecrawl.stopInteraction(scrapeId);
return Response.json({ output: response.output });
}
```
## Script usage
Use Firecrawl in a standalone Bun script:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const app = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const results = await app.search("firecrawl web scraping", { limit: 5 });
console.log(results);
```
```bash theme={null}
bun run search.ts
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Cloudflare Workers
Source: https://docs.firecrawl.dev/quickstarts/cloudflare-workers
Use Firecrawl with Cloudflare Workers to search, scrape, and interact with web data at the edge.
## Prerequisites
* Wrangler CLI (`npm install -g wrangler`)
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
npm create cloudflare@latest my-scraper
cd my-scraper
npm install @mendable/firecrawl-js
```
Add your API key as a secret:
```bash theme={null}
wrangler secret put FIRECRAWL_API_KEY
```
## Search the web
Create a handler that searches the web and returns results with full page content.
Edit `src/index.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
export interface Env {
FIRECRAWL_API_KEY: string;
}
export default {
async fetch(request: Request, env: Env): Promise {
const firecrawl = new Firecrawl({ apiKey: env.FIRECRAWL_API_KEY });
const url = new URL(request.url);
if (request.method === "POST" && url.pathname === "/search") {
const { query } = (await request.json()) as { query: string };
const results = await firecrawl.search(query, { limit: 5 });
return Response.json(results);
}
return new Response("Not found", { status: 404 });
},
};
```
## Scrape a page
Add a `/scrape` route to extract clean markdown from any URL.
```typescript theme={null}
if (request.method === "POST" && url.pathname === "/scrape") {
const { url: targetUrl } = (await request.json()) as { url: string };
const result = await firecrawl.scrape(targetUrl);
return Response.json(result);
}
```
## Interact with a page
Add an `/interact` route to control a live browser session — click buttons, fill forms, and extract dynamic content.
```typescript theme={null}
if (request.method === "POST" && url.pathname === "/interact") {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
await firecrawl.stopInteraction(scrapeId);
return Response.json({ output: response.output });
}
```
## Deploy
```bash theme={null}
wrangler deploy
```
## Test it
```bash theme={null}
curl -X POST https://my-scraper..workers.dev/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Deno Deploy
Source: https://docs.firecrawl.dev/quickstarts/deno-deploy
Use Firecrawl with Deno Deploy to search, scrape, and interact with web data at the edge.
## Prerequisites
* Deno 1.40+ or Deno 2
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
Create `main.ts`:
```typescript theme={null}
import Firecrawl from "npm:@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: Deno.env.get("FIRECRAWL_API_KEY"),
});
```
## Search the web
Add a `/search` route that searches the web and returns results with full page content.
```typescript theme={null}
Deno.serve(async (req) => {
const url = new URL(req.url);
if (req.method === "POST" && url.pathname === "/search") {
const { query } = await req.json();
const results = await firecrawl.search(query, { limit: 5 });
return Response.json(results);
}
return new Response("Not found", { status: 404 });
});
```
## Scrape a page
Add a `/scrape` route to extract clean markdown from any URL.
```typescript theme={null}
if (req.method === "POST" && url.pathname === "/scrape") {
const { url: targetUrl } = await req.json();
const result = await firecrawl.scrape(targetUrl);
return Response.json(result);
}
```
## Interact with a page
Add an `/interact` route to control a live browser session — click buttons, fill forms, and extract dynamic content.
```typescript theme={null}
if (req.method === "POST" && url.pathname === "/interact") {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
console.log(response.output);
await firecrawl.stopInteraction(scrapeId);
return Response.json({ output: response.output });
}
```
## Run locally
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY deno run --allow-net --allow-env main.ts
```
## Deploy
Install the Deno Deploy CLI (`deployctl`) and deploy:
```bash theme={null}
deployctl deploy --project=my-scraper main.ts
```
Set the environment variable in the Deno Deploy dashboard or via CLI:
```bash theme={null}
deployctl env set FIRECRAWL_API_KEY=fc-YOUR-API-KEY --project=my-scraper
```
## Test it
```bash theme={null}
curl -X POST https://my-scraper.deno.dev/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Django
Source: https://docs.firecrawl.dev/quickstarts/django
Use Firecrawl with Django to scrape, search, and interact with web data in your Python web application.
## Prerequisites
* Django 4+ project
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
pip install firecrawl-py
```
Add your API key to your Django settings or environment:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Create views
Add search, scrape, and interact views to your Django app. In `views.py`:
```python theme={null}
import json
import os
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
@csrf_exempt
@require_POST
def search_view(request):
body = json.loads(request.body)
results = firecrawl.search(body["query"], limit=body.get("limit", 5))
return JsonResponse(
[{"title": r.title, "url": r.url} for r in results.web],
safe=False,
)
@csrf_exempt
@require_POST
def scrape_view(request):
body = json.loads(request.body)
result = firecrawl.scrape(body["url"])
return JsonResponse({
"markdown": result.markdown,
"metadata": result.metadata,
})
@csrf_exempt
@require_POST
def interact_start_view(request):
body = json.loads(request.body)
result = firecrawl.scrape(body["url"], formats=["markdown"])
return JsonResponse({"scrape_id": result.metadata.scrape_id})
@csrf_exempt
@require_POST
def interact_view(request):
body = json.loads(request.body)
response = firecrawl.interact(body["scrape_id"], prompt=body["prompt"])
return JsonResponse({"output": response.output})
@csrf_exempt
@require_POST
def interact_stop_view(request):
body = json.loads(request.body)
firecrawl.stop_interaction(body["scrape_id"])
return JsonResponse({"status": "stopped"})
```
## Wire up URLs
In `urls.py`:
```python theme={null}
from django.urls import path
from . import views
urlpatterns = [
path("api/search/", views.search_view),
path("api/scrape/", views.scrape_view),
path("api/interact/start/", views.interact_start_view),
path("api/interact/", views.interact_view),
path("api/interact/stop/", views.interact_stop_view),
]
```
## Test it
```bash theme={null}
python manage.py runserver
# Search the web
curl -X POST http://localhost:8000/api/search/ \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping", "limit": 5}'
# Scrape a page
curl -X POST http://localhost:8000/api/scrape/ \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# Start an interactive session
curl -X POST http://localhost:8000/api/interact/start/ \
-H "Content-Type: application/json" \
-d '{"url": "https://www.amazon.com"}'
```
## Management command
Use Firecrawl in a Django management command for scripts and data pipelines. Create `management/commands/scrape.py`:
```python theme={null}
import os
from django.core.management.base import BaseCommand
from firecrawl import Firecrawl
class Command(BaseCommand):
help = "Scrape a URL and print the markdown"
def add_arguments(self, parser):
parser.add_argument("url", type=str)
def handle(self, *args, **options):
firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
result = firecrawl.scrape(options["url"])
self.stdout.write(result.markdown)
```
```bash theme={null}
python manage.py scrape https://example.com
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, async, and more
# .NET
Source: https://docs.firecrawl.dev/quickstarts/dotnet
Get started with Firecrawl in .NET. Scrape, search, and interact with web data using the REST API.
## Prerequisites
* .NET 6.0+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Search the web
Firecrawl works with .NET through the REST API using `HttpClient`.
```csharp theme={null}
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
var apiKey = Environment.GetEnvironmentVariable("FIRECRAWL_API_KEY");
var client = new HttpClient();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
var content = new StringContent(
JsonSerializer.Serialize(new { query = "firecrawl web scraping", limit = 5 }),
Encoding.UTF8,
"application/json"
);
var response = await client.PostAsync("https://api.firecrawl.dev/v2/search", content);
var json = await response.Content.ReadAsStringAsync();
Console.WriteLine(json);
```
```json theme={null}
{
"success": true,
"data": {
"web": [
{
"url": "https://docs.firecrawl.dev",
"title": "Firecrawl Documentation",
"markdown": "# Firecrawl\n\nFirecrawl is a web scraping API..."
}
]
}
}
```
## Scrape a page
```csharp theme={null}
var scrapeContent = new StringContent(
JsonSerializer.Serialize(new { url = "https://example.com" }),
Encoding.UTF8,
"application/json"
);
var scrapeResponse = await client.PostAsync("https://api.firecrawl.dev/v2/scrape", scrapeContent);
var scrapeJson = await scrapeResponse.Content.ReadAsStringAsync();
using var doc = JsonDocument.Parse(scrapeJson);
var markdown = doc.RootElement.GetProperty("data").GetProperty("markdown").GetString();
Console.WriteLine(markdown);
```
```json theme={null}
{
"success": true,
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
}
```
## Interact with a page
Start a browser session, interact with the page using natural-language prompts, then close the session.
### Step 1 — Scrape to start a session
```csharp theme={null}
var sessionContent = new StringContent(
JsonSerializer.Serialize(new { url = "https://www.amazon.com", formats = new[] { "markdown" } }),
Encoding.UTF8,
"application/json"
);
var sessionResponse = await client.PostAsync("https://api.firecrawl.dev/v2/scrape", sessionContent);
var sessionJson = await sessionResponse.Content.ReadAsStringAsync();
using var sessionDoc = JsonDocument.Parse(sessionJson);
var scrapeId = sessionDoc.RootElement
.GetProperty("data")
.GetProperty("metadata")
.GetProperty("scrapeId")
.GetString();
Console.WriteLine($"scrapeId: {scrapeId}");
```
### Step 2 — Send interactions
```csharp theme={null}
var interactUrl = $"https://api.firecrawl.dev/v2/scrape/{scrapeId}/interact";
// Search for a product
var searchBody = new StringContent(
JsonSerializer.Serialize(new { prompt = "Search for iPhone 16 Pro Max" }),
Encoding.UTF8,
"application/json"
);
var searchResult = await client.PostAsync(interactUrl, searchBody);
Console.WriteLine(await searchResult.Content.ReadAsStringAsync());
// Click on the first result
var clickBody = new StringContent(
JsonSerializer.Serialize(new { prompt = "Click on the first result and tell me the price" }),
Encoding.UTF8,
"application/json"
);
var clickResult = await client.PostAsync(interactUrl, clickBody);
Console.WriteLine(await clickResult.Content.ReadAsStringAsync());
```
### Step 3 — Stop the session
```csharp theme={null}
await client.DeleteAsync(interactUrl);
Console.WriteLine("Session stopped");
```
## Reusable client class
For repeated use, wrap the API in a typed client:
```csharp theme={null}
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
public class FirecrawlClient
{
private readonly HttpClient _http;
private const string BaseUrl = "https://api.firecrawl.dev/v2";
public FirecrawlClient(string apiKey)
{
_http = new HttpClient();
_http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
}
private async Task PostAsync(string endpoint, object payload)
{
var content = new StringContent(
JsonSerializer.Serialize(payload),
Encoding.UTF8,
"application/json"
);
var response = await _http.PostAsync($"{BaseUrl}{endpoint}", content);
response.EnsureSuccessStatusCode();
var json = await response.Content.ReadAsStringAsync();
return JsonDocument.Parse(json);
}
public async Task ScrapeAsync(string url)
{
return await PostAsync("/scrape", new { url });
}
public async Task SearchAsync(string query, int limit = 5)
{
return await PostAsync("/search", new { query, limit });
}
}
// Usage
var firecrawl = new FirecrawlClient(Environment.GetEnvironmentVariable("FIRECRAWL_API_KEY")!);
var result = await firecrawl.SearchAsync("firecrawl web scraping");
Console.WriteLine(result.RootElement);
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Complete REST API documentation
# Elixir
Source: https://docs.firecrawl.dev/quickstarts/elixir
Get started with Firecrawl in Elixir. Search, scrape, and interact with web data using the official SDK.
## Prerequisites
* Elixir 1.14+ and OTP 25+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
Add `firecrawl` to your `mix.exs`:
```elixir theme={null}
defp deps do
[
{:firecrawl, "~> 1.0"}
]
end
```
Configure your API key in `config/config.exs`:
```elixir theme={null}
config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY")
```
## Search the web
```elixir theme={null}
{:ok, result} = Firecrawl.search_and_scrape(query: "firecrawl web scraping", limit: 5)
for entry <- result.body["data"]["web"] do
IO.puts("#{entry["title"]} - #{entry["url"]}")
end
```
## Scrape a page
```elixir theme={null}
{:ok, result} = Firecrawl.scrape_and_extract_from_url(url: "https://example.com")
IO.puts(result.body["data"]["markdown"])
```
```json theme={null}
{
"success": true,
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
}
```
## Interact with a page
Scrape a page, then keep working with it using the browser session API.
```elixir theme={null}
{:ok, scrape} = Firecrawl.scrape_and_extract_from_url(
url: "https://www.amazon.com",
formats: ["markdown"]
)
scrape_id = get_in(scrape.body, ["data", "metadata", "scrapeId"])
# Use the REST API for interact (prompt-based)
headers = [
{"Authorization", "Bearer #{Application.get_env(:firecrawl, :api_key)}"},
{"Content-Type", "application/json"}
]
{:ok, _} = Req.post(
"https://api.firecrawl.dev/v2/scrape/#{scrape_id}/interact",
json: %{prompt: "Search for iPhone 16 Pro Max"},
headers: headers
)
{:ok, response} = Req.post(
"https://api.firecrawl.dev/v2/scrape/#{scrape_id}/interact",
json: %{prompt: "Click on the first result and tell me the price"},
headers: headers
)
IO.inspect(response.body)
# Stop the session
Req.delete(
"https://api.firecrawl.dev/v2/scrape/#{scrape_id}/interact",
headers: headers
)
```
## Environment variable
Set `FIRECRAWL_API_KEY` instead of hardcoding:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Express
Source: https://docs.firecrawl.dev/quickstarts/express
Use Firecrawl with Express to build web scraping and search APIs.
## Prerequisites
* Node.js 18+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
npm install express @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
```javascript theme={null}
import express from "express";
import Firecrawl from "@mendable/firecrawl-js";
const app = express();
app.use(express.json());
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
app.post("/search", async (req, res) => {
try {
const { query } = req.body;
const results = await firecrawl.search(query, { limit: 5 });
res.json(results);
} catch (error) {
res.status(500).json({ error: error.message });
}
});
app.listen(3000, () => console.log("Server running on port 3000"));
```
## Scrape a page
```javascript theme={null}
app.post("/scrape", async (req, res) => {
try {
const { url } = req.body;
const result = await firecrawl.scrape(url);
res.json(result);
} catch (error) {
res.status(500).json({ error: error.message });
}
});
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
```javascript theme={null}
app.post("/interact", async (req, res) => {
try {
const { url } = req.body;
const result = await firecrawl.scrape(url, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await firecrawl.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
await firecrawl.stopInteraction(scrapeId);
res.json({ output: response.output });
} catch (error) {
res.status(500).json({ error: error.message });
}
});
```
## Test it
```bash theme={null}
curl -X POST http://localhost:3000/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# FastAPI
Source: https://docs.firecrawl.dev/quickstarts/fastapi
Use Firecrawl with FastAPI to build async web scraping and search APIs in Python.
## Prerequisites
* Python 3.8+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
pip install fastapi uvicorn firecrawl-py
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Create the API
Create `main.py`:
```python theme={null}
import os
from fastapi import FastAPI
from pydantic import BaseModel
from firecrawl import Firecrawl
app = FastAPI()
firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
class SearchRequest(BaseModel):
query: str
limit: int = 5
class ScrapeRequest(BaseModel):
url: str
class InteractRequest(BaseModel):
scrape_id: str
prompt: str
@app.post("/search")
async def search(req: SearchRequest):
results = firecrawl.search(req.query, limit=req.limit)
return [{"title": r.title, "url": r.url} for r in results.web]
@app.post("/scrape")
async def scrape(req: ScrapeRequest):
result = firecrawl.scrape(req.url)
return {"markdown": result.markdown, "metadata": result.metadata}
@app.post("/interact/start")
async def interact_start(req: ScrapeRequest):
result = firecrawl.scrape(req.url, formats=["markdown"])
return {"scrape_id": result.metadata.scrape_id}
@app.post("/interact")
async def interact(req: InteractRequest):
response = firecrawl.interact(req.scrape_id, prompt=req.prompt)
return {"output": response.output}
@app.post("/interact/stop")
async def interact_stop(req: InteractRequest):
firecrawl.stop_interaction(req.scrape_id)
return {"status": "stopped"}
```
## Run it
```bash theme={null}
uvicorn main:app --reload
```
## Test it
```bash theme={null}
# Search the web
curl -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping", "limit": 5}'
# Scrape a page
curl -X POST http://localhost:8000/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# Start an interactive session, then send prompts
curl -X POST http://localhost:8000/interact/start \
-H "Content-Type: application/json" \
-d '{"url": "https://www.amazon.com"}'
```
FastAPI auto-generates interactive docs at `http://localhost:8000/docs`.
## Async variant
For better concurrency under load, use `AsyncFirecrawl`:
```python theme={null}
from firecrawl import AsyncFirecrawl
async_firecrawl = AsyncFirecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
@app.post("/scrape-async")
async def scrape_async(req: ScrapeRequest):
result = await async_firecrawl.scrape(req.url)
return {"markdown": result.markdown}
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, async, and more
# Fastify
Source: https://docs.firecrawl.dev/quickstarts/fastify
Use Firecrawl with Fastify to build high-performance web scraping and search APIs.
## Prerequisites
* Node.js 18+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
npm install fastify @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
```javascript theme={null}
import Fastify from "fastify";
import Firecrawl from "@mendable/firecrawl-js";
const fastify = Fastify({ logger: true });
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
fastify.post("/search", async (request) => {
const { query } = request.body;
return firecrawl.search(query, { limit: 5 });
});
fastify.listen({ port: 3000 });
```
## Scrape a page
```javascript theme={null}
fastify.post("/scrape", async (request) => {
const { url } = request.body;
return firecrawl.scrape(url);
});
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
```javascript theme={null}
fastify.post("/interact", async (request) => {
const { url } = request.body;
const result = await firecrawl.scrape(url, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await firecrawl.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
await firecrawl.stopInteraction(scrapeId);
return { output: response.output };
});
```
## As a Fastify plugin
Encapsulate the client in a plugin for reuse across routes:
```javascript theme={null}
import fp from "fastify-plugin";
import Firecrawl from "@mendable/firecrawl-js";
export default fp(async function firecrawlPlugin(fastify) {
const client = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
fastify.decorate("firecrawl", client);
});
```
Register the plugin, then use `fastify.firecrawl` in any route:
```javascript theme={null}
fastify.register(firecrawlPlugin);
fastify.post("/search", async function (request) {
const { query } = request.body;
return this.firecrawl.search(query, { limit: 5 });
});
```
## Test it
```bash theme={null}
curl -X POST http://localhost:3000/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Flask
Source: https://docs.firecrawl.dev/quickstarts/flask
Use Firecrawl with Flask to build web scraping and search APIs in Python.
## Prerequisites
* Python 3.8+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
pip install flask firecrawl-py
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Create the app
Create `app.py`:
```python theme={null}
import os
from flask import Flask, request, jsonify
from firecrawl import Firecrawl
app = Flask(__name__)
firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
@app.post("/search")
def search():
data = request.get_json()
results = firecrawl.search(data["query"], limit=data.get("limit", 5))
return jsonify([{"title": r.title, "url": r.url} for r in results.web])
@app.post("/scrape")
def scrape():
data = request.get_json()
result = firecrawl.scrape(data["url"])
return jsonify(markdown=result.markdown, metadata=result.metadata)
@app.post("/interact/start")
def interact_start():
data = request.get_json()
result = firecrawl.scrape(data["url"], formats=["markdown"])
return jsonify(scrape_id=result.metadata.scrape_id)
@app.post("/interact")
def interact():
data = request.get_json()
response = firecrawl.interact(data["scrape_id"], prompt=data["prompt"])
return jsonify(output=response.output)
@app.post("/interact/stop")
def interact_stop():
data = request.get_json()
firecrawl.stop_interaction(data["scrape_id"])
return jsonify(status="stopped")
if __name__ == "__main__":
app.run(debug=True)
```
## Run it
```bash theme={null}
flask run
```
## Test it
```bash theme={null}
# Search the web
curl -X POST http://localhost:5000/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping", "limit": 5}'
# Scrape a page
curl -X POST http://localhost:5000/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# Start an interactive session
curl -X POST http://localhost:5000/interact/start \
-H "Content-Type: application/json" \
-d '{"url": "https://www.amazon.com"}'
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, async, and more
# Go
Source: https://docs.firecrawl.dev/quickstarts/go
Get started with Firecrawl in Go. Scrape, search, and interact with web data using the REST API.
## Prerequisites
* Go 1.21+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Search the web
Firecrawl works with Go through the REST API. Use `net/http` to make requests directly.
```go theme={null}
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
apiKey := os.Getenv("FIRECRAWL_API_KEY")
body, _ := json.Marshal(map[string]interface{}{
"query": "firecrawl web scraping",
"limit": 5,
})
req, _ := http.NewRequest("POST", "https://api.firecrawl.dev/v2/search", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "request failed: %v\n", err)
os.Exit(1)
}
defer resp.Body.Close()
result, _ := io.ReadAll(resp.Body)
fmt.Println(string(result))
}
```
```json theme={null}
{
"success": true,
"data": {
"web": [
{
"url": "https://docs.firecrawl.dev",
"title": "Firecrawl Documentation",
"markdown": "# Firecrawl\n\nFirecrawl is a web scraping API..."
}
]
}
}
```
## Scrape a page
```go theme={null}
body, _ := json.Marshal(map[string]string{
"url": "https://example.com",
})
req, _ := http.NewRequest("POST", "https://api.firecrawl.dev/v2/scrape", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "request failed: %v\n", err)
os.Exit(1)
}
defer resp.Body.Close()
result, _ := io.ReadAll(resp.Body)
fmt.Println(string(result))
```
```json theme={null}
{
"success": true,
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
}
```
## Interact with a page
Start a browser session, interact with the page using natural-language prompts, then close the session.
### Step 1 — Scrape to start a session
```go theme={null}
body, _ := json.Marshal(map[string]interface{}{
"url": "https://www.amazon.com",
"formats": []string{"markdown"},
})
req, _ := http.NewRequest("POST", "https://api.firecrawl.dev/v2/scrape", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "request failed: %v\n", err)
os.Exit(1)
}
defer resp.Body.Close()
var scrapeResult map[string]interface{}
json.NewDecoder(resp.Body).Decode(&scrapeResult)
data := scrapeResult["data"].(map[string]interface{})
metadata := data["metadata"].(map[string]interface{})
scrapeId := metadata["scrapeId"].(string)
fmt.Println("scrapeId:", scrapeId)
```
### Step 2 — Send interactions
```go theme={null}
// Search for a product
interactBody, _ := json.Marshal(map[string]string{
"prompt": "Search for iPhone 16 Pro Max",
})
interactURL := fmt.Sprintf("https://api.firecrawl.dev/v2/scrape/%s/interact", scrapeId)
req, _ = http.NewRequest("POST", interactURL, bytes.NewReader(interactBody))
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err = http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "interact failed: %v\n", err)
os.Exit(1)
}
defer resp.Body.Close()
result, _ := io.ReadAll(resp.Body)
fmt.Println(string(result))
// Click on the first result
interactBody, _ = json.Marshal(map[string]string{
"prompt": "Click on the first result and tell me the price",
})
req, _ = http.NewRequest("POST", interactURL, bytes.NewReader(interactBody))
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err = http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "interact failed: %v\n", err)
os.Exit(1)
}
defer resp.Body.Close()
result, _ = io.ReadAll(resp.Body)
fmt.Println(string(result))
```
### Step 3 — Stop the session
```go theme={null}
req, _ = http.NewRequest("DELETE", interactURL, nil)
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err = http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "delete failed: %v\n", err)
os.Exit(1)
}
defer resp.Body.Close()
fmt.Println("Session stopped")
```
## Reusable helper
For repeated use, wrap the API in a small helper:
```go theme={null}
type FirecrawlClient struct {
APIKey string
BaseURL string
Client *http.Client
}
func NewFirecrawlClient(apiKey string) *FirecrawlClient {
return &FirecrawlClient{
APIKey: apiKey,
BaseURL: "https://api.firecrawl.dev/v2",
Client: &http.Client{},
}
}
func (fc *FirecrawlClient) post(endpoint string, payload interface{}) ([]byte, error) {
body, err := json.Marshal(payload)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", fc.BaseURL+endpoint, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+fc.APIKey)
req.Header.Set("Content-Type", "application/json")
resp, err := fc.Client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return io.ReadAll(resp.Body)
}
func (fc *FirecrawlClient) Scrape(url string) ([]byte, error) {
return fc.post("/scrape", map[string]string{"url": url})
}
func (fc *FirecrawlClient) Search(query string, limit int) ([]byte, error) {
return fc.post("/search", map[string]interface{}{"query": query, "limit": limit})
}
```
A [community Go SDK](https://github.com/mendableai/firecrawl-go) is available for the v1 API. See the [Go SDK docs](/sdks/go) for details.
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Complete REST API documentation
# Hono
Source: https://docs.firecrawl.dev/quickstarts/hono
Use Firecrawl with Hono to build lightweight web scraping and search APIs that run anywhere.
## Prerequisites
* Node.js 18+, Bun, or Deno
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
npm install hono @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
```typescript theme={null}
import { Hono } from "hono";
import Firecrawl from "@mendable/firecrawl-js";
const app = new Hono();
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
app.post("/search", async (c) => {
const { query } = await c.req.json();
const results = await firecrawl.search(query, { limit: 5 });
return c.json(results);
});
export default app;
```
## Scrape a page
```typescript theme={null}
app.post("/scrape", async (c) => {
const { url } = await c.req.json();
const result = await firecrawl.scrape(url);
return c.json(result);
});
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
```typescript theme={null}
app.post("/interact", async (c) => {
const { url } = await c.req.json();
const result = await firecrawl.scrape(url, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await firecrawl.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
await firecrawl.stopInteraction(scrapeId);
return c.json({ output: response.output });
});
```
## Deploy anywhere
Hono runs on multiple runtimes. For Cloudflare Workers, pass the API key from the environment binding:
```typescript theme={null}
import { Hono } from "hono";
import Firecrawl from "@mendable/firecrawl-js";
type Bindings = { FIRECRAWL_API_KEY: string };
const app = new Hono<{ Bindings: Bindings }>();
app.post("/search", async (c) => {
const firecrawl = new Firecrawl({ apiKey: c.env.FIRECRAWL_API_KEY });
const { query } = await c.req.json();
const results = await firecrawl.search(query, { limit: 5 });
return c.json(results);
});
export default app;
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Java
Source: https://docs.firecrawl.dev/quickstarts/java
Get started with Firecrawl in Java. Search, scrape, and interact with web data using the official SDK.
## Prerequisites
* Java 11+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```kotlin theme={null}
dependencies {
implementation("com.firecrawl:firecrawl-java:1.2.0")
}
```
```xml theme={null}
com.firecrawlfirecrawl-java1.2.0
```
## Search the web
```java theme={null}
import com.firecrawl.client.FirecrawlClient;
import com.firecrawl.models.SearchData;
import com.firecrawl.models.SearchOptions;
public class Main {
public static void main(String[] args) {
FirecrawlClient client = FirecrawlClient.builder()
.apiKey("fc-YOUR-API-KEY")
.build();
SearchData results = client.search(
"firecrawl web scraping",
SearchOptions.builder().limit(5).build()
);
if (results.getWeb() != null) {
for (var result : results.getWeb()) {
System.out.println(result.get("title") + " - " + result.get("url"));
}
}
}
}
```
## Scrape a page
```java theme={null}
import com.firecrawl.models.Document;
Document doc = client.scrape("https://example.com");
System.out.println(doc.getMarkdown());
```
```json theme={null}
{
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
```
## Interact with a page
Open a browser session, run Playwright code against it, and close it when done:
```java theme={null}
import com.firecrawl.models.ScrapeOptions;
import com.firecrawl.models.BrowserExecuteResponse;
import java.util.List;
Document doc = client.scrape("https://www.amazon.com",
ScrapeOptions.builder().formats(List.of((Object) "markdown")).build());
String scrapeId = (String) doc.getMetadata().get("scrapeId");
BrowserExecuteResponse run = client.interact(scrapeId,
"const title = await page.title(); console.log(title);");
System.out.println(run.getStdout());
client.stopInteractiveBrowser(scrapeId);
```
## Environment variable
Instead of passing `apiKey` directly, set the `FIRECRAWL_API_KEY` environment variable:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
```java theme={null}
FirecrawlClient client = FirecrawlClient.fromEnv();
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Laravel
Source: https://docs.firecrawl.dev/quickstarts/laravel
Use Firecrawl with Laravel to search, scrape, and interact with web data using the REST API.
## Prerequisites
* Laravel 10+ project
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Configuration
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
Add the config entry to `config/services.php`:
```php theme={null}
'firecrawl' => [
'api_key' => env('FIRECRAWL_API_KEY'),
'base_url' => env('FIRECRAWL_API_URL', 'https://api.firecrawl.dev/v2'),
],
```
## Create a service class
Create `app/Services/FirecrawlService.php`:
```php theme={null}
apiKey = config('services.firecrawl.api_key');
$this->baseUrl = config('services.firecrawl.base_url');
}
public function search(string $query, int $limit = 5): array
{
$response = Http::withToken($this->apiKey)
->post("{$this->baseUrl}/search", [
'query' => $query,
'limit' => $limit,
]);
return $response->json();
}
public function scrape(string $url, array $options = []): array
{
$response = Http::withToken($this->apiKey)
->post("{$this->baseUrl}/scrape", array_merge(['url' => $url], $options));
return $response->json();
}
public function interact(string $url, string $prompt, ?string $followUp = null): array
{
// 1. Scrape to open a browser session
$scrapeResult = $this->scrape($url, ['formats' => ['markdown']]);
$scrapeId = $scrapeResult['data']['metadata']['scrapeId'];
// 2. Send first prompt
Http::withToken($this->apiKey)
->post("{$this->baseUrl}/scrape/{$scrapeId}/interact", [
'prompt' => $prompt,
]);
// 3. Send follow-up prompt
$result = null;
if ($followUp) {
$result = Http::withToken($this->apiKey)
->post("{$this->baseUrl}/scrape/{$scrapeId}/interact", [
'prompt' => $followUp,
])->json();
}
// 4. Close the session
Http::withToken($this->apiKey)
->delete("{$this->baseUrl}/scrape/{$scrapeId}/interact");
return $result ?? $scrapeResult;
}
}
```
## Create a controller
Create `app/Http/Controllers/FirecrawlController.php`:
```php theme={null}
validate(['query' => 'required|string']);
return response()->json(
$this->firecrawl->search($validated['query'], $request->input('limit', 5))
);
}
public function scrape(Request $request)
{
$validated = $request->validate(['url' => 'required|url']);
return response()->json($this->firecrawl->scrape($validated['url']));
}
public function interact(Request $request)
{
$validated = $request->validate([
'url' => 'required|url',
'prompt' => 'required|string',
]);
return response()->json(
$this->firecrawl->interact(
$validated['url'],
$validated['prompt'],
$request->input('followUp')
)
);
}
}
```
## Register routes
In `routes/api.php`:
```php theme={null}
use App\Http\Controllers\FirecrawlController;
Route::post('/search', [FirecrawlController::class, 'search']);
Route::post('/scrape', [FirecrawlController::class, 'scrape']);
Route::post('/interact', [FirecrawlController::class, 'interact']);
```
## Test it
```bash theme={null}
php artisan serve
# Search the web
curl -X POST http://localhost:8000/api/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
# Scrape a page
curl -X POST http://localhost:8000/api/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# Interact with a page
curl -X POST http://localhost:8000/api/interact \
-H "Content-Type: application/json" \
-d '{"url": "https://www.amazon.com", "prompt": "Search for iPhone 16 Pro Max", "followUp": "Click on the first result and tell me the price"}'
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Complete REST API documentation
# NestJS
Source: https://docs.firecrawl.dev/quickstarts/nestjs
Use Firecrawl with NestJS to build structured web scraping and search services.
## Prerequisites
* NestJS project (`@nestjs/cli`)
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
npm install @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Create a Firecrawl service
Create `src/firecrawl/firecrawl.service.ts`:
```typescript theme={null}
import { Injectable } from "@nestjs/common";
import Firecrawl from "@mendable/firecrawl-js";
@Injectable()
export class FirecrawlService {
private readonly client: Firecrawl;
constructor() {
this.client = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
}
async search(query: string, limit = 5) {
return this.client.search(query, { limit });
}
async scrape(url: string) {
return this.client.scrape(url);
}
async interact(url: string, prompts: string[]) {
const result = await this.client.scrape(url, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
const responses = [];
for (const prompt of prompts) {
const response = await this.client.interact(scrapeId, { prompt });
responses.push(response);
}
await this.client.stopInteraction(scrapeId);
return responses;
}
}
```
## Create a controller
Create `src/firecrawl/firecrawl.controller.ts`:
```typescript theme={null}
import { Body, Controller, Post } from "@nestjs/common";
import { FirecrawlService } from "./firecrawl.service";
@Controller("firecrawl")
export class FirecrawlController {
constructor(private readonly firecrawlService: FirecrawlService) {}
@Post("search")
async search(@Body("query") query: string) {
return this.firecrawlService.search(query);
}
@Post("scrape")
async scrape(@Body("url") url: string) {
return this.firecrawlService.scrape(url);
}
@Post("interact")
async interact(@Body("url") url: string, @Body("prompts") prompts: string[]) {
return this.firecrawlService.interact(url, prompts);
}
}
```
## Register the module
Create `src/firecrawl/firecrawl.module.ts`:
```typescript theme={null}
import { Module } from "@nestjs/common";
import { FirecrawlService } from "./firecrawl.service";
import { FirecrawlController } from "./firecrawl.controller";
@Module({
providers: [FirecrawlService],
controllers: [FirecrawlController],
exports: [FirecrawlService],
})
export class FirecrawlModule {}
```
Import `FirecrawlModule` in your `AppModule`.
## Test it
```bash theme={null}
curl -X POST http://localhost:3000/firecrawl/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Next.js
Source: https://docs.firecrawl.dev/quickstarts/nextjs
Use Firecrawl with Next.js to scrape, search, and interact with web data in your React application.
## Prerequisites
* Next.js 14+ project (App Router)
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
npm install @mendable/firecrawl-js
```
## Set your API key
Add your API key to `.env.local`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
The SDK should only run server-side since it requires your API key.
### Route Handler
Create `app/api/search/route.ts`:
```typescript theme={null}
import { NextResponse } from "next/server";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function POST(request: Request) {
const { query } = await request.json();
const results = await firecrawl.search(query, { limit: 5 });
return NextResponse.json(results);
}
```
### Server Action
Create `app/actions.ts` for use from Client Components:
```typescript theme={null}
"use server";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function searchWeb(query: string) {
const results = await firecrawl.search(query, { limit: 5 });
return (results.web || []).map((r) => ({ title: r.title, url: r.url }));
}
```
## Scrape a page
### Route Handler
Create `app/api/scrape/route.ts`:
```typescript theme={null}
import { NextResponse } from "next/server";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function POST(request: Request) {
const { url } = await request.json();
const result = await firecrawl.scrape(url);
return NextResponse.json(result);
}
```
### Server Component
Fetch data directly in a Server Component at `app/page.tsx`:
```tsx theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export default async function Page() {
const result = await firecrawl.scrape("https://example.com");
return (
Scraped Content
{result.markdown}
);
}
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
### Route Handler
Create `app/api/interact/route.ts`:
```typescript theme={null}
import { NextResponse } from "next/server";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function POST(request: Request) {
const { url, prompts } = await request.json();
const result = await firecrawl.scrape(url, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await firecrawl.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
await firecrawl.stopInteraction(scrapeId);
return NextResponse.json({ output: response.output });
}
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Node.js
Source: https://docs.firecrawl.dev/quickstarts/nodejs
Get started with Firecrawl in Node.js. Scrape, search, and interact with web data using the official SDK.
## Prerequisites
* Node.js 18+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
npm install @mendable/firecrawl-js
```
## Environment variable
Instead of passing `apiKey` directly, set the `FIRECRAWL_API_KEY` environment variable:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
```javascript theme={null}
const app = new Firecrawl();
```
## Search the web
```javascript theme={null}
import Firecrawl from '@mendable/firecrawl-js';
const app = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });
const results = await app.search("firecrawl web scraping", { limit: 5 });
for (const result of results.web) {
console.log(result.title, result.url);
}
```
## Scrape a page
```javascript theme={null}
const result = await app.scrape("https://example.com");
console.log(result.markdown);
```
```json theme={null}
{
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
```javascript theme={null}
const result = await app.scrape('https://www.amazon.com', { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await app.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await app.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
console.log(response.output);
await app.stopInteraction(scrapeId);
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Nuxt
Source: https://docs.firecrawl.dev/quickstarts/nuxt
Use Firecrawl with Nuxt to scrape, search, and interact with web data in your Vue application.
## Prerequisites
* Nuxt 3+ project
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
npm install @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
Create `server/api/search.post.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: process.env.FIRECRAWL_API_KEY,
});
export default defineEventHandler(async (event) => {
const { query } = await readBody(event);
const results = await firecrawl.search(query, { limit: 5 });
return results;
});
```
Call it from a Vue component:
```vue theme={null}
```
## Scrape a page
Create `server/api/scrape.post.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: process.env.FIRECRAWL_API_KEY,
});
export default defineEventHandler(async (event) => {
const { url } = await readBody(event);
const result = await firecrawl.scrape(url);
return result;
});
```
Call it from a Vue component:
```vue theme={null}
{{ data.markdown }}
```
## Interact with a page
Create `server/api/interact.post.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: process.env.FIRECRAWL_API_KEY,
});
export default defineEventHandler(async (event) => {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
await firecrawl.stopInteraction(scrapeId);
return { output: response.output };
});
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# PHP
Source: https://docs.firecrawl.dev/quickstarts/php
Get started with Firecrawl in PHP. Scrape, search, and interact with web data using the REST API.
## Prerequisites
* PHP 8.0+ with cURL extension
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Search the web
Firecrawl works with PHP through the REST API using cURL.
```php theme={null}
true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => [
'Authorization: Bearer ' . $apiKey,
'Content-Type: application/json',
],
CURLOPT_POSTFIELDS => json_encode([
'query' => 'firecrawl web scraping',
'limit' => 5,
]),
]);
$response = curl_exec($ch);
curl_close($ch);
$results = json_decode($response, true);
foreach ($results['data']['web'] as $result) {
echo $result['title'] . ' - ' . $result['url'] . "\n";
}
```
```json theme={null}
{
"success": true,
"data": {
"web": [
{
"url": "https://docs.firecrawl.dev",
"title": "Firecrawl Documentation",
"markdown": "# Firecrawl\n\nFirecrawl is a web scraping API..."
}
]
}
}
```
## Scrape a page
```php theme={null}
true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => [
'Authorization: Bearer ' . $apiKey,
'Content-Type: application/json',
],
CURLOPT_POSTFIELDS => json_encode([
'url' => 'https://example.com',
]),
]);
$response = curl_exec($ch);
curl_close($ch);
$data = json_decode($response, true);
echo $data['data']['markdown'];
```
```json theme={null}
{
"success": true,
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
}
```
## Interact with a page
Start a browser session, interact with the page using natural-language prompts, then close the session.
### Step 1 — Scrape to start a session
```php theme={null}
true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => [
'Authorization: Bearer ' . $apiKey,
'Content-Type: application/json',
],
CURLOPT_POSTFIELDS => json_encode([
'url' => 'https://www.amazon.com',
'formats' => ['markdown'],
]),
]);
$response = curl_exec($ch);
curl_close($ch);
$scrapeResult = json_decode($response, true);
$scrapeId = $scrapeResult['data']['metadata']['scrapeId'];
echo "scrapeId: $scrapeId\n";
```
### Step 2 — Send interactions
```php theme={null}
true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_POSTFIELDS => json_encode([
'prompt' => 'Search for iPhone 16 Pro Max',
]),
]);
$response = curl_exec($ch);
curl_close($ch);
echo $response . "\n";
// Click on the first result
$ch = curl_init($interactUrl);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_POSTFIELDS => json_encode([
'prompt' => 'Click on the first result and tell me the price',
]),
]);
$response = curl_exec($ch);
curl_close($ch);
echo $response . "\n";
```
### Step 3 — Stop the session
```php theme={null}
true,
CURLOPT_CUSTOMREQUEST => 'DELETE',
CURLOPT_HTTPHEADER => [
'Authorization: Bearer ' . $apiKey,
],
]);
curl_exec($ch);
curl_close($ch);
echo "Session stopped\n";
```
## Reusable helper class
For repeated use, wrap the API in a class:
```php theme={null}
apiKey = $apiKey;
}
private function post(string $endpoint, array $payload): array
{
$ch = curl_init($this->baseUrl . $endpoint);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => [
'Authorization: Bearer ' . $this->apiKey,
'Content-Type: application/json',
],
CURLOPT_POSTFIELDS => json_encode($payload),
]);
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if ($httpCode >= 400) {
throw new \RuntimeException("Firecrawl API error: HTTP $httpCode");
}
return json_decode($response, true);
}
public function scrape(string $url, array $options = []): array
{
return $this->post('/scrape', array_merge(['url' => $url], $options));
}
public function search(string $query, int $limit = 5): array
{
return $this->post('/search', ['query' => $query, 'limit' => $limit]);
}
}
// Usage
$app = new Firecrawl(getenv('FIRECRAWL_API_KEY'));
$result = $app->scrape('https://example.com');
echo $result['data']['markdown'];
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Complete REST API documentation
# Python
Source: https://docs.firecrawl.dev/quickstarts/python
Get started with Firecrawl in Python. Scrape, search, and interact with web data using the official SDK.
## Prerequisites
* Python 3.8+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
pip install firecrawl-py
```
## Search the web
```python theme={null}
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR-API-KEY")
results = app.search("firecrawl web scraping", limit=5)
for result in results.web:
print(result.title, result.url)
```
## Scrape a page
```python theme={null}
result = app.scrape("https://example.com")
print(result.markdown)
```
```json theme={null}
{
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
```
## Interact with a page
Use Interact to control a live browser session — click buttons, fill forms, and extract dynamic content.
```python theme={null}
result = app.scrape("https://www.amazon.com", formats=["markdown"])
scrape_id = result.metadata.scrape_id
app.interact(scrape_id, prompt="Search for iPhone 16 Pro Max")
response = app.interact(scrape_id, prompt="Click on the first result and tell me the price")
print(response.output)
app.stop_interaction(scrape_id)
```
## Environment variable
Instead of passing `api_key` directly, set the `FIRECRAWL_API_KEY` environment variable:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
```python theme={null}
app = Firecrawl()
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, async, and more
# Rails
Source: https://docs.firecrawl.dev/quickstarts/rails
Use Firecrawl with Ruby on Rails to search, scrape, and interact with web data using the REST API.
## Prerequisites
* Ruby 3.0+ and Rails 7+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Configuration
Add your API key to your Rails credentials or environment:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Create a service
Create `app/services/firecrawl_service.rb`:
```ruby theme={null}
require "net/http"
require "json"
require "uri"
class FirecrawlService
BASE_URL = "https://api.firecrawl.dev/v2"
def initialize(api_key: ENV.fetch("FIRECRAWL_API_KEY"))
@api_key = api_key
end
def search(query, limit: 5)
post("/search", { query: query, limit: limit })
end
def scrape(url, **options)
post("/scrape", { url: url }.merge(options))
end
def interact(url, prompt, follow_up: nil)
# 1. Scrape to open a browser session
scrape_result = scrape(url, formats: ["markdown"])
scrape_id = scrape_result.dig("data", "metadata", "scrapeId")
# 2. Send first prompt
post("/scrape/#{scrape_id}/interact", { prompt: prompt })
# 3. Send follow-up prompt
result = nil
if follow_up
result = post("/scrape/#{scrape_id}/interact", { prompt: follow_up })
end
# 4. Close the session
delete("/scrape/#{scrape_id}/interact")
result || scrape_result
end
private
def post(endpoint, payload)
uri = URI("#{BASE_URL}#{endpoint}")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{@api_key}"
request["Content-Type"] = "application/json"
request.body = payload.to_json
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(request)
end
JSON.parse(response.body)
end
def delete(endpoint)
uri = URI("#{BASE_URL}#{endpoint}")
request = Net::HTTP::Delete.new(uri)
request["Authorization"] = "Bearer #{@api_key}"
Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(request)
end
end
end
```
## Create a controller
Generate a controller:
```bash theme={null}
rails generate controller Firecrawl search scrape interact --skip-routes
```
Edit `app/controllers/firecrawl_controller.rb`:
```ruby theme={null}
class FirecrawlController < ApplicationController
skip_before_action :verify_authenticity_token
def search
service = FirecrawlService.new
result = service.search(params.require(:query), limit: params.fetch(:limit, 5).to_i)
render json: result
end
def scrape
service = FirecrawlService.new
result = service.scrape(params.require(:url))
render json: result
end
def interact
service = FirecrawlService.new
result = service.interact(
params.require(:url),
params.require(:prompt),
follow_up: params[:followUp]
)
render json: result
end
end
```
## Add routes
In `config/routes.rb`:
```ruby theme={null}
Rails.application.routes.draw do
post "api/search", to: "firecrawl#search"
post "api/scrape", to: "firecrawl#scrape"
post "api/interact", to: "firecrawl#interact"
end
```
## Test it
```bash theme={null}
rails server
# Search the web
curl -X POST http://localhost:3000/api/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
# Scrape a page
curl -X POST http://localhost:3000/api/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# Interact with a page
curl -X POST http://localhost:3000/api/interact \
-H "Content-Type: application/json" \
-d '{"url": "https://www.amazon.com", "prompt": "Search for iPhone 16 Pro Max", "followUp": "Click on the first result and tell me the price"}'
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Complete REST API documentation
# Remix
Source: https://docs.firecrawl.dev/quickstarts/remix
Use Firecrawl with Remix to scrape, search, and interact with web data in your full-stack React app.
## Prerequisites
* Remix project
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
npm install @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
Use Firecrawl in an `action` to handle form submissions. Create `app/routes/search.tsx`:
```tsx theme={null}
import { json, type ActionFunctionArgs } from "@remix-run/node";
import { Form, useActionData } from "@remix-run/react";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function action({ request }: ActionFunctionArgs) {
const formData = await request.formData();
const query = formData.get("query") as string;
const results = await firecrawl.search(query, { limit: 5 });
return json({ results: (results.web || []).map((r) => ({ title: r.title, url: r.url })) });
}
export default function SearchPage() {
const data = useActionData();
return (
);
}
```
## Scrape a page
Use Firecrawl in a `loader` to fetch data at request time. Create `app/routes/scrape.tsx`:
```tsx theme={null}
import { json, type LoaderFunctionArgs } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function loader({ request }: LoaderFunctionArgs) {
const url = new URL(request.url);
const target = url.searchParams.get("url");
if (!target) return json({ markdown: null });
const result = await firecrawl.scrape(target);
return json({ markdown: result.markdown });
}
export default function ScrapePage() {
const { markdown } = useLoaderData();
return (
Scraped Content
{markdown ?
{markdown}
:
Pass ?url= to scrape a page
}
);
}
```
## Interact with a page
Use interact to control a live browser session — click buttons, fill forms, and extract dynamic content. Create `app/routes/interact.tsx`:
```tsx theme={null}
import { json, type ActionFunctionArgs } from "@remix-run/node";
import { Form, useActionData } from "@remix-run/react";
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function action({ request }: ActionFunctionArgs) {
const formData = await request.formData();
const url = formData.get("url") as string;
const result = await firecrawl.scrape(url, { formats: ['markdown'] });
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, { prompt: 'Search for iPhone 16 Pro Max' });
const response = await firecrawl.interact(scrapeId, { prompt: 'Click on the first result and tell me the price' });
await firecrawl.stopInteraction(scrapeId);
return json({ output: response.output });
}
export default function InteractPage() {
const data = useActionData();
return (
{data?.output &&
{data.output}
}
);
}
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Ruby
Source: https://docs.firecrawl.dev/quickstarts/ruby
Get started with Firecrawl in Ruby. Search, scrape, and interact with web data using the REST API.
## Prerequisites
* Ruby 3.0+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Search the web
Firecrawl works with Ruby through the REST API using `net/http`.
```ruby theme={null}
require "net/http"
require "json"
require "uri"
api_key = ENV.fetch("FIRECRAWL_API_KEY")
uri = URI("https://api.firecrawl.dev/v2/search")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{api_key}"
request["Content-Type"] = "application/json"
request.body = { query: "firecrawl web scraping", limit: 5 }.to_json
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(request) }
results = JSON.parse(response.body)
results["data"]["web"].each do |result|
puts "#{result['title']} - #{result['url']}"
end
```
## Scrape a page
```ruby theme={null}
uri = URI("https://api.firecrawl.dev/v2/scrape")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{api_key}"
request["Content-Type"] = "application/json"
request.body = { url: "https://example.com" }.to_json
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(request) }
data = JSON.parse(response.body)
puts data.dig("data", "markdown")
```
```json theme={null}
{
"success": true,
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
}
```
## Interact with a page
Scrape a page, then keep working with it using natural language prompts.
```ruby theme={null}
uri = URI("https://api.firecrawl.dev/v2/scrape")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{api_key}"
request["Content-Type"] = "application/json"
request.body = { url: "https://www.amazon.com", formats: ["markdown"] }.to_json
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(request) }
scrape_id = JSON.parse(response.body).dig("data", "metadata", "scrapeId")
interact_uri = URI("https://api.firecrawl.dev/v2/scrape/#{scrape_id}/interact")
interact_req = Net::HTTP::Post.new(interact_uri)
interact_req["Authorization"] = "Bearer #{api_key}"
interact_req["Content-Type"] = "application/json"
interact_req.body = { prompt: "Search for iPhone 16 Pro Max" }.to_json
interact_resp = Net::HTTP.start(interact_uri.hostname, interact_uri.port, use_ssl: true) { |http| http.request(interact_req) }
puts JSON.parse(interact_resp.body)
# Stop the session
delete_uri = URI("https://api.firecrawl.dev/v2/scrape/#{scrape_id}/interact")
delete_req = Net::HTTP::Delete.new(delete_uri)
delete_req["Authorization"] = "Bearer #{api_key}"
Net::HTTP.start(delete_uri.hostname, delete_uri.port, use_ssl: true) { |http| http.request(delete_req) }
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Complete REST API documentation
Click, fill forms, and extract dynamic content
# Rust
Source: https://docs.firecrawl.dev/quickstarts/rust
Get started with Firecrawl in Rust. Search, scrape, and interact with web data using the official SDK.
## Prerequisites
* Rust 1.70+ with Cargo
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the crate
Add `firecrawl` to your `Cargo.toml`:
```toml theme={null}
[dependencies]
firecrawl = "2"
tokio = { version = "1", features = ["full"] }
serde_json = "1"
```
## Search the web
```rust theme={null}
use firecrawl::{Client, SearchOptions};
#[tokio::main]
async fn main() -> Result<(), Box> {
let client = Client::new("fc-YOUR-API-KEY")?;
let results = client.search(
"firecrawl web scraping",
SearchOptions { limit: Some(5), ..Default::default() },
).await?;
if let Some(web) = results.data.web {
for item in web {
if let firecrawl::SearchResultOrDocument::WebResult(r) = item {
println!("{} - {}", r.url, r.title.unwrap_or_default());
}
}
}
Ok(())
}
```
## Scrape a page
```rust theme={null}
let doc = client.scrape("https://example.com", None).await?;
println!("{}", doc.markdown.unwrap_or_default());
```
```json theme={null}
{
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com"
}
}
```
## Interact with a page
Scrape a page to get a `scrapeId`, then use the interact API to control the browser session:
```rust theme={null}
use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions};
let doc = client.scrape(
"https://www.amazon.com",
ScrapeOptions {
formats: Some(vec![Format::Markdown]),
..Default::default()
},
).await?;
let scrape_id = doc.metadata
.as_ref()
.and_then(|m| m.scrape_id.as_deref())
.expect("scrapeId not found");
// Send a prompt to interact with the page
let run = client.interact(
scrape_id,
ScrapeExecuteOptions {
prompt: Some("Search for iPhone 16 Pro Max".to_string()),
..Default::default()
},
).await?;
let run = client.interact(
scrape_id,
ScrapeExecuteOptions {
prompt: Some("Click on the first result and tell me the price".to_string()),
..Default::default()
},
).await?;
println!("{:?}", run.output);
// Close the session
client.stop_interaction(scrape_id).await?;
```
## Environment variable
Set `FIRECRAWL_API_KEY` instead of passing the key directly:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
```rust theme={null}
let api_key = std::env::var("FIRECRAWL_API_KEY")?;
let client = Client::new(api_key)?;
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Spring Boot
Source: https://docs.firecrawl.dev/quickstarts/spring-boot
Use Firecrawl with Spring Boot to search, scrape, and interact with web data using the official Java SDK.
## Prerequisites
* Java 17+ and Spring Boot 3+
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Add the dependency
```kotlin theme={null}
dependencies {
implementation("com.firecrawl:firecrawl-java:1.2.0")
}
```
```xml theme={null}
com.firecrawlfirecrawl-java1.2.0
```
## Configuration
Add your API key to `application.properties`:
```properties theme={null}
firecrawl.api-key=${FIRECRAWL_API_KEY}
```
Or set it as an environment variable:
```bash theme={null}
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Create a configuration bean
Create `FirecrawlConfig.java`:
```java theme={null}
import com.firecrawl.client.FirecrawlClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class FirecrawlConfig {
@Bean
public FirecrawlClient firecrawlClient(
@Value("${firecrawl.api-key}") String apiKey) {
return FirecrawlClient.builder()
.apiKey(apiKey)
.build();
}
}
```
## Create a REST controller
Create `FirecrawlController.java`:
```java theme={null}
import com.firecrawl.client.FirecrawlClient;
import com.firecrawl.models.Document;
import com.firecrawl.models.SearchData;
import com.firecrawl.models.SearchOptions;
import com.firecrawl.models.ScrapeOptions;
import com.firecrawl.models.BrowserExecuteResponse;
import org.springframework.web.bind.annotation.*;
import java.util.List;
import java.util.Map;
@RestController
@RequestMapping("/api")
public class FirecrawlController {
private final FirecrawlClient firecrawl;
public FirecrawlController(FirecrawlClient firecrawl) {
this.firecrawl = firecrawl;
}
@PostMapping("/search")
public SearchData search(@RequestBody Map body) {
return firecrawl.search(
(String) body.get("query"),
SearchOptions.builder()
.limit((int) body.getOrDefault("limit", 5))
.build()
);
}
@PostMapping("/scrape")
public Map scrape(@RequestBody Map body) {
Document doc = firecrawl.scrape(body.get("url"));
return Map.of(
"markdown", doc.getMarkdown(),
"metadata", doc.getMetadata()
);
}
@PostMapping("/interact")
public Map interact(@RequestBody Map body) {
Document doc = firecrawl.scrape(body.get("url"),
ScrapeOptions.builder().formats(List.of((Object) "markdown")).build());
String scrapeId = (String) doc.getMetadata().get("scrapeId");
BrowserExecuteResponse response = firecrawl.interact(scrapeId,
body.getOrDefault("code", "const title = await page.title(); console.log(title);"));
firecrawl.stopInteractiveBrowser(scrapeId);
return Map.of("result", response.getStdout());
}
}
```
## Run it
```bash theme={null}
./gradlew bootRun
```
## Test it
```bash theme={null}
# Search the web
curl -X POST http://localhost:8080/api/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
# Scrape a page
curl -X POST http://localhost:8080/api/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# Interact with a page
curl -X POST http://localhost:8080/api/interact \
-H "Content-Type: application/json" \
-d '{"url": "https://www.amazon.com", "code": "const title = await page.title(); console.log(title);"}'
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Supabase Edge Functions
Source: https://docs.firecrawl.dev/quickstarts/supabase-edge-functions
Use Firecrawl with Supabase Edge Functions to search, scrape, and interact with web data at the edge.
## Prerequisites
* Supabase project with CLI (`supabase init`)
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
supabase functions new firecrawl-search
supabase functions new firecrawl-scrape
supabase functions new firecrawl-interact
```
Set the secret:
```bash theme={null}
supabase secrets set FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
Edit `supabase/functions/firecrawl-search/index.ts`:
```typescript theme={null}
import Firecrawl from "npm:@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: Deno.env.get("FIRECRAWL_API_KEY"),
});
Deno.serve(async (req) => {
const { query } = await req.json();
const results = await firecrawl.search(query, { limit: 5 });
return new Response(JSON.stringify(results), {
headers: { "Content-Type": "application/json" },
});
});
```
## Scrape a page
Edit `supabase/functions/firecrawl-scrape/index.ts`:
```typescript theme={null}
import Firecrawl from "npm:@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: Deno.env.get("FIRECRAWL_API_KEY"),
});
Deno.serve(async (req) => {
const { url } = await req.json();
const result = await firecrawl.scrape(url);
return new Response(JSON.stringify(result), {
headers: { "Content-Type": "application/json" },
});
});
```
## Interact with a page
Edit `supabase/functions/firecrawl-interact/index.ts`:
```typescript theme={null}
import Firecrawl from "npm:@mendable/firecrawl-js";
const firecrawl = new Firecrawl({
apiKey: Deno.env.get("FIRECRAWL_API_KEY"),
});
Deno.serve(async (_req) => {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
console.log(response.output);
await firecrawl.stopInteraction(scrapeId);
return new Response(JSON.stringify({ output: response.output }), {
headers: { "Content-Type": "application/json" },
});
});
```
## Deploy
```bash theme={null}
supabase functions deploy firecrawl-search
supabase functions deploy firecrawl-scrape
supabase functions deploy firecrawl-interact
```
## Test it
```bash theme={null}
curl -X POST https://.supabase.co/functions/v1/firecrawl-search \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# SvelteKit
Source: https://docs.firecrawl.dev/quickstarts/sveltekit
Use Firecrawl with SvelteKit to scrape, search, and interact with web data in your Svelte application.
## Prerequisites
* SvelteKit project
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Install the SDK
```bash theme={null}
npm install @mendable/firecrawl-js
```
Add your API key to `.env`:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
Create a form action in `src/routes/search/+page.server.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
import { FIRECRAWL_API_KEY } from "$env/static/private";
const firecrawl = new Firecrawl({ apiKey: FIRECRAWL_API_KEY });
export const actions = {
default: async ({ request }) => {
const data = await request.formData();
const query = data.get("query") as string;
const results = await firecrawl.search(query, { limit: 5 });
return { results: (results.web || []).map((r) => ({ title: r.title, url: r.url })) };
},
};
```
Wire it up in `src/routes/search/+page.svelte`:
```svelte theme={null}
{#if form?.results}
{#each form.results as result}
{/each}
{/if}
```
## Scrape a page
Fetch data in a load function at `src/routes/scrape/+page.server.ts`:
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
import { FIRECRAWL_API_KEY } from "$env/static/private";
const firecrawl = new Firecrawl({ apiKey: FIRECRAWL_API_KEY });
export async function load({ url }) {
const target = url.searchParams.get("url");
if (!target) return { markdown: null };
const result = await firecrawl.scrape(target);
return { markdown: result.markdown };
}
```
Display it in `src/routes/scrape/+page.svelte`:
```svelte theme={null}
{#if data.markdown}
{data.markdown}
{:else}
Pass ?url= to scrape a page
{/if}
```
## Interact with a page
Create a server endpoint at `src/routes/api/interact/+server.ts`:
```typescript theme={null}
import { json } from "@sveltejs/kit";
import Firecrawl from "@mendable/firecrawl-js";
import { FIRECRAWL_API_KEY } from "$env/static/private";
const firecrawl = new Firecrawl({ apiKey: FIRECRAWL_API_KEY });
export async function POST() {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
await firecrawl.stopInteraction(scrapeId);
return json({ output: response.output });
}
```
## Next steps
All scrape options including formats, actions, and proxies
Search the web and get full page content
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Vercel Functions
Source: https://docs.firecrawl.dev/quickstarts/vercel-functions
Use Firecrawl with Vercel Functions to search, scrape, and interact with web data in serverless deployments.
## Prerequisites
* Vercel project (Next.js, SvelteKit, Nuxt, or standalone)
* A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys)
## Setup
```bash theme={null}
npm install @mendable/firecrawl-js
```
Add `FIRECRAWL_API_KEY` as an environment variable in your Vercel project settings, or in `.env.local` for local development:
```bash theme={null}
FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
## Search the web
Create `api/search.ts` (or `app/api/search/route.ts` for Next.js):
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function POST(request: Request) {
const { query } = await request.json();
const results = await firecrawl.search(query, { limit: 5 });
return new Response(JSON.stringify(results), {
headers: { "Content-Type": "application/json" },
});
}
```
## Scrape a page
Create `api/scrape.ts` (or `app/api/scrape/route.ts` for Next.js):
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function POST(request: Request) {
const { url } = await request.json();
const result = await firecrawl.scrape(url);
return new Response(JSON.stringify(result), {
headers: { "Content-Type": "application/json" },
});
}
```
## Interact with a page
Create `api/interact.ts` (or `app/api/interact/route.ts` for Next.js):
```typescript theme={null}
import Firecrawl from "@mendable/firecrawl-js";
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
export async function POST(request: Request) {
const result = await firecrawl.scrape("https://www.amazon.com", {
formats: ["markdown"],
});
const scrapeId = result.metadata?.scrapeId;
await firecrawl.interact(scrapeId, {
prompt: "Search for iPhone 16 Pro Max",
});
const response = await firecrawl.interact(scrapeId, {
prompt: "Click on the first result and tell me the price",
});
await firecrawl.stopInteraction(scrapeId);
return new Response(JSON.stringify({ output: response.output }), {
headers: { "Content-Type": "application/json" },
});
}
```
## Deploy
```bash theme={null}
vercel deploy
```
## Test it
```bash theme={null}
curl -X POST https://your-project.vercel.app/api/search \
-H "Content-Type: application/json" \
-d '{"query": "firecrawl web scraping"}'
```
Vercel Functions have a default timeout of 10 seconds on the Hobby plan and 60 seconds on Pro. For large crawl jobs, use the Firecrawl async API with webhooks instead.
## Next steps
Search the web and get full page content
All scrape options including formats, actions, and proxies
Click, fill forms, and extract dynamic content
Full SDK reference with crawl, map, batch scrape, and more
# Rate Limits
Source: https://docs.firecrawl.dev/rate-limits
Rate limits for different pricing plans and API requests
## Billing Model
Firecrawl uses subscription-based monthly plans. There is no pure pay-as-you-go model, but the **auto-recharge feature** provides flexible scaling. Once you subscribe to a plan, you can automatically purchase additional credits when you dip below a threshold. Larger auto-recharge packs offer better rates. To test before committing to a larger plan, start with the Free or Hobby tier.
Plan downgrades take effect at the next renewal. Unused-time credits are not issued.
## Concurrent Browser Limits
Concurrent browsers represent how many web pages Firecrawl can process for you at the same time.
Your plan determines how many of these jobs can run simultaneously - if you exceed this limit,
additional jobs will wait in a queue until resources become available. Note that time spent waiting in the queue counts against the request's [`timeout`](/advanced-scraping-guide#timing-and-cache) parameter, so you can set a lower timeout to fail fast instead of waiting. You can also check current availability via the [Queue Status](/api-reference/endpoint/queue-status) endpoint.
### Current Plans
Plan
Concurrent Browsers
Max Queued Jobs
Free
2
50,000
Hobby
5
50,000
Standard
50
100,000
Growth
100
200,000
Scale / Enterprise
150+
300,000+
Each team has a maximum number of jobs that can be waiting in the concurrency queue. If you exceed this limit, new jobs will be rejected with a `429` status code until existing jobs complete. For larger plans with custom concurrency limits, the max queued jobs is 2,000 times your concurrency limit, capped at 2,000,000.
If you require higher concurrency limits, [contact us about enterprise plans](https://firecrawl.dev/enterprise).
### Extract Plans (Legacy)
Plan
Concurrent Browsers
Max Queued Jobs
Free
2
50,000
Starter
50
100,000
Explorer
100
200,000
Pro
200
400,000
## API Rate Limits
Rate limits are measured in requests per minute and are primarily in place to prevent abuse. When configured correctly, your real bottleneck will be concurrent browsers. Rate limits are applied per team, so all API keys on the same team share the same rate limit counters.
### Current Plans
Plan
/scrape
/map
/crawl
/search
/agent
/crawl/status
/agent/status
Free
10
10
1
5
10
1500
500
Hobby
100
100
15
50
100
1500
25000
Standard
500
500
50
250
500
1500
25000
Growth
5000
5000
250
2500
1000
1500
25000
Scale
7500
7500
750
7500
1000
25000
25000
These rate limits are enforced to ensure fair usage and availability of the API for all users. If you require higher limits, please contact us at [help@firecrawl.com](mailto:help@firecrawl.com) to discuss custom plans.
### Extract Endpoints
The extract endpoints share limits with the corresponding /agent rate limits.
### Batch Scrape Endpoints
The batch scrape endpoints share limits with the corresponding /crawl rate limits.
### Browser Sandbox
The browser sandbox endpoints have per-plan rate limits that scale with your subscription:
Plan
/browser
/browser/\{id}/execute
Free
2
10
Hobby
20
100
Standard
100
500
Growth
1,000
5,000
Scale
1,500
7,500
In addition, each team's plan determines how many browser sessions can be active simultaneously (see Concurrent Browser Limits above). If you exceed this limit, new session requests will return a `429` status code until existing sessions are destroyed.
### FIRE-1 Agent
Requests involving the FIRE-1 agent requests have separate rate limits that are counted independently for each endpoint:
Endpoint
Rate Limit (requests/min)
/scrape
10
/extract
10
### Extract Plans (Legacy)
Plan
/extract (requests/min)
/extract/status (requests/min)
Starter
100
25000
Explorer
500
25000
Pro
1000
25000
# Skill + CLI
Source: https://docs.firecrawl.dev/sdks/cli
Firecrawl Skill is an easy way for AI agents such as Claude Code, Antigravity and OpenCode to use Firecrawl through the CLI.
Search, scrape, and interact with the web directly from the terminal. The Firecrawl CLI works standalone or as a skill that AI coding agents like Claude Code, Antigravity, and OpenCode can discover and use automatically.
## Installation
If you are using an AI agent like Claude Code, you can install the Firecrawl skill below and the agent will set it up for you.
```bash theme={null}
npx -y firecrawl-cli@latest init --all --browser
```
* `--all` installs the Firecrawl skill to every detected AI coding agent
* `--browser` opens the browser for Firecrawl authentication automatically
After installing the skill, restart your agent for it to discover the new skill.
You can also manually install the Firecrawl CLI globally using npm:
```bash CLI theme={null}
# Install globally with npm
npm install -g firecrawl-cli
```
## Authentication
Before using the CLI, you need to authenticate with your Firecrawl API key.
### Login
```bash CLI theme={null}
# Interactive login (opens browser or prompts for API key)
firecrawl login
# Login with browser authentication (recommended for agents)
firecrawl login --browser
# Login with API key directly
firecrawl login --api-key fc-YOUR-API-KEY
# Or set via environment variable
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
### View Configuration
```bash CLI theme={null}
# View current configuration and authentication status
firecrawl view-config
```
### Logout
```bash CLI theme={null}
# Clear stored credentials
firecrawl logout
```
### Self-Hosted / Local Development
For self-hosted Firecrawl instances or local development, use the `--api-url` option:
```bash CLI theme={null}
# Use a local Firecrawl instance (no API key required)
firecrawl --api-url http://localhost:3002 scrape https://example.com
# Or set via environment variable
export FIRECRAWL_API_URL=http://localhost:3002
firecrawl scrape https://example.com
# Configure and persist the custom API URL
firecrawl config --api-url http://localhost:3002
```
When using a custom API URL (anything other than `https://api.firecrawl.dev`), API key authentication is automatically skipped, allowing you to use local instances without an API key.
### Check Status
Verify installation, authentication, and view rate limits:
```bash CLI theme={null}
firecrawl --status
```
Output when ready:
```
🔥 firecrawl cli v1.1.1
● Authenticated via FIRECRAWL_API_KEY
Concurrency: 0/100 jobs (parallel scrape limit)
Credits: 500,000 remaining
```
* **Concurrency**: Maximum parallel jobs. Run parallel operations close to this limit but not above.
* **Credits**: Remaining API credits. Each scrape/crawl consumes credits.
## Commands
### Scrape
Scrape a single URL and extract its content in various formats.
Use `--only-main-content` to get clean output without navigation, footers, and ads. This is recommended for most use cases where you want just the article or main page content.
```bash CLI theme={null}
# Scrape a URL (default: markdown output)
firecrawl https://example.com
# Or use the explicit scrape command
firecrawl scrape https://example.com
# Recommended: use --only-main-content for clean output without nav/footer
firecrawl https://example.com --only-main-content
```
#### Output Formats
```bash CLI theme={null}
# Get HTML output
firecrawl https://example.com --html
# Multiple formats (returns JSON)
firecrawl https://example.com --format markdown,links
# Get images from a page
firecrawl https://example.com --format images
# Get a summary of the page content
firecrawl https://example.com --format summary
# Track changes on a page
firecrawl https://example.com --format changeTracking
# Available formats: markdown, html, rawHtml, links, screenshot, json, images, summary, changeTracking, attributes, branding
```
#### Scrape Options
```bash CLI theme={null}
# Extract only main content (removes navs, footers)
firecrawl https://example.com --only-main-content
# Wait for JavaScript rendering
firecrawl https://example.com --wait-for 3000
# Take a screenshot
firecrawl https://example.com --screenshot
# Include/exclude specific HTML tags
firecrawl https://example.com --include-tags article,main
firecrawl https://example.com --exclude-tags nav,footer
# Save output to file
firecrawl https://example.com -o output.md
# Pretty print JSON output
firecrawl https://example.com --format markdown,links --pretty
# Force JSON output even with single format
firecrawl https://example.com --json
# Show request timing information
firecrawl https://example.com --timing
```
**Available Options:**
| Option | Short | Description |
| ----------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--url ` | `-u` | URL to scrape (alternative to positional argument) |
| `--format ` | `-f` | Output formats (comma-separated): `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `json`, `images`, `summary`, `changeTracking`, `attributes`, `branding` |
| `--html` | `-H` | Shortcut for `--format html` |
| `--only-main-content` | | Extract only main content |
| `--wait-for ` | | Wait time in milliseconds for JS rendering |
| `--screenshot` | | Take a screenshot |
| `--include-tags ` | | HTML tags to include (comma-separated) |
| `--exclude-tags ` | | HTML tags to exclude (comma-separated) |
| `--output ` | `-o` | Save output to file |
| `--json` | | Force JSON output even with single format |
| `--pretty` | | Pretty print JSON output |
| `--timing` | | Show request timing and other useful information |
***
### Search
Search the web and optionally scrape the results.
```bash CLI theme={null}
# Search the web
firecrawl search "web scraping tutorials"
# Limit results
firecrawl search "AI news" --limit 10
# Pretty print results
firecrawl search "machine learning" --pretty
```
#### Search Options
```bash CLI theme={null}
# Search specific sources
firecrawl search "AI" --sources web,news,images
# Search with category filters
firecrawl search "react hooks" --categories github
firecrawl search "machine learning" --categories research,pdf
# Time-based filtering
firecrawl search "tech news" --tbs qdr:h # Last hour
firecrawl search "tech news" --tbs qdr:d # Last day
firecrawl search "tech news" --tbs qdr:w # Last week
firecrawl search "tech news" --tbs qdr:m # Last month
firecrawl search "tech news" --tbs qdr:y # Last year
# Location-based search
firecrawl search "restaurants" --location "Berlin,Germany" --country DE
# Search and scrape results
firecrawl search "documentation" --scrape --scrape-formats markdown
# Save to file
firecrawl search "firecrawl" --pretty -o results.json
```
**Available Options:**
| Option | Description |
| ---------------------------- | ------------------------------------------------------------------------------------------- |
| `--limit ` | Maximum results (default: 5, max: 100) |
| `--sources ` | Sources to search: `web`, `images`, `news` (comma-separated) |
| `--categories ` | Filter by category: `github`, `research`, `pdf` (comma-separated) |
| `--tbs ` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |
| `--location ` | Geo-targeting (e.g., "Berlin,Germany") |
| `--country ` | ISO country code (default: US) |
| `--timeout ` | Timeout in milliseconds (default: 60000) |
| `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints |
| `--scrape` | Scrape search results |
| `--scrape-formats ` | Formats for scraped content (default: markdown) |
| `--only-main-content` | Include only main content when scraping (default: true) |
| `--json` | Output as JSON |
| `--output ` | Save output to file |
| `--pretty` | Pretty print JSON output |
***
### Map
Discover all URLs on a website quickly.
```bash CLI theme={null}
# Discover all URLs on a website
firecrawl map https://example.com
# Output as JSON
firecrawl map https://example.com --json
# Limit number of URLs
firecrawl map https://example.com --limit 500
```
#### Map Options
```bash CLI theme={null}
# Filter URLs by search query
firecrawl map https://example.com --search "blog"
# Include subdomains
firecrawl map https://example.com --include-subdomains
# Control sitemap usage
firecrawl map https://example.com --sitemap include # Use sitemap
firecrawl map https://example.com --sitemap skip # Skip sitemap
firecrawl map https://example.com --sitemap only # Only use sitemap
# Ignore query parameters (dedupe URLs)
firecrawl map https://example.com --ignore-query-parameters
# Wait for map to complete with timeout
firecrawl map https://example.com --wait --timeout 60
# Save to file
firecrawl map https://example.com -o urls.txt
firecrawl map https://example.com --json --pretty -o urls.json
```
**Available Options:**
| Option | Description |
| --------------------------- | ----------------------------------------------- |
| `--url ` | URL to map (alternative to positional argument) |
| `--limit ` | Maximum URLs to discover |
| `--search ` | Filter URLs by search query |
| `--sitemap ` | Sitemap handling: `include`, `skip`, `only` |
| `--include-subdomains` | Include subdomains |
| `--ignore-query-parameters` | Treat URLs with different params as same |
| `--wait` | Wait for map to complete |
| `--timeout ` | Timeout in seconds |
| `--json` | Output as JSON |
| `--output ` | Save output to file |
| `--pretty` | Pretty print JSON output |
***
### Browser
Have your agents interact with the web using a secure browser sandbox.
Launch cloud browser sessions and execute Python, JavaScript, or bash code remotely. Each session runs a full Chromium instance — no local browser install required. Code runs server-side with a pre-configured [Playwright](https://playwright.dev/) `page` object ready to use.
```bash CLI theme={null}
# Launch a cloud browser session
firecrawl browser launch-session
# Execute agent-browser commands (default - "agent-browser" is auto-prefixed)
firecrawl browser execute "open https://example.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "click @e5"
firecrawl browser execute "scrape"
# Execute Playwright Python code
firecrawl browser execute --python 'await page.goto("https://example.com")
print(await page.title())'
# Execute Playwright JavaScript code
firecrawl browser execute --node 'await page.goto("https://example.com"); console.log(await page.title());'
# List all sessions (or: list active / list destroyed)
firecrawl browser list
# Close the active session
firecrawl browser close
```
#### Browser Options
```bash CLI theme={null}
# Launch with custom TTL (10 minutes) and live view
firecrawl browser launch-session --ttl 600 --stream
# Launch with inactivity timeout
firecrawl browser launch-session --ttl 120 --ttl-inactivity 60
# agent-browser commands (default - "agent-browser" is auto-prefixed)
firecrawl browser execute "open https://news.ycombinator.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "click @e3"
firecrawl browser execute "scrape"
# Playwright Python - navigate, interact, extract
firecrawl browser execute --python '
await page.goto("https://news.ycombinator.com")
items = await page.query_selector_all(".titleline > a")
for item in items[:5]:
print(await item.text_content())
'
# Playwright JavaScript - same page object
firecrawl browser execute --node '
await page.goto("https://example.com");
const title = await page.title();
console.log(title);
'
# Explicit bash mode - runs in the sandbox
firecrawl browser execute --bash "agent-browser snapshot"
# Target a specific session
firecrawl browser execute --session --python 'print(await page.title())'
# Save output to file
firecrawl browser execute "scrape" -o result.txt
# Close a specific session
firecrawl browser close --session
# List sessions (all / active / destroyed)
firecrawl browser list
firecrawl browser list active --json
```
**Subcommands:**
| Subcommand | Description |
| ---------------- | ----------------------------------------------------------------------------------- |
| `launch-session` | Launch a new cloud browser session (returns session ID, CDP URL, and live view URL) |
| `execute ` | Execute Playwright Python/JS code or bash commands in a session |
| `list [status]` | List browser sessions (filter by `active` or `destroyed`) |
| `close` | Close a browser session |
**Execute Options:**
| Option | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--bash` | Execute bash commands remotely in the sandbox (default). [agent-browser](https://github.com/vercel-labs/agent-browser) (40+ commands) is pre-installed and auto-prefixed. `CDP_URL` is auto-injected so agent-browser connects to your session automatically. Best approach for AI agents. |
| `--python` | Execute as Playwright Python code. A Playwright `page` object is available — use `await page.goto()`, `await page.title()`, etc. |
| `--node` | Execute as Playwright JavaScript code. Same `page` object available. |
| `--session ` | Target a specific session (default: active session) |
**Launch Options:**
| Option | Description |
| ---------------------------- | ------------------------------------------------------------------- |
| `--ttl ` | Total session TTL (default: 600, range: 30–3600) |
| `--ttl-inactivity ` | Auto-close after inactivity (range: 10–3600) |
| `--profile ` | Name for a profile (saves and reuses browser state across sessions) |
| `--no-save-changes` | Load existing profile data without saving changes back |
| `--stream` | Enable live view streaming |
**Common Options:**
| Option | Description |
| ----------------- | --------------------- |
| `--output ` | Save output to file |
| `--json` | Output as JSON format |
***
### Crawl
Crawl an entire website starting from a URL.
```bash CLI theme={null}
# Start a crawl (returns job ID immediately)
firecrawl crawl https://example.com
# Wait for crawl to complete
firecrawl crawl https://example.com --wait
# Wait with progress indicator
firecrawl crawl https://example.com --wait --progress
```
#### Check Crawl Status
```bash CLI theme={null}
# Check crawl status using job ID
firecrawl crawl
# Example with a real job ID
firecrawl crawl 550e8400-e29b-41d4-a716-446655440000
```
#### Crawl Options
```bash CLI theme={null}
# Limit crawl depth and pages
firecrawl crawl https://example.com --limit 100 --max-depth 3 --wait
# Include only specific paths
firecrawl crawl https://example.com --include-paths /blog,/docs --wait
# Exclude specific paths
firecrawl crawl https://example.com --exclude-paths /admin,/login --wait
# Include subdomains
firecrawl crawl https://example.com --allow-subdomains --wait
# Crawl entire domain
firecrawl crawl https://example.com --crawl-entire-domain --wait
# Rate limiting
firecrawl crawl https://example.com --delay 1000 --max-concurrency 2 --wait
# Custom polling interval and timeout
firecrawl crawl https://example.com --wait --poll-interval 10 --timeout 300
# Save results to file
firecrawl crawl https://example.com --wait --pretty -o results.json
```
**Available Options:**
| Option | Description |
| --------------------------- | ------------------------------------------------- |
| `--url ` | URL to crawl (alternative to positional argument) |
| `--wait` | Wait for crawl to complete |
| `--progress` | Show progress indicator while waiting |
| `--poll-interval ` | Polling interval (default: 5) |
| `--timeout ` | Timeout when waiting |
| `--status` | Check status of existing crawl job |
| `--limit ` | Maximum pages to crawl |
| `--max-depth ` | Maximum crawl depth |
| `--include-paths ` | Paths to include (comma-separated) |
| `--exclude-paths ` | Paths to exclude (comma-separated) |
| `--sitemap ` | Sitemap handling: `include`, `skip`, `only` |
| `--allow-subdomains` | Include subdomains |
| `--allow-external-links` | Follow external links |
| `--crawl-entire-domain` | Crawl entire domain |
| `--ignore-query-parameters` | Treat URLs with different params as same |
| `--delay ` | Delay between requests |
| `--max-concurrency ` | Maximum concurrent requests |
| `--output ` | Save output to file |
| `--pretty` | Pretty print JSON output |
***
### Agent
Search and gather data from the web using natural language prompts.
```bash CLI theme={null}
# Basic usage - URLs are optional
firecrawl agent "Find the top 5 AI startups and their funding amounts" --wait
# Focus on specific URLs
firecrawl agent "Compare pricing plans" --urls https://slack.com/pricing,https://teams.microsoft.com/pricing --wait
# Use a schema for structured output
firecrawl agent "Get company information" --urls https://example.com --schema '{"name": "string", "founded": "number"}' --wait
# Use schema from a file
firecrawl agent "Get product details" --urls https://example.com --schema-file schema.json --wait
```
#### Agent Options
```bash CLI theme={null}
# Use Spark 1 Pro for higher accuracy
firecrawl agent "Competitive analysis across multiple domains" --model spark-1-pro --wait
# Set max credits to limit costs
firecrawl agent "Gather contact information from company websites" --max-credits 100 --wait
# Check status of an existing job
firecrawl agent --status
# Custom polling interval and timeout
firecrawl agent "Summarize recent blog posts" --wait --poll-interval 10 --timeout 300
# Save output to file
firecrawl agent "Find pricing information" --urls https://example.com --wait -o pricing.json --pretty
```
**Available Options:**
| Option | Description |
| --------------------------- | -------------------------------------------------------------------------------------- |
| `--urls ` | Optional list of URLs to focus the agent on (comma-separated) |
| `--model ` | Model to use: `spark-1-mini` (default, 60% cheaper) or `spark-1-pro` (higher accuracy) |
| `--schema ` | JSON schema for structured output (inline JSON string) |
| `--schema-file ` | Path to JSON schema file for structured output |
| `--max-credits ` | Maximum credits to spend (job fails if limit reached) |
| `--status` | Check status of existing agent job |
| `--wait` | Wait for agent to complete before returning results |
| `--poll-interval ` | Polling interval when waiting (default: 5) |
| `--timeout ` | Timeout when waiting (default: no timeout) |
| `--output ` | Save output to file |
| `--json` | Output as JSON format |
***
### Credit Usage
Check your team's credit balance and usage.
```bash CLI theme={null}
# View credit usage
firecrawl credit-usage
# Output as JSON
firecrawl credit-usage --json --pretty
```
***
### Version
Display the CLI version.
```bash CLI theme={null}
firecrawl version
# or
firecrawl --version
```
## Global Options
These options are available for all commands:
| Option | Short | Description |
| ----------------- | ----- | ------------------------------------------------------ |
| `--status` | | Show version, auth, concurrency, and credits |
| `--api-key ` | `-k` | Override stored API key for this command |
| `--api-url ` | | Use custom API URL (for self-hosted/local development) |
| `--help` | `-h` | Show help for a command |
| `--version` | `-V` | Show CLI version |
## Output Handling
The CLI outputs to stdout by default, making it easy to pipe or redirect:
```bash CLI theme={null}
# Pipe markdown to another command
firecrawl https://example.com | head -50
# Redirect to a file
firecrawl https://example.com > output.md
# Save JSON with pretty formatting
firecrawl https://example.com --format markdown,links --pretty -o data.json
```
### Format Behavior
* **Single format**: Outputs raw content (markdown text, HTML, etc.)
* **Multiple formats**: Outputs JSON with all requested data
```bash CLI theme={null}
# Raw markdown output
firecrawl https://example.com --format markdown
# JSON output with multiple formats
firecrawl https://example.com --format markdown,links
```
## Examples
### Quick Scrape
```bash CLI theme={null}
# Get markdown content from a URL (use --only-main-content for clean output)
firecrawl https://docs.firecrawl.dev --only-main-content
# Get HTML content
firecrawl https://example.com --html -o page.html
```
### Full Site Crawl
```bash CLI theme={null}
# Crawl a docs site with limits
firecrawl crawl https://docs.example.com --limit 50 --max-depth 2 --wait --progress -o docs.json
```
### Site Discovery
```bash CLI theme={null}
# Find all blog posts
firecrawl map https://example.com --search "blog" -o blog-urls.txt
```
### Research Workflow
```bash CLI theme={null}
# Search and scrape results for research
firecrawl search "machine learning best practices 2024" --scrape --scrape-formats markdown --pretty
```
### Agent
```bash CLI theme={null}
# URLs are optional
firecrawl agent "Find the top 5 AI startups and their funding amounts" --wait
# Focus on specific URLs
firecrawl agent "Compare pricing plans" --urls https://slack.com/pricing,https://teams.microsoft.com/pricing --wait
```
### Browser Automation
```bash CLI theme={null}
# Launch a session, scrape a page, and close
firecrawl browser launch-session
firecrawl browser execute "open https://news.ycombinator.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "scrape"
firecrawl browser close
# Use agent-browser via bash mode (default — recommended for AI agents)
firecrawl browser launch-session
firecrawl browser execute "open https://example.com"
firecrawl browser execute "snapshot"
# snapshot returns @ref IDs — use them to interact
firecrawl browser execute "click @e5"
firecrawl browser execute "fill @e3 'search query'"
firecrawl browser execute "scrape"
# Run --help to see all 40+ commands
firecrawl browser execute --bash "agent-browser --help"
firecrawl browser close
```
### Combine with Other Tools
```bash CLI theme={null}
# Extract URLs from search results
jq -r '.data.web[].url' search-results.json
# Get titles from search results
jq -r '.data.web[] | "\(.title): \(.url)"' search-results.json
# Extract links and process with jq
firecrawl https://example.com --format links | jq '.links[].url'
# Count URLs from map
firecrawl map https://example.com | wc -l
```
## Telemetry
The CLI collects anonymous usage data during authentication to help improve the product:
* CLI version, OS, and Node.js version
* Development tool detection (e.g., Cursor, VS Code, Claude Code)
**No command data, URLs, or file contents are collected via the CLI.**
To disable telemetry, set the environment variable:
```bash CLI theme={null}
export FIRECRAWL_NO_TELEMETRY=1
```
## Open Source
The Firecrawl CLI and Skill are open source and available on GitHub: [firecrawl/cli](https://github.com/firecrawl/cli)
> Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.
# AI Platforms
Source: https://docs.firecrawl.dev/use-cases/ai-platforms
Power AI assistants and let customers build AI apps
AI platform builders and teams use Firecrawl to power knowledge bases, chatbots, and enable customers to build AI applications with web data.
## Start with a Template
Instant AI chatbots for websites with web knowledge integration
**Get started with templates and examples.** Build AI-powered applications with web data.
## How It Works
Transform websites into AI-ready data. Power chatbots with real-time web knowledge, build RAG systems with up-to-date documentation, and enable your users to connect their AI applications to web sources.
## Why AI Platforms Choose Firecrawl
### Reduce Hallucinations with Real-Time Data
Your AI assistants need current information, not outdated training data. Whether it's domain-specific knowledge, technical documentation, or industry-specific content, Firecrawl ensures your knowledge bases stay synchronized with the latest updates-reducing hallucinations and improving response accuracy.
## Customer Stories
**Replit**
Learn how Replit leverages Firecrawl to keep Replit Agent up-to-date with the latest API documentation and web content.
**Stack AI**
Discover how Stack AI uses Firecrawl to seamlessly feed agentic AI workflows with high-quality web data.
## FAQs
Firecrawl provides simple APIs and SDKs that integrate directly into AI platforms. Whether you're building with LangChain, using no-code tools like n8n, or custom frameworks, Firecrawl delivers clean, structured web data ready for AI consumption.
Yes. Firecrawl is designed for enterprise-scale data extraction, processing millions of pages for AI training datasets. Our infrastructure scales automatically to meet your needs.
Firecrawl delivers data in AI-friendly formats including clean markdown, structured JSON, raw HTML, extracted images, screenshots, and news content. This flexibility ensures compatibility with any AI platform's data ingestion requirements.
Yes! Our API supports real-time data extraction, enabling AI applications to access fresh web data on-demand. This is perfect for AI agents that need current information to make decisions.
## Related Use Cases
* [Deep Research](/use-cases/deep-research) - Advanced research capabilities
* [Content Generation](/use-cases/content-generation) - AI-powered content creation
* [Developers & MCP](/use-cases/developers-mcp) - Developer integrations
# Competitive Intelligence
Source: https://docs.firecrawl.dev/use-cases/competitive-intelligence
Monitor competitor websites and track changes in real-time
Business intelligence teams use Firecrawl to monitor competitors and get alerts on strategic changes.
## Start with a Template
Real-time website monitoring with intelligent alerts
Research and analyze competitor strategies with AI
**Choose from monitoring and research templates.** Track competitors and analyze their strategies.
## How It Works
Set up a pipeline that scrapes competitor sites on a schedule, extracts the data you care about, and alerts your team when something changes.
Crawl or scrape competitor websites with Firecrawl to get clean, structured content from product pages, pricing tables, and blog posts.
Pull out specific fields like pricing tiers, feature lists, job openings, or partnership announcements using structured extraction.
Store each extraction and diff it against previous snapshots to pinpoint what changed and when.
Trigger notifications when you detect important updates such as a new product launch, a pricing shift, or a change in positioning.
## What You Can Track
* **Products**: New launches, features, specs, pricing, documentation
* **Marketing**: Messaging changes, campaigns, case studies, testimonials
* **Business**: Job postings, partnerships, funding, press releases
* **Strategy**: Positioning, target markets, pricing approaches, go-to-market
* **Technical**: API changes, integrations, technology stack updates
## FAQs
Firecrawl extracts current page content whenever called. Build your own monitoring system to check competitors at intervals that match your needs - from hourly for critical updates to daily for routine tracking.
Yes, Firecrawl can access region-specific content. You can monitor different versions of competitor sites across multiple countries and languages.
When building your monitoring system, implement filters to ignore minor changes like timestamps or dynamic content. Compare extracted data over time and use your own logic to determine what constitutes a meaningful change.
Yes. Extract data from competitor press releases, blog posts, and public social media pages. Build systems to analyze announcement patterns, messaging changes, and campaign launches over time.
Extract data from multiple competitor sites using Firecrawl's APIs. Build your own system to organize and compare this data - many users create databases with competitor profiles and custom dashboards for analysis.
## Related Use Cases
* [Product & E-commerce](/use-cases/product-ecommerce) - Track competitor products
* [Investment & Finance](/use-cases/investment-finance) - Market intelligence
* [SEO Platforms](/use-cases/seo-platforms) - search competitor tracking
# Content Generation
Source: https://docs.firecrawl.dev/use-cases/content-generation
Generate AI content based on website data, images, and news
Turn live website data into presentations, emails, marketing copy, and newsletters. Firecrawl scrapes pages, extracts clean text and images, and feeds structured content to your AI pipeline so every piece you publish is grounded in real, up-to-date sources.
## Start with a Template
Clone and recreate any website as a modern React app
**Get started with the Open Lovable template.** Transform websites into content and applications.
## How It Works
Point Firecrawl at any URL or set of URLs. It returns clean, structured content you can pipe directly into your generation workflow.
### Multiple Output Formats
Extract content as structured HTML, clean Markdown, JSON, or full-page screenshots, whichever format best fits your content pipeline.
### Image and Visual Extraction
Capture images and take screenshots from source pages to enrich presentations, emails, and social posts with real visuals.
### News and Trending Content
Surface relevant news stories as part of your request to keep generated content current and timely.
## What You Can Create
* **Sales Decks**: Custom presentations with prospect data
* **Email Campaigns**: Personalized outreach at scale
* **Marketing Content**: Data-driven blog posts and reports
* **Social Media**: Trending topic and news-driven content generation
* **Documentation**: Auto-updated technical content
* **Newsletters**: Curated updates from industry and competitor news
* **Visual Content**: Posts and reports enriched with extracted images and screenshots
## FAQs
Firecrawl extracts data directly from source websites, preserving the original content structure and context. All extracted data includes source URLs and timestamps for verification.
Firecrawl provides clean markdown, structured JSON, HTML, images, and screenshots from websites. This extracted data serves as the factual foundation for your content generation workflows.
Yes. Firecrawl can extract images, capture screenshots, and pull content from news sites. This enables you to create visually rich content and stay current with industry developments.
Firecrawl excels at extracting from company websites, news sites, blogs, and documentation. Sites with structured HTML and clear content hierarchies yield the cleanest extraction results.
Use Firecrawl's batch scraping and crawl APIs to extract data from multiple websites efficiently. Process hundreds of URLs in parallel to build comprehensive datasets for your content workflows.
## Related Use Cases
* [AI Platforms](/use-cases/ai-platforms) - Build AI-powered content tools
* [Lead Enrichment](/use-cases/lead-enrichment) - Personalize with prospect data
* [SEO Platforms](/use-cases/seo-platforms) - Optimize generated content
# Data Migration
Source: https://docs.firecrawl.dev/use-cases/data-migration
Transfer web data efficiently between platforms and systems
Extract content, structure, and metadata from any website and transform it for import into a new platform. Firecrawl handles the scraping so you can focus on mapping data to your target system.
## Start with a Template
Efficiently migrate data between platforms and systems
**Get started with the Firecrawl Migrator template.** Extract and transform data for platform migrations.
## How It Works
Point Firecrawl at your source website to crawl every page and return clean, structured data. From there, you transform the output to match your target platform's schema and import it using that platform's API or bulk-import tools.
## What You Can Migrate
* **Content**: Pages, posts, articles, media files, metadata
* **Structure**: Hierarchies, categories, tags, taxonomies
* **Users**: Profiles and user-related data where publicly accessible
* **Settings**: Configurations, custom fields, workflows
* **E-commerce**: Products, catalogs, inventory, orders
## Common Migration Use Cases
Users build migration tools with Firecrawl to extract data from various platforms:
Extract content from WordPress, Drupal, and Joomla sites or custom CMS platforms. Preserve content structure and metadata, then export for import into new systems like Contentful, Strapi, or Sanity.
Extract product catalogs from Magento and WooCommerce stores including inventory, pricing, descriptions, and specifications. Format data for import into Shopify, BigCommerce, or other platforms.
## FAQs
Our infrastructure scales automatically to handle large migrations. We support incremental processing with batching and parallel extraction, allowing you to migrate millions of pages by breaking them into manageable chunks with progress tracking.
Yes! Extract all SEO metadata including URLs, titles, descriptions, and implement proper redirects. We help maintain your search rankings through the migration.
Firecrawl can extract and catalog all media files. You can download them for re-upload to your new platform or reference them directly if keeping the same CDN.
We provide detailed extraction reports and support comparison tools. You can verify content completeness, check broken links, and validate data integrity.
Yes, you can extract publicly visible user-generated content including comments, reviews, and forum posts. Private user data requires appropriate authentication and permissions.
## Related Use Cases
* [Product & E-commerce](/use-cases/product-ecommerce) - Catalog migrations
* [Content Generation](/use-cases/content-generation) - Content transformation
* [AI Platforms](/use-cases/ai-platforms) - Knowledge base migration
# Deep Research
Source: https://docs.firecrawl.dev/use-cases/deep-research
Build agentic research tools with deep web search capabilities
Build automated research agents that search the web, scrape full-page content, and synthesize findings with an LLM. Firecrawl handles source discovery and content extraction so you can focus on analysis, not parsing HTML.
## Start with a Template
Blazing-fast AI search with real-time citations
Deep research agent with LangGraph and answer validation
Visual AI research assistant for comprehensive analysis
**Choose from multiple research templates.** Clone, configure your API key, and start researching.
## How It Works
Build powerful research tools that transform scattered web data into comprehensive insights. The core pattern is a **search → scrape → analyze → repeat** loop: use Firecrawl's search API to discover relevant sources, scrape each source for full content, then feed the results into an LLM to synthesize findings and identify follow-up queries.
Use the `/search` endpoint to find relevant pages for your research topic.
```python Python theme={null}
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
results = firecrawl.search(
"recent advances in quantum computing",
limit=5,
scrape_options={"formats": ["markdown", "links"]}
)
```
```js Node.js theme={null}
import Firecrawl from '@mendable/firecrawl-js';
const firecrawl = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });
const results = await firecrawl.search(
'recent advances in quantum computing',
{ limit: 5, scrapeOptions: { formats: ['markdown', 'links'] } }
);
```
Extract full content from each result to get detailed information with citations.
```python Python theme={null}
for result in results:
doc = firecrawl.scrape(result["url"], formats=["markdown"])
# Feed doc content into your LLM for analysis
```
```js Node.js theme={null}
for (const result of results) {
const doc = await firecrawl.scrape(result.url, { formats: ['markdown'] });
// Feed doc content into your LLM for analysis
}
```
Use an LLM to synthesize findings, identify gaps, and generate follow-up queries. Repeat the loop until your research question is fully answered.
## Why Researchers Choose Firecrawl
### Accelerate Research from Weeks to Hours
Build automated research systems that discover, read, and synthesize information from across the web. Create tools that deliver comprehensive reports with full citations, eliminating manual searching through hundreds of sources.
### Ensure Research Completeness
Reduce the risk of missing critical information. Build systems that follow citation chains, discover related sources, and surface insights that traditional search methods miss.
## Research Tool Capabilities
* **Iterative Exploration**: Build tools that automatically discover related topics and sources
* **Multi-Source Synthesis**: Combine information from hundreds of websites
* **Citation Preservation**: Maintain full source attribution in your research outputs
* **Intelligent Summarization**: Extract key findings and insights for analysis
* **Trend Detection**: Identify patterns across multiple sources
## FAQs
Use Firecrawl's crawl and search APIs to build iterative research systems. Start with search results, extract content from relevant pages, follow citation links, and aggregate findings. Combine with LLMs to synthesize comprehensive research reports.
Yes. Firecrawl can extract data from open-access research papers, academic websites, and publicly available scientific publications. It preserves formatting, citations, and technical content critical for research work.
Firecrawl maintains source attribution and extracts content exactly as presented on websites. All data includes source URLs and timestamps, ensuring full traceability for research purposes.
Yes. Set up scheduled crawls to track how information changes over time. This is perfect for monitoring trends, policy changes, or any research requiring temporal data analysis.
Our crawling infrastructure scales to handle thousands of sources simultaneously. Whether you're analyzing entire industries or tracking global trends, Firecrawl provides the data pipeline you need.
## Related Use Cases
* [AI Platforms](/use-cases/ai-platforms) - Build AI research assistants
* [Content Generation](/use-cases/content-generation) - Research-based content
* [Competitive Intelligence](/use-cases/competitive-intelligence) - Market research
# Developers & MCP
Source: https://docs.firecrawl.dev/use-cases/developers-mcp
Build powerful integrations with Model Context Protocol support
Give your AI coding assistant the ability to scrape, crawl, and search the web in real time. Firecrawl's MCP server connects to Claude Desktop, Cursor, and other Model Context Protocol clients so your assistant can pull live documentation, discover site structures, and extract structured data on demand.
## Start with a Template
Official MCP server - Add web scraping to Claude Desktop and Cursor
Build complete applications from any website instantly
**Get started with MCP in minutes.** Follow our [setup guide](https://github.com/firecrawl/firecrawl-mcp-server#installation) to integrate Firecrawl into Claude Desktop or Cursor.
## How It Works
Integrate Firecrawl directly into your AI coding workflow through Model Context Protocol. Once configured, your AI assistant gains access to a set of web scraping tools it can call on your behalf:
| Tool | What it does |
| ---------------- | ---------------------------------------------------------- |
| **Scrape** | Extract content or structured data from a single URL |
| **Batch Scrape** | Extract content from multiple known URLs in parallel |
| **Map** | Discover all indexed URLs on a website |
| **Crawl** | Walk a site section and extract content from every page |
| **Search** | Search the web and optionally extract content from results |
Your assistant picks the right tool automatically. Ask it to "read the Next.js docs" and it will scrape. Ask it to "find all blog posts on example.com" and it will map then batch scrape.
## Why Developers Choose Firecrawl MCP
### Build Smarter AI Assistants
Give your AI real-time access to documentation, APIs, and web resources. Reduce outdated information and hallucinations by providing your assistant with the latest data.
### Zero Infrastructure Required
No servers to manage, no crawlers to maintain. Just configure once and your AI assistant can access websites instantly through the Model Context Protocol.
## Customer Stories
**Botpress**
Discover how Botpress uses Firecrawl to streamline knowledge base population and improve developer experience.
**Answer HQ**
Learn how Answer HQ uses Firecrawl to help businesses import website data and build intelligent support assistants.
## FAQs
Currently, Claude Desktop and Cursor have native MCP support. More AI assistants are adding support regularly. You can also use the MCP SDK to build custom integrations.
VS Code and other IDEs can use MCP through community extensions or terminal integrations. Native support varies by IDE. Check our [GitHub repository](https://github.com/firecrawl/firecrawl-mcp-server) for IDE-specific setup guides.
The MCP server automatically caches responses for 15 minutes. You can configure cache duration in your MCP server settings or implement custom caching logic.
MCP requests use your standard Firecrawl API rate limits. We recommend batching related requests and using caching for frequently accessed documentation.
Follow our [setup guide](https://github.com/firecrawl/firecrawl-mcp-server#installation) to configure MCP. You'll need to add your Firecrawl API key to your MCP configuration file. The process takes just a few minutes.
## Related Use Cases
* [AI Platforms](/use-cases/ai-platforms) - Build AI-powered dev tools
* [Deep Research](/use-cases/deep-research) - Complex technical research
* [Content Generation](/use-cases/content-generation) - Generate documentation
# Investment & Finance
Source: https://docs.firecrawl.dev/use-cases/investment-finance
Track companies and extract financial insights from web data
Turn public web data into structured financial signals. Firecrawl scrapes company websites, news sites, job boards, and regulatory filings, then returns clean JSON you can feed directly into due diligence workflows, earnings prep, or ongoing portfolio surveillance.
## Start with a Template
Monitor portfolio companies for material changes and trigger events
## What You Can Track
* **Company Metrics**: growth indicators, team changes, product launches, funding rounds
* **Market Signals**: industry trends, competitor moves, sentiment shifts, regulatory changes
* **Risk Indicators**: leadership changes, legal issues, regulatory mentions, customer complaints
* **Financial Data**: pricing updates, revenue signals, partnership announcements
* **Alternative Data**: job postings, web traffic, social signals, news mentions
## Customer Stories
**Athena Intelligence**
Discover how Athena Intelligence leverages Firecrawl to fuel its AI-native analytics platform for enterprise analysts.
**Cargo**
See how Cargo uses Firecrawl to analyze market data and power revenue intelligence workflows.
## FAQs
Yes, you can monitor publicly available information about private companies from their websites, news mentions, job postings, and social media presence.
Firecrawl extracts data in real-time when called. Build your own monitoring system to fetch data at intervals that match your investment strategy - from minute-by-minute for critical events to daily for routine tracking.
Public web sources such as company websites, news sites, job boards, review sites, forums, social media, government filings, and open-access industry data.
Extract data from company ESG reports, sustainability pages, news mentions of environmental initiatives, and regulatory filings. Build tracking systems to identify changes in sustainability commitments or ESG-related developments.
Yes. Extract recent company updates, product launches, executive changes, and industry trends before earnings calls. Combine with competitor data to anticipate questions and identify key discussion points.
## Related Use Cases
* [Competitive Intelligence](/use-cases/competitive-intelligence) - Track market competitors
* [Deep Research](/use-cases/deep-research) - Comprehensive market analysis
* [Lead Enrichment](/use-cases/lead-enrichment) - B2B investment opportunities
# Lead Enrichment
Source: https://docs.firecrawl.dev/use-cases/lead-enrichment
Extract and filter leads from websites to power your sales pipeline
Scrape business directories for leads, enrich your CRM with live company data, and automate account research. Firecrawl handles the extraction so you can focus on closing deals.
## Start with a Template
AI-powered lead enrichment and data extraction from websites
## How It Works
1. **Point Firecrawl at a source.** Pass a directory URL, company website, or list of domains to the API.
2. **Extract structured data.** Firecrawl scrapes each page and returns fields like company name, contact details, team members, and recent news.
3. **Send the data to your CRM.** Push enriched records into Salesforce, HubSpot, or any other tool through the API or a Zapier integration.
## Why Sales Teams Choose Firecrawl
### Transform Directories into Pipeline
Every industry directory is a goldmine of potential customers. Firecrawl extracts thousands of qualified leads from business directories, trade associations, and conference attendee lists-complete with company details and contact information.
### Enrich CRM Data Automatically
Stop paying for stale data from traditional providers. Firecrawl pulls real-time information directly from company websites, ensuring your sales team always has the latest company news, team changes, and growth signals.
## Customer Stories
**Zapier**
Discover how Zapier uses Firecrawl to empower customers with custom knowledge in their chatbots.
**Cargo**
See how Cargo uses Firecrawl to instantly analyze webpage content and power Go-To-Market workflows.
## Lead Sources
### Business Directories
* Industry-specific directories
* Chamber of commerce listings
* Trade association members
* Conference attendee lists
### Company Websites
* About pages and team sections
* Press releases and news
* Job postings for growth signals
* Customer case studies
## FAQs
Firecrawl automatically extracts company information, contact details, product offerings, and recent news from prospect websites. This enriches your CRM with accurate, up-to-date information for better sales outreach.
Yes! Firecrawl extracts publicly available contact information including emails and phone numbers from company websites, team pages, and contact sections.
Since Firecrawl extracts data directly from live websites, you get the most current information available. This is more accurate than static databases that quickly become outdated.
Yes. Use our API or Zapier integration to automatically enrich leads in Salesforce, HubSpot, Pipedrive, and other CRMs. Keep your lead data fresh without manual research.
Extract detailed company information, recent updates, and trigger events from target account websites. This intelligence helps personalize outreach and identify the perfect timing for engagement.
## Related Use Cases
* [AI Platforms](/use-cases/ai-platforms) - Build AI sales assistants
* [Competitive Intelligence](/use-cases/competitive-intelligence) - Track competitors
* [Investment & Finance](/use-cases/investment-finance) - Investment opportunities
# Observability & Monitoring
Source: https://docs.firecrawl.dev/use-cases/observability
Monitor websites, track uptime, and detect changes in real-time
DevOps and SRE teams use Firecrawl to monitor websites, track availability, and detect critical changes across their digital infrastructure.
## Start with a Template
Real-time website monitoring and intelligent change detection
**Get started with the Firecrawl Observer template.** Monitor websites and track changes in real-time.
## How It Works
Use Firecrawl's extraction capabilities to build observability systems for your websites. Extract page content, analyze changes over time, validate deployments, and create monitoring workflows that ensure your sites function correctly.
## What You Can Monitor
* **Availability**: Uptime, response times, error rates
* **Content**: Text changes, image updates, layout shifts
* **Performance**: Page load times, resource sizes, Core Web Vitals
* **Security**: SSL certificates, security headers, misconfigurations
* **SEO Health**: Meta tags, structured data, sitemap validity
## Monitoring Types
### Synthetic Monitoring
* User journey validation
* Transaction monitoring
* Multi-step workflows
* Cross-browser testing
### Content Monitoring
* Text change detection
* Visual regression testing
* Dynamic content validation
* Internationalization checks
## FAQs
Firecrawl extracts website content and structure on demand. Build monitoring systems that call Firecrawl's API to check pages, compare extracted data against baselines, and trigger your own alerts when changes occur.
Yes! Firecrawl fully renders JavaScript, making it perfect for monitoring modern SPAs, React apps, and dynamic content. We capture the page as users see it, not just the raw HTML.
Firecrawl extracts data in real-time when called. Build your monitoring system to check sites at whatever frequency you need - from minute-by-minute for critical pages to daily for routine checks.
Yes. Use the extract API to pull specific elements like prices, inventory levels, or critical content. Build validation logic in your monitoring system to verify that important information is present and correct.
Firecrawl provides webhooks that you can use to build integrations with your alerting tools. Send extracted data to PagerDuty, Slack, email, or any monitoring platform by building connectors that process Firecrawl's responses.
## Related Use Cases
* [Competitive Intelligence](/use-cases/competitive-intelligence) - Monitor competitor changes
* [Product & E-commerce](/use-cases/product-ecommerce) - Track inventory and pricing
* [Data Migration](/use-cases/data-migration) - Validate migrations
# Use Cases
Source: https://docs.firecrawl.dev/use-cases/overview
Transform web data into powerful features for your applications
Turn any website into structured data for your product. Whether you are building RAG pipelines, enriching leads, or monitoring competitors, Firecrawl handles the scraping so you can focus on what you do with the data.
Browse the use cases below to find guides, code samples, and architecture patterns for your workflow.
Add web knowledge to your RAG chatbots and AI assistants.
Extract and filter leads from websites to enrich your sales pipeline.
Monitor search rankings and optimize content strategy.
Build agentic research tools with deep web search capabilities.
Monitor pricing and track inventory across e-commerce sites.
Generate AI content based on website data and structure.
Build powerful integrations with Model Context Protocol support.
Track companies and extract financial insights from web data.
Monitor competitor websites and track changes in real-time.
Transfer web data seamlessly between platforms and systems.
Monitor websites, track uptime, and detect changes in real-time.
# Product & E-commerce
Source: https://docs.firecrawl.dev/use-cases/product-ecommerce
Monitor pricing and track inventory across e-commerce sites
Turn any e-commerce website into structured product data. Firecrawl extracts pricing, inventory, and catalog information so you can monitor competitors, track stock levels, and migrate products between platforms.
## Start with a Template
Migrate product catalogs and e-commerce data between platforms
**Get started with the Firecrawl Migrator template.** Extract and migrate e-commerce data efficiently.
## What You Can Extract
* **Product Data**: Title, SKU, specs, descriptions, categories
* **Pricing**: Current price, discounts, shipping, tax
* **Inventory**: Stock levels, availability, lead times
* **Reviews**: Ratings, customer feedback, Q\&A sections
## Use Cases in Action
**Price Monitoring**
Track competitor pricing across multiple e-commerce sites, receive alerts on price changes, and optimize your pricing strategy based on real-time market data.
**Catalog Migration**
Seamlessly migrate thousands of products between e-commerce platforms, preserving all product data, variants, images, and metadata.
## FAQs
Build a monitoring system using Firecrawl's API to extract prices at regular intervals. Compare extracted data over time to identify pricing trends, promotions, and competitive positioning.
Yes, Firecrawl can extract all product variants including size, color, and other options. Structure the data with custom schemas to capture all variant information.
For dynamic pricing, you can use Firecrawl's JavaScript rendering to capture prices after they load. For user-specific pricing, configure authentication headers in your requests.
Yes. Firecrawl can extract data from any publicly accessible e-commerce website. Users successfully extract from Shopify, WooCommerce, Magento, BigCommerce, and custom-built stores.
Yes. Firecrawl can navigate through paginated product listings and handle infinite scroll mechanisms to extract complete product catalogs, ensuring no products are missed during extraction.
## Related Use Cases
* [Lead Enrichment](/use-cases/lead-enrichment) - Enrich B2B e-commerce leads
* [Competitive Intelligence](/use-cases/competitive-intelligence) - Track competitor strategies
* [Data Migration](/use-cases/data-migration) - Migrate between platforms
# SEO Platforms
Source: https://docs.firecrawl.dev/use-cases/seo-platforms
Optimize websites for AI assistants and search engines
Audit entire websites for SEO and AI readability at scale. Firecrawl extracts meta tags, header structures, internal links, and content across thousands of pages, so you can spot optimization gaps that sampling-based tools miss.
## Start with a Template
GEO-powered SEO monitoring and multi-region rank tracking
**Get started with the FireGEO template.** Optimize for both search engines and AI assistants.
## How It Works
Crawl a site with Firecrawl to get structured markdown and metadata for every page. Use that output to analyze content quality, check technical SEO elements, and evaluate how well each page communicates with AI assistants.
## Why SEO Platforms Choose Firecrawl
### Optimize for AI Discovery, Not Just Google
The future of search is AI-powered. While competitors focus on traditional SEO, forward-thinking platforms use Firecrawl to increase their clients' visibility in AI assistant responses-the new frontier of organic discovery.
### Complete Site Intelligence at Scale
Analyze entire websites, not just sample pages. Extract every meta tag, header structure, internal link, and content element across thousands of pages simultaneously. Identify optimization opportunities your competitors miss.
## What You Can Build
* **AI Readability Audit**: Optimize for AI comprehension
* **Content Analysis**: Structure and semantic optimization
* **Technical SEO**: Site performance and crawlability
* **Search Tracking**: Monitor search engine positions
## FAQs
Firecrawl helps you structure content for optimal AI comprehension, extract semantic signals, and ensure your site follows best practices for AI discovery. This includes generating experimental formats like llms.txt (an emerging convention for AI crawler guidance).
Firecrawl extracts complete site structures, meta tags, headers, internal links, and content to perform comprehensive SEO audits. Identify optimization opportunities and track improvements over time.
Yes. Analyze competitor site structures, keyword usage, content strategies, and technical SEO implementations. Understand what's working in your industry to inform your strategy.
Crawl competitor sites to identify topics they cover that you don't. Extract their content categories, blog topics, and page structures to find opportunities for new content.
Yes. Identify broken links, track redirect chains, extract canonical tags, and monitor meta tag implementation. Regular crawls help identify technical SEO issues across your site.
## Related Use Cases
* [AI Platforms](/use-cases/ai-platforms) - Build AI-powered SEO tools
* [Competitive Intelligence](/use-cases/competitive-intelligence) - Track competitor SEO
* [Content Generation](/use-cases/content-generation) - Create SEO content
# Event Types
Source: https://docs.firecrawl.dev/webhooks/events
Webhook event reference
Firecrawl sends webhook events at each stage of a job's lifecycle, so you can track progress, capture results, and handle failures in real time without polling.
## Quick Reference
| Event | Trigger |
| ------------------------ | ---------------------------------------------------- |
| `crawl.started` | Crawl job begins processing |
| `crawl.page` | A page is scraped during a crawl |
| `crawl.completed` | Crawl job finishes and all pages have been processed |
| `batch_scrape.started` | Batch scrape job begins processing |
| `batch_scrape.page` | A URL is scraped during a batch scrape |
| `batch_scrape.completed` | All URLs in the batch have been processed |
| `extract.started` | Extract job begins processing |
| `extract.completed` | Extraction finishes successfully |
| `extract.failed` | Extraction fails |
| `agent.started` | Agent job begins processing |
| `agent.action` | Agent executes a tool (scrape, search, etc.) |
| `agent.completed` | Agent finishes successfully |
| `agent.failed` | Agent encounters an error |
| `agent.cancelled` | Agent job is cancelled by the user |
## Payload Structure
All webhook events share this structure:
```json theme={null}
{
"success": true,
"type": "crawl.page",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [...],
"metadata": {}
}
```
| Field | Type | Description |
| ---------- | ------- | ----------------------------------------- |
| `success` | boolean | Whether the operation succeeded |
| `type` | string | Event type (e.g. `crawl.page`) |
| `id` | string | Job ID |
| `data` | array | Event-specific data (see examples below) |
| `metadata` | object | Custom metadata from your webhook config |
| `error` | string | Error message (when `success` is `false`) |
## Crawl Events
### `crawl.started`
Sent when the crawl job begins processing.
```json theme={null}
{
"success": true,
"type": "crawl.started",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"metadata": {}
}
```
### `crawl.page`
Sent for each page scraped. The `data` array contains the page content and metadata.
```json theme={null}
{
"success": true,
"type": "crawl.page",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"markdown": "# Page content...",
"metadata": {
"title": "Page Title",
"description": "Page description",
"url": "https://example.com/page",
"statusCode": 200,
"contentType": "text/html",
"scrapeId": "550e8400-e29b-41d4-a716-446655440001",
"sourceURL": "https://example.com/page",
"proxyUsed": "basic",
"cacheState": "hit",
"cachedAt": "2025-09-03T21:11:25.636Z",
"creditsUsed": 1
}
}
],
"metadata": {}
}
```
### `crawl.completed`
Sent when the crawl job finishes and all pages have been processed.
```json theme={null}
{
"success": true,
"type": "crawl.completed",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"metadata": {}
}
```
## Batch Scrape Events
### `batch_scrape.started`
Sent when the batch scrape job begins processing.
```json theme={null}
{
"success": true,
"type": "batch_scrape.started",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"metadata": {}
}
```
### `batch_scrape.page`
Sent for each URL scraped. The `data` array contains the page content and metadata.
```json theme={null}
{
"success": true,
"type": "batch_scrape.page",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"markdown": "# Page content...",
"metadata": {
"title": "Page Title",
"description": "Page description",
"url": "https://example.com",
"statusCode": 200,
"contentType": "text/html",
"scrapeId": "550e8400-e29b-41d4-a716-446655440001",
"sourceURL": "https://example.com",
"proxyUsed": "basic",
"cacheState": "miss",
"cachedAt": "2025-09-03T23:30:53.434Z",
"creditsUsed": 1
}
}
],
"metadata": {}
}
```
### `batch_scrape.completed`
Sent when all URLs in the batch have been processed.
```json theme={null}
{
"success": true,
"type": "batch_scrape.completed",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"metadata": {}
}
```
## Extract Events
### `extract.started`
Sent when the extract job begins processing.
```json theme={null}
{
"success": true,
"type": "extract.started",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"metadata": {}
}
```
### `extract.completed`
Sent when extraction finishes successfully. The `data` array contains the extracted data and usage info.
```json theme={null}
{
"success": true,
"type": "extract.completed",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"success": true,
"data": { "siteName": "Example Site", "category": "Technology" },
"extractId": "550e8400-e29b-41d4-a716-446655440000",
"llmUsage": 0.0020118,
"totalUrlsScraped": 1,
"sources": {
"siteName": ["https://example.com"],
"category": ["https://example.com"]
}
}
],
"metadata": {}
}
```
### `extract.failed`
Sent when extraction fails. The `error` field contains the failure reason.
```json theme={null}
{
"success": false,
"type": "extract.failed",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"error": "Failed to extract data: timeout exceeded",
"metadata": {}
}
```
## Agent Events
### `agent.started`
Sent when the agent job begins processing.
```json theme={null}
{
"success": true,
"type": "agent.started",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [],
"metadata": {}
}
```
### `agent.action`
Sent after each tool execution (scrape, search, etc.).
```json theme={null}
{
"success": true,
"type": "agent.action",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"creditsUsed": 5,
"action": "mcp__tools__scrape",
"input": {
"url": "https://example.com"
}
}
],
"metadata": {}
}
```
The `creditsUsed` value in `action` events is an **estimate** of the total
credits used so far. The final accurate credit count is only
available in the `completed`, `failed`, or `cancelled` events.
### `agent.completed`
Sent when the agent finishes successfully. The `data` array contains the extracted data and total credits used.
```json theme={null}
{
"success": true,
"type": "agent.completed",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"creditsUsed": 15,
"data": {
"company": "Example Corp",
"industry": "Technology",
"founded": 2020
}
}
],
"metadata": {}
}
```
### `agent.failed`
Sent when the agent encounters an error. The `error` field contains the failure reason.
```json theme={null}
{
"success": false,
"type": "agent.failed",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"creditsUsed": 8
}
],
"error": "Max credits exceeded",
"metadata": {}
}
```
### `agent.cancelled`
Sent when the agent job is cancelled by the user.
```json theme={null}
{
"success": false,
"type": "agent.cancelled",
"id": "550e8400-e29b-41d4-a716-446655440000",
"data": [
{
"creditsUsed": 3
}
],
"metadata": {}
}
```
## Event Filtering
By default, you receive all events. To subscribe to specific events only, use the `events` array in your webhook config:
```json theme={null}
{
"url": "https://your-app.com/webhook",
"events": ["completed", "failed"]
}
```
This is useful if you only care about job completion and don't need per-page updates.
# Overview
Source: https://docs.firecrawl.dev/webhooks/overview
Real-time notifications for your Firecrawl operations
Get notified the moment a crawl, batch scrape, extract, or agent job starts, progresses, or finishes. Instead of polling for status, you provide an HTTPS endpoint and Firecrawl delivers events to it in real time.
## Supported Operations
| Operation | Events |
| ------------ | ------------------------------------------------------- |
| Crawl | `started`, `page`, `completed` |
| Batch Scrape | `started`, `page`, `completed` |
| Extract | `started`, `completed`, `failed` |
| Agent | `started`, `action`, `completed`, `failed`, `cancelled` |
See [Event Types](/webhooks/events) for full payload details and examples.
## Configuration
Add a `webhook` object to your request:
```json JSON theme={null}
{
"webhook": {
"url": "https://your-domain.com/webhook",
"metadata": {
"any_key": "any_value"
},
"events": ["started", "page", "completed", "failed"]
}
}
```
| Field | Type | Required | Description |
| ---------- | ------ | -------- | ------------------------------------- |
| `url` | string | Yes | Your endpoint URL (HTTPS) |
| `headers` | object | No | Custom headers to include |
| `metadata` | object | No | Custom data included in payloads |
| `events` | array | No | Event types to receive (default: all) |
## Usage
### Crawl with Webhook
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/crawl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://docs.firecrawl.dev",
"limit": 100,
"webhook": {
"url": "https://your-domain.com/webhook",
"metadata": {
"any_key": "any_value"
},
"events": ["started", "page", "completed"]
}
}'
```
### Batch Scrape with Webhook
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/batch/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"urls": [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3"
],
"webhook": {
"url": "https://your-domain.com/webhook",
"metadata": {
"any_key": "any_value"
},
"events": ["started", "page", "completed"]
}
}'
```
## Timeouts & Retries
Your endpoint must respond with a `2xx` status within **10 seconds**.
If delivery fails (timeout, non-2xx, or network error), Firecrawl retries automatically:
| Retry | Delay after failure |
| ----- | ------------------- |
| 1st | 1 minute |
| 2nd | 5 minutes |
| 3rd | 15 minutes |
After 3 failed retries, the webhook is marked as failed and no further attempts are made.
# Security
Source: https://docs.firecrawl.dev/webhooks/security
Verify webhook authenticity
Firecrawl signs every webhook request using HMAC-SHA256. Verifying signatures ensures requests are authentic and haven't been tampered with.
## Secret Key
Your webhook secret is available in the [Advanced tab](https://www.firecrawl.dev/app/settings?tab=advanced) of your account settings. Each account has a unique secret used to sign all webhook requests.
Keep your webhook secret secure and never expose it publicly. If you believe your secret has been compromised, regenerate it immediately from your account settings.
## Signature Verification
Each webhook request includes an `X-Firecrawl-Signature` header:
```
X-Firecrawl-Signature: sha256=abc123def456...
```
### How to Verify
1. Extract the signature from the `X-Firecrawl-Signature` header
2. Get the raw request body (before parsing)
3. Compute HMAC-SHA256 using your secret key
4. Compare signatures using a timing-safe function
### Implementation
```js Node/Express theme={null}
import crypto from 'crypto';
import express from 'express';
const app = express();
// Use raw body parser for signature verification
app.use('/webhook/firecrawl', express.raw({ type: 'application/json' }));
app.post('/webhook/firecrawl', (req, res) => {
const signature = req.get('X-Firecrawl-Signature');
const webhookSecret = process.env.FIRECRAWL_WEBHOOK_SECRET;
if (!signature || !webhookSecret) {
return res.status(401).send('Unauthorized');
}
// Extract hash from signature header
const [algorithm, hash] = signature.split('=');
if (algorithm !== 'sha256') {
return res.status(401).send('Invalid signature algorithm');
}
// Compute expected signature
const expectedSignature = crypto
.createHmac('sha256', webhookSecret)
.update(req.body)
.digest('hex');
// Verify signature using timing-safe comparison
if (!crypto.timingSafeEqual(Buffer.from(hash, 'hex'), Buffer.from(expectedSignature, 'hex'))) {
return res.status(401).send('Invalid signature');
}
// Parse and process verified webhook
const event = JSON.parse(req.body);
console.log('Verified Firecrawl webhook:', event);
res.status(200).send('ok');
});
app.listen(3000, () => console.log('Listening on 3000'));
```
```python Python/Flask theme={null}
import hmac
import hashlib
from flask import Flask, request, abort
app = Flask(__name__)
WEBHOOK_SECRET = 'your-webhook-secret-here' # Get from Firecrawl dashboard
@app.post('/webhook/firecrawl')
def webhook():
signature = request.headers.get('X-Firecrawl-Signature')
if not signature:
abort(401, 'Missing signature header')
# Extract hash from signature header
try:
algorithm, hash_value = signature.split('=', 1)
if algorithm != 'sha256':
abort(401, 'Invalid signature algorithm')
except ValueError:
abort(401, 'Invalid signature format')
# Compute expected signature
expected_signature = hmac.new(
WEBHOOK_SECRET.encode('utf-8'),
request.data,
hashlib.sha256
).hexdigest()
# Verify signature using timing-safe comparison
if not hmac.compare_digest(hash_value, expected_signature):
abort(401, 'Invalid signature')
# Parse and process verified webhook
event = request.get_json(force=True)
print('Verified Firecrawl webhook:', event)
return 'ok', 200
if __name__ == '__main__':
app.run(port=3000)
```
## Best Practices
* **Verify every request.** Always check the signature before processing a webhook payload. Reject any request that fails verification with a `401` status.
* **Use timing-safe comparisons.** Standard string comparison can leak timing information. Use `crypto.timingSafeEqual()` in Node.js or `hmac.compare_digest()` in Python.
* **Serve your endpoint over HTTPS.** This ensures webhook payloads are encrypted in transit.
# Testing
Source: https://docs.firecrawl.dev/webhooks/testing
Test and debug webhooks
Verify that your webhook integration works before deploying to production. This page covers how to receive webhooks on your local machine and how to diagnose common delivery and verification failures.
## Local Development
Webhooks require a publicly reachable URL, so you need to expose your local server to the internet during development.
### Using Cloudflare Tunnels
[Cloudflare Tunnels](https://github.com/cloudflare/cloudflared/releases) provide a free way to expose your local server without opening firewall ports:
```bash theme={null}
cloudflared tunnel --url localhost:3000
```
You'll get a public URL like `https://abc123.trycloudflare.com`. Use this in your webhook config:
```json theme={null}
{
"url": "https://abc123.trycloudflare.com/webhook"
}
```
## Troubleshooting
### Webhooks Not Arriving
* **Endpoint not accessible** - Verify your server is publicly reachable and firewalls allow incoming connections
* **Using HTTP** - Webhook URLs must use HTTPS
* **Wrong events** - Check the `events` filter in your webhook config
### Signature Verification Failing
The most common cause is using the parsed JSON body instead of the raw request body.
```javascript theme={null}
// Wrong - using parsed body
const signature = crypto
.createHmac('sha256', secret)
.update(JSON.stringify(req.body))
.digest('hex');
// Correct - using raw body
app.use('/webhook', express.raw({ type: 'application/json' }));
app.post('/webhook', (req, res) => {
const signature = crypto
.createHmac('sha256', secret)
.update(req.body) // Raw buffer
.digest('hex');
});
```
### Other Issues
* **Wrong secret** - Verify you're using the correct secret from your [account settings](https://www.firecrawl.dev/app/settings?tab=advanced)
* **Timeout errors** - Ensure your endpoint responds within 10 seconds
# Advanced Scraping Guide
Source: https://docs.firecrawl.dev/advanced-scraping-guide
Configure scrape options, browser actions, crawl, map, and the agent endpoint with Firecrawl's full API surface.
Reference for every option across Firecrawl's scrape, crawl, map, and agent endpoints.
## Basic scraping
To scrape a single page and get clean markdown content, use the `/scrape` endpoint.
```python Python theme={null}
# pip install firecrawl-py
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
doc = firecrawl.scrape("https://firecrawl.dev")
print(doc.markdown)
```
```js Node theme={null}
// npm install @mendable/firecrawl-js
import { Firecrawl } from 'firecrawl-js';
const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' });
const doc = await firecrawl.scrape('https://firecrawl.dev');
console.log(doc.markdown);
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://docs.firecrawl.dev"
}'
```
## Scraping PDFs
Firecrawl supports PDFs. Use the `parsers` option (e.g., `parsers: ["pdf"]`) when you want to ensure PDF parsing. You can control the parsing strategy with the `mode` option:
* **`auto`** (default) — attempts fast text-based extraction first, then falls back to OCR if needed.
* **`fast`** — text-based parsing only (embedded text). Fastest, but skips scanned/image-heavy pages.
* **`ocr`** — forces OCR parsing on every page. Use for scanned documents or when `auto` misclassifies a page.
`{ type: "pdf" }` and `"pdf"` both default to `mode: "auto"`.
```json theme={null}
"parsers": [{ "type": "pdf", "mode": "fast", "maxPages": 50 }]
```
## Scrape options
When using the `/scrape` endpoint, you can customize the request with the following options.
### Formats (`formats`)
The `formats` array controls which output types the scraper returns. Default: `["markdown"]`.
**String formats**: pass the name directly (e.g. `"markdown"`).
| Format | Description |
| ---------- | ---------------------------------------------------------------------------- |
| `markdown` | Page content converted to clean Markdown. |
| `html` | Processed HTML with unnecessary elements removed. |
| `rawHtml` | Original HTML exactly as returned by the server. |
| `links` | All links found on the page. |
| `images` | All images found on the page. |
| `summary` | An LLM-generated summary of the page content. |
| `branding` | Extracts brand identity (colors, fonts, typography, spacing, UI components). |
**Object formats**: pass an object with `type` and additional options.
| Format | Options | Description |
| ---------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `json` | `prompt?: string`, `schema?: object` | Extract structured data using an LLM. Provide a JSON schema and/or a natural-language prompt (max 10,000 characters). |
| `screenshot` | `fullPage?: boolean`, `quality?: number`, `viewport?: { width, height }` | Capture a screenshot. Max one per request. Viewport max resolution is 7680×4320. Screenshot URLs expire after 24 hours. |
| `changeTracking` | `modes?: ("json" \| "git-diff")[]`, `tag?: string`, `schema?: object`, `prompt?: string` | Track changes between scrapes. Requires `"markdown"` to also be in the formats array. |
| `attributes` | `selectors: [{ selector: string, attribute: string }]` | Extract specific HTML attributes from elements matching CSS selectors. |
### Content filtering
These parameters control which parts of the page appear in the output. When `onlyMainContent` is `true` (the default), boilerplate (nav, footer, etc.) is stripped. `includeTags` and `excludeTags` are applied against the original page DOM, not the post-filtered result, so your selectors should target elements as they appear in the source HTML. Set `onlyMainContent: false` to use the full page as the starting point for tag filtering.
| Parameter | Type | Default | Description |
| ----------------- | --------- | ------- | ---------------------------------------------------------------------------- |
| `onlyMainContent` | `boolean` | `true` | Return only the main content. Set `false` for the full page. |
| `includeTags` | `array` | — | HTML tags, classes, or IDs to include (e.g. `["h1", "p", ".main-content"]`). |
| `excludeTags` | `array` | — | HTML tags, classes, or IDs to exclude (e.g. `["#ad", "#footer"]`). |
### Timing and cache
| Parameter | Type | Default | Description |
| --------- | -------------- | ----------- | ------------------------------------------------------------------------------------------------------ |
| `waitFor` | `integer` (ms) | `0` | Extra wait time before scraping, on top of smart-wait. Use sparingly. |
| `maxAge` | `integer` (ms) | `172800000` | Return a cached version if fresher than this value (default is 2 days). Set `0` to always fetch fresh. |
| `timeout` | `integer` (ms) | `60000` | Max request duration before aborting (default is 60 seconds). Minimum is 1000 (1 second). |
### PDF parsing
| Parameter | Type | Default | Description |
| --------- | ------- | --------- | -------------------------------------------------------------------------------- |
| `parsers` | `array` | `["pdf"]` | Controls PDF processing. `[]` to skip parsing and return base64 (1 credit flat). |
```json theme={null}
{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": 10 }
```
| Property | Type | Default | Description |
| ---------- | --------------------------- | ------------ | ------------------------------------------------------------------------------------- |
| `type` | `"pdf"` | *(required)* | Parser type. |
| `mode` | `"fast" \| "auto" \| "ocr"` | `"auto"` | `fast`: text-based extraction only. `auto`: fast with OCR fallback. `ocr`: force OCR. |
| `maxPages` | `integer` | — | Cap the number of pages to parse. |
### Actions
Run browser actions before scraping. This is useful for dynamic content, navigation, or user-gated pages. You can include up to 50 actions per request, and the combined wait time across all `wait` actions and `waitFor` must not exceed 60 seconds.
| Action | Parameters | Description |
| ------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `wait` | `milliseconds?: number`, `selector?: string` | Wait for a fixed duration **or** until an element is visible (provide one, not both). When using `selector`, times out after 30 seconds. |
| `click` | `selector: string`, `all?: boolean` | Click an element matching the CSS selector. Set `all: true` to click every match. |
| `write` | `text: string` | Type text into the currently focused field. You must focus the element with a `click` action first. |
| `press` | `key: string` | Press a keyboard key (e.g. `"Enter"`, `"Tab"`, `"Escape"`). |
| `scroll` | `direction?: "up" \| "down"`, `selector?: string` | Scroll the page or a specific element. Direction defaults to `"down"`. |
| `screenshot` | `fullPage?: boolean`, `quality?: number`, `viewport?: { width, height }` | Capture a screenshot. Max viewport resolution is 7680×4320. |
| `scrape` | *(none)* | Capture the current page HTML at this point in the action sequence. |
| `executeJavascript` | `script: string` | Run JavaScript code in the page. Return values are available in the `actions.javascriptReturns` array of the response. |
| `pdf` | `format?: string`, `landscape?: boolean`, `scale?: number` | Generate a PDF. Supported formats: `"A0"` through `"A6"`, `"Letter"`, `"Legal"`, `"Tabloid"`, `"Ledger"`. Defaults to `"Letter"`. |
```python Python theme={null}
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key='fc-YOUR-API-KEY')
doc = firecrawl.scrape('https://example.com', {
'actions': [
{ 'type': 'wait', 'milliseconds': 1000 },
{ 'type': 'click', 'selector': '#accept' },
{ 'type': 'scroll', 'direction': 'down' },
{ 'type': 'click', 'selector': '#q' },
{ 'type': 'write', 'text': 'firecrawl' },
{ 'type': 'press', 'key': 'Enter' },
{ 'type': 'wait', 'milliseconds': 2000 }
],
'formats': ['markdown']
})
print(doc.markdown)
```
```js Node theme={null}
import { Firecrawl } from 'firecrawl-js';
const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' });
const doc = await firecrawl.scrape('https://example.com', {
actions: [
{ type: 'wait', milliseconds: 1000 },
{ type: 'click', selector: '#accept' },
{ type: 'scroll', direction: 'down' },
{ type: 'click', selector: '#q' },
{ type: 'write', text: 'firecrawl' },
{ type: 'press', key: 'Enter' },
{ type: 'wait', milliseconds: 2000 }
],
formats: ['markdown']
});
console.log(doc.markdown);
```
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "wait", "milliseconds": 1000 },
{ "type": "click", "selector": "#accept" },
{ "type": "scroll", "direction": "down" },
{ "type": "click", "selector": "#q" },
{ "type": "write", "text": "firecrawl" },
{ "type": "press", "key": "Enter" },
{ "type": "wait", "milliseconds": 2000 }
],
"formats": ["markdown"]
}'
```
#### Action execution notes
* **Write** requires a preceding `click` to focus the target element.
* **Scroll** accepts an optional `selector` to scroll a specific element instead of the page.
* **Wait** accepts either `milliseconds` (fixed delay) or `selector` (wait until visible).
* Actions run **sequentially**: each step completes before the next begins.
* Actions are **not supported for PDFs**. If the URL resolves to a PDF the request will fail.
#### Advanced action examples
**Taking a screenshot:**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "click", "selector": "#load-more" },
{ "type": "wait", "milliseconds": 1000 },
{ "type": "screenshot", "fullPage": true, "quality": 80 }
]
}'
```
**Clicking multiple elements:**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "click", "selector": ".expand-button", "all": true },
{ "type": "wait", "milliseconds": 500 }
],
"formats": ["markdown"]
}'
```
**Generating a PDF:**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "pdf", "format": "A4", "landscape": false }
]
}'
```
**Executing JavaScript (e.g. extracting embedded page data):**
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://example.com",
"actions": [
{ "type": "executeJavascript", "script": "document.querySelector(\"#__NEXT_DATA__\").textContent" }
],
"formats": ["markdown"]
}'
```
The return value of each `executeJavascript` action is captured in the `actions.javascriptReturns` array of the response.
### Full scrape example
The following request combines multiple scrape options:
```bash cURL theme={null}
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer fc-YOUR-API-KEY' \
-d '{
"url": "https://docs.firecrawl.dev",
"formats": [
"markdown",
"links",
"html",
"rawHtml",
{ "type": "screenshot", "fullPage": true, "quality": 80 }
],
"includeTags": ["h1", "p", "a", ".main-content"],
"excludeTags": ["#ad", "#footer"],
"onlyMainContent": false,
"waitFor": 1000,
"timeout": 15000,
"parsers": ["pdf"]
}'
```
This request returns markdown, HTML, raw HTML, links, and a full-page screenshot. It scopes content to `
Each team has a maximum number of jobs that can be waiting in the concurrency queue. If you exceed this limit, new jobs will be rejected with a `429` status code until existing jobs complete. For larger plans with custom concurrency limits, the max queued jobs is 2,000 times your concurrency limit, capped at 2,000,000.
If you require higher concurrency limits, [contact us about enterprise plans](https://firecrawl.dev/enterprise).
### Extract Plans (Legacy)
Plan
Concurrent Browsers
Max Queued Jobs
Free
2
50,000
Starter
50
100,000
Explorer
100
200,000
Pro
200
400,000
## API Rate Limits
Rate limits are measured in requests per minute and are primarily in place to prevent abuse. When configured correctly, your real bottleneck will be concurrent browsers. Rate limits are applied per team, so all API keys on the same team share the same rate limit counters.
### Current Plans
Plan
/scrape
/map
/crawl
/search
/agent
/crawl/status
/agent/status
Free
10
10
1
5
10
1500
500
Hobby
100
100
15
50
100
1500
25000
Standard
500
500
50
250
500
1500
25000
Growth
5000
5000
250
2500
1000
1500
25000
Scale
7500
7500
750
7500
1000
25000
25000
These rate limits are enforced to ensure fair usage and availability of the API for all users. If you require higher limits, please contact us at [help@firecrawl.com](mailto:help@firecrawl.com) to discuss custom plans.
### Extract Endpoints
The extract endpoints share limits with the corresponding /agent rate limits.
### Batch Scrape Endpoints
The batch scrape endpoints share limits with the corresponding /crawl rate limits.
### Browser Sandbox
The browser sandbox endpoints have per-plan rate limits that scale with your subscription:
Plan
/browser
/browser/\{id}/execute
Free
2
10
Hobby
20
100
Standard
100
500
Growth
1,000
5,000
Scale
1,500
7,500
In addition, each team's plan determines how many browser sessions can be active simultaneously (see Concurrent Browser Limits above). If you exceed this limit, new session requests will return a `429` status code until existing sessions are destroyed.
### FIRE-1 Agent
Requests involving the FIRE-1 agent requests have separate rate limits that are counted independently for each endpoint:
Endpoint
Rate Limit (requests/min)
/scrape
10
/extract
10
### Extract Plans (Legacy)
Plan
/extract (requests/min)
/extract/status (requests/min)
Starter
100
25000
Explorer
500
25000
Pro
1000
25000
# Skill + CLI
Source: https://docs.firecrawl.dev/sdks/cli
Firecrawl Skill is an easy way for AI agents such as Claude Code, Antigravity and OpenCode to use Firecrawl through the CLI.
Search, scrape, and interact with the web directly from the terminal. The Firecrawl CLI works standalone or as a skill that AI coding agents like Claude Code, Antigravity, and OpenCode can discover and use automatically.
## Installation
If you are using an AI agent like Claude Code, you can install the Firecrawl skill below and the agent will set it up for you.
```bash theme={null}
npx -y firecrawl-cli@latest init --all --browser
```
* `--all` installs the Firecrawl skill to every detected AI coding agent
* `--browser` opens the browser for Firecrawl authentication automatically
After installing the skill, restart your agent for it to discover the new skill.
You can also manually install the Firecrawl CLI globally using npm:
```bash CLI theme={null}
# Install globally with npm
npm install -g firecrawl-cli
```
## Authentication
Before using the CLI, you need to authenticate with your Firecrawl API key.
### Login
```bash CLI theme={null}
# Interactive login (opens browser or prompts for API key)
firecrawl login
# Login with browser authentication (recommended for agents)
firecrawl login --browser
# Login with API key directly
firecrawl login --api-key fc-YOUR-API-KEY
# Or set via environment variable
export FIRECRAWL_API_KEY=fc-YOUR-API-KEY
```
### View Configuration
```bash CLI theme={null}
# View current configuration and authentication status
firecrawl view-config
```
### Logout
```bash CLI theme={null}
# Clear stored credentials
firecrawl logout
```
### Self-Hosted / Local Development
For self-hosted Firecrawl instances or local development, use the `--api-url` option:
```bash CLI theme={null}
# Use a local Firecrawl instance (no API key required)
firecrawl --api-url http://localhost:3002 scrape https://example.com
# Or set via environment variable
export FIRECRAWL_API_URL=http://localhost:3002
firecrawl scrape https://example.com
# Configure and persist the custom API URL
firecrawl config --api-url http://localhost:3002
```
When using a custom API URL (anything other than `https://api.firecrawl.dev`), API key authentication is automatically skipped, allowing you to use local instances without an API key.
### Check Status
Verify installation, authentication, and view rate limits:
```bash CLI theme={null}
firecrawl --status
```
Output when ready:
```
🔥 firecrawl cli v1.1.1
● Authenticated via FIRECRAWL_API_KEY
Concurrency: 0/100 jobs (parallel scrape limit)
Credits: 500,000 remaining
```
* **Concurrency**: Maximum parallel jobs. Run parallel operations close to this limit but not above.
* **Credits**: Remaining API credits. Each scrape/crawl consumes credits.
## Commands
### Scrape
Scrape a single URL and extract its content in various formats.
Use `--only-main-content` to get clean output without navigation, footers, and ads. This is recommended for most use cases where you want just the article or main page content.
```bash CLI theme={null}
# Scrape a URL (default: markdown output)
firecrawl https://example.com
# Or use the explicit scrape command
firecrawl scrape https://example.com
# Recommended: use --only-main-content for clean output without nav/footer
firecrawl https://example.com --only-main-content
```
#### Output Formats
```bash CLI theme={null}
# Get HTML output
firecrawl https://example.com --html
# Multiple formats (returns JSON)
firecrawl https://example.com --format markdown,links
# Get images from a page
firecrawl https://example.com --format images
# Get a summary of the page content
firecrawl https://example.com --format summary
# Track changes on a page
firecrawl https://example.com --format changeTracking
# Available formats: markdown, html, rawHtml, links, screenshot, json, images, summary, changeTracking, attributes, branding
```
#### Scrape Options
```bash CLI theme={null}
# Extract only main content (removes navs, footers)
firecrawl https://example.com --only-main-content
# Wait for JavaScript rendering
firecrawl https://example.com --wait-for 3000
# Take a screenshot
firecrawl https://example.com --screenshot
# Include/exclude specific HTML tags
firecrawl https://example.com --include-tags article,main
firecrawl https://example.com --exclude-tags nav,footer
# Save output to file
firecrawl https://example.com -o output.md
# Pretty print JSON output
firecrawl https://example.com --format markdown,links --pretty
# Force JSON output even with single format
firecrawl https://example.com --json
# Show request timing information
firecrawl https://example.com --timing
```
**Available Options:**
| Option | Short | Description |
| ----------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--url ` | `-u` | URL to scrape (alternative to positional argument) |
| `--format ` | `-f` | Output formats (comma-separated): `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `json`, `images`, `summary`, `changeTracking`, `attributes`, `branding` |
| `--html` | `-H` | Shortcut for `--format html` |
| `--only-main-content` | | Extract only main content |
| `--wait-for ` | | Wait time in milliseconds for JS rendering |
| `--screenshot` | | Take a screenshot |
| `--include-tags ` | | HTML tags to include (comma-separated) |
| `--exclude-tags ` | | HTML tags to exclude (comma-separated) |
| `--output ` | `-o` | Save output to file |
| `--json` | | Force JSON output even with single format |
| `--pretty` | | Pretty print JSON output |
| `--timing` | | Show request timing and other useful information |
***
### Search
Search the web and optionally scrape the results.
```bash CLI theme={null}
# Search the web
firecrawl search "web scraping tutorials"
# Limit results
firecrawl search "AI news" --limit 10
# Pretty print results
firecrawl search "machine learning" --pretty
```
#### Search Options
```bash CLI theme={null}
# Search specific sources
firecrawl search "AI" --sources web,news,images
# Search with category filters
firecrawl search "react hooks" --categories github
firecrawl search "machine learning" --categories research,pdf
# Time-based filtering
firecrawl search "tech news" --tbs qdr:h # Last hour
firecrawl search "tech news" --tbs qdr:d # Last day
firecrawl search "tech news" --tbs qdr:w # Last week
firecrawl search "tech news" --tbs qdr:m # Last month
firecrawl search "tech news" --tbs qdr:y # Last year
# Location-based search
firecrawl search "restaurants" --location "Berlin,Germany" --country DE
# Search and scrape results
firecrawl search "documentation" --scrape --scrape-formats markdown
# Save to file
firecrawl search "firecrawl" --pretty -o results.json
```
**Available Options:**
| Option | Description |
| ---------------------------- | ------------------------------------------------------------------------------------------- |
| `--limit ` | Maximum results (default: 5, max: 100) |
| `--sources ` | Sources to search: `web`, `images`, `news` (comma-separated) |
| `--categories ` | Filter by category: `github`, `research`, `pdf` (comma-separated) |
| `--tbs ` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |
| `--location ` | Geo-targeting (e.g., "Berlin,Germany") |
| `--country ` | ISO country code (default: US) |
| `--timeout ` | Timeout in milliseconds (default: 60000) |
| `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints |
| `--scrape` | Scrape search results |
| `--scrape-formats ` | Formats for scraped content (default: markdown) |
| `--only-main-content` | Include only main content when scraping (default: true) |
| `--json` | Output as JSON |
| `--output ` | Save output to file |
| `--pretty` | Pretty print JSON output |
***
### Map
Discover all URLs on a website quickly.
```bash CLI theme={null}
# Discover all URLs on a website
firecrawl map https://example.com
# Output as JSON
firecrawl map https://example.com --json
# Limit number of URLs
firecrawl map https://example.com --limit 500
```
#### Map Options
```bash CLI theme={null}
# Filter URLs by search query
firecrawl map https://example.com --search "blog"
# Include subdomains
firecrawl map https://example.com --include-subdomains
# Control sitemap usage
firecrawl map https://example.com --sitemap include # Use sitemap
firecrawl map https://example.com --sitemap skip # Skip sitemap
firecrawl map https://example.com --sitemap only # Only use sitemap
# Ignore query parameters (dedupe URLs)
firecrawl map https://example.com --ignore-query-parameters
# Wait for map to complete with timeout
firecrawl map https://example.com --wait --timeout 60
# Save to file
firecrawl map https://example.com -o urls.txt
firecrawl map https://example.com --json --pretty -o urls.json
```
**Available Options:**
| Option | Description |
| --------------------------- | ----------------------------------------------- |
| `--url ` | URL to map (alternative to positional argument) |
| `--limit ` | Maximum URLs to discover |
| `--search ` | Filter URLs by search query |
| `--sitemap ` | Sitemap handling: `include`, `skip`, `only` |
| `--include-subdomains` | Include subdomains |
| `--ignore-query-parameters` | Treat URLs with different params as same |
| `--wait` | Wait for map to complete |
| `--timeout ` | Timeout in seconds |
| `--json` | Output as JSON |
| `--output ` | Save output to file |
| `--pretty` | Pretty print JSON output |
***
### Browser
Have your agents interact with the web using a secure browser sandbox.
Launch cloud browser sessions and execute Python, JavaScript, or bash code remotely. Each session runs a full Chromium instance — no local browser install required. Code runs server-side with a pre-configured [Playwright](https://playwright.dev/) `page` object ready to use.
```bash CLI theme={null}
# Launch a cloud browser session
firecrawl browser launch-session
# Execute agent-browser commands (default - "agent-browser" is auto-prefixed)
firecrawl browser execute "open https://example.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "click @e5"
firecrawl browser execute "scrape"
# Execute Playwright Python code
firecrawl browser execute --python 'await page.goto("https://example.com")
print(await page.title())'
# Execute Playwright JavaScript code
firecrawl browser execute --node 'await page.goto("https://example.com"); console.log(await page.title());'
# List all sessions (or: list active / list destroyed)
firecrawl browser list
# Close the active session
firecrawl browser close
```
#### Browser Options
```bash CLI theme={null}
# Launch with custom TTL (10 minutes) and live view
firecrawl browser launch-session --ttl 600 --stream
# Launch with inactivity timeout
firecrawl browser launch-session --ttl 120 --ttl-inactivity 60
# agent-browser commands (default - "agent-browser" is auto-prefixed)
firecrawl browser execute "open https://news.ycombinator.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "click @e3"
firecrawl browser execute "scrape"
# Playwright Python - navigate, interact, extract
firecrawl browser execute --python '
await page.goto("https://news.ycombinator.com")
items = await page.query_selector_all(".titleline > a")
for item in items[:5]:
print(await item.text_content())
'
# Playwright JavaScript - same page object
firecrawl browser execute --node '
await page.goto("https://example.com");
const title = await page.title();
console.log(title);
'
# Explicit bash mode - runs in the sandbox
firecrawl browser execute --bash "agent-browser snapshot"
# Target a specific session
firecrawl browser execute --session --python 'print(await page.title())'
# Save output to file
firecrawl browser execute "scrape" -o result.txt
# Close a specific session
firecrawl browser close --session
# List sessions (all / active / destroyed)
firecrawl browser list
firecrawl browser list active --json
```
**Subcommands:**
| Subcommand | Description |
| ---------------- | ----------------------------------------------------------------------------------- |
| `launch-session` | Launch a new cloud browser session (returns session ID, CDP URL, and live view URL) |
| `execute ` | Execute Playwright Python/JS code or bash commands in a session |
| `list [status]` | List browser sessions (filter by `active` or `destroyed`) |
| `close` | Close a browser session |
**Execute Options:**
| Option | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--bash` | Execute bash commands remotely in the sandbox (default). [agent-browser](https://github.com/vercel-labs/agent-browser) (40+ commands) is pre-installed and auto-prefixed. `CDP_URL` is auto-injected so agent-browser connects to your session automatically. Best approach for AI agents. |
| `--python` | Execute as Playwright Python code. A Playwright `page` object is available — use `await page.goto()`, `await page.title()`, etc. |
| `--node` | Execute as Playwright JavaScript code. Same `page` object available. |
| `--session ` | Target a specific session (default: active session) |
**Launch Options:**
| Option | Description |
| ---------------------------- | ------------------------------------------------------------------- |
| `--ttl ` | Total session TTL (default: 600, range: 30–3600) |
| `--ttl-inactivity ` | Auto-close after inactivity (range: 10–3600) |
| `--profile ` | Name for a profile (saves and reuses browser state across sessions) |
| `--no-save-changes` | Load existing profile data without saving changes back |
| `--stream` | Enable live view streaming |
**Common Options:**
| Option | Description |
| ----------------- | --------------------- |
| `--output ` | Save output to file |
| `--json` | Output as JSON format |
***
### Crawl
Crawl an entire website starting from a URL.
```bash CLI theme={null}
# Start a crawl (returns job ID immediately)
firecrawl crawl https://example.com
# Wait for crawl to complete
firecrawl crawl https://example.com --wait
# Wait with progress indicator
firecrawl crawl https://example.com --wait --progress
```
#### Check Crawl Status
```bash CLI theme={null}
# Check crawl status using job ID
firecrawl crawl
# Example with a real job ID
firecrawl crawl 550e8400-e29b-41d4-a716-446655440000
```
#### Crawl Options
```bash CLI theme={null}
# Limit crawl depth and pages
firecrawl crawl https://example.com --limit 100 --max-depth 3 --wait
# Include only specific paths
firecrawl crawl https://example.com --include-paths /blog,/docs --wait
# Exclude specific paths
firecrawl crawl https://example.com --exclude-paths /admin,/login --wait
# Include subdomains
firecrawl crawl https://example.com --allow-subdomains --wait
# Crawl entire domain
firecrawl crawl https://example.com --crawl-entire-domain --wait
# Rate limiting
firecrawl crawl https://example.com --delay 1000 --max-concurrency 2 --wait
# Custom polling interval and timeout
firecrawl crawl https://example.com --wait --poll-interval 10 --timeout 300
# Save results to file
firecrawl crawl https://example.com --wait --pretty -o results.json
```
**Available Options:**
| Option | Description |
| --------------------------- | ------------------------------------------------- |
| `--url ` | URL to crawl (alternative to positional argument) |
| `--wait` | Wait for crawl to complete |
| `--progress` | Show progress indicator while waiting |
| `--poll-interval ` | Polling interval (default: 5) |
| `--timeout ` | Timeout when waiting |
| `--status` | Check status of existing crawl job |
| `--limit ` | Maximum pages to crawl |
| `--max-depth ` | Maximum crawl depth |
| `--include-paths ` | Paths to include (comma-separated) |
| `--exclude-paths ` | Paths to exclude (comma-separated) |
| `--sitemap ` | Sitemap handling: `include`, `skip`, `only` |
| `--allow-subdomains` | Include subdomains |
| `--allow-external-links` | Follow external links |
| `--crawl-entire-domain` | Crawl entire domain |
| `--ignore-query-parameters` | Treat URLs with different params as same |
| `--delay ` | Delay between requests |
| `--max-concurrency ` | Maximum concurrent requests |
| `--output ` | Save output to file |
| `--pretty` | Pretty print JSON output |
***
### Agent
Search and gather data from the web using natural language prompts.
```bash CLI theme={null}
# Basic usage - URLs are optional
firecrawl agent "Find the top 5 AI startups and their funding amounts" --wait
# Focus on specific URLs
firecrawl agent "Compare pricing plans" --urls https://slack.com/pricing,https://teams.microsoft.com/pricing --wait
# Use a schema for structured output
firecrawl agent "Get company information" --urls https://example.com --schema '{"name": "string", "founded": "number"}' --wait
# Use schema from a file
firecrawl agent "Get product details" --urls https://example.com --schema-file schema.json --wait
```
#### Agent Options
```bash CLI theme={null}
# Use Spark 1 Pro for higher accuracy
firecrawl agent "Competitive analysis across multiple domains" --model spark-1-pro --wait
# Set max credits to limit costs
firecrawl agent "Gather contact information from company websites" --max-credits 100 --wait
# Check status of an existing job
firecrawl agent --status
# Custom polling interval and timeout
firecrawl agent "Summarize recent blog posts" --wait --poll-interval 10 --timeout 300
# Save output to file
firecrawl agent "Find pricing information" --urls https://example.com --wait -o pricing.json --pretty
```
**Available Options:**
| Option | Description |
| --------------------------- | -------------------------------------------------------------------------------------- |
| `--urls ` | Optional list of URLs to focus the agent on (comma-separated) |
| `--model ` | Model to use: `spark-1-mini` (default, 60% cheaper) or `spark-1-pro` (higher accuracy) |
| `--schema ` | JSON schema for structured output (inline JSON string) |
| `--schema-file ` | Path to JSON schema file for structured output |
| `--max-credits ` | Maximum credits to spend (job fails if limit reached) |
| `--status` | Check status of existing agent job |
| `--wait` | Wait for agent to complete before returning results |
| `--poll-interval ` | Polling interval when waiting (default: 5) |
| `--timeout ` | Timeout when waiting (default: no timeout) |
| `--output ` | Save output to file |
| `--json` | Output as JSON format |
***
### Credit Usage
Check your team's credit balance and usage.
```bash CLI theme={null}
# View credit usage
firecrawl credit-usage
# Output as JSON
firecrawl credit-usage --json --pretty
```
***
### Version
Display the CLI version.
```bash CLI theme={null}
firecrawl version
# or
firecrawl --version
```
## Global Options
These options are available for all commands:
| Option | Short | Description |
| ----------------- | ----- | ------------------------------------------------------ |
| `--status` | | Show version, auth, concurrency, and credits |
| `--api-key ` | `-k` | Override stored API key for this command |
| `--api-url ` | | Use custom API URL (for self-hosted/local development) |
| `--help` | `-h` | Show help for a command |
| `--version` | `-V` | Show CLI version |
## Output Handling
The CLI outputs to stdout by default, making it easy to pipe or redirect:
```bash CLI theme={null}
# Pipe markdown to another command
firecrawl https://example.com | head -50
# Redirect to a file
firecrawl https://example.com > output.md
# Save JSON with pretty formatting
firecrawl https://example.com --format markdown,links --pretty -o data.json
```
### Format Behavior
* **Single format**: Outputs raw content (markdown text, HTML, etc.)
* **Multiple formats**: Outputs JSON with all requested data
```bash CLI theme={null}
# Raw markdown output
firecrawl https://example.com --format markdown
# JSON output with multiple formats
firecrawl https://example.com --format markdown,links
```
## Examples
### Quick Scrape
```bash CLI theme={null}
# Get markdown content from a URL (use --only-main-content for clean output)
firecrawl https://docs.firecrawl.dev --only-main-content
# Get HTML content
firecrawl https://example.com --html -o page.html
```
### Full Site Crawl
```bash CLI theme={null}
# Crawl a docs site with limits
firecrawl crawl https://docs.example.com --limit 50 --max-depth 2 --wait --progress -o docs.json
```
### Site Discovery
```bash CLI theme={null}
# Find all blog posts
firecrawl map https://example.com --search "blog" -o blog-urls.txt
```
### Research Workflow
```bash CLI theme={null}
# Search and scrape results for research
firecrawl search "machine learning best practices 2024" --scrape --scrape-formats markdown --pretty
```
### Agent
```bash CLI theme={null}
# URLs are optional
firecrawl agent "Find the top 5 AI startups and their funding amounts" --wait
# Focus on specific URLs
firecrawl agent "Compare pricing plans" --urls https://slack.com/pricing,https://teams.microsoft.com/pricing --wait
```
### Browser Automation
```bash CLI theme={null}
# Launch a session, scrape a page, and close
firecrawl browser launch-session
firecrawl browser execute "open https://news.ycombinator.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "scrape"
firecrawl browser close
# Use agent-browser via bash mode (default — recommended for AI agents)
firecrawl browser launch-session
firecrawl browser execute "open https://example.com"
firecrawl browser execute "snapshot"
# snapshot returns @ref IDs — use them to interact
firecrawl browser execute "click @e5"
firecrawl browser execute "fill @e3 'search query'"
firecrawl browser execute "scrape"
# Run --help to see all 40+ commands
firecrawl browser execute --bash "agent-browser --help"
firecrawl browser close
```
### Combine with Other Tools
```bash CLI theme={null}
# Extract URLs from search results
jq -r '.data.web[].url' search-results.json
# Get titles from search results
jq -r '.data.web[] | "\(.title): \(.url)"' search-results.json
# Extract links and process with jq
firecrawl https://example.com --format links | jq '.links[].url'
# Count URLs from map
firecrawl map https://example.com | wc -l
```
## Telemetry
The CLI collects anonymous usage data during authentication to help improve the product:
* CLI version, OS, and Node.js version
* Development tool detection (e.g., Cursor, VS Code, Claude Code)
**No command data, URLs, or file contents are collected via the CLI.**
To disable telemetry, set the environment variable:
```bash CLI theme={null}
export FIRECRAWL_NO_TELEMETRY=1
```
## Open Source
The Firecrawl CLI and Skill are open source and available on GitHub: [firecrawl/cli](https://github.com/firecrawl/cli)
> Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.
# .NET
Source: https://docs.firecrawl.dev/sdks/dotnet
Firecrawl .NET SDK is a wrapper around the Firecrawl API to help you easily turn websites into markdown.
## Installation
The official .NET SDK is maintained in the Firecrawl monorepo at [apps/.net-sdk](https://github.com/firecrawl/firecrawl/tree/main/apps/.net-sdk).
To install the Firecrawl .NET SDK, add the NuGet package:
```bash theme={null}
dotnet add package firecrawl-sdk
```
```powershell theme={null}
Install-Package firecrawl-sdk
```
```xml theme={null}
```
Requires .NET 8.0 or later.
## Usage
1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
2. Set the API key as an environment variable named `FIRECRAWL_API_KEY`, or pass it to the `FirecrawlClient` constructor
Here is a quick example using the current SDK API surface:
```csharp theme={null}
using Firecrawl;
using Firecrawl.Models;
var client = new FirecrawlClient("fc-your-api-key");
// Scrape a single page
var doc = await client.ScrapeAsync("https://firecrawl.dev",
new ScrapeOptions { Formats = new List