Searching the web and scraping the results with Firecrawl

Firecrawl integrates its SERP (Search Engine Results Page) API with its robust scraping infrastructure to provide a seamless search and scrape functionality through a single endpoint. Here’s why:

  1. Unified Search Query: Users submit a search query via the SERP endpoint.

  2. Automated Result Scraping: Firecrawl automatically processes the search results and utilizes its scraping capabilities to extract data from each result page.

  3. Data Delivery: The scraped data from all result pages is compiled and delivered in a clean markdown - ready to use.

This integration allows users to efficiently perform web searches and obtain comprehensive, scraped data from multiple sources with minimal effort.

For more details, refer to the Search Endpoint Documentation.

Search any query

/search endpoint

Used to search the web, get the most relevant results, scrape each page and return the markdown.

curl -X POST https://api.firecrawl.dev/v0/search \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "query": "firecrawl",
      "pageOptions": {
        "fetchPageContent": true // false for a fast serp api
      }
    }'
{
  "success": true,
  "data": [
    {
      "url": "https://mendable.ai",
      "markdown": "# Markdown Content",
      "provider": "web-scraper",
      "metadata": {
        "title": "Mendable | AI for CX and Sales",
        "description": "AI for CX and Sales",
        "language": null,
        "sourceURL": "https://www.mendable.ai/"
      }
    }
  ]
}

With Python SDK

Installing Python SDK

pip install firecrawl-py

Search a query

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="YOUR_API_KEY")

result = app.search(query="What is firecrawl?")

The response will be similar to the one shown in the curl command above.

With JavaScript SDK

Installing JavaScript SDK

npm install @mendable/firecrawl-js

Search a query

import FirecrawlApp from '@mendable/firecrawl-js';

// Initialize the FirecrawlApp with your API key
const app = new FirecrawlApp({ apiKey: 'YOUR_API_KEY' });

// Perform a search
const result = await app.search('What is firecrawl?');

The response will be similar to the one shown in the curl command above.