CrewAI

​

Using Firecrawl with CrewAI

​

Installing Firecrawl Tools inside of CrewAI

• Get an API key from your firecrawl.dev dashboard and set it in environment variables (FIRECRAWL_API_KEY).
• Install the Firecrawl SDK along with crewai[tools] package:

pip install firecrawl-py 'crewai[tools]'

​

Tools

​

FirecrawlCrawlWebsiteTool

​

Example

from crewai_tools import FirecrawlCrawlWebsiteTool

tool = FirecrawlCrawlWebsiteTool(config={"limit": 5})

# Agents call the tool with a URL. You can also run it directly:
result = tool.run(url="https://firecrawl.dev")
print(result.status, len(result.data))

​

Arguments

• api_key: Optional. Specifies Firecrawl API key. Defaults is the FIRECRAWL_API_KEY environment variable.
• config: Optional. Crawl parameters in snake_case. Passing config replaces the defaults below.
  • limit: Maximum number of pages to crawl. Default: 10.
  • max_discovery_depth: Maximum depth for discovering pages. Default: 2.
  • ignore_sitemap: Whether to ignore the sitemap. Default: True.
  • allow_external_links: Allow crawling external links. Default: False.
  • allow_subdomains: Allow crawling subdomains. Default: False.
  • delay: Delay between requests in milliseconds. Default: None.
  • scrape_options: Options for each crawled page. Default: {"formats": ["markdown"], "only_main_content": True, "timeout": 10000}.

​

FirecrawlScrapeWebsiteTool

​

Example

from crewai_tools import FirecrawlScrapeWebsiteTool

tool = FirecrawlScrapeWebsiteTool(config={"formats": ["markdown"], "only_main_content": True})

result = tool.run(url="https://firecrawl.dev")
print(result.markdown)

​

Arguments

• api_key: Optional. Specifies Firecrawl API key. Defaults is the FIRECRAWL_API_KEY environment variable.
• config: Optional. Scrape parameters in snake_case. Passing config replaces the defaults below.
  • formats: Output formats, such as "markdown", "html", "links", or a JSON format object for structured extraction. Default: ["markdown"].
  • only_main_content: Only return the main content of the page excluding headers, navs, footers, etc. Default: True.
  • include_tags / exclude_tags: HTML tags to include or exclude. Default: [].
  • max_age: Return a cached version if it is younger than this many milliseconds. Default: 172800000 (2 days).
  • wait_for: Delay in milliseconds before fetching content. Default: 0.
  • mobile: Emulate a mobile device. Default: False.
  • timeout: Timeout in milliseconds for the request. Default: None.

​

FirecrawlSearchTool

​

Example

from crewai_tools import FirecrawlSearchTool

tool = FirecrawlSearchTool(config={"limit": 3})

result = tool.run(query="what is firecrawl?")
print([item.url for item in result.web])

​

Arguments

• api_key: Optional. Specifies Firecrawl API key. Defaults is the FIRECRAWL_API_KEY environment variable.
• config: Optional. Search parameters in snake_case. Passing config replaces the defaults below.
  • limit: Maximum number of search results to return. Default: 5.
  • tbs: Time-based search filter, for example "qdr:d" for the past day. Default: None.
  • location: Location for search results. Default: None.
  • timeout: Timeout in milliseconds for the request. Default: None.
  • scrape_options: Scrape each result and return its content. Default: {"formats": ["markdown"], "only_main_content": True}. Leave it out to get only titles, URLs, and descriptions.