Skip to main content
Website monitoring watches a whole site instead of a fixed list of URLs. Each check runs a crawl for the target url, scrapes every discovered page, and reconciles the result against the last retained snapshot. That catches added, changed, and removed pages, not just edits to pages you already named. It’s the right choice for docs sites, blogs, changelogs, help centers, and competitor sites. This page covers the crawl target. Scheduling, goals and judging, change tracking, notifications, and pricing are shared across all monitor types. See the Monitoring overview.

​
Create a website monitor

Create a monitor with a crawl target to diff every page discovered by a crawl on each check: 
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

monitor = firecrawl.create_monitor(
    name="Docs monitor",
    schedule={"cron": "7-59/15 * * * *", "timezone": "UTC"},
    goal="Notify me when docs pages add, remove, or materially change API behavior",
    targets=[
        {
            "type": "crawl",
            "url": "https://example.com/docs",
            "crawlOptions": {
                "limit": 100,
                "maxDiscoveryDepth": 3,
            },
        }
    ],
    webhook={
        "url": "https://example.com/webhooks/firecrawl",
        "events": ["monitor.page", "monitor.check.completed"],
    },
)

print(monitor.id)

​
Crawl target

A crawl target requires type and a single url. Use crawlOptions for crawl behavior and scrapeOptions for how each discovered page is scraped:
Crawl target
{
  "type": "crawl",
  "url": "https://example.com/docs",
  "crawlOptions": {
    "limit": 100,
    "includePaths": ["/docs"]
  },
  "scrapeOptions": {
    "formats": ["markdown"]
  }
}
Common crawlOptions fields:
  • limit: Maximum number of pages a check will crawl.
  • maxDiscoveryDepth: How many links deep from the starting url to discover pages.
  • maxDepth: Maximum crawl depth.
  • includePaths: Only monitor URLs matching these path patterns (for example, /docs).
  • excludePaths: Skip URLs matching these path patterns.
As with page monitors, monitor-triggered scrapes default maxAge to 0, so each check re-scrapes discovered pages fresh unless you set a different maxAge in scrapeOptions.

​
What each check reports

A crawl check reconciles every discovered page against the previous check and records a per-page status:
  • same: The page was discovered again and did not change.
  • changed: The page was discovered again and changed.
  • new: The page was discovered for the first time.
  • removed: A page from the previous check was no longer discovered.
  • error: The page could not be checked.
To alert on specific structured fields across the crawled pages, add a changeTracking format to scrapeOptions. See Change tracking.

​
Shared configuration