​
How It Works

Get your scraping results 500% faster when you don’t need the absolute freshest data. Firecrawl keeps an index of previously scraped pages. Add the maxAge parameter to your request and we’ll:

  1. Return instantly if we have a recent version of the page
  2. Scrape fresh only if our version is older than your specified age
  3. Save you time - results come back in milliseconds instead of seconds

​
When to Use This

Great for:

  • Documentation, articles, product pages
  • Bulk processing jobs
  • Development and testing
  • Building knowledge bases

Skip for:

  • Real-time data (stock prices, live scores, breaking news)
  • Frequently updated content
  • Time-sensitive applications

​
Usage

Simply add the maxAge parameter to your scrape request. The value is in milliseconds - for example, 3600000 means “use cached data if it’s less than 1 hour old.”

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Use cached data if it's less than 1 hour old (3600000 ms)
# This can be 500% faster than a fresh scrape!
scrape_result = app.scrape_url(
    'https://firecrawl.dev', 
    params={
        'formats': ['markdown'],
        'maxAge': 3600000  # 1 hour in milliseconds
    }
)

print(scrape_result['markdown'])

​
Common maxAge Values

Here are some helpful reference values:

  • 5 minutes: 300000 - For semi-dynamic content
  • 1 hour: 3600000 - For content that updates hourly
  • 1 day: 86400000 - For daily-updated content
  • 1 week: 604800000 - For relatively static content

​
Performance Impact

With maxAge enabled:

  • 500% faster response times for recent content
  • Instant results instead of waiting for fresh scrapes

​
Important Notes

  • Default: maxAge defaults to 0, which means always scrape fresh
  • Fresh when needed: If our data is older than maxAge, we scrape fresh automatically
  • No stale data: You’ll never get data older than your specified maxAge

​
Faster Crawling

The same speed benefits apply when crawling multiple pages. Use maxAge within scrapeOptions to get cached results for pages we’ve seen recently.

from firecrawl import FirecrawlApp, ScrapeOptions

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Crawl with cached scraping - 500% faster for pages we've seen recently
crawl_result = app.crawl_url(
    'https://firecrawl.dev', 
    limit=100,
    scrape_options=ScrapeOptions(
        formats=['markdown'],
        maxAge=3600000  # Use cached data if less than 1 hour old
    )
)

for page in crawl_result['data']:
    print(f"URL: {page['metadata']['sourceURL']}")
    print(f"Content: {page['markdown'][:200]}...")

When crawling with maxAge, each page in your crawl will benefit from the 500% speed improvement if we have recent cached data for that page.

Start using maxAge today for dramatically faster scrapes and crawls!