Verifying Freshness and Liveness

A successful scrape tells you what the page returned. It does not prove that the state represented by the page is current. These are two separate questions.

Freshness → Is this content recent, or a copy reused from Firecrawl’s cache? Controlled by maxAge.
Liveness → Is the underlying thing still real and active? Your application decides this from the available evidence.

This guide explains the difference, walks through the maxAge tradeoff, and gives a checklist and worked example for freshness-sensitive actions.

Quick Comparison

	Freshness	Liveness
Question	Is this content recent, or reused from cache?	Is the object the page describes still active?
You control it with	The `maxAge` request parameter	Your own domain logic
Firecrawl reports	`metadata.cacheState` (`"hit"` or `"miss"`), and `metadata.cachedAt` on a hit	Nothing directly — only page evidence
Evidence you get	Whether the response came from cache	Page content, `metadata.statusCode`, and `metadata.url` vs `metadata.sourceURL`
Settled by an HTTP 200 with content?	No — a 200 says nothing about how recent the content is	No — a 200 only describes the page response

The Freshness Tradeoff (`maxAge`)

Firecrawl caches previously scraped pages and returns a recent copy when one is available, which cuts latency. maxAge is the maximum age, in milliseconds, of a cached copy that Firecrawl may return instead of retrieving the page again.

Omit maxAge: Firecrawl may return recently cached content. The default window is 2 days; Firecrawl may use a different window for some sites.
Set maxAge: 0: Firecrawl skips the cache for that request and retrieves the page. This trades latency and reliability for a fresher retrieval.

Keep caching on by default. Pay the maxAge: 0 latency cost only for the reads where staleness would cause a wrong or costly decision — it does not change what the page costs you in credits. metadata.cacheState is returned when Firecrawl considered its cache for the request, so it is a useful check while you tune maxAge. It is not part of a maxAge: 0 response, because that request skips the cache altogether. For caching mechanics, common maxAge values, cache-hit matching rules, and the request options that bypass caching automatically, see Faster Scraping.

Where `maxAge` Applies

Endpoint	Behavior
`/scrape`	`maxAge` is honored on the request body
`/crawl`, `/batch/scrape`	`maxAge` is honored inside `scrapeOptions`
`/search`	Search applies its own freshness window to the pages it scrapes, so `maxAge` in `scrapeOptions` does not take effect
`/parse`	`/parse` always processes the file you supply and never serves or stores cached content, so `maxAge` and `storeInCache` have no effect

If you need a fresh retrieval of a page you found through /search, scrape that URL again with /scrape and maxAge: 0.

Freshness Is Not Liveness

Even with maxAge: 0, the result only tells you what the page returned on that retrieval. A page can return HTTP 200 with content while representing an outdated, unavailable, or otherwise changed state. So neither the status code nor the presence of content settles liveness. Liveness is a conclusion your application draws from source-specific evidence.

Freshness-Sensitive Action Checklist

Before an action that depends on current state, treat scrape output as evidence, not proof:

Use maxAge: 0 for the final retrieval so the response is not served from cache.
Do not treat HTTP 200 or non-empty content as proof of liveness.
Inspect rendered content and redirect evidence. metadata.sourceURL is the URL you requested; metadata.url is the URL the engine reports for the response. When the two differ, it can indicate a redirect to a different resource. Matching values do not prove that no redirect occurred.
Prefer source-specific APIs or identifiers where available — they often expose an explicit status that a rendered page hides.
Treat inconclusive evidence as unknown, and stop before the expensive or irreversible step, rather than assuming active.

Worked Example: Collect Current Page Evidence

Skip the cache, then collect the rendered content and response metadata for your application’s own validation rules. The scrape supplies evidence; it does not decide the domain-specific state.

import os

from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])

def collect_current_page_evidence(url: str) -> dict:
    # max_age=0 skips Firecrawl's cache for this request.
    doc = firecrawl.scrape(url, formats=["markdown"], max_age=0)

    metadata = doc.metadata
    requested_url = (metadata.source_url if metadata else None) or url
    response_url = metadata.url if metadata else None

    return {
        "markdown": doc.markdown or "",
        "status_code": metadata.status_code if metadata else None,
        "requested_url": requested_url,
        "response_url": response_url,
        "possible_redirect": bool(response_url and response_url != requested_url),
    }


evidence = collect_current_page_evidence("https://example.com/resource")
# Apply source-specific content, status, API, or identifier checks here.
# If they are inconclusive, keep the state unknown.

import { Firecrawl } from "firecrawl";

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

async function collectCurrentPageEvidence(url) {
  // maxAge: 0 skips Firecrawl's cache for this request.
  const doc = await firecrawl.scrape(url, {
    formats: ["markdown"],
    maxAge: 0,
  });

  const requestedURL = doc.metadata?.sourceURL ?? url;
  const responseURL = doc.metadata?.url;

  return {
    markdown: doc.markdown ?? "",
    statusCode: doc.metadata?.statusCode,
    requestedURL,
    responseURL,
    possibleRedirect: Boolean(responseURL && responseURL !== requestedURL),
  };
}

const evidence = await collectCurrentPageEvidence("https://example.com/resource");
// Apply source-specific content, status, API, or identifier checks here.
// If they are inconclusive, keep the state unknown.

curl -s -X POST "https://api.firecrawl.dev/v2/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/resource",
    "formats": ["markdown"],
    "maxAge": 0
  }'

The important boundary is after collection: Firecrawl supplies page evidence; your application interprets that evidence using source-specific rules. If those rules are inconclusive, keep the state unknown.

Recommendations by Scenario

Scenario	Recommended approach
Read product copy, docs, or reference content	Omit `maxAge` and use the default cache window
Dashboard or report refreshed on a schedule	Non-zero `maxAge` sized to your refresh interval
Final check before an action that depends on current state	`maxAge: 0` to skip the cache + the checklist above
Confirming an object is truly still active	Prefer the source’s API or status field; treat a scrape as evidence only
Ambiguous rendered page (200 but no positive signal)	Classify as `unknown`; stop before the irreversible step

Key Takeaways

Freshness and liveness are different questions. maxAge controls freshness; liveness is a decision you make from evidence.
HTTP 200 plus content does not prove that the represented state is current.
For freshness-sensitive actions, use maxAge: 0 and follow the checklist. Inspect the rendered content, compare metadata.url against metadata.sourceURL for possible redirect evidence, and prefer source-specific APIs.
Treat inconclusive evidence as unknown. A scrape alone should never upgrade an object to active; stop before expensive or irreversible steps.
Firecrawl has no liveness field. Your application makes that determination in its own domain terms.

Get Started

Core Endpoints

More

Quickstarts

Developer Guides

Webhooks

Use Cases

Other

Contributing

Verifying Freshness and Liveness

Quick Comparison

The Freshness Tradeoff (`maxAge`)

Where `maxAge` Applies

Freshness Is Not Liveness

Freshness-Sensitive Action Checklist

Worked Example: Collect Current Page Evidence

Recommendations by Scenario

Key Takeaways

Further Reading

​Quick Comparison

​The Freshness Tradeoff (maxAge)

​Where maxAge Applies

​Freshness Is Not Liveness

​Freshness-Sensitive Action Checklist

​Worked Example: Collect Current Page Evidence

​Recommendations by Scenario

​Key Takeaways

​Further Reading

Quick Comparison

The Freshness Tradeoff (`maxAge`)

Where `maxAge` Applies

Freshness Is Not Liveness

Freshness-Sensitive Action Checklist

Worked Example: Collect Current Page Evidence

Recommendations by Scenario

Key Takeaways

Further Reading