Firecrawl allows you to turn entire websites into LLM-ready markdown
wait_until_done
to false
.
This will return a jobId.
For cURL, /crawl will always return a jobId where you can use to check the status of the crawl.
scrape_url
method. It takes the URL as a parameter and returns the scraped data as a dictionary.