Batch scrape multiple URLs
/crawl
endpoint works. It submits a batch scrape job and returns a job ID to check the status of the batch scrape.
The sdk provides 2 methods, synchronous and asynchronous. The synchronous method will return the results of the batch scrape job, while the asynchronous method will return a job ID that you can use to check the status of the batch scrape.
/batch/scrape/{id}
endpoint. This endpoint is meant to be used while the job is still running or right after it has completed as batch scrape jobs expire after 24 hours.
batch_scrape.started
- When the batch scrape beginsbatch_scrape.page
- For each URL successfully scrapedbatch_scrape.completed
- When all URLs are processedbatch_scrape.failed
- If the batch scrape encounters an error