Generate LLMs.txt files from any website for LLM training and analysis
/llmstxt
endpoint allows you to transform any website into clean, LLM-ready text files. Simply provide a URL, and Firecrawl will crawl the site and generate both llms.txt
and llms-full.txt
files that can be used for training or analysis with any LLM.
llms.txt
: Concise summaries and key informationllms-full.txt
: Complete text content with more detailmaxUrls
parameter