Generate LLMs.txt with an API
Generate LLMs.txt files from any website for LLM training and analysis
This API is being deprecated in favor of our main endpoints. Here is an example repo that generates LLMs.txt files: https://github.com/mendableai/create-llmstxt-py. This API endpoint will still remain active but we will no longer be maintaining it after June 30, 2025.
Introducing LLMs.txt Generator Endpoint (Alpha) 📃
The /llmstxt
endpoint allows you to transform any website into clean, LLM-ready text files. Simply provide a URL, and Firecrawl will crawl the site and generate both llms.txt
and llms-full.txt
files that can be used for training or analysis with any LLM.
How It Works
The LLMs.txt Generator:
- Crawls the provided website URL and its linked pages
- Extracts clean, meaningful text content
- Generates two formats:
llms.txt
: Concise summaries and key informationllms-full.txt
: Complete text content with more detail
Example Usage
Key Parameters:
- url: The website URL to generate LLMs.txt files from
- maxUrls (Optional): Maximum number of pages to crawl (1-100, default: 10)
- showFullText (Optional): Generate llms-full.txt in addition to llms.txt (default: false)
See API Reference for more details.
Checking Generation Status
LLMs.txt generation runs asynchronously. Make the aync call and monitor the status with:
Status Examples
In Progress
Completed
Known Limitations (Alpha)
-
Access Restrictions
Only publicly accessible pages can be processed. Login-protected or paywalled content is not supported. -
Site Size
We are only are allowing processing for up to 5000 URLs during the alpha stage. -
Alpha State
As an Alpha feature, the output format and processing may evolve based on feedback.
Billing and Usage
Billing is based on the number of URLs processed:
- Base cost: 1 credit per URL processed
- Control URL costs with
maxUrls
parameter
Have feedback or need help? Email help@firecrawl.com.