> ## Documentation Index
> Fetch the complete documentation index at: https://docs.firecrawl.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Generate LLMs.txt with an API

> Generate LLMs.txt files from any website for LLM training and analysis

<Warning>
  This API is being deprecated in favor of our main endpoints. Here is an example repo that generates LLMs.txt files: [https://github.com/firecrawl/create-llmstxt-py](https://github.com/firecrawl/create-llmstxt-py). This API endpoint will still remain active but we will no longer be maintaining it after June 30, 2025.
</Warning>

## Introducing LLMs.txt Generator Endpoint (Alpha) 📃

The `/llmstxt` endpoint allows you to transform any website into clean, [LLM-ready text files](https://www.firecrawl.dev/blog/How-to-Create-an-llms-txt-File-for-Any-Website). Simply provide a URL, and Firecrawl will crawl the site and generate both `llms.txt` and `llms-full.txt` files that can be used for training or analysis with any LLM.

## How It Works

The LLMs.txt Generator:

1. Crawls the provided website URL and its linked pages
2. Extracts clean, meaningful text content
3. Generates two formats:
   * `llms.txt`: Concise summaries and key information
   * `llms-full.txt`: Complete text content with more detail

### Example Usage

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp

  # Initialize the client
  firecrawl = FirecrawlApp(api_key="your_api_key")

  # Generate LLMs.txt with polling
  results = firecrawl.generate_llms_text(
      url="https://example.com",
      max_urls=2,
      show_full_text=True
  )

  # Access generation results
  if results.success:
      print(f"Status: {results.status}")
      print(f"Generated Data: {results.data}")
  else:
      print(f"Error: {results.error}")
  ```

  ```js Node theme={null}
  import { Firecrawl } from "firecrawl";

  // Initialize the client
  const firecrawl = new Firecrawl({ apiKey: "your_api_key" });

  // Define generation parameters
  const params = {
    maxUrls: 2, // Maximum URLs to analyze
    showFullText: true, // Include full text in results
  };

  // Generate LLMs.txt with polling
  const results = await firecrawl.generateLLMsText("https://example.com", params);

  // Access generation results
  if (results.success) {
    console.log(`Status: ${results.status}`);
    console.log(`Generated Data:`, results.data);
  } else {
    console.error(`Error: ${results.error || "Unknown error"}`);
  }
  ```

  ```bash cURL theme={null}
  # Start LLMs.txt generation
  curl -X POST "https://api.firecrawl.dev/v1/llmstxt" \
    -H "Authorization: Bearer your_api_key" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com",
      "maxUrls": 2,
      "showFullText": true
    }'

  # Check generation status
  curl -X GET "https://api.firecrawl.dev/v1/llmstxt/job_id" \
    -H "Authorization: Bearer your_api_key"
  ```
</CodeGroup>

**Key Parameters:**

* **url**: The website URL to generate LLMs.txt files from
* **maxUrls** (Optional): Maximum number of pages to crawl (1-100, default: 10)
* **showFullText** (Optional): Generate llms-full.txt in addition to llms.txt (default: false)

See [API Reference](/api-reference/endpoint/llmstxt) for more details.

## Checking Generation Status

LLMs.txt generation runs asynchronously. Make the aync call and monitor the status with:

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import FirecrawlApp

  # Initialize the client
  firecrawl = FirecrawlApp(api_key="your_api_key")

  # Create async job
  job = firecrawl.async_generate_llms_text(
      url="https://example.com",
  )

  if job.success:
      job_id = job.id

  # Check LLMs.txt generation status
  status = firecrawl.check_generate_llms_text_status("job_id")

  # Print current status
  print(f"Status: {status.status}")

  if status.status == 'completed':
      print("LLMs.txt Content:", status.data.llmstxt)
      if 'llmsfulltxt' in status.data:
          print("Full Text Content:", status.data.llmsfulltxt)
      print(f"Processed URLs: {len(status.data.processed_urls)}")
  ```

  ```js Node theme={null}
  import { Firecrawl } from "firecrawl";

  // Initialize the client
  const firecrawl = new Firecrawl({ apiKey: "your_api_key" });

  // Create async job
  const job = await firecrawl.asyncGenerateLLMsText("https://example.com", {
    maxUrls: 50,
    showFullText: true
  });

  // Check LLMs.txt generation status
  const status = await firecrawl.checkGenerateLLMsTextStatus(job.jobId);

  // Print current status
  console.log(`Status: ${status.status}`);

  if (status.status === "completed") {
    console.log("LLMs.txt Content:", status.data.llmstxt);
    if (status.data.llmsfulltxt) {
      console.log("Full Text Content:", status.data.llmsfulltxt);
    }
    console.log(`Processed URLs: ${status.data.processedUrls.length}`);
  }
  ```

  ```bash theme={null}
  curl "https://api.firecrawl.dev/v1/llmstxt/job_id" \
    -H "Authorization: Bearer your_api_key"
  ```
</CodeGroup>

### Status Examples

#### In Progress

```json theme={null}
{
  "success": true,
  "data": {
    "llmstxt": "# Firecrawl.dev llms.txt\n\n- [Web Data Extraction Tool](https://www.firecrawl.dev/)...",
    "llmsfulltxt": "# Firecrawl.dev llms-full.txt\n\n"
  },
  "status": "processing",
  "expiresAt": "2025-03-03T23:19:18.000Z"
}
```

#### Completed

```json theme={null}
{
  "success": true,
  "data": {
    "llmstxt": "# http://firecrawl.dev llms.txt\n\n- [Web Data Extraction Tool](https://www.firecrawl.dev/): Transform websites into clean, LLM-ready data effortlessly.\n- [Flexible Web Scraping Pricing](https://www.firecrawl.dev/pricing): Flexible pricing plans for web scraping and data extraction.\n- [Web Scraping and AI](https://www.firecrawl.dev/blog): Explore tutorials and articles on web scraping and AI...",
    "llmsfulltxt": "# http://firecrawl.dev llms-full.txt\n\n## Web Data Extraction Tool\nIntroducing /extract - Get web data with a prompt [Try now](https://www.firecrawl.dev/extract)\n\n[💥Get 2 months free with yearly plan](https://www.firecrawl.dev/pricing)..."
  },
  "status": "completed",
  "expiresAt": "2025-03-03T22:45:50.000Z"
}
```

## Known Limitations (Alpha)

1. **Access Restrictions**\
   Only publicly accessible pages can be processed. Login-protected or paywalled content is not supported.

2. **Site Size**\
   We are only allowing processing for up to 5000 URLs during the alpha stage.

3. **Alpha State**\
   As an Alpha feature, the output format and processing may evolve based on feedback.

## Billing and Usage

Billing is based on the number of URLs processed:

* Base cost: 1 credit per URL processed
* Control URL costs with `maxUrls` parameter