Running locally

Welcome to Firecrawl 🔥! Here are some instructions on how to get the project locally so you can run it on your own and contribute.

If you’re contributing, note that the process is similar to other open-source repos, i.e., fork Firecrawl, make changes, run tests, PR.

If you have any questions or would like help getting on board, join our Discord community here for more information or submit an issue on Github here!

​

Running the project locally

First, start by installing dependencies

1. node.js instructions
2. pnpm instructions
3. redis instructions

Set environment variables in a .env file in the /apps/api/ directory. You can copy over the template in .env.example.

To start, we won’t set up authentication, or any optional sub services (pdf parsing, JS blocking support, AI features)

# ./apps/api/.env

# ===== Required ENVS ======
NUM_WORKERS_PER_QUEUE=8 
PORT=3002
HOST=0.0.0.0
REDIS_URL=redis://localhost:6379
REDIS_RATE_LIMIT_URL=redis://localhost:6379

## To turn on DB authentication, you need to set up supabase.
USE_DB_AUTHENTICATION=false

# ===== Optional ENVS ======

# Supabase Setup (used to support DB authentication, advanced logging, etc.)
SUPABASE_ANON_TOKEN= 
SUPABASE_URL= 
SUPABASE_SERVICE_TOKEN=

# Other Optionals
TEST_API_KEY= # use if you've set up authentication and want to test with a real API key
SCRAPING_BEE_API_KEY= #Set if you'd like to use scraping Be to handle JS blocking
OPENAI_API_KEY= # add for LLM dependednt features (image alt generation, etc.)
BULL_AUTH_KEY= #
LOGTAIL_KEY= # Use if you're configuring basic logging with logtail
PLAYWRIGHT_MICROSERVICE_URL=  # set if you'd like to run a playwright fallback
LLAMAPARSE_API_KEY= #Set if you have a llamaparse key you'd like to use to parse pdfs

​

Installing dependencies

First, install the dependencies using pnpm.

pnpm install

​

Running the project

You’re going to need to open 3 terminals for running the services (optional: 4 terminals for running the services and testing).

​

Terminal 1 - setting up redis

Run the command anywhere within your project

redis-server

​

Terminal 2 - setting up workers

Now, navigate to the apps/api/ directory and run:

pnpm run workers

This will start the workers who are responsible for processing crawl jobs.

​

Terminal 3 - setting up the main server

To do this, navigate to the apps/api/ directory. If you haven’t installed pnpm already, you can do so here: https://pnpm.io/installation

Next, run your server with:

pnpm run start

​

(Optional) Terminal 4 - sending our first request

Alright, now let’s send our first request.

curl -X GET http://localhost:3002/test

This should return the response Hello, world!

If you’d like to test the crawl endpoint, you can run this

curl -X POST http://localhost:3002/v0/crawl \
    -H 'Content-Type: application/json' \
    -d '{
      "url": "https://docs.firecrawl.dev"
    }'

​

Tests:

The best way to do this is run the test with npm run test:local-no-auth if you’d like to run the tests without authentication.

If you’d like to run the tests with authentication, run npm run test:prod