Setup
- First, obtain your Firecrawl API key from your Firecrawl dashboard
- Configure your SourceSync.ai namespace to use Firecrawl as the web scraping provider:
curl -X PATCH https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID \
-H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"webScraperConfig": {
"provider": "FIRECRAWL",
"apiKey": "YOUR_FIRECRAWL_API_KEY"
}
}'
const response = await fetch(
'https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID',
{
method: 'PATCH',
headers: {
Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
webScraperConfig: {
provider: 'FIRECRAWL',
apiKey: 'YOUR_FIRECRAWL_API_KEY',
},
}),
}
)
import requests
response = requests.patch(
'https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID',
headers={
'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
'Content-Type': 'application/json',
},
json={
'webScraperConfig': {
'provider': 'FIRECRAWL',
'apiKey': 'YOUR_FIRECRAWL_API_KEY',
},
},
)
Usage
Once configured, you can use SourceSync.ai’s web scraping endpoints with Firecrawl’s capabilities. Here are the main ingestion methods:URL List Ingestion
Scrape specific URLs:curl -X POST https://api.sourcesync.ai/v1/ingest/urls \
-H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespaceId": "YOUR_NAMESPACE_ID",
"ingestConfig": {
"source": "URLS_LIST",
"config": {
"urls": [
"https://example.com/page1",
"https://example.com/page2"
],
"scrapeOptions": {
"includeSelectors": ["article", "main"],
"excludeSelectors": [".navigation", ".footer"]
}
}
}
}'
const response = await fetch('https://api.sourcesync.ai/v1/ingest/urls', {
method: 'POST',
headers: {
Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
namespaceId: 'YOUR_NAMESPACE_ID',
ingestConfig: {
source: 'URLS_LIST',
config: {
urls: ['https://example.com/page1', 'https://example.com/page2'],
scrapeOptions: {
includeSelectors: ['article', 'main'],
excludeSelectors: ['.navigation', '.footer'],
},
},
},
}),
})
response = requests.post(
'https://api.sourcesync.ai/v1/ingest/urls',
headers={
'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
'Content-Type': 'application/json',
},
json={
'namespaceId': 'YOUR_NAMESPACE_ID',
'ingestConfig': {
'source': 'URLS_LIST',
'config': {
'urls': ['https://example.com/page1', 'https://example.com/page2'],
'scrapeOptions': {
'includeSelectors': ['article', 'main'],
'excludeSelectors': ['.navigation', '.footer'],
},
},
},
},
)
Website Crawling
Crawl an entire website with custom rules:curl -X POST https://api.sourcesync.ai/v1/ingest/website \
-H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespaceId": "YOUR_NAMESPACE_ID",
"ingestConfig": {
"source": "WEBSITE",
"config": {
"url": "https://example.com",
"maxDepth": 3,
"maxLinks": 100,
"includePaths": ["/docs", "/blog"],
"excludePaths": ["/admin"],
"scrapeOptions": {
"includeSelectors": ["article", "main"],
"excludeSelectors": [".navigation", ".footer"]
}
}
}
}'
const response = await fetch('https://api.sourcesync.ai/v1/ingest/website', {
method: 'POST',
headers: {
Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
namespaceId: 'YOUR_NAMESPACE_ID',
ingestConfig: {
source: 'WEBSITE',
config: {
url: 'https://example.com',
maxDepth: 3,
maxLinks: 100,
includePaths: ['/docs', '/blog'],
excludePaths: ['/admin'],
scrapeOptions: {
includeSelectors: ['article', 'main'],
excludeSelectors: ['.navigation', '.footer'],
},
},
},
}),
})
response = requests.post(
'https://api.sourcesync.ai/v1/ingest/website',
headers={
'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
'Content-Type': 'application/json',
},
json={
'namespaceId': 'YOUR_NAMESPACE_ID',
'ingestConfig': {
'source': 'WEBSITE',
'config': {
'url': 'https://example.com',
'maxDepth': 3,
'maxLinks': 100,
'includePaths': ['/docs', '/blog'],
'excludePaths': ['/admin'],
'scrapeOptions': {
'includeSelectors': ['article', 'main'],
'excludeSelectors': ['.navigation', '.footer'],
},
},
},
},
)
Sitemap Processing
Process all URLs from a sitemap:curl -X POST https://api.sourcesync.ai/v1/ingest/sitemap \
-H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespaceId": "YOUR_NAMESPACE_ID",
"ingestConfig": {
"source": "SITEMAP",
"config": {
"url": "https://example.com/sitemap.xml",
"scrapeOptions": {
"includeSelectors": ["article", "main"],
"excludeSelectors": [".navigation", ".footer"]
}
}
}
}'
const response = await fetch('https://api.sourcesync.ai/v1/ingest/sitemap', {
method: 'POST',
headers: {
Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
namespaceId: 'YOUR_NAMESPACE_ID',
ingestConfig: {
source: 'SITEMAP',
config: {
url: 'https://example.com/sitemap.xml',
scrapeOptions: {
includeSelectors: ['article', 'main'],
excludeSelectors: ['.navigation', '.footer'],
},
},
},
}),
})
response = requests.post(
'https://api.sourcesync.ai/v1/ingest/sitemap',
headers={
'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
'Content-Type': 'application/json',
},
json={
'namespaceId': 'YOUR_NAMESPACE_ID',
'ingestConfig': {
'source': 'SITEMAP',
'config': {
'url': 'https://example.com/sitemap.xml',
'scrapeOptions': {
'includeSelectors': ['article', 'main'],
'excludeSelectors': ['.navigation', '.footer'],
},
},
},
},
)
Features
When using Firecrawl with SourceSync.ai, you get access to:- JavaScript rendering support
- Automatic rate limiting
- CSS selector-based content extraction
- Recursive crawling with depth control
- Sitemap processing
Resources
For additional support:- Email: support@sourcesync.ai
- Discord: Join our community