Amazon 是被抓取最频繁的电商网站之一。本指南将演示如何利用 Firecrawl 的强大功能，高效提取商品数据、价格、评价以及搜索结果。
配置
npm install firecrawl zod
概述
- 提取产品信息 (标题、价格、库存情况)
- 获取用户评论和评分
- 监控价格变化
- 以编程方式搜索产品
- 跟踪竞争对手的商品列表
使用 JSON 模式进行抓取
import { Firecrawl } from 'firecrawl';
import { z } from 'zod';
// 定义 Zod schema
const ProductSchema = z.object({
title: z.string(),
price: z.string(),
rating: z.number(),
availability: z.string(),
features: z.array(z.string())
});
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const result = await firecrawl.scrape('https://www.amazon.com/dp/B0DZZWMB2L', {
formats: [{
type: 'json',
schema: z.toJSONSchema(ProductSchema)
}],
});
// 使用 Zod 解析和验证
const jsonData = typeof result.json === 'string' ? JSON.parse(result.json) : result.json;
const validated = ProductSchema.parse(jsonData);
console.log('✅ 已验证的产品数据:');
console.log(validated);
搜索
import { Firecrawl } from 'firecrawl';
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const searchResult = await firecrawl.search('gaming laptop site:amazon.com', {
limit: 10,
sources: [{ type: 'web' }], // { type: 'news' }（新闻）, { type: 'images' }（图片）
scrapeOptions: {
formats: ['markdown']
}
});
console.log(searchResult);
抓取
import { Firecrawl } from 'firecrawl';
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const result = await firecrawl.scrape('https://www.amazon.com/ASUS-ROG-Strix-Gaming-Laptop/dp/B0DZZWMB2L', {
formats: ['markdown'], // 即 html、links 等
onlyMainContent: true
});
console.log(result);
Map
import { Firecrawl } from 'firecrawl';
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const mapResult = await firecrawl.map('https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics');
console.log(mapResult.links);
// 返回 URL 数组,不包含页面内容
爬取
import { Firecrawl } from 'firecrawl';
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const crawlResult = await firecrawl.crawl('https://www.amazon.com/s?k=mechanical+keyboards', {
limit: 10,
scrapeOptions: {
formats: ['markdown']
}
});
console.log(crawlResult.data);
批量抓取
import { Firecrawl } from 'firecrawl';
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
// 等待完成
const job = await firecrawl.batchScrape([
'https://www.amazon.com/ASUS-ROG-Strix-Gaming-Laptop/dp/B0DZZWMB2L',
'https://www.amazon.com/Razer-Blade-Gaming-Laptop-Lightweight/dp/B0FP47DNFQ',
'https://www.amazon.com/HP-2025-Omen-Gaming-Laptop/dp/B0FL4RMGSH'],
{
options: {
formats: ['markdown']
},
pollInterval: 2,
timeout: 120
}
);
console.log(job.status, job.completed, job.total);
console.log(job);