Nouveautés de la v2
Meilleur contrôle des sitemaps
"include"
- Utiliser le sitemap et trouver d’autres pages (par défaut)"skip"
- Ignorer complètement le sitemap"only"
- Renvoyer uniquement les URL du sitemap
Format de réponse modifié
links
avec des métadonnées enrichies.
Authorizations
Bearer authentication header of the form Bearer <token>
, where <token>
is your auth token.
Body
The base URL to start crawling from
Specify a search query to order the results by relevance. Example: 'blog' will return URLs that contain the word 'blog' in the URL ordered by relevance.
Sitemap mode when mapping. If you set it to skip
, the sitemap won't be used to find URLs. If you set it to only
, only URLs that are in the sitemap will be returned. By default (include
), the sitemap and other methods will be used together to find URLs.
skip
, include
, only
Include subdomains of the website
Do not return URLs with query parameters
Maximum number of links to return
x <= 30000
Timeout in milliseconds. There is no timeout by default.
Location settings for the request. When specified, this will use an appropriate proxy if available and emulate the corresponding language and timezone settings. Defaults to 'US' if not specified.