Images and assets
Control image handling, external styles and scripts, sitemap seeds, and crawl pacing.
Image strategies
--placeholder accepts three values:
| Value | Behaviour |
|---|---|
real |
Download original images and rewrite their references locally. |
local |
Generate local gray PNG placeholders. |
external |
Replace images with placehold.co URLs. This is the non-interactive default. |
site-scraper https://www.example.com --placeholder localExternal CSS and JavaScript
External stylesheets and scripts are downloaded by default. Preserve their remote references with
--no-allow-external-assets.
Sitemap discovery
Sitemap URLs are included as seeds by default — from Sitemap: lines in robots.txt and from sitemap.xml, sitemap_index.xml and wp-sitemap.xml, including gzipped sitemaps and nested sitemap indexes. Use --no-sitemap to crawl only from links
discovered below the starting URL.
Pacing
The default is four concurrent downloads with a 300 ms request delay. Tune both explicitly when a site or server needs a different load profile:
site-scraper https://www.example.com --concurrency 2 --delay-ms 750