Showcase

Each example shows real output produced by site-scraper itself, from files in the project repository. Input on the left, output on the right.

Mirror a real site

examples/crawl-casoon.txtDetail →

site-scraper https://www.casoon.de --max-depth 1 --no-sitemap --placeholder local

ℹ Crawling https://www.casoon.de/
✔ https://www.casoon.de/ → index.html
✔ https://www.casoon.de/ratgeber/ → ratgeber/index.html
✔ https://www.casoon.de/joern-seidel/ → joern-seidel/index.html
✔ https://www.casoon.de/arbeitsweise/ → arbeitsweise/index.html
✔ https://www.casoon.de/ki/ → ki/index.html
✔ https://www.casoon.de/hotellerie/rostock/ → hotellerie/rostock/index.html
✔ https://www.casoon.de/impressum/ → impressum/index.html
✔ https://www.casoon.de/tool-engineering/ → tool-engineering/index.html
✔ https://www.casoon.de/technologien/ → technologien/index.html
✔ https://www.casoon.de/handwerker/rostock/ → handwerker/rostock/index.html
✔ https://www.casoon.de/praxis/rostock/ → praxis/rostock/index.html
✔ https://www.casoon.de/ki-automatisierung/ → ki-automatisierung/index.html
✔ https://www.casoon.de/kanzlei/rostock/ → kanzlei/rostock/index.html
✔ https://www.casoon.de/kollaboration/ → kollaboration/index.html
✔ https://www.casoon.de/agb/ → agb/index.html
✔ https://www.casoon.de/leistungskatalog/ → leistungskatalog/index.html
✔ https://www.casoon.de/kontakt/ → kontakt/index.html
✔ https://www.casoon.de/strategieberatung/ → strategieberatung/index.html
✔ https://www.casoon.de/existenzgruendung-webdesign-foerderung/ → existenzgruendung-webdesign-foerderung/index.html
✔ https://www.casoon.de/cloud-entwicklung/ → cloud-entwicklung/index.html
✔ https://www.casoon.de/gastronomie/rostock/ → gastronomie/rostock/index.html
✔ https://www.casoon.de/usp/ → usp/index.html
✔ https://www.casoon.de/dienstleister/rostock/ → dienstleister/rostock/index.html
✔ https://www.casoon.de/datenschutz/ → datenschutz/index.html
✔ https://www.casoon.de/webentwicklung/ → webentwicklung/index.html
✔ https://www.casoon.de/e-commerce/ → e-commerce/index.html
✔ https://www.casoon.de/mittelstand/rostock/ → mittelstand/rostock/index.html
✔ https://www.casoon.de/wartung-und-support/ → wartung-und-support/index.html
✔ https://www.casoon.de/ferienwohnung/rostock/ → ferienwohnung/rostock/index.html
✔ https://www.casoon.de/plattform-apps/ → plattform-apps/index.html
✔ https://www.casoon.de/picks/ → picks/index.html
✔ https://www.casoon.de/ferienwohnung/mecklenburg-vorpommern/ → ferienwohnung/mecklenburg-vorpommern/index.html
✔ https://www.casoon.de/seo-marketing/ → seo-marketing/index.html
✔ https://www.casoon.de/ → output/www.casoon.de

  Pages: 33   Failed: 0   Warnings: 0   Duration: 11s

crawl
output

Crawl command

examples/crawl-help.txtDetail →

site-scraper --help

CLI utility to mirror a website (HTML + CSS) into a local folder.

Usage: site-scraper [OPTIONS] <URL>
       site-scraper <COMMAND>

Commands:
  screenshot  Only take full-page screenshots of one URL or a list of URLs (requires Chrome)
  help        Print this message or the help of the given subcommand(s)

Arguments:
  <URL>  URL to scrape

Options:
      --max-depth <MAX_DEPTH>      Maximum crawl depth relative to the start page
      --concurrency <CONCURRENCY>  Number of parallel downloads [default: 4]
      --delay-ms <DELAY_MS>        Delay between requests in milliseconds [default: 300]
      --placeholder <PLACEHOLDER>  Image placeholder strategy: "real" (download originals), "local" (gray PNG), or "external" (placehold.co)
      --sitemap                    Include sitemap.xml URLs as seeds (default)
      --no-sitemap                 Don't use sitemap.xml URLs as seeds
      --allow-external-assets      Download external CSS/JS (default)
      --no-allow-external-assets   Leave external CSS/JS as-is instead of downloading
      --bot                        Identify as bot/crawler instead of simulating a browser
      --headless                   Use a headless Chrome/Chromium browser to render JavaScript (requires Chrome installed)
      --screenshot                 Save a full-page screenshot for each crawled page (requires --headless)
      --user-agent <USER_AGENT>    Custom User-Agent header
      --referer <REFERER>          Custom Referer header
  -h, --help                       Print help
  -V, --version                    Print version

crawl
options

Screenshot command

examples/screenshot-help.txtDetail →

site-scraper screenshot --help

Only take full-page screenshots of one URL or a list of URLs (requires Chrome)

Usage: site-scraper screenshot [OPTIONS] [URL]

Arguments:
  [URL]  URL to screenshot

Options:
      --file <FILE>                Text file with one URL per line (blank lines and # comments are ignored)
      --output <OUTPUT>            Output directory (existing screenshots are kept) [default: screenshots]
      --concurrency <CONCURRENCY>  Number of parallel browser pages [default: 2]
  -h, --help                       Print help

screenshot
headless

Invalid URL

examples/invalid-url.txtDetail →

site-scraper not-a-url

[FAIL] Invalid URL provided

error
validation