site-scraperv1.4.6

CASOON Open Source

Take a website with you.

site-scraper copies a website into a folder that opens without a server: pages, stylesheets, scripts, fonts and images, with every link pointing to the local files. It also takes full-page screenshots of one URL or a whole list.

curl -fsSL https://raw.githubusercontent.com/casoon/site-scraper/main/install.sh | bash
cargo install --git https://github.com/casoon/site-scraper --features headless
Free · MITv1.4.6Rust CLImacOS · Linux
Terminal: site-scraper https://casoon.dev --max-depth 1 --no-sitemap prints 'Crawling https://casoon.dev/', saves https://casoon.dev/ as index.html and https://casoon.dev/en/ as en/index.html, fills a progress bar to 3/3 and ends with 'https://casoon.dev/ → output/casoon.dev' and 'Pages: 3, Failed: 0, Warnings: 0, Duration: 2s'. ls output/casoon.dev then lists _astro, en, fonts and index.html.

What it is for

pages of casoon.de copied in 14 s, 0 failed
36
ways to handle images: originals, local or external placeholders
3
screenshot sizes: desktop, tablet, mobile
3
prebuilt releases: Linux x86_64 and arm64, macOS arm64
3

How it works

You give site-scraper a start URL. It reads the site's sitemap as well, fetches each page, downloads what the page needs and rewrites every reference so that it points to the local copy. The page lands in output/<domain>/ under the same path it had on the web.

Links to other pages of the same site go back into the queue, one level deeper each time, until the depth you chose is reached. A short pause between requests keeps the load on the site low, and a summary at the end tells you what was saved and what failed.

How site-scraper workssite-scraper starts at one URL and, unless switched off, the URLs in the site's sitemap. It fetches each page, downloads its stylesheets, scripts and images, rewrites every reference to a relative local path and saves the page under output/<domain>/. Links to the same site go back into the queue until the maximum depth is reached.How site-scraper worksStart URL → Fetch pageFetch page → Download assetsDownload assets → Rewrite referencesRewrite references → output/<domain>/: saveRewrite references → Fetch page: same-site linkssavesame-site linksStart URL – plus sitemap.xmlStart URLplus sitemap.xmlFetch page – HTTP or headless ChromeFetch pageHTTP or headless ChromeDownload assets – CSS, JS, images, fontsDownload assetsCSS, JS, images, fontsRewrite references – to relative pathsRewrite referencesto relative pathsoutput/<domain>/ – opens offlineoutput/<domain>/opens offline
How site-scraper workssite-scraper starts at one URL and, unless switched off, the URLs in the site's sitemap. It fetches each page, downloads its stylesheets, scripts and images, rewrites every reference to a relative local path and saves the page under output/<domain>/. Links to the same site go back into the queue until the maximum depth is reached.How site-scraper worksStart URL → Fetch pageFetch page → Download assetsDownload assets → Rewrite referencesRewrite references → output/<domain>/: saveRewrite references → Fetch page: same-site linkssavesame-site linksStart URL – plus sitemap.xmlStart URLplus sitemap.xmlFetch page – HTTP or headless ChromeFetch pageHTTP or headless ChromeDownload assets – CSS, JS, images, fontsDownload assetsCSS, JS, images, fontsRewrite references – to relative pathsRewrite referencesto relative pathsoutput/<domain>/ – opens offlineoutput/<domain>/opens offline
How site-scraper workssite-scraper starts at one URL and, unless switched off, the URLs in the site's sitemap. It fetches each page, downloads its stylesheets, scripts and images, rewrites every reference to a relative local path and saves the page under output/<domain>/. Links to the same site go back into the queue until the maximum depth is reached.How site-scraper worksStart URL → Fetch pageFetch page → Download assetsDownload assets → Rewrite referencesRewrite references → output/<domain>/: saveRewrite references → Fetch page: same-site linkssavesame-site linksStart URL – plus sitemap.xmlStart URLplus sitemap.xmlFetch page – HTTP or headless ChromeFetch pageHTTP or headlessChromeDownload assets – CSS, JS, images, fontsDownloadassetsCSS, JS, images,fontsRewrite references – to relative pathsRewritereferencesto relative pathsoutput/<domain>/ – opens offlineoutput/<domain>/opens offline
How site-scraper workssite-scraper starts at one URL and, unless switched off, the URLs in the site's sitemap. It fetches each page, downloads its stylesheets, scripts and images, rewrites every reference to a relative local path and saves the page under output/<domain>/. Links to the same site go back into the queue until the maximum depth is reached.How site-scraper worksStart URL → Fetch pageFetch page → Download assetsDownload assets → Rewrite referencesRewrite references → output/<domain>/: saveRewrite references → Fetch page: same-site linkssavesame-site linksStart URL – plus sitemap.xmlStart URLplus sitemap.xmlFetch page – HTTP or headless ChromeFetch pageHTTP or headlessChromeDownload assets – CSS, JS, images, fontsDownloadassetsCSS, JS, images,fontsRewrite references – to relative pathsRewritereferencesto relative pathsoutput/<domain>/ – opens offlineoutput/<domain>/opens offline
How a crawl works in detailHeadless mode and screenshotsAll options →

Quickstart

Install it, point it at a site and open the copy. Given only a URL, site-scraper asks for depth, images and mode in a short menu.

  1. Install it with one of the lines above; headless mode and screenshots also need Chrome or Chromium.
  2. Run site-scraper with the URL of the site.
  3. Open output/<domain>/index.html in your browser.
Read the documentation
Two commands
site-scraper https://www.example.com --max-depth 2
open output/www.example.com/index.html

New in v1.4.6

Changelog →

Frequently asked questions

Do I need Chrome or Chromium installed?

Only for --headless and for the screenshot subcommand, which both render with a local Chrome or Chromium. The default crawl is plain HTTP and needs nothing beyond site-scraper itself.

How do I verify a downloaded release binary is genuine?

Check it against the release's SHA256SUMS and its signed build provenance attestation with gh attestation verify. install.sh does both automatically before installing.

Does the crawl follow a site's sitemap?

By default, yes: robots.txt, sitemap.xml, sitemap_index.xml and wp-sitemap.xml are read as extra seed URLs, including gzipped sitemaps, and can be turned off with --no-sitemap.

What happens if a page or screenshot fails?

It's reported with its status and the run continues; the overall exit code is 1 if anything failed, so scripts and CI can still detect it.

A local copy in two commands.

Read the quickstart