site-scraperv1.4.6

How a crawl works

The steps of one run, from the start URL to the summary, and what each step does with the site.

A crawl runs in rounds. The first round holds the start URL and, with the sitemap switched on, the pages listed in the site’s sitemaps. Every page that is fetched can add its same-site links to the next round, until no unseen URL is left or the maximum depth is reached.

Focus
One crawl run
One crawl runsite-scraper follows redirects of the start URL, clears output/<domain>/ and, when the sitemap is on and the depth is at least 1, reads the sitemaps listed in robots.txt and at the well-known paths. Each round fetches all unseen URLs in parallel. Responses that are not HTML are skipped; HTML pages get their stylesheets, scripts and images downloaded, references rewritten and the page written to disk. Same-origin links go into the next round while the page's depth is below the maximum depth. At the end a summary lists pages, failures, warnings and duration.site-scraperThe websitesite-scraper <URL> → Follow redirectsFollow redirects → Clear output/<domain>/Clear output/<domain>/ → Sitemap on, depth ≥ 1?Sitemap on, depth ≥ 1? → robots.txt, sitemap.xml: yesSitemap on, depth ≥ 1? → Next round: norobots.txt, sitemap.xml → Next roundNext round → GET pageGET page → text/html?text/html? → CSS, JS, images: yesCSS, JS, images → Rewrite referencesRewrite references → <path>/index.html<path>/index.html → Below max depth?Below max depth? → Next round: yes: same-origin linksBelow max depth? → Summary: queue emptyyesnoyesyes: same-origin linksqueue emptysite-scraper <URL> – flags or guided setupsite-scraper <URL>flags or guided setupFollow redirects – e.g. www to bare domainFollow redirectse.g. www to bare domainClear output/<domain>/Clear output/<domain>/Sitemap on, depth ≥ 1?Sitemap on, depth ≥ 1?robots.txt, sitemap.xml – same-origin URLsrobots.txt, sitemap.xmlsame-origin URLsNext round – unseen URLs, in parallelNext roundunseen URLs, in parallelGET page – delay, 3 triesGET pagedelay, 3 triestext/html?text/html?CSS, JS, images – per placeholder modeCSS, JS, imagesper placeholder modeRewrite references – relative local pathsRewrite referencesrelative local paths<path>/index.html<path>/index.htmlBelow max depth?Below max depth?Summary – pages, failed, durationSummarypages, failed, duration

Source: src/cli.rs, src/crawler.rs, src/parsers/sitemap.rs, src/parsers/links.rs, src/processors/html.rs, src/processors/stylesheet.rs, src/processors/image.rs

Show chart data
Data for One crawl run
StepKindLaneLeads to
site-scraper <URL>startsite-scraperFollow redirects
Follow redirectsexternalThe websiteClear output/<domain>/
Clear output/<domain>/data storesite-scraperSitemap on, depth ≥ 1?
Sitemap on, depth ≥ 1?decisionsite-scraperrobots.txt, sitemap.xml (yes), Next round (no)
robots.txt, sitemap.xmlexternalThe websiteNext round
Next roundstepsite-scraperGET page
GET pageexternalThe websitetext/html?
text/html?decisionsite-scraperCSS, JS, images (yes)
CSS, JS, imagesexternalThe websiteRewrite references
Rewrite referencesstepsite-scraper<path>/index.html
<path>/index.htmldata storesite-scraperBelow max depth?
Below max depth?decisionsite-scraperNext round (yes: same-origin links), Summary (queue empty)
Summaryendsite-scraper
Focus
One crawl run
One crawl runsite-scraper follows redirects of the start URL, clears output/<domain>/ and, when the sitemap is on and the depth is at least 1, reads the sitemaps listed in robots.txt and at the well-known paths. Each round fetches all unseen URLs in parallel. Responses that are not HTML are skipped; HTML pages get their stylesheets, scripts and images downloaded, references rewritten and the page written to disk. Same-origin links go into the next round while the page's depth is below the maximum depth. At the end a summary lists pages, failures, warnings and duration.site-scraperThe websitesite-scraper <URL> → Follow redirectsFollow redirects → Clear output/<domain>/Clear output/<domain>/ → Sitemap on, depth ≥ 1?Sitemap on, depth ≥ 1? → robots.txt, sitemap.xml: yesSitemap on, depth ≥ 1? → Next round: norobots.txt, sitemap.xml → Next roundNext round → GET pageGET page → text/html?text/html? → CSS, JS, images: yesCSS, JS, images → Rewrite referencesRewrite references → <path>/index.html<path>/index.html → Below max depth?Below max depth? → Next round: yes: same-origin linksBelow max depth? → Summary: queue emptyyesnoyesyes: same-origin linksqueue emptysite-scraper <URL> – flags or guided setupsite-scraper <URL>flags or guided setupFollow redirects – e.g. www to bare domainFollow redirectse.g. www to bare domainClear output/<domain>/Clear output/<domain>/Sitemap on, depth ≥ 1?Sitemap on, depth ≥ 1?robots.txt, sitemap.xml – same-origin URLsrobots.txt, sitemap.xmlsame-origin URLsNext round – unseen URLs, in parallelNext roundunseen URLs, in parallelGET page – delay, 3 triesGET pagedelay, 3 triestext/html?text/html?CSS, JS, images – per placeholder modeCSS, JS, imagesper placeholder modeRewrite references – relative local pathsRewrite referencesrelative local paths<path>/index.html<path>/index.htmlBelow max depth?Below max depth?Summary – pages, failed, durationSummarypages, failed, duration

Source: src/cli.rs, src/crawler.rs, src/parsers/sitemap.rs, src/parsers/links.rs, src/processors/html.rs, src/processors/stylesheet.rs, src/processors/image.rs

Show chart data
Data for One crawl run
StepKindLaneLeads to
site-scraper <URL>startsite-scraperFollow redirects
Follow redirectsexternalThe websiteClear output/<domain>/
Clear output/<domain>/data storesite-scraperSitemap on, depth ≥ 1?
Sitemap on, depth ≥ 1?decisionsite-scraperrobots.txt, sitemap.xml (yes), Next round (no)
robots.txt, sitemap.xmlexternalThe websiteNext round
Next roundstepsite-scraperGET page
GET pageexternalThe websitetext/html?
text/html?decisionsite-scraperCSS, JS, images (yes)
CSS, JS, imagesexternalThe websiteRewrite references
Rewrite referencesstepsite-scraper<path>/index.html
<path>/index.htmldata storesite-scraperBelow max depth?
Below max depth?decisionsite-scraperNext round (yes: same-origin links), Summary (queue empty)
Summaryendsite-scraper

Before the first request

  • Redirects: the start URL is requested once and its redirects are followed, so a crawl of https://www.example.com that ends up on https://example.com uses the bare domain for the output directory and for the same-site check.
  • Output directory: output/<domain>/ is removed and created again. Each run leaves exactly one snapshot per domain.
  • Sitemap seeds: with --sitemap (the default) and a depth of at least 1, site-scraper reads the Sitemap: lines in robots.txt and the well-known paths /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. Sitemap indexes and gzipped sitemaps are unpacked; only URLs on the same origin are kept, and they count as depth 1.

One page

  1. Fetch: a GET with browser-like headers (or --bot headers), after the configured delay plus a little jitter, with up to three tries. In headless mode a Chrome tab loads and renders the page instead; everything after this step is the same.
  2. HTML only: responses that are not text/html, such as a linked sitemap.xml, are counted but not saved and add no links.
  3. Assets: stylesheets, scripts and images are downloaded in parallel. CSS files are rewritten as well, so fonts and background images referenced with url() come along. External CSS and JavaScript follow --allow-external-assets, images follow --placeholder (Images and assets).
  4. Rewrite: every reference, including links to other pages of the site, becomes a relative local path, so the copy opens straight from disk.
  5. Save: the page is written to the path of its URL, with index.html for directories, for example https://example.com/blog/ to blog/index.html.
  6. Links: if the page is below --max-depth, its <a href> links on the same origin go into the next round. mailto:, tel: and javascript: links and fragments are dropped, and a URL is only fetched once.

The summary

When the queue is empty, site-scraper prints the output directory and a summary with pages, failures, warnings and duration. A page that returns 404 but is listed in the sitemap only counts as a warning, because stale sitemap entries are common. Any other failed page makes the exit code 1 (Output and exit codes).

Edit this page on GitHub · Docs for v1.4.6