astro-site-filesv0.7.2

Migrating from astro-crawler-policy

Replace crawlerPolicy({ preset }) with siteFiles({ robots: { preset } }) in three steps, and what no longer exists.

@casoon/astro-crawler-policy is archived. Its robots.txt and llms.txt features live on in this package; the preset names are the same.

Three steps

1. Swap the package

npm rm @casoon/astro-crawler-policy
npm install @casoon/astro-site-files

Node.js 22.12.0 and Astro 6 are required, see installation.

2. Move the config

// before
import crawlerPolicy from '@casoon/astro-crawler-policy'
export default defineConfig({
  integrations: [crawlerPolicy({ preset: 'citationFriendly' })],
})

// after
import siteFiles from '@casoon/astro-site-files'
export default defineConfig({
  site: 'https://example.com',
  integrations: [siteFiles({ robots: { preset: 'citationFriendly' } })],
})

The preset moves under robots. The Sitemap: line is derived from Astro’s site, so set it.

3. Compare the output

Keep a copy of the old dist/robots.txt, build, and diff:

cp dist/robots.txt robots.old.txt
npx astro build
diff robots.old.txt dist/robots.txt

Expect more bot blocks (the registry is larger) and no Content-Signal line.

`sitemap` is on by default, so the integration also writes `sitemap.xml`. If you generate one with `@astrojs/sitemap` as well, switch one of the two off, for example with `siteFiles({ sitemap: false })`.

What is gone

Before Now
Content-Signal line Not written, see robots.txt
Merge strategy replace, prepend, append with public/robots.txt No merge option. The file is written to the build output after the build, so it replaces a public/robots.txt; move any hand-written rules into disallow, allow or agents
env option for per-environment overrides Plain JavaScript in the Astro config, see staging recipe

Edit this page on GitHub · Docs for v0.7.2