Migrating from astro-crawler-policy
Replace crawlerPolicy({ preset }) with siteFiles({ robots: { preset } }) in three steps, and what no longer exists.
@casoon/astro-crawler-policy is archived. Its robots.txt and llms.txt features live on in this
package; the preset names are the same.
Three steps
1. Swap the package
npm rm @casoon/astro-crawler-policy
npm install @casoon/astro-site-files
Node.js 22.12.0 and Astro 6 are required, see installation.
2. Move the config
// before
import crawlerPolicy from '@casoon/astro-crawler-policy'
export default defineConfig({
integrations: [crawlerPolicy({ preset: 'citationFriendly' })],
})
// after
import siteFiles from '@casoon/astro-site-files'
export default defineConfig({
site: 'https://example.com',
integrations: [siteFiles({ robots: { preset: 'citationFriendly' } })],
})
The preset moves under robots. The Sitemap: line is derived from Astro’s site, so set it.
3. Compare the output
Keep a copy of the old dist/robots.txt, build, and diff:
cp dist/robots.txt robots.old.txt
npx astro build
diff robots.old.txt dist/robots.txt
Expect more bot blocks (the registry is larger) and no Content-Signal line.
What is gone
| Before | Now |
|---|---|
Content-Signal line |
Not written, see robots.txt |
Merge strategy replace, prepend, append with public/robots.txt |
No merge option. The file is written to the build output after the build, so it replaces a public/robots.txt; move any hand-written rules into disallow, allow or agents |
env option for per-environment overrides |
Plain JavaScript in the Astro config, see staging recipe |