CASOON Open Source

One typed config, five generated files.

@casoon/astro-site-files writes robots.txt, llms.txt, sitemap.xml, security.txt and humans.txt into the build output when astro build runs. The files below are the ones this build wrote.

astro.config.tsTypeScript

siteFiles({
  robots: { preset: 'seoOnly' },
  llms: { title: 'Example', description: 'An example website.' },
  sitemap: {},
  security: { contact: 'mailto:security@example.com' },
  humans: { team: [{ name: 'Alice', role: 'Development' }] },
})

astro build

User-agent: *
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: OAI-AdsBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: YouBot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: GrokBot
Disallow: /

User-agent: xAI-Grok
Allow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: KimiBot
Disallow: /

User-agent: Kimi-SearchBot
Allow: /

User-agent: Kimi-User
Allow: /

User-agent: QwenBot
Disallow: /

User-agent: TongyiBot
Allow: /

User-agent: Hunyuan
Disallow: /

User-agent: ERNIEBot
Disallow: /

User-agent: PanguBot
Disallow: /

User-agent: PetalBot
Allow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: cohere-ai
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: TikTokSpider
Disallow: /

User-agent: DoubaoBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: bedrockbot
Allow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /

User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: kagi-fetcher
Allow: /

User-agent: Diffbot
Disallow: /

User-agent: Omgilibot
User-agent: omgili
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: FirecrawlAgent
Disallow: /

User-agent: Crawlspace
Disallow: /

User-agent: VelenPublicWebCrawler
Disallow: /

Sitemap: https://casoon.github.io/astro-site-files/sitemap.xml

dist/robots.txt

# astro-site-files

> Astro integration that generates robots.txt, llms.txt, sitemap.xml, security.txt and humans.txt at build time from typed configuration.

Published on npm as @casoon/astro-site-files. Requires Node.js 22.12 or newer and Astro 6 or newer. MIT licensed.

## Documentation

- [Installation](https://casoon.github.io/astro-site-files/docs/getting-started/installation/)
- [Quickstart](https://casoon.github.io/astro-site-files/docs/getting-started/quickstart/)
- [robots.txt](https://casoon.github.io/astro-site-files/docs/configuration/robots-txt/): Presets, bot groups and path rules
- [llms.txt](https://casoon.github.io/astro-site-files/docs/configuration/llms-txt/)
- [sitemap.xml](https://casoon.github.io/astro-site-files/docs/configuration/sitemap-xml/): Discovery, priorities, i18n and sources
- [RSS feed](https://casoon.github.io/astro-site-files/docs/configuration/rss/)
- [security.txt](https://casoon.github.io/astro-site-files/docs/configuration/security-txt/)
- [humans.txt](https://casoon.github.io/astro-site-files/docs/configuration/humans-txt/)
- [API overview](https://casoon.github.io/astro-site-files/docs/reference/api/)

## Project

- [Source code](https://github.com/casoon/astro-site-files)
- [npm package](https://www.npmjs.com/package/@casoon/astro-site-files)
- [Changelog](https://casoon.github.io/astro-site-files/changelog/)

dist/llms.txt

<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated by @casoon/astro-site-files -->
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://casoon.github.io/astro-site-files/changelog/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/configuration/humans-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/configuration/llms-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/configuration/robots-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/configuration/rss/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/configuration/security-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/configuration/sitemap-xml/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/getting-started/installation/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/getting-started/quickstart/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/guides/audit/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/guides/site-url-and-base/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/docs/reference/api/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/showcase/robots-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/showcase/llms-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/showcase/sitemap-xml/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/showcase/security-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/showcase/humans-txt/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/showcase/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://casoon.github.io/astro-site-files/</loc>
    <lastmod>2026-11-11</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
</urlset>

dist/sitemap.xml

Contact: mailto:info@casoon.de
Expires: 2027-05-10T18:11:26.409Z
Preferred-Languages: en, de
Canonical: https://casoon.github.io/astro-site-files/.well-known/security.txt

dist/.well-known/security.txt

/* TEAM */
    Name: Jörn Seidel
    Role: Author and maintainer

/* SITE LAST UPDATED */
    2026-11-11

/* TECHNOLOGY COLOPHON */
    Astro, TypeScript, @casoon/astro-site-files, @casoon/pages-theme

/* NOTE */
    This file, robots.txt, llms.txt, sitemap.xml and security.txt are generated by @casoon/astro-site-files during the site build.

dist/humans.txt

npmpnpm

npm install @casoon/astro-site-files

pnpm add @casoon/astro-site-files

MITnpm 0.7.2Astro ≥ 6Node.js ≥ 22.12

file types written at build time: 6
crawlers in the built-in bot registry: 64
tests in the suite: 162
runtime dependencies: 0

What it does

robots.txt from presets
Five presets from seoOnly to lockdown decide per bot group; single bots and whole groups can be overridden. Global rules are repeated in every allow group, because a crawler obeys only its most specific group.
Sitemap from the build output
Pages come from what Astro built. Redirect and noindex pages are dropped, hreflang alternates follow the Astro i18n config, and more than 50,000 URLs switch to a sitemap index.
Typed per file
Each file has its own options interface (RobotsOptions, LlmsOptions, SitemapOptions, SecurityOptions, HumansOptions), exported for your own config modules.
Audited during the build
15 audit rules report problems in the build log, among them an expired security.txt, a contact without URI scheme and legal pages blocked in robots.txt.

Generated at build time

All examples →

/* TEAM */
    Name: Jörn Seidel
    Role: Author and maintainer

/* SITE LAST UPDATED */
    2026-11-11

/* TECHNOLOGY COLOPHON */
    Astro, TypeScript, @casoon/astro-site-files, @casoon/pages-theme

/* NOTE */
    This file, robots.txt, llms.txt, sitemap.xml and security.txt are generated by @casoon/astro-site-files during the site build.

Generated by @casoon/astro-site-files while this site was built, from the configuration in site/src/site-files.ts. The showcase shows all five files.

dist/humans.txt

Quickstart

Three steps from install to generated files. The full walkthrough lives in the documentation.

Install the package and set site in your Astro config.
Add siteFiles() to integrations with the files you want.
Run astro build. The files are written to dist/.

Read the documentation

astro.config.tsTypeScript

import { defineConfig } from 'astro/config'
import siteFiles from '@casoon/astro-site-files'

export default defineConfig({
  site: 'https://example.com',
  integrations: [
    siteFiles({
      robots: { preset: 'seoOnly' },
      llms: { title: 'Example', description: 'An example website.' },
      security: { contact: 'mailto:security@example.com', expires: '2027-03-01T00:00:00.000Z' },
      humans: { team: [{ name: 'Alice', role: 'Development' }] },
    }),
  ],
})

New in v0.7.2

Changelog →

Pages: the project site uses the current shared theme (0.12.6).
Site: the footer follows the page language instead of always showing German text, and the start page now carries a release teaser, an FAQ and a closing link to the quickstart (theme 0.12.2).
Robots: the global User-agent: * group is no longer written without a rule; when no allow, disallow or preset rule applies, it gets Allow: /. Parsers that merge consecutive User-agent lines handed it the rules of the next bot block before. Details: Presets.

Frequently asked questions

Which files does it generate, and which ones are on by default?

robots.txt and sitemap.xml are generated by default; llms.txt, rss.xml, security.txt and humans.txt are opt-in and only written once their option is configured.

Does robots.txt actually stop AI crawlers from scraping my site?

No — robots.txt is voluntary, and many scrapers since 2025-2026 no longer identify themselves as bots. Blocking crawlers that do comply still requires pairing a preset like blockTraining or lockdown with real infrastructure such as rate limiting or bot detection.

What does the 0.6.0 registry update change for an existing config?

The bot registry grew from 25 to 64 crawlers, and the citationFriendly and blockTraining presets now derive their training rules from the registry categories instead of a hand-maintained bot list, so newly added bots are covered automatically.

Does this replace @casoon/astro-crawler-policy or @casoon/astro-sitemap?

Yes. @casoon/astro-site-files is the successor to both — it covers robots.txt and llms.txt from astro-crawler-policy, and sitemap.xml and rss.xml from astro-sitemap, which are no longer actively maintained.

Five generated files in three steps.

Read the quickstart

One typed config, five generated files.

What it does

robots.txt from presets

Sitemap from the build output

Typed per file

Audited during the build

Generated at build time

Quickstart

New in v0.7.2

Frequently asked questions

Five generated files in three steps.