Add Content-Signal directives to robots.txt

[fzipi]

Adds layouts/robots.txt so Hugo renders a real robots.txt instead of its default stub (enableRobotsTXT was already enabled in config/production/hugo.yaml).

Declared signals:

Content-Signal: search=yes, ai-input=yes, ai-train=no

Search indexing and AI-assistant retrieval are allowed so people can find CRS docs; use of the content for model training is not. The file also emits the sitemap URL via absURL.

Note: subsite/docs is a separate repo (submodule) and serves its own robots.txt — the same file needs to be added there for docs.coreruleset.org.

Reference: https://contentsignals.org/

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added a robots.txt policy with content signals for search, AI input, and AI training.
  • Enabled site access for all search crawlers.
  • Added a dynamically generated sitemap reference.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 04080834-dd18-4dc0-a729-7e8aee65e423

📥 Commits

Reviewing files that changed from the base of the PR and between 24a6d14 and 38d37c5.

📒 Files selected for processing (1)

• layouts/robots.txt

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The robots template adds content signals for search and AI use, allows all user agents to crawl the site, and emits a dynamic sitemap URL.

Changes

Robots policy

Layer / File(s)	Summary
Robots policy template layouts/robots.txt	Adds Content-Signal directives, unrestricted crawling, and a templated sitemap reference.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 38d37

This change adds the intended robots.txt directives and sitemap URL without introducing an actionable merge-blocking risk.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding Content-Signal directives to robots.txt.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cloudflare-workers-and-pages]

Deploying website with    Cloudflare Pages

Latest commit:	38d37c5
Status:	✅  Deploy successful!
Preview URL:	https://6cf465a4.website-1u6.pages.dev
Branch Preview URL:	https://feat-robots-content-signals.website-1u6.pages.dev

View logs