You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This phase moved early and it blocks phase 5. ADR 0057 makes docs.nemar.org the canonical
retrieval surface and takes repository access out of the retrieval contract. Rewriting AGENTS.md
to point agents at docs URLs is only honest once those URLs are consumable, so this lands first.
The gap
Today the docs site serves HTML plus llms.txt. llms.txt is an index of titles and URLs with no
retrievable bodies, and its own generator comment is candid that there is no evidence it is
consumed. So an agent told to "read the docs" has two options: parse Starlight's HTML, or clone the
repo. The second is what ADR 0057 removes, which leaves scraping unless this phase happens.
Scope
A markdown representation per public page, addressable from the HTML URL by a predictable
rule (for example /cli/commands.md beside /cli/commands/). Generated from the same content
collection the HTML is built from, never hand-maintained, for the reason llms.txt already gives.
llms.txt stays the index, and gains bodies or an llms-full.txt companion. Decide which
after measuring size; nm000281's index was 12.8 MB before source_key moved out of it, so "one
big file" is not free.
Bot and cache controls. A machine-readable surface is bot-attracting by construction. Reuse
the pattern the zarr plane already proved: cache rules plus rate limiting that counts without
blocking legitimate clients (observeOnly in backend/src/middleware/rateLimit.ts). Rate limits
belong here, not bolted on later.
The gated half is phase 3's, not this one's. This phase covers the public surface only.
Acceptance
A named public page is retrievable as markdown by URL with no credentials and no HTML parsing.
The build fails if a gated page appears in any public machine-readable document, the way scripts/check-admin-gating.ts --built already enforces for the sitemap, llms.txt and the
search index.
bun run probe:gate still passes, and public pages still serve.
All 67 pages are served as Markdown at <path>.md; llms.txt links those rather than the HTML. entry.id is the published path, so the mirrors track the sitemap one for one and the one slug: override lands correctly at platform/zarr/mental-model.md.
Probing the gate before writing the route found a real hole.isGatedPath matched /admin, /admin/ and /admin/*; /admin.md is the admin section index's mirror and is none of those, so that one file would have been served to anyone. It now strips a trailing extension generically, so the next representation inherits the gate. The build gate checks this by PATH and never by content, because the home page links /admin/commands/ and platform/api.md lists a dozen /admin/ API routes on another host, so a content scan would fail the build for the wrong reason. It asks the middleware's own isGatedPath rather than keeping a second copy of the rule, and reverting the fix makes the build fail naming admin.md.
Rate limiting was not built and the code says so. The docs Pages project has no KV, D1 or rate-limit binding, and an in-Function counter is per-isolate. What shipped is edge caching, which is not a limit. The owner's call is to enable Cloudflare bot fighting if the surface is abused.
The cache header was checked against a cache-busted request: the first response showed a stale edge value (s-maxage=604800) that would have been reported as fact.
Next: phase 3 (#1341), rescoped by ADR 0059 to one private docs repo rather than an overlay.
Activity
neuromechanist commentedon Sep 13, 2026
Done in nemarOrg/docs#35 (squashed to
ca5e5b5d), verified in production.All 67 pages are served as Markdown at
<path>.md;llms.txtlinks those rather than the HTML.entry.idis the published path, so the mirrors track the sitemap one for one and the oneslug:override lands correctly atplatform/zarr/mental-model.md.Probing the gate before writing the route found a real hole.
isGatedPathmatched/admin,/admin/and/admin/*;/admin.mdis the admin section index's mirror and is none of those, so that one file would have been served to anyone. It now strips a trailing extension generically, so the next representation inherits the gate. The build gate checks this by PATH and never by content, because the home page links/admin/commands/andplatform/api.mdlists a dozen/admin/API routes on another host, so a content scan would fail the build for the wrong reason. It asks the middleware's ownisGatedPathrather than keeping a second copy of the rule, and reverting the fix makes the build fail namingadmin.md.Rate limiting was not built and the code says so. The docs Pages project has no KV, D1 or rate-limit binding, and an in-Function counter is per-isolate. What shipped is edge caching, which is not a limit. The owner's call is to enable Cloudflare bot fighting if the surface is abused.
The cache header was checked against a cache-busted request: the first response showed a stale edge value (
s-maxage=604800) that would have been reported as fact.Next: phase 3 (#1341), rescoped by ADR 0059 to one private docs repo rather than an overlay.