Skip to content

Phase 2: The agent retrieval surface #1377

Closed

Description

@neuromechanist

Part of epic #1336. Governed by ADR 0057.

This phase moved early and it blocks phase 5. ADR 0057 makes docs.nemar.org the canonical
retrieval surface and takes repository access out of the retrieval contract. Rewriting AGENTS.md
to point agents at docs URLs is only honest once those URLs are consumable, so this lands first.

The gap

Today the docs site serves HTML plus llms.txt. llms.txt is an index of titles and URLs with no
retrievable bodies, and its own generator comment is candid that there is no evidence it is
consumed. So an agent told to "read the docs" has two options: parse Starlight's HTML, or clone the
repo. The second is what ADR 0057 removes, which leaves scraping unless this phase happens.

Scope

  • A markdown representation per public page, addressable from the HTML URL by a predictable
    rule (for example /cli/commands.md beside /cli/commands/). Generated from the same content
    collection the HTML is built from, never hand-maintained, for the reason llms.txt already gives.
  • llms.txt stays the index, and gains bodies or an llms-full.txt companion. Decide which
    after measuring size; nm000281's index was 12.8 MB before source_key moved out of it, so "one
    big file" is not free.
  • Bot and cache controls. A machine-readable surface is bot-attracting by construction. Reuse
    the pattern the zarr plane already proved: cache rules plus rate limiting that counts without
    blocking legitimate clients (observeOnly in backend/src/middleware/rateLimit.ts). Rate limits
    belong here, not bolted on later.
  • The gated half is phase 3's, not this one's. This phase covers the public surface only.

Acceptance

  • A named public page is retrievable as markdown by URL with no credentials and no HTML parsing.
  • The build fails if a gated page appears in any public machine-readable document, the way
    scripts/check-admin-gating.ts --built already enforces for the sitemap, llms.txt and the
    search index.
  • bun run probe:gate still passes, and public pages still serve.

Activity

  1. neuromechanist commented on Sep 13, 2026

    @neuromechanist
    ContributorAuthor

    Done in nemarOrg/docs#35 (squashed to ca5e5b5d), verified in production.

    /cli/commands.md   200  text/markdown; charset=utf-8  public, max-age=300, s-maxage=3600
    /admin.md          302  -> the sign-in handoff
    probe:gate         30/30 (was 27)
    

    All 67 pages are served as Markdown at <path>.md; llms.txt links those rather than the HTML. entry.id is the published path, so the mirrors track the sitemap one for one and the one slug: override lands correctly at platform/zarr/mental-model.md.

    Probing the gate before writing the route found a real hole. isGatedPath matched /admin, /admin/ and /admin/*; /admin.md is the admin section index's mirror and is none of those, so that one file would have been served to anyone. It now strips a trailing extension generically, so the next representation inherits the gate. The build gate checks this by PATH and never by content, because the home page links /admin/commands/ and platform/api.md lists a dozen /admin/ API routes on another host, so a content scan would fail the build for the wrong reason. It asks the middleware's own isGatedPath rather than keeping a second copy of the rule, and reverting the fix makes the build fail naming admin.md.

    Rate limiting was not built and the code says so. The docs Pages project has no KV, D1 or rate-limit binding, and an in-Function counter is per-isolate. What shipped is edge caching, which is not a limit. The owner's call is to enable Cloudflare bot fighting if the surface is abused.

    The cache header was checked against a cache-busted request: the first response showed a stale edge value (s-maxage=604800) that would have been reported as fact.

    Next: phase 3 (#1341), rescoped by ADR 0059 to one private docs repo rather than an overlay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agentsdocumentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions