Repository navigation
Description
The idea
AGENTS.md is 633 lines and it is explicitly a map, not a manual: almost every section ends in a pointer at something under .context/. Those pointees are the actual reference material, and today they are readable only by whoever has this checkout open. Content in nemarOrg/docs is different in kind: it is one git pull away for any agent or person in any of our repos, and it is also published at docs.nemar.org, so it is reachable by URL with no checkout at all.
So the proposal is to move what belongs in a published reference out of this repo's agent files and into nemarOrg/docs, and leave AGENTS.md as a short set of hard rules plus pointers at docs pages. The same question applies to every other repo's AGENTS.md (website, docs, nemar-observability, nemar-tools); this repo is the biggest and should go first, and whatever pattern it settles becomes the pattern for the rest.
Related: nemarOrg/docs#28 is the first piece of this in practice. Nobody could answer "which hostname serves what" from any published page, so it was answered by guessing a URL, and the guess shipped in OpenScience-Collective/osa. The facts existed, in four repos' configuration.
What makes this non-trivial
Not everything can move, and the sorting is the real work. Three buckets:
Publishable reference. Belongs on docs.nemar.org. The dataset ID bands, the release pipeline, the user flow and account kinds, the S3 layout, the Zarr serving contract, the CLI command surface. Much of this is ALREADY on the docs site in some form, which means we currently maintain two copies and they can drift; harmonizing removes a copy rather than adding a page.
Admin-gated runbook. Belongs under docs /admin/*, which is gated by NEMAR's own admin-only session (admin or owner role) through a Cloudflare Pages Function. edge-gated by Cloudflare Access was wrong when this epic was written and is what phase 0 existed to fix: the Access application on that Pages project covers preview deployments only, so every admin page answered 200 to anyone on production. See nemarOrg/docs ADR 0002 and nemar-cli ADR 0056. Verify a deploy with bun run probe:gate against production, never a preview. Disaster recovery is already there. The candidates here are parts of .context/systems-inventory.md and .context/validated_workflows.md.
Must stay in this repo. The docs repo's own AGENTS.md says it plainly: keep genuinely internal material out of the docs site, even behind the gate. That covers the webhook contracts, the blast-radius fences (dev D1 holds real users, the nemarDatasets org is shared with prod, the live nm000103-nm000107 datasets), and anything that is an instruction to an agent editing THIS checkout rather than a fact about the platform.
ADRs stay here regardless. They are binding decisions about this codebase, test/adr-index.unit.test.ts enforces that the index and the files on disk agree, and the never-delete-only-supersede rule depends on them being version-controlled next to the code they bind. This repo is public, so docs pages can link to them at their GitHub paths.
Reframed by ADR 0057 (2026-09-13)
Phase 0 built the admin gate and, in doing so, broke two of this epic's premises.
The gate controls who is SERVED, not who can read. nemarOrg/docs is public. The rule meant to
cover that gap says keep genuinely internal material in nemar-cli, and nemar-cli is public too.
So the original phase 3, "move the operational runbooks behind the gate", would have moved them from
one public repository to another and called the result gated.
"One git pull away" assumed repository access is how agents retrieve docs. It does not have to
be. The docs URL serves that, and serving it that way is what makes a public/private source split
possible at all.
So the epic is now: make docs.nemar.org the canonical retrieval surface for humans and agents,
with the public/gated split enforced at the source rather than only at serving.
Calibration, which is part of the decision: no credential value is in any of the public material,
nothing needs rotating, and this is not an incident. NEMAR is an open project and its public
documentation stays public on purpose. The bar is publish good examples, not a recipe for
attacking the platform — a single fact about where something runs is fine, an assembled map of
host aliases, state paths, cron windows and deploy mechanism is the artifact that moves. Historical
exposure in git is accepted because operational detail ages out; that reasoning would not cover a
key.
Phases
| Phase | Issue | |
|---|---|---|
| 0 | #1338 | Gate the admin docs behind a NEMAR admin session — done, verified 27/27 in production |
| 1 | #1339 | Inventory, the classification rule, and the docs-link drift guard |
| 2 | #1377 | The agent retrieval surface: per-page markdown, llms.txt, bot and cache controls |
| 3 | #1341 | Private admin source, the credential exchange, and the move out of public repos |
| 4 | #1340 | Move the publishable reference into docs |
| 5 | #1342 | Rewrite AGENTS.md to hard rules, docs URLs, and escalation |
| 6 | #1378 | Apply the pattern to the other repos |
Two orderings are load-bearing. Phase 2 precedes phase 5, because pointing agents at docs URLs
is only honest once those URLs are retrievable as markdown. Phase 3 precedes phase 4, so the
public/private split is settled before anything moves.
Retrieval and access
An admin or service account (ADR 0048) retrieves gated docs by exchanging an API key for a
short-lived docs-scoped token, never by presenting the key to the docs edge. That keeps ADR 0056's
separation working in both directions.
Losing read access must not cost a contributor the ability to report a problem: a non-admin who hits
an admin-adjacent issue opens it anyway and tags the admin team. No GitHub team exists in
nemarOrg today, so one has to be created before phase 5 can point at it.
Not in scope
Rewriting the docs site's information architecture, and moving the ADRs.
Activity
neuromechanist commentedon Sep 10, 2026
Revised scope after investigation, 2026-09-09
Two things came out of checking the assumptions in this issue before starting, and both change the plan.
1. The docs
/admin/*gate does not exist. The docs repo'sAGENTS.mdsays access control is applied at the edge by Cloudflare Access ondocs.nemar.org/admin/*. It is not. All 12 pages under/admin/answer 200 anonymously to any user agent, and all 12 are listed in the public sitemap, so they are crawlable rather than merely reachable.Nothing confidential is behind it, so this is not a leak: the 4,731 lines there are procedures plus identifiers, never credential values (secret NAMES and setup steps in
github-app-setup.md; the UUIDs inrestoration-guide.mdare DataLad dataset ids next to public Zenodo DOIs, and one already appears in nemar-cli's publicscripts/nemar-restore-dataset.sh). Both repos are public, so the text was already published.It is worse than a leak in one specific way: it is a false safety claim that invites one. Anyone reading that file believes the section is protected and writes accordingly. This epic's original phase 3 would have been the first victim, moving
systems-inventory.md(hosts, paths, crons) behind a control that is not there.2. The gate will be NEMAR's own ORCID-backed login, not Cloudflare Access. Owner's call: one identity system for everything rather than a second one for the docs site.
The constraint that makes this real work rather than a config flip:
WEB_SESSION_COOKIE_DOMAIN = "app.nemar.org"(backend/wrangler-sccn.toml:148), scoped that narrowly on purpose so the session never rides along withdata.nemar.orgbyte-range fetches orapi.nemar.orgsearch. A cookie scoped to one host cannot authenticate another, sodocs.nemar.orgneeds a signed cross-host handoff, not cookie sharing. Widening the cookie to.nemar.orgis not an option; it would undo the reason the scope exists.Phases
docs.nemar.org/admin/*behind the ORCID-backed session with an admin-role check, verified by an anonymous fetch afterwardsAGENTS.mdto hard rules, the ADR index pointer, and docs pointersThe guard lands in phase 1, before any content moves, so every pointer added in phases 2 to 4 is validated as it lands.
Settled, not open
.rules/*(six files, 585 lines) stays in this repo untouched. Those are standards for editing this checkout, not facts about the platform, which is exactly the must-stay bucket. The drift guard reads them only to check any docs URLs they cite.Integration branch for this epic is
dev, notmain.Move the four runbooks out and point everything at the docs URLs (#1389)
Rewrite AGENTS.md to hard rules, docs URLs, and escalation (#1390)
Rewrite AGENTS.md to hard rules, docs URLs, and escalation (#332)
Add the retrieval contract and escalation to AGENTS.md (#215)
Add the retrieval contract and escalation to AGENTS.md (#43)
Add the retrieval contract and escalation to AGENTS.md (#2)
Merge pull request #1394 from nemarOrg/feature/issue-1336-epic-agent-…
neuromechanist commentedon Sep 14, 2026
Merged to
devase5ccda05, with per-phase history preserved.What this epic set out to do, and did
docs.nemar.orgis now the canonical surface for anything about the NEMAR platform. Repositories keep the rules that bind someone editing them and point at URLs for everything else. Twenty-three pull requests across seven repositories.nemarOrg/docsprivate at source; runbooks moved in;nemar admin docsAGENTS.md669 to 313 linesThe premise changed twice, and both corrections were the valuable part
Phase 0 found that Cloudflare Access was never protecting the admin docs. The application covered preview deployments only, so all twelve pages answered 200 anonymously and sat in the public sitemap. Nothing confidential leaked, but the claim invited someone to put a real secret behind a gate that was not there. A false safety claim is worse than a known gap.
That broke the epic's own design. The original plan moved internal material into
nemar-clibecause the docs repo was public. Butnemar-cliis public too, so the rule was a surface rule wearing the language of a secrecy control. ADR 0057 and ADR 0059 replaced it: one docs repo, private at source, public at the URL.Three defects found along the way, none of them this epic's subject
/and/nemar, middleware sees the full path, so everyAUTH_PATHSentry was unthrottled through the prefix. Long pre-existing.import-weekly-summary-sweepcompared a SQL timestamp against an injected clock and passed only while the wall clock sat in the same ISO week as a hardcoded literal. Fixed ondevin Fix the weekly-summary audit-gate tests, red on every branch since 2026-09-14 #1385 because it blocked every branch.nemar-toolswas stale and is archived; its Zenodo flow had been superseded by EZID per ADR 0007.What the seam review caught before it shipped further
Commissioned to hunt for content deleted in one repository that never landed in the other. That risk was clean -- all four runbook moves verified by structural inventory, zero loss. The real damage was elsewhere:
unit-pure, a required check, failed on every fork pull request, because the private docs checkout needs a secret forks never receive. Exactly what ADR 0057 says must not happen.AGENTS.mdrewrite, including "revocation cascades", removed by the same change that violated it.The lesson worth keeping: checking the thing you are worried about is not the same as checking. The losses were findable only by inventorying every claim against three buckets -- survives here, exists on the docs site, or nowhere -- not by reading the diff.
Calibration, recorded because it shaped the work
This is an open-access project. The restrictions exist to protect resources, not knowledge. When the Zarr verification rules were restored they went on a public page, not behind the gate, because they explain results a data consumer can observe and disclose nothing that costs anything. Gating them by reflex would have been over-restriction.
Follow-ups, not blockers
nemarOrg/websitestagingcarries 17 pre-existing test failures (webSigninEnabled,/api/auth/keys,dedupingFetch), unrelated to this epic and flagged on Rewrite AGENTS.md to hard rules, docs URLs, and escalation website#332.Merge dev: the runbooks moved to the docs site