Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 72 additions & 35 deletions .context/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,55 +18,92 @@ Documents that record completed work carry a `STATUS: HISTORICAL` banner. They a
`test/adr-index.unit.test.ts`. Read before designing, and before "fixing" something that looks
odd — several oddities here are deliberate.

## Where each document is headed

Every row below carries a **Destination**, the classification ADR 0058 assigns it, as groundwork for
epic #1336 phase 3. Nothing has moved yet.

| Value | Meaning |
|---|---|
| `public` | Stays here, in this public repo, and may be published to docs.nemar.org. |
| `private` | Moves to the private content repo in phase 3; served gated at docs.nemar.org. |
| `stays-with-code` | Belongs next to the code it binds and is not a docs-site candidate. |

`decisions/` is `stays-with-code` as a whole, as are `.rules/*` and `AGENTS.md`'s hard rules.
Destination only: the reasoning is in ADR 0058 as a general rule and deliberately not restated
per file.

## Current reference

Trust these for how things work today.

| Document | What it is |
|---|---|
| [systems-inventory.md](systems-inventory.md) | Every host and service, with paths, cron schedules, and deploy procedures. Expands the map in `AGENTS.md`. |
| [validated_workflows.md](validated_workflows.md) | Workflows proven by prototype, with the gotchas. The git-annex and staging-to-final recipes live here. |
| [release-safety-playbook.md](release-safety-playbook.md) | Environments, promotion path, pre-release checks. |
| [recover-runbook.md](recover-runbook.md) | Recovering 0-byte imports (epic #967 Phase 5). |
| [access_control.md](access_control.md) | How collaborator-based access is implemented. |
| [dataset_workflow.md](dataset_workflow.md) | Dataset lifecycle: IDs, upload, download, versioning. |
| [pr_architecture.md](pr_architecture.md) | PR + staging mechanics. **The branch-protection payload it described is superseded** — see ADR 0001. |
| [phase5-cross-repo-owner-deploys.md](phase5-cross-repo-owner-deploys.md) | Known cross-repo gaps needing owner action. |
| [ideas.md](ideas.md) | Exploratory design notes. Frequently rewritten; promote settled items to an ADR. |
| [research.md](research.md) | Prior art from the `nemar-tools` scripts. Describes the **pre-NEMAR** Zenodo flow — DOIs are EZID now (ADR 0007). |
| Document | What it is | Destination |
|---|---|---|
| [systems-inventory.md](systems-inventory.md) | Every host and service, with paths, cron schedules, and deploy procedures. Expands the map in `AGENTS.md`. | private |
| [validated_workflows.md](validated_workflows.md) | Workflows proven by prototype, with the gotchas. The git-annex and staging-to-final recipes live here. | private |
| [release-safety-playbook.md](release-safety-playbook.md) | Environments, promotion path, pre-release checks. | public |
| [recover-runbook.md](recover-runbook.md) | Recovering 0-byte imports (epic #967 Phase 5). | public |
| [access_control.md](access_control.md) | How collaborator-based access is implemented. | public |
| [dataset_workflow.md](dataset_workflow.md) | Dataset lifecycle: IDs, upload, download, versioning. | public |
| [pr_architecture.md](pr_architecture.md) | PR + staging mechanics. **The branch-protection payload it described is superseded**; see ADR 0001. | public |
| [phase5-cross-repo-owner-deploys.md](phase5-cross-repo-owner-deploys.md) | Known cross-repo gaps needing owner action. | public |
| [ideas.md](ideas.md) | Exploratory design notes. Frequently rewritten; promote settled items to an ADR. | public |
| [research.md](research.md) | Prior art from the `nemar-tools` scripts. Describes the **pre-NEMAR** Zenodo flow; DOIs are EZID now (ADR 0007). | public |

## Design notes

Longer-form design documents. Found unlisted by the phase 1 inventory sweep (#1339) and added
here, since an index that misses a document cannot classify it either.

| Document | What it is | Destination |
|---|---|---|
| [mcp-server-design.md](mcp-server-design.md) | The MCP server design, the largest document here. | public |
| [draft-zarr-inference-ecosystem-plan.md](draft-zarr-inference-ecosystem-plan.md) | Draft plan for inference over the Zarr serving copy. | public |
| [draft-817-submission-minimums-policy.md](draft-817-submission-minimums-policy.md) | Draft submission-minimums policy. | public |
| [draft-1036-validator-3x-evaluation.md](draft-1036-validator-3x-evaluation.md) | Evaluation of bids-validator 3.x. | public |
| [draft-eegdash-zarr-issues.sh](draft-eegdash-zarr-issues.sh) | Script drafting the EEGDash Zarr issues. The one non-markdown document here, which is why a markdown-only sweep missed it. | public |
| [plans/spec-pr1.md](plans/spec-pr1.md) | Column-budget rebuild spec (issue #1182). In `plans/`, which a top-level sweep missed. | public |

## Research — point-in-time findings

Measurements and investigations. Accurate as of their date; re-verify before relying on numbers.

| Document | Question it answered |
|---|---|
| [research-catalog-consolidation.md](research-catalog-consolidation.md) | How to collapse the two dataset tables (-> ADR 0003). |
| [research-d1-backup-655.md](research-d1-backup-655.md) | How to back up D1 (-> ADR 0004). |
| [research-submission-minimums-deskreject.md](research-submission-minimums-deskreject.md) | How many datasets would trip each proposed reject rule (-> ADR 0014). |
| [research-archive-import-candidates.md](research-archive-import-candidates.md) | Survey of ~90 non-OpenNeuro archives (-> ADR 0013). |
| [research-openneuro-import-forensics.md](research-openneuro-import-forensics.md) | Why a 5-dataset import batch failed. |
| [openneuro-support-403-report.md](openneuro-support-403-report.md) | Objects inaccessible upstream, for reporting. |
| [security-fix-dataset-visibility.md](security-fix-dataset-visibility.md) | The 2026-01 private-dataset leak (-> ADR 0017). |
| [plan-multi-archive-importer.md](plan-multi-archive-importer.md) | Architecture RFC for multi-archive import (-> ADR 0013, **proposed**). |
| [research-make-vs-take-audit.md](research-make-vs-take-audit.md) | 2026-09-03 audit of bespoke code an established library or platform feature already covers, with replace / wrap / keep verdicts (-> ADR 0037). |
| [research-agent-findability.md](research-agent-findability.md) | 2026-09-03: how AI agents and crawlers actually find datasets (robots tokens, sitemaps, JSON-LD, registries, DataCite); evidence that llms.txt and markdown mirrors are not read; Zarr guidance for agents (-> OSCAR epic). |
| Document | Question it answered | Destination |
|---|---|---|
| [research-catalog-consolidation.md](research-catalog-consolidation.md) | How to collapse the two dataset tables (-> ADR 0003). | public |
| [research-d1-backup-655.md](research-d1-backup-655.md) | How to back up D1 (-> ADR 0004). | private |
| [research-submission-minimums-deskreject.md](research-submission-minimums-deskreject.md) | How many datasets would trip each proposed reject rule (-> ADR 0014). | public |
| [research-archive-import-candidates.md](research-archive-import-candidates.md) | Survey of ~90 non-OpenNeuro archives (-> ADR 0013). | public |
| [research-openneuro-import-forensics.md](research-openneuro-import-forensics.md) | Why a 5-dataset import batch failed. | public |
| [openneuro-support-403-report.md](openneuro-support-403-report.md) | Objects inaccessible upstream, for reporting. | public |
| [security-fix-dataset-visibility.md](security-fix-dataset-visibility.md) | The 2026-01 private-dataset leak (-> ADR 0017). | public |
| [plan-multi-archive-importer.md](plan-multi-archive-importer.md) | Architecture RFC for multi-archive import (-> ADR 0013, **proposed**). | public |
| [research-make-vs-take-audit.md](research-make-vs-take-audit.md) | 2026-09-03 audit of bespoke code an established library or platform feature already covers, with replace / wrap / keep verdicts (-> ADR 0037). | public |
| [research-agent-findability.md](research-agent-findability.md) | 2026-09-03: how AI agents and crawlers actually find datasets (robots tokens, sitemaps, JSON-LD, registries, DataCite); evidence that llms.txt and markdown mirrors are not read; Zarr guidance for agents (-> OSCAR epic). | public |

## Historical — completed work

Records of what was done. Do not use as a current-state reference.

[plan.md](plan.md) · [prototyping_plan.md](prototyping_plan.md) ·
[architecture_review.md](architecture_review.md) ·
[epic_central_manifest_state.md](epic_central_manifest_state.md) ·
[sprint_review_publication_workflow.md](sprint_review_publication_workflow.md) ·
[deleted-datasets-incident.md](deleted-datasets-incident.md) ·
[blast-radius-catalog-fold.md](blast-radius-catalog-fold.md) ·
[plan-923-test-staging.md](plan-923-test-staging.md) ·
[plan-legacy-separation-793.md](plan-legacy-separation-793.md) ·
[plan-import-robustness.md](plan-import-robustness.md) ·
[plan-phase2-retry-engine.md](plan-phase2-retry-engine.md) ·
[draft-1023-service-access-endpoints-plan.md](draft-1023-service-access-endpoints-plan.md)
A table rather than a list, so each carries a destination like every other document here.

| Document | Destination |
|---|---|
| [plan.md](plan.md) | public |
| [prototyping_plan.md](prototyping_plan.md) | public |
| [architecture_review.md](architecture_review.md) | public |
| [epic_central_manifest_state.md](epic_central_manifest_state.md) | public |
| [sprint_review_publication_workflow.md](sprint_review_publication_workflow.md) | public |
| [deleted-datasets-incident.md](deleted-datasets-incident.md) | public |
| [blast-radius-catalog-fold.md](blast-radius-catalog-fold.md) | public |
| [plan-923-test-staging.md](plan-923-test-staging.md) | private |
| [plan-legacy-separation-793.md](plan-legacy-separation-793.md) | public |
| [plan-import-robustness.md](plan-import-robustness.md) | public |
| [plan-phase2-retry-engine.md](plan-phase2-retry-engine.md) | public |
| [draft-1023-service-access-endpoints-plan.md](draft-1023-service-access-endpoints-plan.md) | public |

Historical does not mean harmless: a completed plan can still name things that are current, which
is why these are classified individually rather than as a block.

---

Expand Down
83 changes: 83 additions & 0 deletions .context/decisions/0058-what-moves-to-the-private-side.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# ADR 0058: What moves to the private side, and what deliberately does not

**Status:** accepted
**Date:** 2026-09-13
**Owner:** Seyed Yahya Shirazi

## Context

ADR 0057 decided that admin-only material becomes private at source rather than merely gated at
serving. It did not say what counts as admin-only, and that judgment is about to be applied to
roughly 19,400 committed lines of agent-facing material in this repo alone, and about 23,100 across
the public repos this epic touches (`nemar-cli`, `docs`, `website`, `nemar-observability`). A rule that lives in
one person's head produces a different answer every time it is applied, and the two failure modes
point in opposite directions: collect too little and the gate stays cosmetic, collect too much and
the project stops being open for no security gain.

The evaluation behind this rule is on issue #1339. Its calibration is binding: no credential value
is in any of the public material, nothing needs rotating, this is not an incident, and NEMAR's
public documentation stays public because that is what the project is for. A determined reader
digging through public history will find most of it. The goal is that they should not find an
assembled map on the first click.

## Decision

Move an item to the private side if it does at least one of:

1. **Names a code-execution or credential path concretely enough to act on** -- host, plus path,
plus ref, plus cadence.
2. **Names where a credential or a full copy of the data lives.**
3. **Describes an incomplete security migration**, which is the one case where a roadmap is more
sensitive than the end state.
4. **Is an assembled operational map rather than a single fact.**

**Assembly is the trigger, and it is the part people get wrong.** That the Zarr converter runs on an
SDSC host is worth documenting and stays public. Host alias, plus state path, plus cron window, plus
the deploy mechanism, gathered into one table, is a different artifact than the sum of those facts
published separately.

## What does NOT move, and this half is load-bearing

- **ADRs, architecture, contracts, schemas, and the API and CLI surface.** This is the project's
value and most of what makes contribution possible.
- **Every blast-radius warning**, and this is the one that will look tempting later. The dev
database holds real users, the `nemarDatasets` org is shared with production, `nm000103`-`nm000107`
are live, the version is owned by CI. Those warnings are what prevent the LIKELIER harm, which is
a contributor breaking something by accident rather than an attacker exploiting anything. Moving
them behind the gate trades a small reconnaissance gain for a real increase in accidents. Anyone
tidying them into the private repo later is making the system less safe, not more.
- **Material whose only protection would be obscurity against someone who already holds an admin
key.** Dataset ID bands, the S3 layout, which endpoints cascade: someone with a key runs
`nemar admin --help` and has the same map in a minute. The control there is key hygiene, not
document classification, and collecting these pages buys nothing while costing contributors.

## Consequences

- The expected result is a **small** move. Applied to this repo it is four documents --
`systems-inventory.md`, `validated_workflows.md`, `research-d1-backup-655.md` and
`plan-923-test-staging.md` -- out of the 99 files under `.context/`. Named individually rather
than described, because an earlier draft of this line double-counted one file and referred to a
document that does not exist.
- Classification is recorded as a destination column on `.context/README.md`, which is already the
curated index. Destination only, with no per-file rationale: a list of destinations is not a
target list, and the reasoning belongs here as a general rule rather than beside each file.
- The rule has to be applied by reading, not by grep. A mechanical scan for host, path, cron and
credential shapes is useful triage and it ranked the same two files far above everything else,
but "assembled map" is a judgment a regex cannot make.
- Anything that moves stays in public git history. Accepted: no credential value is involved, and
operational detail ages out on its own as paths and schedules drift. This reasoning does not
extend to a key, and must not be cited for one.
- Applying the rule to a repo is cheap; changing it later is not, because material that has been
made private and then republished was public all along. Prefer leaving an item public when the
rule is genuinely ambiguous, and revisit.

## Alternatives considered

- **Classify by feel, per file, as each phase reaches it.** Rejected: the sorting is the actual work
of this epic, it will be done by more than one person across four repos, and an unwritten rule
produces an inconsistent result that nobody can review.
- **Move everything that mentions infrastructure.** Rejected as over-collection. It would sweep up
the blast-radius warnings and most of the architecture, which is the majority of what makes the
repos useful, in exchange for hiding facts an admin key already grants.
- **Move nothing and rely on the serving gate.** Rejected by ADR 0057: a gate in front of a public
source is cosmetic.
1 change: 1 addition & 0 deletions .context/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ invisible one — which is why it is checked rather than trusted.
- [ADR 0055](0055-a-reconcile-reports-a-disagreement-and-files-nothing.md) - A reconcile reports a disagreement and files nothing, and coverage is looser than ownership
- [ADR 0056](0056-the-admin-docs-are-gated-by-nemars-own-session.md) - The admin docs are gated by NEMAR's own session, handed to the docs host by a one-time code
- [ADR 0057](0057-docs-are-the-canonical-surface-and-gated-content-is-private-at-source.md) - Docs are the canonical surface, and gated content is private at source
- [ADR 0058](0058-what-moves-to-the-private-side.md) - What moves to the private side, and what deliberately does not

## Backfill note (2026-07-31)

Expand Down
Loading
Loading