Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
dbe1c7c
Pin biosigio 1.2.6 and add jsonschema
neuromechanist Sep 2, 2026
d22e6c8
Add zarr index v3 and manifest schemas
neuromechanist Sep 2, 2026
1a6b395
Publish zarr index v3 with coverage data
neuromechanist Sep 2, 2026
c4eddfa
Requeue done datasets with pending retries
neuromechanist Sep 2, 2026
8a755fc
Pass contract base and api base to the driver
neuromechanist Sep 2, 2026
ede91be
Count pending zarr recordings in the backend
neuromechanist Sep 2, 2026
00eb798
Serve the zarr index schema from the API
neuromechanist Sep 2, 2026
b0af3ec
Test index v3 consumers and the schema route
neuromechanist Sep 2, 2026
26b556a
Document index v3 and the engine-bump steps
neuromechanist Sep 2, 2026
ef370ad
Pass the sidecar to both zarr exporters
neuromechanist Sep 2, 2026
cfa4c8f
chore: bump zarr engine to 3
neuromechanist Sep 2, 2026
3df05db
ci: install jsonschema for the zarr python tests
neuromechanist Sep 2, 2026
a60a8e1
Type the index entries and build failures once
neuromechanist Sep 2, 2026
333085a
Keep legacy non-raw stores in the index
neuromechanist Sep 2, 2026
cb3d28e
Redact credentials from published failure detail
neuromechanist Sep 2, 2026
8043dcb
Drop non-raw stores loudly and report the count
neuromechanist Sep 2, 2026
974bcab
Test that refused publishes still call back
neuromechanist Sep 2, 2026
0424c10
Spare untried recordings from the retry budget
neuromechanist Sep 2, 2026
7ec8fd1
Close the schemas and publish the layout recipe
neuromechanist Sep 2, 2026
1ca0d8c
Validate the zarr callback and cross-check coverage
neuromechanist Sep 2, 2026
3e47fcc
Cover the clean-run, entry assembly and guard paths
neuromechanist Sep 2, 2026
5b500aa
Document the MCP layout recipe
neuromechanist Sep 2, 2026
1e8d568
Reconcile the non-raw purge wording
neuromechanist Sep 2, 2026
b9ebbff
test: pin the CLI default limit; redaction false positives
neuromechanist Sep 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 153 additions & 0 deletions .context/validated_workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ This document contains workflows that have been **tested and validated** through
6. [IAM Eventual Consistency Fix](#6-iam-eventual-consistency-fix)
7. [GitHub Invitation Auto-Accept](#7-github-invitation-auto-accept)
8. [Commit Authorship for NEMAR Users](#8-commit-authorship-for-nemar-users)
9. [Zarr Test-Instance Bootstrap (`hallu-zarr.sh --test`)](#9-zarr-test-instance-bootstrap-hallu-zarrsh---test)
10. [Zarr Engine-Version Bump (index format v3)](#10-zarr-engine-version-bump-index-format-v3)

---

Expand Down Expand Up @@ -1143,6 +1145,157 @@ curl -s -o /dev/null -w '%{http_code}\n' https://api.nemar.org/datasets/xx099905

---

## 10. Zarr Engine-Version Bump (index format v3)

**Validated:** 2026-09-02 -- `ZARR_ENGINE_VERSION = "3"` IS armed by phase 7;
staging re-converts itself automatically, production still needs the two-step ack.
**Prototype:** nemarOrg/nemar-cli#1059 / #1197 / #1064, epic #1181 phase 7 (ADR 0033)
**Purpose:** Reach the back catalogue with a producer-side change. Index format v3,
structured store provenance, coverage accounting, and the channels.tsv units change
are all producer-side,
so an already-converted dataset only picks them up when `reconcile` re-queues it --
and the only thing that re-queues a `done` row on an unchanged dataset version
is a change to `zarr_queue.py`'s `ZARR_ENGINE_VERSION` (ADR 0033).

### Why the floor is biosigio>=1.2.7 and not 1.2.6

The units half of this bump changes the BYTES: adopting a channels.tsv `units`
value converts the channel's samples into that unit rather than relabelling them
(biosigio#125). On 1.2.6 only the in-memory exporter could do it --
`stream_to_zarr` had no `bids_channels` parameter at all -- so a dataset's small
recordings would have converted under the sidecar's units and its large ones under
the importer's, and a full re-conversion would have baked that split into the
serving copy. 1.2.7 (biosigio#128) gives both exporters the same parameter, which
is what makes one coherent re-conversion possible.

Worth knowing, because no capability probe saves you from it: 1.2.6's
`Recording.from_file` **accepts** `bids_channels=<path>` and silently treats
anything that is not `"auto"` as `"off"` (measured -- the units report comes back
`None`). A driver that passed the path form on 1.2.6 would believe the sidecar had
been applied and serve importer units. Only the floor prevents that.

### Staging proves the wave first, at no cost

Merging this into the epic branch is enough to exercise the whole bump on the
`--test` instance before production sees it:

- Production tracks `main` (`ZARR_DRIVER_REF=main`), so it does not run this code
until the release.
- The staging instance tracks the epic branch nightly and its queue holds only the
7 curated `xx0999NN` exemplars. That is under `ENGINE_REQUEUE_LIMIT` (25), so
`reconcile` requeues them WITHOUT an ack and the fleet re-converts through
engine 3 on its own -- eeg / ieeg / emg / meg / multi-modal / HED coverage,
which is the point of the fleet.
- Verify with the served artifact rather than the queue:

```bash
curl -s https://zarr-test.nemar.org/xx099905/zarr/index.json | python3 -m json.tool | head -40
# Expect: format_version 3, engine_version "3", biosigio_version "1.2.7",
# contract_base https://zarr-test.nemar.org/xx099905/zarr/,
# discovered_count == store_count + failure_count + pending_count,
# and units_report on the store entries whose recordings ship a channels.tsv.
curl -s https://zarr-test.nemar.org/xx099905/zarr/manifest.json | python3 -m json.tool | head -20
```

### What an MCP broker reads (ADR 0025)

Index v3 is meant to make a serving recipe computable from `index.json` plus ONE
array-metadata fetch, with no probing -- the broker is stateless, so anything it
cannot read from the index it must discover by request, and discovery-by-404 is
what #1178 item 2 removed. The index therefore carries, per group,
`n_view_levels` / `view_chunk_columns` / `chunk_samples` / `shard_samples` /
`source_rate_hz`, and at the top level a `layout` object of `const` templates:

```json
"layout": {
"level0": "<zarr>/<group>/0",
"view": "<zarr>/<group>/view/<L>",
"view_levels": "1..n_view_levels from the group attrs",
"scale_offset": "level-0 array attrs scale[] and offset[]; physical = digital * scale + offset"
}
```

Paths are relative to `contract_base`. `<zarr>` is a store's `zarr`, `<group>` a
`groups[].name`, `<L>` a view level in `1..n_view_levels`. The templates are
`const` in the schema, so a client may hardcode them once it has checked
`format_version` -- and changing the layout becomes a schema change rather than a
silent one. Dataset-level `doi` / `license` / `citation` / `hed_version` are
published at the top level too (already fetched once per run for the store
attrs), so a citation tool needs this document and nothing else.

```bash
# The whole recipe, from one fetch:
curl -s https://zarr-test.nemar.org/xx099905/zarr/index.json |
python3 -c 'import json,sys; d=json.load(sys.stdin); s=d["stores"][0]; g=s["groups"][0]; \
print(d["contract_base"] + d["layout"]["level0"].replace("<zarr>", s["zarr"]).replace("<group>", g["name"]))'
```

### The production bump, at release

```bash
# 1. Release to main as usual. The pin and the engine constant are ALREADY in the
# tree -- phase 7 carries them; do NOT re-bump anything by hand.
# scripts/zarr/requirements.txt: biosigio[zarr,meg,mef3,hdf5]>=1.2.7
# scripts/zarr/hallu-zarr.sh: BIOSIGIO_SPEC fallback floor >=1.2.7
# scripts/zarr/zarr_queue.py: ZARR_ENGINE_VERSION = "3"

# 2. Merging to main DEPLOYS it: setup() resets the Hallu clone to
# origin/$DRIVER_REF every run, so the next hourly tick runs the new constant.
# It re-queues nothing yet -- the guard below holds it, because ~800 rows is
# far over ENGINE_REQUEUE_LIMIT.

# 3. Preview the cost before paying it. Read-only, no network, no writes.
ssh hallu '/mnt/local/zarr-state/nemar-cli/scripts/zarr/hallu-zarr.sh --preview-engine-bump'
# Expect: engine-preview: current=3 ... stale=<~800>, i.e. the whole catalogue.

# 4. Arm EXACTLY ONE run. The script consumes the file; re-touch it to re-arm.
ssh hallu 'touch /mnt/local/zarr-state/.zarr-engine-bump-ack'

# 5. Watch the next tick's reconcile line for engine_requeued=<n>, then the drain.
ssh hallu 'tail -f /mnt/local/zarr-state/.nm-zarr.log'
```

### Test Results

| Test | Result | Notes |
|------|--------|-------|
| `reconcile` requeues on a stamp change | PASS | `EngineStampRequeueTest`, unchanged by phase 7 |
| A bump over `--engine-requeue-limit` (25) requeues nothing until acked | PASS | `EngineBumpGuardTest`, unchanged |
| Pending-driven requeue is NOT blocked by an unacked bump | PASS | `PendingRetryTest::test_the_engine_bump_guard_does_not_block_a_pending_requeue` |
| Coverage invariant holds on a full and a partial run | PASS | `TestCoverageInvariant` |
| A carried-over non-raw store is dropped, counted and logged | PASS | `test_a_non_raw_store_is_dropped_and_counted` |
| Index and manifest validate against their schemas before upload | PASS | `TestIndexSchemaSelfCheck` |

### Gotchas and Warnings

1. **Bump only when discovery WIDENS -- or when what a store CONTAINS does.**
Index v3 finds the same recordings; what changed is what a store says
(provenance, view geometry) and, for the units change, what it contains.
That is what makes a re-conversion worth ~800 datasets of Hallu compute.
A narrowing must never bump it (ADR 0033).
2. **Until step 4, every reconcile logs `ENGINE BUMP PENDING ACK`** and requeues
nothing for the stamp; new datasets keep converting normally throughout.
The shell re-raises that notice as its own ERROR line, so it is not something
you find by reading to the end of a reconcile summary.
3. **`pending_count` / `retry_round` arrive by `ALTER TABLE`,** not by
`CREATE TABLE IF NOT EXISTS`, and default to a constant 0 --
deliberately, so the ~800 pre-existing `done` rows do NOT look outstanding on
the first reconcile after deploy and re-queue the archive by accident
(the same trap `engine_version`'s NULL seeding exists to avoid).
4. **The re-conversion rewrites every index and writes a new `manifest.json`.**
`source_key` disappears from index.json as datasets are re-converted, so a
consumer reading it (there is none on nemar.org) would break gradually rather
than all at once.

### Architecture Implications

- One engine bump carries every producer-side change in this phase, which is why
they were bundled: the archive is re-converted once, not once per feature.
- `contract_base` in the index is the only URL clients are told to hardcode, so a
later move of the bytes needs no engine bump at all -- just a re-publish.

---

## Template for New Validated Workflows

```markdown
Expand Down
11 changes: 11 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,17 @@ jobs:
bash -n scripts/zarr/hallu-zarr.sh
shellcheck --severity=warning scripts/zarr/hallu-zarr.sh

# jsonschema ONLY -- deliberately not the whole requirements.txt. This job
# is the fast, pure-Python tier: it has no biosigIO/MNE/zarr and the tests
# that need them self-skip (`patch-duration-test` below is the job that
# installs the real stack). But the index/manifest schema self-check is
# pure Python, and without the validator installed those tests do not
# fail -- they ERROR on the import, and `validate_document`'s deliberate
# "warn and continue" degradation makes the positive cases pass vacuously.
# A schema gate nobody can see failing is not a gate (#1059).
- name: Install jsonschema
run: uv pip install --system "jsonschema>=4.21"

- name: Converter helper tests
run: python scripts/zarr/test_generate_zarr.py

Expand Down
60 changes: 57 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,10 +360,64 @@ Environments and pre-release checks: [`.context/release-safety-playbook.md`](.co
and an embedded classic `.set` loads fully even with `preload=False`.
Discovery and dispatch are raw-only (ADR 0027):
nothing under `derivatives/`, `sourcedata/`, or `code/` becomes a *new* store.
Stores published under those trees before that landed are a separate,
explicitly authorized purge — some are still served until it completes,
so do not read the rule as a description of what is currently in the bucket.
Stores published under those trees before that landed are being deleted by
`scripts/zarr/purge_non_raw_stores.py`; index v3 stops republishing them at the
same time (see the coverage paragraph below), so the index describes what is
served rather than what the bucket still happens to hold mid-purge.
A `--clean` rebuild reconciles rather than wiping first (ADR 0023).
**`index.json` is format v3 and is the mandatory entry point** — anonymous
in-prefix ListBucket is denied, so it is how a client learns what is served.
Schema: `shared/zarr-index.schema.json`, validated by the converter before
upload and served at `GET /schemas/zarr-index-v3.json`.
`contract_base` is the only URL a client may hardcode; `data_base` / `s3_uri`
say where the bytes are today and are per-dataset so one dataset can be
mirrored without rewriting anyone's client.
`source_commit` is always a 40-hex SHA — the converter refuses to publish an
index without one (on008083 shipped `""`).
**Coverage balances by construction:**
`discovered_count == store_count + failure_count + pending_count`.
A carried-over store under `derivatives/`/`sourcedata/`/`code/` is DROPPED from
the index rather than republished — those stores are being deleted by
`purge_non_raw_stores.py`, not served — and each drop is logged with its cause
and counted as `non_raw_dropped` on the callback, never in the index. So
`source_tree` is always `raw`.
`failures[]` is what will not convert without a change to the data or the
converter, and each entry now carries a `detail` (exception class plus first
message line, local paths stripped) so an opaque `file_read_error` is
diagnosable from the public index. `pending[]` is new and is the other half:
recordings with no store that are still expected to convert
(`infra_failure` / `memory_budget` / `not_attempted`), with an `attempts`
count. These used to be omitted "so they retry next run" and then never did,
because a run that converted anything is marked `done`; `zarr_queue` now
re-queues such a dataset on its own. **Only the ATTEMPTED pendings
(`infra_failure`, `memory_budget`) drive the backoff table (1 h, 6 h, 24 h,
then weekly) and the 5-round exhaustion cap, after which the recording becomes
a typed `retry_exhausted` failure. A `not_attempted` pending is re-queued at
the shortest delay (1 h) and does NOT advance `retry_round`** — nothing has
failed for it, so a dataset merely too large to finish in one run must not burn
its rounds on recordings nobody has tried yet. The converter reports the split
as `pending_count` and `not_attempted_count` on the callback.
Per-store `source_key` moved OUT of the index into a sibling `manifest.json`
(nothing on the website read it; it was 18 percent of nm000281's 12.8 MB index).
The index also publishes dataset-level `doi` / `license` / `citation` /
`hed_version` and a `layout` object of `const` path templates, so an MCP recipe
(ADR 0025) is computable from `index.json` plus one array-metadata fetch with no
probing.
Stores also carry a structured `nemar` root attribute — dataset id, DOI,
license, citation, source commit, source tree, derived, HED version, engine
version, contract URL — read once per run from `GET /datasets/<id>`, so a
client that has the store has the attribution.
Served samples carry the unit the recording's `channels.tsv` declares, on
BOTH conversion paths (biosigio>=1.2.7). The sidecar is resolved by BIDS
inheritance — the same resolution the channel-count fidelity gate uses — and
passed EXPLICITLY as `bids_channels`, never biosigIO's `"auto"`: the file an
exporter is handed is a scratch materialisation, and on the MaxShield path it
is the filtered copy at `work/sss_<basename>`, so sibling detection is the
wrong question. `units_report` on a store entry is present exactly when a
sidecar was applied. **Index v3 is what `ZARR_ENGINE_VERSION = "3"` carries
to the back catalogue**, and the units change is why that bump needed the
1.2.7 floor: below it the streaming exporter could not apply the sidecar at
all, so the two paths would have disagreed.
**A widening of discovery reaches the back catalog only through the engine
stamp** (ADR 0033): `reconcile` re-queues on a version change, and an engine
upgrade bumps no version, so `zarr_queue.py`'s `ZARR_ENGINE_VERSION` is what
Expand Down
7 changes: 7 additions & 0 deletions backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import { authWebRoutes } from "./routes/auth-web";
import { catalogIndexResponse, dataRoutes } from "./routes/data";
import { datasetRoutes } from "./routes/datasets";
import { sandboxRoutes } from "./routes/sandbox";
import { schemaRoutes } from "./routes/schemas";
import { userRoutes } from "./routes/users";
import webhooks from "./routes/webhooks";
import { zarrDataRoutes } from "./routes/zarr-data";
Expand Down Expand Up @@ -124,6 +125,7 @@ api.get("/", (c) => {
datasets: "/datasets/*",
sandbox: "/sandbox/*",
webhooks: "/webhooks/*",
schemas: "/schemas/*",
},
});
});
Expand Down Expand Up @@ -153,6 +155,11 @@ api.route("/admin", adminRoutes);
api.route("/datasets", datasetRoutes);
api.route("/sandbox", sandboxRoutes);
api.route("/webhooks", webhooks);
// Public JSON Schema documents (issue #1059). Static, D1-free, cached: the
// bytes are the repo's own shared/*.schema.json, the same files the Zarr
// converter validates against before upload, so the served contract cannot
// drift from the one the producer enforces.
api.route("/schemas", schemaRoutes);
// Path-based mount of the data sub-app so it's reachable on every hostname
// (api.nemar.org, *.workers.dev dev fallback, etc.). The Worker also serves
// the same handlers at the root path when the request hits data.nemar.org;
Expand Down
Loading
Loading