Skip to content

Phase 7: index v3, provenance, coverage - #1204

Merged
neuromechanist merged 24 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1059-phase7-index-v3
Sep 2, 2026
Merged

neuromechanist merged 24 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1059-phase7-index-v3

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Bundles every producer-side Zarr change that needs a re-conversion into one phase,
so the back catalogue is converted once rather than once per feature.

Index format v3 (shared/zarr-index.schema.json, format_version: 3).
index.json is the mandatory entry point — anonymous in-prefix ListBucket is denied on
the serving bucket — and until now it was undocumented and said nothing about where the
bytes are. It now declares contract_base (the only URL a client may hardcode),
data_base / data_base_kind / s3_uri / s3_region / s3_anonymous (where the bytes
are today, per-dataset so one dataset can be mirrored without rewriting anyone's client),
engine_version, biosigio_version, and the coverage counts. source_commit is now
always a 40-hex SHA: the converter refuses to build an index without one (on008083
published "" while D1 held the real commit).

Coverage accounting (#1197). Infra failures are no longer dropped on the floor "so
they retry next run" — a run that converted anything is marked done, so they never did.
They are published in a new pending[] (infra_failure / memory_budget /
not_attempted) with an attempts count, and zarr_queue re-queues such a dataset after
a backoff (1 h, 6 h, 24 h, then weekly) for at most 5 rounds, after which the recording
becomes a typed retry_exhausted failure with its last error as detail. Typed failures
also gained detail — exception class plus first message line, local paths stripped — so
on008083's 36 identical file_read_error entries become diagnosable from the public index
without SSH access to Hallu.

The invariant, enforced in merge_index (by construction), asserted by
check_index_invariant before upload, and tested on full and partial runs:

discovered_count == store_count + failure_count + pending_count

discovered_count is every raw recording the walker finds after the ADR 0027 exclusions,
computed through the same _buildable_primaries the worklist uses — not a second walk that
could drift. A partial run balances because discovered paths in none of the three lists
become not_attempted pending entries with attempts: 0.

Store provenance (#1064). Every store gains a structured nemar root attribute
(dataset id, DOI, license, citation, source commit, source tree, derived, HED version,
engine version, contract URL), read once per dataset from GET /datasets/<id>.
biosigIO's own attrs are untouched. Prose provenance is useless to the machines that
increasingly read these stores, and a client that has the store now has the attribution.

Index weight (#1178 item 5). Per-store source_key left the index for a sibling
manifest.json (shared/zarr-manifest.schema.json). Nothing on the website read it and it
was 2.3 MB of nm000281's 12.8 MB index, fetched on every dataset-page visit.

The index nonetheless grows per store, and that is accepted. Measured on a synthetic
100-store index against the v1 shape: +49 B/store for a minimal store (no events, no
channels.tsv) and +287 B/store for a typical one (events summary plus units_report),
i.e. source_key's ~90 B is repaid several times over by the view geometry, the coverage
fields and the per-store provenance. That is the trade: the fields exist so a client stops
probing (a zoomed-out read went from 594 requests to 3) and so coverage is checkable at all.
The size levers are the edge cache (phase 1) and the manifest split, not omitting fields
consumers then have to discover by request.

Units. Served samples now carry the unit the recording's BIDS channels.tsv declares
(biosigio#125 converts samples rather than relabelling them), on both conversion paths,
and biosigIO's per-file report is republished as the store entry's units_report. The
sidecar is resolved by BIDS inheritance — the same resolution the channel-count fidelity
gate uses — and passed explicitly as bids_channels to Recording.from_file and
stream_to_zarr, never as biosigIO's "auto". Two reasons, and each alone is sufficient:
the file an exporter is handed is a scratch materialisation, and on the ADR 0028 MaxShield
path it is the filtered copy at work/sss_<basename> where no channels.tsv is adjacent; and
BIDS inheritance means the applicable sidecar may live in an ancestor directory rather than
beside the recording at all. "off" is passed when nothing applies, so no store can ever be
shaped by a sidecar this driver did not choose.

Schema. Draft 2020-12 schemas for both documents, validated by the converter before
upload and served from the Worker at GET /schemas/zarr-index-v3.json (static, cached, no
D1 — the served bytes are the same file the producer enforces). This is the first instance
of #1198's process-report envelope; the liftable fields are engine_version,
source_commit, updated_utc, discovered_count, the store_count / failure_count /
pending_count counts with errors, and the per-record
status / code / reason / detail / attempts.

Consumers. getZarrIndex and aggregateRecordingStats read v1 and v3 through one code
path (v3 is additive for everything they read); pending recordings now count toward
recordingCount and recordingsUnavailable, so on008083 reports 43 rather than 38.
zarr-ready carries pending_count / discovered_count inside the existing bounded
zarr_data_failures summary — {count, detail_ref, pending, discovered} — with no new
datasets column
(ADR 0034 / 0036). The website is untouched: parseZarrIndex reads
dataset_id, format, stores[], failures[], all unchanged.

Engine bump: armed, and staging proves the wave

ZARR_ENGINE_VERSION = "3" (final commit). biosigio 1.2.7 is released and is now the
floor, which is what makes the bump safe: below it stream_to_zarr had no bids_channels
parameter at all, so a dataset's small recordings would have converted under the sidecar's
units and its large ones under the importer's, and a full re-conversion would have baked
that split into the serving copy. 1.2.7 (biosigio#128) gives both exporters the same
parameter. Worth recording because no capability probe saves you from it: 1.2.6's
Recording.from_file accepts bids_channels=<path> and silently treats anything that is
not "auto" as "off" (measured — the units report comes back None), so a driver passing
the path form on 1.2.6 would believe the sidecar was applied and serve importer units. Only
the floor prevents that.

Merging this into the epic branch is safe and is the point. Production tracks main
(ZARR_DRIVER_REF=main), so it does not run this code until the release. The staging
instance on Hallu tracks the epic branch nightly and its queue holds only the 7 curated
xx0999NN exemplars — under ENGINE_REQUEUE_LIMIT (25), so reconcile requeues them
without an ack and the fleet re-converts through engine 3 on its own. That exercises the
whole wave across eeg / ieeg / emg / meg / multi-modal / HED before production sees it.
Verify from the served artifact rather than the queue:

curl -s https://zarr-test.nemar.org/xx099905/zarr/index.json | python3 -m json.tool | head -40
# format_version 3, engine_version "3", biosigio_version "1.2.7",
# contract_base https://zarr-test.nemar.org/xx099905/zarr/,
# discovered_count == store_count + failure_count + pending_count,
# units_report on the entries whose recordings ship a channels.tsv.

Production still needs the two-step ack at release (~800 stale rows is far over the
guard, so nothing re-queues unattended):

  1. Release to main as usual — the pin and the engine constant are already in the tree; do
    not re-bump anything by hand.
  2. Merging to main deploys it (setup() resets the Hallu clone to origin/$DRIVER_REF
    every run). It re-queues nothing yet.
  3. hallu-zarr.sh --preview-engine-bump — read-only, no network, no writes.
  4. touch /mnt/local/zarr-state/.zarr-engine-bump-ack arms exactly one run (the script
    consumes the file), then watch reconcile's engine_requeued=<n>.

Until step 4 every reconcile logs ENGINE BUMP PENDING ACK and the shell re-raises it as
its own ERROR line; new datasets keep converting throughout (ADR 0033).
.context/validated_workflows.md §10 carries all of this with test results and gotchas.

Test plan

  • Converter (cd scripts/zarr && uv run --with-requirements requirements.txt --with pytest pytest -q),
    against biosigio 1.2.7: 664 passed, 2 skipped, 35 subtests. New: TestIndexFormatV3, TestCoverageInvariant,
    TestPendingRetries, TestIndexSchemaSelfCheck, TestRealRecordingV3Fields,
    TestFailureDetail, TestEventsSummary, TestDatasetProvenanceAttrs,
    TestFetchDatasetRow, TestSourceTree.
  • Real data, no mocks. TestRealRecordingV3Fields writes a spec-compliant EDF+ with
    pyedflib (the repo's bids-minimal .edf is a 1 KB stub pyedflib refuses to open) and
    runs the real conversion, then reads back the attrs the index republishes. It covers the
    store-level functions rather than convert_one, which uploads with aws s3 sync and so
    needs S3 credentials — stated here as the brief asks. fetch_dataset_row is tested
    against a real http.server on a real socket. No recorded calls remain: now that 1.2.7
    applies the sidecar on the streaming path, the assertion is an observable property of a
    real streamed store (units_report present) rather than a note about what the exporter
    was called with.
  • Queue: PendingRetryTest (backoff table, round accumulation, exhaustion, reset on a
    clean run, a new version winning over the backoff, the engine guard NOT blocking a
    pending requeue, and the real CLI parser threading --pending-count) and
    PendingColumnMigrationTest (the columns arrive by ALTER TABLE and default to a
    constant 0, so the ~800 pre-existing done rows are not re-queued on first deploy).
  • Backend (cd backend && bun test): 1538 passed, 0 failed. New
    backend/test/zarr-index-v3.test.ts (31 tests) drives the real Hono route against a real
    D1, asserts the v1 fixture still yields exactly its old numbers, and reads raw v1/v3 JSON
    through getZarrIndex over a real socket (only the hostname is redirected, as a DNS
    override would).
  • New in the review pass: TestMainRefusesToPublish and
    TestMainCleanRunAgainstPriorIndexes drive main() end to end over a real git repo and a
    real stub aws executable; TestConvertOneEndToEnd drives convert_one in local mode over
    a real EDF; TestDetailRedaction covers each credential shape; PendingRetryTest covers
    the not_attempted policy; the engine-requeue guard is exercised at its real default (25 vs
    26 stale rows), and --print-config asserts BIOSIGIO_SPEC matches requirements.txt.
  • Mutation-verified four times. Reverting the non-raw drop guard fails
    test_a_non_raw_store_is_dropped_and_counted; removing write_failed_callback from the
    index guard fails test_a_refused_index_writes_a_failed_callback with the exact message
    about D1 sticking at pending; and the two below. All reverted and confirmed clean. Dropping the in-memory sidecar application fails
    test_units_report_is_published_when_channels_tsv_was_applied; dropping
    bids_channels from the stream_to_zarr call fails
    test_a_streamed_recording_with_no_sidecar_beside_it with KeyError: 'units_report'
    while the sibling-adjacent streaming test still passes via biosigIO's "auto" fallback —
    which is exactly why the empty-directory fixture is the load-bearing one. Both reverted
    and confirmed clean.
  • bunx tsc --noEmit and -p backend clean; Biome clean on every file touched; Ruff adds
    no finding beyond new instances of patterns already accepted in these files
    (noqa: N802 on do_GET, noqa: S310 on urlopen, a class-attribute fixture dict).
  • Not run: the root bun test CLI/staging integration tests, which need
    test/.env.test API keys not present in this environment (51 pre-existing failures there,
    none in backend/test).

Review pass (four reviewers)

Correctness: carried-over non-raw stores (derivatives//sourcedata//code/) are now
dropped loudly rather than silently — each is logged with the tree that excluded it and
the run reports non_raw_dropped on the callback, never in the index (they are being deleted
by purge_non_raw_stores.py, not served, so source_tree stays const: "raw"). Both
refuse-to-publish guards now write a status: "failed" callback with an error string
before returning 1, so D1 can never stick at pending with nothing running. A failed catalog
read is recorded as provenance_fetch_failed on every store and on the callback, so null
DOI/license is distinguishable from an unreachable catalog; an unreadable-but-applicable
channels.tsv sets channels_tsv_read_error; a failed manifest PUT sets
manifest_upload_failed. not_attempted pendings are reported separately and re-queued at
the shortest delay without spending a retry round, so a dataset too large to finish in one
run cannot exhaust itself on recordings nothing has tried. The detail/last_error
sanitiser now redacts Authorization/Bearer, X-Amz-*, AKIA/ASIA key ids and
?token=/?key= values — these strings are published on a public bucket and a presigned-URL
error is not a filesystem path.

Schema and MCP readiness: every sub-object is additionalProperties: false (so "no
source_key in the index" is now checkable, with a mutation test proving it), group
requires name, contract_base/data_base use pattern: "^https://", failure/pending
zarr must end .zarr, and both schemas carry a $comment stating the stability policy
(closed at v3; additive-within-v3 ships as v3 with new optional fields; breaking is v4 at a
new path). Dataset-level doi/license/citation/hed_version are hoisted to the top
level and a const layout object publishes the path templates, so an MCP recipe (ADR 0025)
is computable from index.json plus one array-metadata fetch.

Backend: aggregateRecordingStats cross-checks discovered_count against the sum and falls
back to the sum with a console.warn on disagreement; ZarrIndexInfo gains engineVersion;
the webhook body is validated with zod — malformed fields are dropped and logged per field
(so one bad value cannot discard the store count), unknown fields pass through, and a garbage
body is a 400 rather than a 500 on a fire-and-forget POST.

Python: StoreEntry/FailureEntry/PendingEntry TypedDicts and a _failure_entry
constructor mirroring _pending_entry, used at every construction site.

Closes #1059
Closes #1197
Addresses #1064 (store side), #1178 (item 5)
Part of epic #1181

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1204

Version: 0.9.12-PR1204.b9ebbff
Commit: b9ebbff


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 0a25f90 into feature/issue-1181-epic-zarr-serving Sep 2, 2026
12 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-1059-phase7-index-v3 branch September 2, 2026 20:37

This branch was previously deployed

1 inactive deployment
npm — b9ebbff9 Deployed Sep 2, 2026 by neuromechanist via publish #2191
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant