Repository navigation
Phase 7: index v3, provenance, coverage - #1204
Merged
neuromechanist merged 24 commits intoSep 2, 2026
Merged
neuromechanist merged 24 commits into
neuromechanist merged 24 commits into
Conversation
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1204Version: 🤖 Updated on each push. |
5 tasks
neuromechanist
force-pushed
the
feature/issue-1059-phase7-index-v3
branch
from
September 2, 2026 20:12
f454998 to
1e8d568
Compare
neuromechanist
merged commit Sep 2, 2026
0a25f90
into
feature/issue-1181-epic-zarr-serving
12 checks passed
This was referenced Sep 2, 2026
Closed
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bundles every producer-side Zarr change that needs a re-conversion into one phase,
so the back catalogue is converted once rather than once per feature.
Index format v3 (
shared/zarr-index.schema.json,format_version: 3).index.jsonis the mandatory entry point — anonymous in-prefix ListBucket is denied onthe serving bucket — and until now it was undocumented and said nothing about where the
bytes are. It now declares
contract_base(the only URL a client may hardcode),data_base/data_base_kind/s3_uri/s3_region/s3_anonymous(where the bytesare today, per-dataset so one dataset can be mirrored without rewriting anyone's client),
engine_version,biosigio_version, and the coverage counts.source_commitis nowalways a 40-hex SHA: the converter refuses to build an index without one (on008083
published
""while D1 held the real commit).Coverage accounting (#1197). Infra failures are no longer dropped on the floor "so
they retry next run" — a run that converted anything is marked
done, so they never did.They are published in a new
pending[](infra_failure/memory_budget/not_attempted) with anattemptscount, andzarr_queuere-queues such a dataset aftera backoff (1 h, 6 h, 24 h, then weekly) for at most 5 rounds, after which the recording
becomes a typed
retry_exhaustedfailure with its last error asdetail. Typed failuresalso gained
detail— exception class plus first message line, local paths stripped — soon008083's 36 identical
file_read_errorentries become diagnosable from the public indexwithout SSH access to Hallu.
The invariant, enforced in
merge_index(by construction), asserted bycheck_index_invariantbefore upload, and tested on full and partial runs:discovered_countis every raw recording the walker finds after the ADR 0027 exclusions,computed through the same
_buildable_primariesthe worklist uses — not a second walk thatcould drift. A partial run balances because discovered paths in none of the three lists
become
not_attemptedpending entries withattempts: 0.Store provenance (#1064). Every store gains a structured
nemarroot attribute(dataset id, DOI, license, citation, source commit, source tree, derived, HED version,
engine version, contract URL), read once per dataset from
GET /datasets/<id>.biosigIO's own attrs are untouched. Prose provenance is useless to the machines that
increasingly read these stores, and a client that has the store now has the attribution.
Index weight (#1178 item 5). Per-store
source_keyleft the index for a siblingmanifest.json(shared/zarr-manifest.schema.json). Nothing on the website read it and itwas 2.3 MB of nm000281's 12.8 MB index, fetched on every dataset-page visit.
The index nonetheless grows per store, and that is accepted. Measured on a synthetic
100-store index against the v1 shape: +49 B/store for a minimal store (no events, no
channels.tsv) and +287 B/store for a typical one (events summary plus
units_report),i.e.
source_key's ~90 B is repaid several times over by the view geometry, the coveragefields and the per-store provenance. That is the trade: the fields exist so a client stops
probing (a zoomed-out read went from 594 requests to 3) and so coverage is checkable at all.
The size levers are the edge cache (phase 1) and the manifest split, not omitting fields
consumers then have to discover by request.
Units. Served samples now carry the unit the recording's BIDS
channels.tsvdeclares(biosigio#125 converts samples rather than relabelling them), on both conversion paths,
and biosigIO's per-file report is republished as the store entry's
units_report. Thesidecar is resolved by BIDS inheritance — the same resolution the channel-count fidelity
gate uses — and passed explicitly as
bids_channelstoRecording.from_fileandstream_to_zarr, never as biosigIO's"auto". Two reasons, and each alone is sufficient:the file an exporter is handed is a scratch materialisation, and on the ADR 0028 MaxShield
path it is the filtered copy at
work/sss_<basename>where no channels.tsv is adjacent; andBIDS inheritance means the applicable sidecar may live in an ancestor directory rather than
beside the recording at all.
"off"is passed when nothing applies, so no store can ever beshaped by a sidecar this driver did not choose.
Schema. Draft 2020-12 schemas for both documents, validated by the converter before
upload and served from the Worker at
GET /schemas/zarr-index-v3.json(static, cached, noD1 — the served bytes are the same file the producer enforces). This is the first instance
of #1198's process-report envelope; the liftable fields are
engine_version,source_commit,updated_utc,discovered_count, thestore_count/failure_count/pending_countcounts witherrors, and the per-recordstatus/code/reason/detail/attempts.Consumers.
getZarrIndexandaggregateRecordingStatsread v1 and v3 through one codepath (v3 is additive for everything they read); pending recordings now count toward
recordingCountandrecordingsUnavailable, so on008083 reports 43 rather than 38.zarr-readycarriespending_count/discovered_countinside the existing boundedzarr_data_failuressummary —{count, detail_ref, pending, discovered}— with no newdatasetscolumn (ADR 0034 / 0036). The website is untouched:parseZarrIndexreadsdataset_id,format,stores[],failures[], all unchanged.Engine bump: armed, and staging proves the wave
ZARR_ENGINE_VERSION = "3"(final commit). biosigio 1.2.7 is released and is now thefloor, which is what makes the bump safe: below it
stream_to_zarrhad nobids_channelsparameter at all, so a dataset's small recordings would have converted under the sidecar's
units and its large ones under the importer's, and a full re-conversion would have baked
that split into the serving copy. 1.2.7 (biosigio#128) gives both exporters the same
parameter. Worth recording because no capability probe saves you from it: 1.2.6's
Recording.from_fileacceptsbids_channels=<path>and silently treats anything that isnot
"auto"as"off"(measured — the units report comes backNone), so a driver passingthe path form on 1.2.6 would believe the sidecar was applied and serve importer units. Only
the floor prevents that.
Merging this into the epic branch is safe and is the point. Production tracks
main(
ZARR_DRIVER_REF=main), so it does not run this code until the release. The staginginstance on Hallu tracks the epic branch nightly and its queue holds only the 7 curated
xx0999NNexemplars — underENGINE_REQUEUE_LIMIT(25), soreconcilerequeues themwithout an ack and the fleet re-converts through engine 3 on its own. That exercises the
whole wave across eeg / ieeg / emg / meg / multi-modal / HED before production sees it.
Verify from the served artifact rather than the queue:
Production still needs the two-step ack at release (~800 stale rows is far over the
guard, so nothing re-queues unattended):
not re-bump anything by hand.
setup()resets the Hallu clone toorigin/$DRIVER_REFevery run). It re-queues nothing yet.
hallu-zarr.sh --preview-engine-bump— read-only, no network, no writes.touch /mnt/local/zarr-state/.zarr-engine-bump-ackarms exactly one run (the scriptconsumes the file), then watch
reconcile'sengine_requeued=<n>.Until step 4 every reconcile logs
ENGINE BUMP PENDING ACKand the shell re-raises it asits own ERROR line; new datasets keep converting throughout (ADR 0033).
.context/validated_workflows.md§10 carries all of this with test results and gotchas.Test plan
cd scripts/zarr && uv run --with-requirements requirements.txt --with pytest pytest -q),against biosigio 1.2.7: 664 passed, 2 skipped, 35 subtests. New:
TestIndexFormatV3,TestCoverageInvariant,TestPendingRetries,TestIndexSchemaSelfCheck,TestRealRecordingV3Fields,TestFailureDetail,TestEventsSummary,TestDatasetProvenanceAttrs,TestFetchDatasetRow,TestSourceTree.TestRealRecordingV3Fieldswrites a spec-compliant EDF+ withpyedflib (the repo's
bids-minimal.edfis a 1 KB stub pyedflib refuses to open) andruns the real conversion, then reads back the attrs the index republishes. It covers the
store-level functions rather than
convert_one, which uploads withaws s3 syncand soneeds S3 credentials — stated here as the brief asks.
fetch_dataset_rowis testedagainst a real
http.serveron a real socket. No recorded calls remain: now that 1.2.7applies the sidecar on the streaming path, the assertion is an observable property of a
real streamed store (
units_reportpresent) rather than a note about what the exporterwas called with.
PendingRetryTest(backoff table, round accumulation, exhaustion, reset on aclean run, a new version winning over the backoff, the engine guard NOT blocking a
pending requeue, and the real CLI parser threading
--pending-count) andPendingColumnMigrationTest(the columns arrive byALTER TABLEand default to aconstant 0, so the ~800 pre-existing
donerows are not re-queued on first deploy).cd backend && bun test): 1538 passed, 0 failed. Newbackend/test/zarr-index-v3.test.ts(31 tests) drives the real Hono route against a realD1, asserts the v1 fixture still yields exactly its old numbers, and reads raw v1/v3 JSON
through
getZarrIndexover a real socket (only the hostname is redirected, as a DNSoverride would).
TestMainRefusesToPublishandTestMainCleanRunAgainstPriorIndexesdrivemain()end to end over a real git repo and areal stub
awsexecutable;TestConvertOneEndToEnddrivesconvert_onein local mode overa real EDF;
TestDetailRedactioncovers each credential shape;PendingRetryTestcoversthe
not_attemptedpolicy; the engine-requeue guard is exercised at its real default (25 vs26 stale rows), and
--print-configassertsBIOSIGIO_SPECmatchesrequirements.txt.test_a_non_raw_store_is_dropped_and_counted; removingwrite_failed_callbackfrom theindex guard fails
test_a_refused_index_writes_a_failed_callbackwith the exact messageabout D1 sticking at
pending; and the two below. All reverted and confirmed clean. Dropping the in-memory sidecar application failstest_units_report_is_published_when_channels_tsv_was_applied; droppingbids_channelsfrom thestream_to_zarrcall failstest_a_streamed_recording_with_no_sidecar_beside_itwithKeyError: 'units_report'while the sibling-adjacent streaming test still passes via biosigIO's
"auto"fallback —which is exactly why the empty-directory fixture is the load-bearing one. Both reverted
and confirmed clean.
bunx tsc --noEmitand-p backendclean; Biome clean on every file touched; Ruff addsno finding beyond new instances of patterns already accepted in these files
(
noqa: N802ondo_GET,noqa: S310onurlopen, a class-attribute fixture dict).bun testCLI/staging integration tests, which needtest/.env.testAPI keys not present in this environment (51 pre-existing failures there,none in
backend/test).Review pass (four reviewers)
Correctness: carried-over non-raw stores (
derivatives//sourcedata//code/) are nowdropped loudly rather than silently — each is logged with the tree that excluded it and
the run reports
non_raw_droppedon the callback, never in the index (they are being deletedby
purge_non_raw_stores.py, not served, sosource_treestaysconst: "raw"). Bothrefuse-to-publish guards now write a
status: "failed"callback with anerrorstringbefore returning 1, so D1 can never stick at
pendingwith nothing running. A failed catalogread is recorded as
provenance_fetch_failedon every store and on the callback, so nullDOI/license is distinguishable from an unreachable catalog; an unreadable-but-applicable
channels.tsv sets
channels_tsv_read_error; a failed manifest PUT setsmanifest_upload_failed.not_attemptedpendings are reported separately and re-queued atthe shortest delay without spending a retry round, so a dataset too large to finish in one
run cannot exhaust itself on recordings nothing has tried. The
detail/last_errorsanitiser now redacts
Authorization/Bearer,X-Amz-*,AKIA/ASIAkey ids and?token=/?key=values — these strings are published on a public bucket and a presigned-URLerror is not a filesystem path.
Schema and MCP readiness: every sub-object is
additionalProperties: false(so "nosource_keyin the index" is now checkable, with a mutation test proving it),grouprequires
name,contract_base/data_baseusepattern: "^https://", failure/pendingzarrmust end.zarr, and both schemas carry a$commentstating the stability policy(closed at v3; additive-within-v3 ships as v3 with new optional fields; breaking is v4 at a
new path). Dataset-level
doi/license/citation/hed_versionare hoisted to the toplevel and a
constlayoutobject publishes the path templates, so an MCP recipe (ADR 0025)is computable from
index.jsonplus one array-metadata fetch.Backend:
aggregateRecordingStatscross-checksdiscovered_countagainst the sum and fallsback to the sum with a
console.warnon disagreement;ZarrIndexInfogainsengineVersion;the webhook body is validated with zod — malformed fields are dropped and logged per field
(so one bad value cannot discard the store count), unknown fields pass through, and a garbage
body is a 400 rather than a 500 on a fire-and-forget POST.
Python:
StoreEntry/FailureEntry/PendingEntryTypedDicts and a_failure_entryconstructor mirroring
_pending_entry, used at every construction site.Closes #1059
Closes #1197
Addresses #1064 (store side), #1178 (item 5)
Part of epic #1181