Skip to content

Phase 8: zarr fidelity verification sweep - #1203

Merged
neuromechanist merged 15 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1068-phase8-fidelity-sweep
Sep 2, 2026
Merged

neuromechanist merged 15 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1068-phase8-fidelity-sweep

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

  • Adds the standing Zarr fidelity verification sweep (backend/src/services/zarr-fidelity-sweep.ts, runZarrFidelitySweep): re-derives ground truth per sampled recording from the dataset's own channels.tsv and modality sidecar (*_eeg.json/*_ieeg.json/*_meg.json/*_emg.json), fetched via the public, credential-free raw.githubusercontent.com content host (never the GitHub API/App/PAT, which is what keeps this dev-cron-safe), and compares it against the published index.json.
  • Writes a per-dataset verdict (verified / failed / unverifiable) plus a bounded mismatch-examples list into the sweep_stamps JSON column only (ADR 0034/0035 -- no new column), and re-arms verification on a re-conversion (zarr_source_commit change).
  • Decision revised after the phase 2 type review: has_zarr keeps its existing meaning (converted); the new, separate has_zarr_verified filter (backend + CLI --has-zarr-verified) is "converted AND verified". zarr_verify_status/zarr_verified_at are additive on the catalog/detail contract and on every zarr-catalog.json entry, with a .superRefine() guarding zarr_verified_at against a null zarr_verify_status.
  • New admin endpoint POST /admin/datasets/zarr-fidelity-sweep and CLI nemar admin zarr-fidelity-sweep [--limit N], both sharing the one sweep implementation with the daily cron (added to DEV_CRON_ALLOWLIST -- it only reads S3 + the public content host and writes this env's own D1).
  • On a failed verdict, writes an audit_log row (zarr_fidelity_failed) and includes the dataset in the sweep's response/cron log line for the observability dashboard; #1068 is not closed (the re-measurement after the engine bump remains).

Verdict rules

  • channel_count_mismatch: total store channels (summed across a store's groups, matching generate_zarr.py's store_total_channels) fall short of channels.tsv's data-row count.
  • duration_mismatch: the store's longest group duration_s disagrees with the sidecar's RecordingDuration by more than 1 second.
  • rate_mismatch: a group's served rate disagrees with min(SamplingFrequency, MODALITY_RATES[modality]) (EEG/MEG 250 Hz, IEEG/EMG 1000 Hz).
  • Sample: every store when a dataset has <= 40, else 40 spread evenly by path order (sampleEvenly) plus every store with a n_channels === 1 group. Verdict is unverifiable when no ground truth was reachable for any sampled store, failed when any mismatch was found, else verified.

Test plan

  • backend/test/zarr-fidelity-sweep.test.ts: pure-function unit tests (bidsSidecarCandidates, zarrFidelityStoreChannelTotal/Duration, zarrFidelitySelectSample, pushZarrFidelityExample) plus real-engine tests (freshDb/realD1 + a real local Bun.serve() answering both the signed S3 index GET and the raw-content sidecar GET) covering verified / each mismatch code / unverifiable / v1-v3 index tolerance / the 20-entry-4KB example cap / re-conversion re-arming / an absent index / an invalid source_commit / the limit clamp. Mutation-proved (flipped the channel-count comparison; the two dependent tests failed as expected, then reverted).
  • backend/test/zarr-fidelity-sweep-route.test.ts: real admin route tests (auth 401/403, a thrown dependency becomes 500, and a real success path against the same local receiver with the stamp verified in D1).
  • backend/test/cron-sweep-wiring.test.ts: the sweep is registered all-envs, wired outside the prod-only block, in DEV_CRON_ALLOWLIST, wrapped in ctx.waitUntil, and its chain has a .catch.
  • backend/test/catalog-has-zarr.test.ts: has_zarr=1 is unchanged (still every converted row regardless of verify status); has_zarr_verified=1 returns only verified rows; both fields present on list and detail rows.
  • test/contract-schemas.test.ts: the two new fields parse; the zarr_verified_at-without-zarr_verify_status pair is rejected on both catalogItemSchema and datasetDetailSchema.
  • backend/test/zarr-catalog.test.ts: the candidate SQL projects the two fields from sweep_stamps; buildZarrCatalog carries them per entry without filtering the population.
  • test/zarr-fidelity-sweep-cli.test.ts and test/has-zarr-cli-flag.test.ts: real subprocess CLI invocations (bun run src/index.ts admin zarr-fidelity-sweep, ... dataset list/search --has-zarr-verified) against a local capture server, covering auth, request shape, output, and exit codes.
  • Full backend suite (cd backend && bun test): 1495 pass, 0 fail.
  • Every root test file touched, run individually: all pass.
  • Biome and tsc --noEmit clean on every touched/new file.

Addresses #1068 (verification sweep and has_zarr semantics)
Part of epic #1181

Docs pointer (for nemarOrg/docs)

  • has_zarr is unchanged; document has_zarr_verified as the strict "converted AND verified" filter, on both the REST API (GET /datasets?has_zarr_verified=1) and the CLI (--has-zarr-verified).
  • Document zarr_verify_status (verified/failed/unverifiable/null) and zarr_verified_at on the dataset catalog/detail response and on zarr-catalog.json entries -- null until the daily sweep (or POST /admin/datasets/zarr-fidelity-sweep) reaches a freshly-converted dataset.
  • Document nemar admin zarr-fidelity-sweep [--limit N].

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1203

Version: 0.9.12-PR1203.54cc805
Commit: 54cc805


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 360c2c7 into feature/issue-1181-epic-zarr-serving Sep 2, 2026
12 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-1068-phase8-fidelity-sweep branch September 2, 2026 20:07

This branch was previously deployed

1 inactive deployment
npm — 54cc805c Deployed Sep 2, 2026 by neuromechanist via publish #2187
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant