Skip to content

Release v0.9.12: zarr serving next phase - #1209

Merged
neuromechanist merged 15 commits into
mainfrom
dev
Sep 3, 2026
Merged

neuromechanist merged 15 commits into
mainfrom
dev

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Releases epic #1181 (zarr serving next phase, merged into dev as #1206) to production. This is the dev to main promotion as-is; CI strips the -devN suffix and tags v0.9.12.

The epic tightens the Zarr serving system ahead of the MCP server (#1065, ADR 0025): Range-keyed edge caching with tokened metadata TTLs and index purge (#1199), catalog exposure of has_zarr and the zarr failure summary contract plus zarr-catalog.json (#1201), the staging Zarr pipeline on Hallu against api-test / nemar-dev (#1200), the redirect contract that sends non-browser clients a 302 to bytes and never redirects HEAD (#1202), converter and index format v3 with the biosigio 1.2.7 pin, data-plane fields, provenance attrs and the engine bump to 3 (#1204), the fidelity verification sweep so has_zarr means converted and verified (#1203), events.parquet with converter-computed sample_index (#1205), and the whole-epic review fixes (#1208).

The biosigio side (view levels chunked at constant columns, BrainVision iEEG one-channel collapse, channels.tsv units applied without rescaling, MemoryError re-typed as permanent, stream_to_zarr parity) shipped as biosigio 1.2.6 and 1.2.7 on PyPI. Docs land via nemarOrg/docs#15 and the website via nemarOrg/website#278 (staging).

What changes in production on merge

  • The worker serves index v3 fields, the edge cache rules, and the redirect contract immediately after the deploy.
  • The Hallu cron tracks main, so its next setup() pulls the v3 converter. With the engine bump over the 25-row guard, reconcile logs ENGINE BUMP PENDING ACK and re-queues nothing until /mnt/local/zarr-state/.zarr-engine-bump-ack is touched for exactly one run (ADR 0033). New datasets keep converting normally in the meantime.
  • No new migrations, no wrangler-sccn.toml changes, no new runtime env vars.

Pre-merge checklist (release-safety-playbook)

  • Required checks green on this PR: lint, unit-pure, integration-dev
  • Migrations additive and idempotent: none newly landing on main
  • Prod wrangler config completeness: no diff in backend/wrangler-sccn.toml; no new env.* reads
  • Deploy path self-consistent: shared/ changed; the deploy job runs root bun install before the backend install
  • Env-gating prod-safe: no ENVIRONMENT === "production" patterns in the diff
  • Scoped prod-safety review done: whole-epic six-lens review on Epic: zarr serving next phase #1206, findings fixed on Epic review fixes for #1206 #1208
  • Fresh prod D1 snapshot: hourly backup succeeded at 2026-09-03T04:27Z
  • Rollback plan understood: worker rollback via wrangler; no destructive migration
  • A human is watching the deploy

Post-merge watch

  1. deploy-backend.yml on main deploys twice (pre-strip and stripped tag); both smokes must be green.
  2. api.nemar.org/health reports 0.9.12.
  3. auto-tag.yml tags v0.9.12, npm-publish.yml publishes, sync-dev.yml advances dev to 0.9.13-dev0.
  4. Spot-check /datasets and a dataset page on nemar.org.
  5. Then on Hallu: hallu-zarr.sh --preview-engine-bump, and ack for one run.

nemar-bot and others added 13 commits September 2, 2026 12:59
* Phase 1: edge cache for zarr data plane

Range-keyed cache entries (single bytes-range, synthetic 200 storage
for the 206 the Cache API refuses), a canonical origin+path+v cache
key, tokened Cache-Control (zarr.json/index.json get a day when
?v= is present, index.json's untokened TTL rises 60s -> 3600s),
edge-cached missing-object 404s, and a deps seam (cache/fetch/s3Base)
so backend/test/zarr-data-cache.test.ts can drive the real route
against a real Bun.serve() upstream and an in-memory cache.

Addresses #1178 (items 3, 4) and #1035. Part of epic #1181.

* Address PR #1199 five-reviewer follow-up

Correctness: gate isPublicDataset() before every cache lookup, not
only on miss, so a dataset flipped private stops serving immediately
instead of up to a day later; a ranged 404 lookup now falls back to
the canonical negative-cache entry so a ranged probe storm converges;
a 206 with no Content-Range fails closed (stream through, warn, don't
guess); cached entries get a size cap (8 MiB range / 32 MiB full),
checked from Content-Length before any clone/put; 416 now passes
through as 416 with Content-Range instead of collapsing into 502;
cache.put failures are caught and logged instead of risking an
unhandled rejection under waitUntil; app.onError logs and still
applies CORS instead of a bare 500.

Cache key and TTLs: canonicalCacheUrl now derives the key from the
normalised S3 key (each segment encodeURIComponent'd) instead of the
raw request pathname, so differently percent-encoded spellings of the
same object collapse onto one entry; untokened index.json drops to
300s/3600s (was 3600s/86400s) with a corrected comment on why (the
zone id is unset in prod today, so the purge is a no-op and the TTL
is the only bound); the chunk-purge comment now says zarrPurgeTargets
never targets a chunk URL.

Types: NormalizedRange/CanonicalCacheUrl are branded strings;
cacheControlFor takes an options object; the test InMemoryCache
throws on a 206 (mirroring the real Cache API) with a comment on why
only 206.

Tests: 45 (was 29) in zarr-data-cache.test.ts, covering every item
above plus a v-token round trip, telemetry via a real binding-shaped
collector, four concurrent identical range requests, HEAD with a
Range header, and an upstream that ignores a well-formed single range.

Part of epic #1181
Addresses #1178 (items 3, 4) and #1035

* test: update cacheControlFor route test to the new contract

* fix: absorb synchronous cache.put throws
* Add --test and --print-config to hallu-zarr.sh

* Add config-resolution tests for hallu-zarr.sh --test

* Run hallu-zarr.sh config tests in CI

* Thread TEST_API_URL through --test for nemar CLI

* Document the Zarr test conversion instance

* Add validated recipe for hallu-zarr.sh --test

* Point AGENTS.md Zarr paragraph at test instance

* Harden --test guard rails against review findings

* Add tests for hardened --test guard rails

* Fix guard-rail count in CI comment

* Amend ADR 0029 on hallu-zarr.sh self-deploy

* Fix Zarr test-instance pointer wording

* Correct 3.3/3.4 ground truth and wording

* Add bootstrap-copy gotcha; reflow section 9 prose

* fix: strip DNS root dot in prod host guard

* fix: require bash 4+ for the host guards
* Add has_zarr dataset filter

* Expose zarr fields in catalog responses

* Add zarr-catalog document service

* Add admin route to publish zarr catalog

* Publish zarr catalog on daily cron

* Serve zarr-catalog.json on zarr host

* Add CLI has-zarr flag and publish command

* Test has_zarr filter and catalog zarr fields

* Test the zarr-catalog document service

* Add contract tests for zarr fields

* Add CLI test for has-zarr flag

* Log malformed zarr_data_failures input

* Defend zarr-catalog service invariants

* Remove test-only endpointUrl from S3 type

* Require zarr_index_url only when ready

* Name the zarr-catalog dev-cron allowlist

* Add publish DI seam to admin zarr route

* Clarify has_zarr filter's stable meaning

* Test hardened zarr catalog fields

* Test zarr-catalog service invariants

* Test the zarr-catalog publish route

* Pin publishZarrCatalog cron wiring

* Test the zarr_index_url invariant

* Let fetchZarrCatalogObject take a fetch impl

* Wrap serveCatalog signature (biome)

* Test GET /catalog.json via the phase 1 harness
* feat: redirect zarr bytes for non-browser clients

Non-browser GET requests to a zarr store object (no allowlisted
Origin) now 302 to the public S3 object instead of proxying through
the Worker, never touching D1, the edge cache, or an upstream fetch.
index.json stays proxied/edge-cached/D1-gated, HEAD is never
redirected, and the rate limiter exempts redirect candidates from the
data-ip bucket via the same predicate serve() uses to route.

Closes #1061. Part of epic #1181.

* fix: encode S3 key segments in s3PublicUrl

s3PublicUrl interpolated the decoded key unescaped, so a literal '#'
truncated the redirect Location at the fragment, '?'/space broke the
request line, and a non-Latin-1 character made workerd refuse to send
the header at all (Bun tolerated it, so tests stayed green). It now
shares canonicalCacheUrl's per-segment encodeURIComponent treatment
via a new encodeS3KeyPath() helper, so the fetch target, the cache
key, and the Location all agree.

Exported s3PublicUrl for a direct unit test of its real (non-s3Base)
branch. PR #1202 review item 1.

* fix: anchor redirect path regex and validate dataset id

ZARR_OBJECT_PATH_RE's \/zarr\/?(.*)$ matched "bogus" as rest for
/on000001/zarrbogus, exempting a path that never reaches serve()'s
route from the rate limiter. Anchored on \/zarr(?:\/(.*))? like
ZARR_PATH_RE, requiring end-of-string or a literal slash after zarr.

isRedirectCandidate never validated the dataset id either, so
zz000001 (wrong prefix), nm1234567 (too many digits), and nm999999
(over the 99999 cap) got redirected to a garbage S3 URL and wrote an
invalid id into Analytics Engine, while the same request with a
browser Origin correctly 404'd. The regex now captures the id and
isValidDatasetId gates it; a malformed id falls through to the normal
proxied 404 path instead of a separate D1-free branch.

Also drops the stale "catalog.json will need the same exemption"
comment: catalog.json is a separate route with no <id>/zarr/ segment
and can never match this regex at all.

PR #1202 review items 2, 3, 7 (comment only; item 7's tests follow).

* fix: bytesFromRangeHeader never returns negative bytes

"bytes=100-50" is syntactically a valid single range (parseCacheableRange
only checks digit shape, not end >= start), so it computed -49 and that
landed in the Analytics Engine data point. Real S3 would 416 an inverted
range like any other out-of-range request, so there is never a real
byte count for one; fall back to the same "unknown" 0 as every other
unmeasurable case.

PR #1202 review item 4.

* feat: observe-only rate-limit counting for zarr redirects

Replaces the unconditional per-request console.info the zarr
redirect exemption used to emit with an observe-only mode on
rateLimiter itself (a new RateLimiterOptions.observeOnly). A
redirect candidate now counts toward the SAME IP-keyed data-ip
bucket a proxied request from that client would be enforced
against, but never blocks -- so a later proxied request from a
client that's been hammering the bucket via redirects still 429s
correctly. Logs at most one console.warn per window, the first
time the bucket crosses the point enforcement would have tripped,
with a hash of the IP (not the raw address) rather than a log line
per exempted request.

Updates AGENTS.md's zarr paragraph to match, and corrects a related
inaccuracy: catalog.json is proxied and edge-cached but has no D1
gate of its own (it lists only already-public datasets), unlike
index.json.

PR #1202 review item 5.

* docs: document chunk-redirect in access-metrics detail contract

AccessEvent.detail's zarr contract listed only index/metadata/chunk;
the redirect branch's "chunk-redirect" marker (#1181 phase 6) was
undocumented at the source of truth for the field.

PR #1202 review item 6.

* test: catalog.json is proxied, not a redirect candidate

Route-level companion to the isRedirectCandidate unit tests added
earlier: a bare GET /catalog.json still gets a 200 from S3 via
serveCatalog, never a 302 -- this route has no <id>/zarr/ segment
and never consults isRedirectCandidate at all.

PR #1202 review item 7 (tests).

* test: cover redirect edge cases from review

- /zarrproxy + allowed Origin + store object: proxied bytes with CORS
  (item 8)
- HEAD tests assert the fake upstream actually received method HEAD,
  not just a matching path/range (item 9)
- Private dataset redirect safety: follow the 302 Location for real
  and confirm S3 itself 403s an unauthenticated request under the
  private prefix (item 10)
- 8 concurrent redirect-candidate GETs: all 302, zero upstream, zero
  D1 (item 13)
- Re-verify range-key hit/miss, the full-object size cap, and the
  206-without-Content-Range fail-closed path via a no-Origin
  index.json request, since those mechanics moved to allowed-Origin
  store objects earlier in this PR (item 12)

Items 1-7 and 11 landed in earlier commits on this branch.

PR #1202 review items 8, 9, 10, 12, 13.

* refactor: hash observe-only IPs with hashIp
* feat: zarr fidelity verification sweep service

* feat: admin route for zarr fidelity sweep

* feat: wire fidelity sweep into daily cron

* feat: add has_zarr_verified filter

* feat: add zarr verify fields to contract

* feat: expose verify fields in zarr catalog

* feat: add zarr-fidelity-sweep CLI command

* feat: add --has-zarr-verified CLI flag

* docs: document fidelity sweep in AGENTS.md

* fix: harden zarr fidelity sweep per PR 1203 review

* fix: 502 when every sweep candidate errors

* feat: surface new sweep fields in the CLI

* test: has_zarr_verified filters GET /datasets/search

* docs: update fidelity sweep paragraph for review fixes

* fix: scope budget_exhausted to the sweep-wide budget
* Pin biosigio 1.2.6 and add jsonschema

* Add zarr index v3 and manifest schemas

* Publish zarr index v3 with coverage data

* Requeue done datasets with pending retries

* Pass contract base and api base to the driver

* Count pending zarr recordings in the backend

* Serve the zarr index schema from the API

* Test index v3 consumers and the schema route

* Document index v3 and the engine-bump steps

* Pass the sidecar to both zarr exporters

* chore: bump zarr engine to 3

* ci: install jsonschema for the zarr python tests

* Type the index entries and build failures once

* Keep legacy non-raw stores in the index

* Redact credentials from published failure detail

* Drop non-raw stores loudly and report the count

* Test that refused publishes still call back

* Spare untried recordings from the retry budget

* Close the schemas and publish the layout recipe

* Validate the zarr callback and cross-check coverage

* Cover the clean-run, entry assembly and guard paths

* Document the MCP layout recipe

* Reconcile the non-raw purge wording

* test: pin the CLI default limit; redaction false positives
* Publish events.parquet with sample indices

One columnar file per dataset at <id>/zarr/events.parquet, one row per
(event, channel group), with the level-0 sample index computed by the
converter -- the only party that knows the exact resampling relation
(#1060). index.json gains events_parquet + events_row_count and a layout
entry; n_events/trial_types now come from the same parse as the rows.

Rows are staged per store on disk as recordings finish and written store
by store into a ParquetWriter, so a 25k-store dataset never builds one
table. An incremental run carries an untouched store's rows forward from
the published file exactly as the index carries its entry.

Best-effort like manifest.json: a missing pyarrow or a failed build/upload
warns, leaves the index naming no file, and reports events_upload_failed
on the callback rather than failing a good conversion.

Tested: converter suite (438), incl. a main()-level clean + incremental
run over a real EDF, and mutation-checked on the formula, the ordering,
the flush, the carry-forward and the schema.

* Verify sample indices on real recordings

nm000329 (1000 Hz served at 250 Hz) is #1060's acceptance case, and a
fixture built at the serving rate cannot check it: the error class the
column removes only exists when the rates differ. Downloads the recording
and its events.tsv on demand into a cache outside the repo (skipped, never
failed, when offline; NEMAR_ZARR_SKIP_REAL_DATA opts out), converts it
with the real exporter, and checks three things that do not reuse the
formula: level-0 n_samples == round(n_native * target/native), zero filter
delay (served level 0 correlates with native[::4] peaking at lag 0), and
every onset within one sample of the level-0 grid -- exact except on 13
events that sit exactly half a sample between two, where no rule can do
better. The dataset's own native `sample` column agrees on the same terms.

Adds a synthetic 512 -> 250 case (ratio 125/256), since nm000329's exact
4 cannot tell a formula that assumes integer decimation from one that
does not.

* Add pyarrow to the converter dependencies

The events.parquet writer needs it (#1060), and the schema/ordering/
bounded-write tests are pure Python, so the fast zarr-python-test tier
installs it too -- otherwise those tests do not fail, they skip, and the
converter's "warn and publish no events file" degradation makes a broken
writer look like a dataset with no events. hallu-zarr.sh's fallback spec
is unchanged: the venv installs from requirements.txt.

* Record the events file in the producer manifest

manifest.json now lists the events.parquet a run uploaded with its byte
size and row count, so "the object on S3 is the one this conversion
wrote" is a HEAD away rather than a download. Not carried over from the
prior manifest: a run that published no file must not claim the previous
run's bytes.

* Type the events fields on the zarr callback

events_row_count (nullable) and events_upload_failed were reaching the
backend through passthrough(), so nothing said what they mean or that
null is a real value. Declared, documented, and tested; deliberately not
persisted -- the row count is served by the published index, and a column
would spend the datasets budget (ADR 0034) on a number already available.

* Cache and purge the zarr documents as one set

manifest.json was getting the 24-hour chunk TTL and no purge, so a
re-conversion's manifest could disagree with the fresh index at the edge
for a day; events.parquet would have inherited the same gap. One
ZARR_DATASET_DOCUMENTS list now drives both cacheControlFor's short
untokened TTL and zarrPurgeTargets, so the two cannot cover different
files. Only index.json stays always-proxied: the other two redirect to S3
for non-browser clients like any store object.

* Say why the telemetry vocabulary is narrower

zarrObjectType's comment points at cacheControlFor, which now covers three
documents while this still labels only index.json -- on purpose, since
"index.json is 95% of egress" is a measurement of index.json.

* Update the purge-target tests for the document set

The canonical zarrPurgeTargets tests live in test/, not backend/test/,
and asserted the old index.json-only list. They now assert all three
documents against the shared constant; the backend cache file keeps only
the cross-module claim (every purged document has the short TTL, no chunk
does), which is the half that belongs beside cacheControlFor.

* Drop the events pointer when a purge rewrites the index

The published events.parquet still holds rows for the stores the purge
just removed, so events_row_count would overstate it and events_parquet
would name a file describing stores the index no longer lists. Both go
until the next conversion republishes them, which is the same rule the
converter follows.

* Fix the events step's carry-forward and ordering

Three defects in one code path (they do not build apart), all from
review of #1205:

Carry-forward was defined as "produced no rows this run", so a store
that WAS reconverted and legitimately has no events -- its events.tsv
deleted or emptied -- was treated as untouched and had its stale rows
republished from the prior file forever. On a first --clean run the same
definition reported every event-less store as an unrecoverable
carry-over. `reconverted` (every store this run rebuilt, rows or not) is
now threaded from `record` through to the writer, which never falls back
to prior rows for one.

Ordering: the index is now validated BEFORE anything is uploaded, in the
shape it will take (the events pair filled with the deterministic URL
and a `minimum: 0` placeholder count), so a refused index no longer
leaves a freshly overwritten events.parquet that the live index does not
describe. The one interleaving that can still overwrite -- a document
that passes with the placeholder and fails on the real count -- now
reports events_row_count/events_upload_failed on the failed callback,
which is the only place an operator would learn the object was replaced.

Per-store signals: a store whose events cannot be positioned is named
(`::warning::`) and counted into a new `events_stores_without_rows`
callback field -- either every sample_index is null, or the store has
events and no channel group to attach them to. The decision is a pure
helper because the second case cannot be reached through a real
conversion.

Tested: 470 converter tests, incl. a mixed dataset through main() twice,
the refused-index interleavings, an unclamped past-the-end onset, a
failed events upload, duplicate onsets, and a run with pyarrow absent
from sys.modules. Mutation-checked per fix.

* Log the sibling documents on the zarr callback

The summary line now carries events_row_count, events_upload_failed and
manifest_upload_failed, and an unpublished sibling gets its own warn.
Neither flag has a column and neither fails the run, so without this the
condition -- index.json published while manifest.json or events.parquet
was not -- was visible only in a cron log on one box.

* Write the purged index conditionally

purge_non_raw_stores read index.json, spent minutes deleting objects,
then PUT the rewrite unconditionally -- so a converter run that
published in that window was silently reverted, taking every store it
had just added with it. The live document is now re-read (with its ETag)
immediately before the write, the rewrite is recomputed from it, and the
PUT carries --if-match; a 412 is reported as index_rewrite_conflict for
that dataset and the newer document is left alone. An absent index writes
with --if-none-match, so a first index published meanwhile is safe too.

Both modes now share one planner (plan_live_index_rewrite, renamed from
plan_snapshot_index_rewrite): the snapshot path was already doing this,
the live path had the same exposure on a shorter fuse.

Tested end to end against a real stub aws with md5 ETag semantics: an
uncontested rewrite, a concurrent publish detected and not clobbered, a
store published between the reads surviving, and the absent-index case.

* Log events_stores_without_rows on the zarr-ready callback
* Validate zarr-ready counts as integers

Every number on the zarr-ready callback body is a count, so the schema
primitive carries .int(): a fractional store_count validated cleanly and
was bound into an INTEGER column as a REAL. .catch(undefined) makes it
behave like any other malformed field -- dropped and logged, the rest of
the body still persisted.

Also retypes data_failures as unknown[] to match its validator (nothing
reads an element's fields, only the length), and puts not_attempted_count
and non_raw_dropped on the summary log line: both were validated and then
read by nothing, and neither has a column.

* Note the zarr index trust boundary and one cast

The index.json shapes in s3.ts and zarr-fidelity-sweep.ts are all-unknown
because the document is fetched from the serving bucket, written by a cron
on another host: the coverage equation is the producer's invariant
(check_index_invariant), not something these types assert. Says so, and
says where each file re-derives rather than trusts it.

Also tags zarr-data.ts's rateLimiter context cast with SAFETY: it widens
the static Variables map on the same runtime object, and rateLimiter reads
no Variable at all.

* Correct the dev-D1 cron allowlist rationale

The comment called dev D1 a partial production mirror; the catalog was
purged to the curated fixtures. The allowlist survives that purge for the
reasons AGENTS.md gives: the users table was not purged, the dev worker
holds a real RESEND_API_KEY, and the nemarDatasets org is shared.

* Publish index.json as a conditional write

Two processes write <id>/zarr/index.json -- a converter run and
purge_non_raw_stores.py -- and both read it, work for minutes to hours,
then write back what they merged from that read. The converter's PUT was
unconditional, so whichever finished second silently reverted the other,
both exiting 0.

The PUT now carries --if-match with the ETag read at the top of the run
(--if-none-match '*' when there was no document). A 412 is re-read,
re-merged onto the newer document and retried once; a second 412 abandons
the publish through write_failed_callback so the driver posts a failed
callback rather than leaving D1 at converting forever. An upload failure
now takes that same path instead of escaping as an uncaught RuntimeError.

read_index_with_etag/write_index move into generate_zarr.py and the purge
imports them: the two writers of one object must agree on how the ETag is
read and sent back. The ETag now comes from the PUT's own response rather
than a follow-up head-object, which could observe another writer's version.

The stub aws in test_generate_zarr.py gains the real ETag semantics the
purge test stub already had.

Same file, same pass: PendingEntry.reason becomes the schema's closed
three-value enum (normalized through _pending_reason when read off a
published index, so a bad value cannot fail validation at the END of a
conversion), EventsPublication.file becomes a ManifestFileEntry mirroring
the manifest schema, and a duplicate import is dropped.

* Reconcile the staging cron and two ops claims

systems-inventory and validated_workflows contradicted each other on the
staging nightly cron; both now carry the installed crontab line verbatim,
including the explicit ZARR_DRIVER_REF override and the note that it flips
to dev once the epic merges.

Also corrects the --test guard-rail claim (only API_BASE and TEST_API_URL
are case-folded; the dirs are path-normalized and the bucket/profile are
exact equality) and hallu-zarr.sh's 'no writes' claim for
--preview-engine-bump, which connects and so runs migrate_schema.

* Serve the zarr summary keys D1 already stores

The callback wrote pending/discovered into zarr_data_failures (#1197), the
catalog projection kept only count/detail_ref, and the contract stripped
what it did not declare -- so counts that existed in D1 reached nobody and
every layer's own tests passed.

Projects them, declares them, and adds events_upload_failed and
manifest_upload_failed to the same bounded object: an index published
without its sibling was previously visible only as a console.warn in the
Worker log. Both are written only when true, so a later run that publishes
both clears the claim. No new columns (ADR 0034).

Tested end to end -- real callback route into a real D1, out through the
real detail route, parsed by the shipped contract -- which is the only
shape that fails when one of the three layers is missed.

* Stop erroring datasets starving the fidelity sweep

Candidacy clears only on a verdict, and the batch was ordered by
dataset_id, so ~25 alphabetically early datasets that error every run
(unreachable index, rotated credentials, their own fetch budget spent)
held the whole 25-row quota forever and nothing behind them was ever
swept -- while each run reported errors and looked like progress.

Every attempt now stamps zarr_verify_attempted_at, verdict or not, and the
batch is ordered by it (never-attempted first, then oldest attempt). An
unreachable dataset still records NO verdict and stays a candidate; it
just costs one slot per cycle instead of all of them.

The sweep_stamps JSON paths move into one module the sweep and its three
readers import: json_extract returns NULL for a path that matches nothing,
so a typo read as 'never swept' forever with no error anywhere. A seam
test now runs the real sweep and then both readers' real SQL.

* Fail loudly when an S3 listing fails

_s3_child_prefixes returned [] for both 'no subdirectories' and 'the call
failed'. _rm_recursive is correct either way (it falls back to one
unsharded delete, and keeps that tolerance at the call site), but
list_dataset_ids read the same [] as 'this bucket holds no datasets' and
discover_excluded_stores as 'no stranded stores here' -- a clean empty
result for a run that could not see the bucket.

It now raises, and purge --all with zero datasets exits 2 like the
snapshot-directory check: the bucket holds ~800 prefixes, so an empty
listing is a misconfiguration, not an all-clean verdict.

Also adds the missing purge test for a non-412 write failure (classified
as an error, never as a conflict and never as a rewrite), and fixes the
editor diagnostics on the converter: typed containers where the values
really are TypedDicts, a str head_commit, and two unused bindings.

* Say what a 403 and a redirect row actually mean

The fidelity sweep reported an unfetchable index as 'index.json is
absent'. S3 also answers 403 for a missing key when the caller cannot list
the bucket, so a rotated credential or a changed bucket policy produces
that same message for every candidate at once; the message now names both
readings so an incident is not read as a per-dataset converter bug.

zarr-data.ts's chunk-redirect access rows carry the byte count the Range
header ASKED for, recorded before the 302, so they are intent and not
delivery -- said so at the call site. shared/facets.ts now lists
hasZarrVerified among the bespoke filters it excludes.

Pins the redirect-before-cache ordering: a chunk already primed by an
allowlisted Origin must still 302 for a library request. Nothing else in
that file fails if the order inverts.

* Compile the published zarr schemas in CI

shared/zarr-index.schema.json and zarr-manifest.schema.json are what the
converter validates every document against before upload, and nothing on
the TypeScript side ever compiled them: a broken schema DOCUMENT (a typo'd
keyword, a dangling $ref) failed at the next Hallu run with every TS test
green.

Adds an ajv test that compiles both, validates checked-in v3 fixtures, and
proves the schemas discriminate by mutating one field each. The consumer
fixture in backend/test/zarr-index-v3.test.ts now reads the same file: the
inline one carried v1's source_key and neither layout nor updated_utc, so
every consumer assertion about 'a v3 index' was made against a document
the producer would have refused to publish.

* Run the converter suite CI actually skips

zarr-python-test installs only jsonschema and pyarrow, so five classes --
43 tests covering real-recording conversion, events.parquet publication,
convert_one and the sample-index rate cases -- raised SkipTest on every CI
run and were reported as 'skipped=9' in a green job. patch-duration-test
already installs the real stack, so the full suite runs there now, under
pytest, with a tripwire that fails the job above two skips (the real-data
download and the bash-3.2 test).

Adds the missing shared/** paths filter: a PR touching only shared/** --
the wire contract, the facet vocabulary, the two JSON Schemas -- matched no
filter and ran zero jobs. A tripwire test now asserts every filter is
exported as an output and that shared gates the jobs that compile it.

Also covers the engine-bump ack file in bash: armed for exactly one run,
consumed before reconcile, and the pending-ack notice re-raised as its own
error line.

* State the real sample_index formula

Both the docstring and AGENTS.md said round(onset_s * rate); the code is
math.floor(onset_s * rate + 0.5). They differ at an exact .5 tie, which is
not exotic (any onset landing on a half sample at the serving rate hits
it): Python's round() is banker's rounding and ties to even, this ties up.
A client reimplementing the column from the docs would disagree with the
published value on exactly those rows, which is the failure the column
exists to prevent. Written out verbatim in both places.

* Log provenance_fetch_failed, fix four comments

provenance_fetch_failed was validated and read by nothing, like the two
counts the earlier pass added: it says this wave's stores carry null
doi/license/citation/hed_version because the CATALOG READ failed, not
because the data lacks them, and it has no column. On the summary log line
now, with the test extended.

Comment corrections: the garbled 'on008083 shape' becomes
'pending-with-no-failures shape'; application/schema+json is registered by
RFC 9358, published, not a draft; the purge script's 'PR #98' is qualified
with the repo it was filed in (nemarDatasets/.github, before ADR 0029
repatriated the converter) and a truncated docstring sentence is finished.

The ADR README now says what an 'Amendment YYYY-MM-DD:' block is for -- a
clarification that leaves the verdict standing -- and that anything
changing the verdict supersedes instead.

* Restore the on008083 provenance in a comment

on008083 is the real dataset behind #1197 -- 43 raw recordings, 2 stores,
36 failures, and five in no published list at all -- not a garbled token.
Dropping it cost the comment its provenance and left a shape named after
nothing. s3.ts and zarr-index-v3.test.ts both still cite it by id.

* Type the arrow rows, catch the listing failure

merge_index's failures param is list[FailureEntry]; converted, pending and
prior_pending stay list[dict] with a note, since two of them are inputs
this function does not validate and one is a worker-built store entry.

PriorEventRows._row_group returned object and was then indexed and
filtered, so both were unchecked accesses. Typed with an ArrowTable
Protocol declaring the three members used: pyarrow is an optional lazy
import, so there is no module-scope name to annotate with and a
TYPE_CHECKING import would still depend on it being installed.

purge --all no longer hands an operator a traceback when the listing
fails. list_dataset_ids raising is right; letting it out of main was not.
Same message shape and exit 2 as the empty case, with a test driving the
stub aws failure injection.

Also corrects a comment that described an 'or head' expression where the
code is a ternary.

* Correct two docs the earlier rounds left stale

validated_workflows still called the engine-bump preview read-only with no
writes; hallu-zarr.sh's own comment was corrected in round 1 and this was
not. migrate_schema runs for every subcommand and its ALTER plus
engine_version seed are idempotent, so a preview on a node that has never
run the current schema writes those. Nothing is enqueued, parked or reset.

catalog.ts said the contract declares zarr_data_failures CLOSED; it is
.passthrough() since the B5 fix. The three-edit rule still holds -- a key
this projection never reads is never in the object to begin with -- so the
sentence now says that rather than resting on a strictness that is gone.

This branch was previously deployed

2 inactive deployments
development — ee3e796c Deployed Sep 3, 2026 by neuromechanist via deploy-dev #831
npm — ee3e796c Deployed Sep 3, 2026 by neuromechanist via publish #2208
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant