Repository navigation
Conversation
* Pin read_window output with a golden test * Move sharding to shared/zarr * Move blosc decoder to shared/zarr/codecs * Move isV3Index to shared/zarr * Point path comments at shared/zarr * Make golden capture runs fail loudly * Type isV3Index with ZarrIndexAny * Rewrap comments that name moved paths
* Rewrap ragged comment in get-events * Add Fetcher and JsonCache to shared/zarr * Move shard footer loading to shared/zarr * Move array metadata loading to shared/zarr * Move level-0 read loop to shared/zarr * Move value assembly to shared/zarr * Check shared/zarr read against zarr-python * Point moved-function comments at shared/zarr * Pin read_window error-stage texts * Test shared/zarr cache contracts * Soften shared/zarr throw guarantees * Align shared/zarr loader docs
* Tighten footer cache overwrite check * Add engine protocol schema and zod mirror * Check protocol schema against zod mirror * Add frame codec and cross-language fixtures * Publish engine protocol schema as v0 * Record read core and protocol ADR * Allow null id on unreadable requests * Open responses, keep requests strict * Add scale, offset and payload limit * Specify window and range semantics * Harden frame decoding and its tests * Compile protocol schema strictly in test * Fix schema route cache comment * Align the engine protocol with neuroschema Squash of the neuroschema alignment fix PR (a2a47e3..d07a787), reviewed in five rounds. Fork CI cannot run on this PR (its base is a feature branch); local gates: targeted root and backend tests, typecheck, Biome, byte-identical fixtures, and a full unit-pure run on d07a787 compared with base 785c172, its new failures rerun in isolation (clean by the pre-registered rule).
Sync the Zarr epic with upstream dev
Sync the Zarr epic with upstream dev
|
Note from the portable compute program (owner decision Q8, option A, time-boxed): the branch |
Review of the reader epic at head 98d273bThis is a review, not a change request. Short version. The read core, the engine and its client are careful work. 1. What keeps the Tests run red: three tests, all test-sideThe run on this head failed in 1a. test("every window, at every alignment, fits incompressible chunks and its footer", () => {
const failures: string[] = [];
// ... the same loops, with the two expects replaced by:
// if (ceiling < footer) failures.push(`footer ${where}`);
// if (ceiling < worst) failures.push(`start ${start} ${where} (shard ${shardSamples})`);
expect(failures).toEqual([]);
});1b and 1c. import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
/** A frame written to a file in this test's temporary directory, for a child to read: a large
* frame cannot travel in the `-e` argument, which Linux caps at 128 KiB. */
const frameFile = (header: Record<string, unknown>, payload?: Uint8Array): string => {
const file = join(tmp, "reply.bin");
writeFileSync(file, encodeFrame(header, payload));
return JSON.stringify(file);
};Then, at line 449, build 2. The client's frame reader copies a large reply once per chunk
3. The merge into
|
The CLI can now read a time window of a public dataset's Zarr recording itself, through a local engine that shares one read core with the MCP server. Decisions are recorded in ADR 0120. CI on this PR can't go green without secrets, since it comes from a fork: see #2142.
What's in it
The work was done in five phases on my fork, each reviewed on its own PR and squash-merged into this branch, with
devmerged in twice along the way (the two merge commits).shared/zarr/. The MCP server's sharding, blosc/zstd decoder and v3 index check move out ofbackend/src/mcp/andbackend/src/services/, with a golden test pinningread_window's output first.shared/zarr/: range fetching, shard footers, array metadata, window planning and channel-major assembly. The MCP tools now call it;read_window's golden result and request log are unchanged byte for byte, apart from the server version stamp.shared/contract/zarr-engine-protocol.ts,shared/zarr-engine-protocol.schema.json, frames inshared/zarr/protocol-frame.ts), served atGET /schemas/zarr-engine-protocol-v0.json. v0 is unstable and may change in place; requests are open, per theshared/contractrule.src/lib/zarr-engine/): a stdio process that opens a public dataset's store and answers window requests, with range-size and body caps and a host rule.nemar zarr read <dataset> <recording> --start --duration, with a summary,--out(raw little-endian, channel-major) or--json;nemar engine statusandnemar engine cache clear;nemar doctor(exit code unchanged);zarr-testforapi-test).Also: ADR 0120 (new, added to the index), an amendment to ADR 0050 (the blosc decoder's new path, decision unchanged), a CHANGELOG entry under
## Unreleased, and one README line.Production impact
GET /schemas/zarr-engine-protocol-v0.json.fzstd0.1.1, the zstd decoder the backend already uses.package.jsonversion untouched.Tests
Bun.servestand-ins on loopback, no mocks. Each new test was shown to fail when the line it guards was changed.contract-schemas,adr-index,changelog-format, the engine and CLI suites, the backend MCP golden test,mcp-smoke.shunder workerd, and the Python frame fixtures regenerated byte-identically.bun test(macOS 15.1, Bun 1.4.2, no credentials, isolated config dir; 100 minutes): 1,026 files, 26,187 pass, 758 skip, 615 fail. Each of the 47 failing files was then run alone on this branch and ondev04b9d0b:identifier-screen-ci,pr-review-calibrate,pr-review-hostile,s3-presence-classification,neurobagel-node-deploy.unit); environment gaps on this machine (cli.test.tsneeds Deno, two*.integrationfiles need credentials,completion-argv-guardexpects the update check this run turned off); and three I haven't diagnosed (data-host-clone,git-annex-remote-nemar,update-check);pr-review-evidence);release-version-flag's help-tree test, becausezarrandenginewere registered insrc/index.tsbut not in the test's command tree. Fixed in the last commit; the file passes 30/30.mcp-smoke.shpasses.nm000103: a repeated read in a fresh process sent zero range requests; physical values equal digital × scale + offset exactly over 64,500 values;xx099901(MEG, 340 channels): read through the derivedzarr-testhost, and a repeat read sent zero range requests.Review
Every commit was reviewed before it merged into this branch: per-phase reviews (up to three reviewers on the larger phases, with each fix commit reviewed again until clean), then one integration-scoped review of the whole diff against
dev. That review found no Critical or Important issues and confirmed the MCP server and the engine share one core; its eight Moderate and Minor points were fixed and reviewed.CI on this PR
Jobs that need secrets (
api-test,integration-dev, the docs checkout) will fail here at authentication, as they do on my fork, where the results matched the base commit test by test. Details and options in #2142. The unit tier and the other jobs run normally.Known limits
Docs
User docs live in nemarOrg/docs. A draft page for
nemar zarr read,nemar engineand the settings is below; I can open it there once this is reviewed.Draft docs page
Reading Zarr recordings from the command line
nemar zarr readreads a time window of one recording of a public dataset straight from its Zarr store. Nothing is downloaded but the chunks the window needs, and they are cached on disk, so reading the same window again sends no data request (the dataset's index and records are read again each time the engine starts, which is how a re-conversion is noticed). The dataset must be public and have a Zarr conversion (a dataset awaiting re-conversion answers "not ready").nemar zarr read
<recording>is the recording's BIDS path (or its zarr path). Channels are numbered from 0.--start <seconds>--duration <seconds>--channels 0,1,5--units physical|digitalphysical(default): float64, digital x scale + offset.digital: the stored int16.--representation <id>--out <file>--json--outpath) as JSON.Without
--outor--jsonit prints a summary: dataset, recording, representation, sampling rate, channels, samples, dtype, filled ranges, and the first values of up to four channels. A filled range is a stretch the store does not hold; it reads as each channel's offset in physical units (0 with--units digital) and is always reported.Loading a file written with
--out(the summary prints these lines too):Errors exit with status 1 and a hint: not found (check the id and the path), not ready (awaiting re-conversion), out of range (check
--start,--duration,--channels), too large (shorten the window or name fewer channels), upstream error (the data host failed; try again).nemar engine status
Starts the local Zarr engine and checks that it answers, then shows the engine's version and protocol, the largest window it sends, the Zarr host and data plane in use, and the disk cache (directory, size, entries, limit).
--jsonprints the same as JSON.The Zarr host follows your account's API host (
zarr.nemar.orgforapi.nemar.org,zarr-test.nemar.orgforapi-test.nemar.org; any other API host uses production).NEMAR_ZARR_BASEoverrides it.nemar engine cache clear
Removes the disk cache's entries and reports the bytes freed (
--jsonavailable). The cache holds public data only, at<config dir>/zarr-cache/public/v1/, 2 GiB by default. The cache directory, including one named byNEMAR_ZARR_DISK_CACHE_DIR, is made private (mode 0700), andnemar engine cache clearremoves only files the cache wrote and refuses a directory the cache did not create. A dataset that is made private later leaves its already cached public bytes until you clear the cache.nemar doctor
nemar doctorhas a "Zarr engine" section: the engine starts and answers within 10 seconds, and the cache directory is writable. It does not change doctor's exit code.Environment
NEMAR_ZARR_BASENEMAR_ZARR_DATA_PLANE/datamount (default: your API host +/data)NEMAR_ZARR_DISK_CACHE_DIRNEMAR_ZARR_DISK_CACHE_BYTES0turns it off)NEMAR_ZARR_MAX_PAYLOAD_BYTESNEMAR_ZARR_RANGE_CACHE_BYTES🤖 Generated with Claude Code