Skip to content

Add a local Zarr reader: shared read core, engine and nemar zarr read - #2144

Open
marordyan wants to merge 14 commits into
nemarOrg:devfrom
marordyan:feature/issue-1-epic-zarr-reader-core
Open

marordyan wants to merge 14 commits into
nemarOrg:devfrom
marordyan:feature/issue-1-epic-zarr-reader-core

Conversation

@marordyan

@marordyan marordyan commented Nov 13, 2026 •

Copy link
Copy Markdown

The CLI can now read a time window of a public dataset's Zarr recording itself, through a local engine that shares one read core with the MCP server. Decisions are recorded in ADR 0120. CI on this PR can't go green without secrets, since it comes from a fork: see #2142.

What's in it

The work was done in five phases on my fork, each reviewed on its own PR and squash-merged into this branch, with dev merged in twice along the way (the two merge commits).

  1. The read core moves to shared/zarr/. The MCP server's sharding, blosc/zstd decoder and v3 index check move out of backend/src/mcp/ and backend/src/services/, with a golden test pinning read_window's output first.
  2. Core I/O in shared/zarr/: range fetching, shard footers, array metadata, window planning and channel-major assembly. The MCP tools now call it; read_window's golden result and request log are unchanged byte for byte, apart from the server version stamp.
  3. A versioned protocol for the local engine (shared/contract/zarr-engine-protocol.ts, shared/zarr-engine-protocol.schema.json, frames in shared/zarr/protocol-frame.ts), served at GET /schemas/zarr-engine-protocol-v0.json. v0 is unstable and may change in place; requests are open, per the shared/contract rule.
  4. The engine (src/lib/zarr-engine/): a stdio process that opens a public dataset's store and answers window requests, with range-size and body caps and a host rule.
  5. The CLI:
    • nemar zarr read <dataset> <recording> --start --duration, with a summary, --out (raw little-endian, channel-major) or --json;
    • a public disk cache under the config dir (2 GiB by default, keyed by the store's identity, so a re-converted store is fetched again);
    • nemar engine status and nemar engine cache clear;
    • a "Zarr engine" section in nemar doctor (exit code unchanged);
    • the Zarr host follows the account's API host (zarr-test for api-test).

Also: ADR 0120 (new, added to the index), an amendment to ADR 0050 (the blosc decoder's new path, decision unchanged), a CHANGELOG entry under ## Unreleased, and one README line.

Production impact

  • No migration, no Worker variable or secret, no wrangler change.
  • The MCP tools answer as before (golden test above); the other MCP tools changed only imports and comments.
  • One new public route: GET /schemas/zarr-engine-protocol-v0.json.
  • One new CLI runtime dependency: fzstd 0.1.1, the zstd decoder the backend already uses.
  • package.json version untouched.

Tests

  • New tests use real files, real child processes and Bun.serve stand-ins on loopback, no mocks. Each new test was shown to fail when the line it guards was changed.
  • Targeted gates at each phase and at the final head: typecheck (root and backend), lint, Biome on touched files, contract-schemas, adr-index, changelog-format, the engine and CLI suites, the backend MCP golden test, mcp-smoke.sh under workerd, and the Python frame fixtures regenerated byte-identically.
  • Full local bun test (macOS 15.1, Bun 1.4.2, no credentials, isolated config dir; 100 minutes): 1,026 files, 26,187 pass, 758 skip, 615 fail. Each of the 47 failing files was then run alone on this branch and on dev 04b9d0b:
    • 35 fail identically on both, with the same pass/fail counts: OIDC tests (21 backend files, 2 root files); five files that already failed before this work (identifier-screen-ci, pr-review-calibrate, pr-review-hostile, s3-presence-classification, neurobagel-node-deploy.unit); environment gaps on this machine (cli.test.ts needs Deno, two *.integration files need credentials, completion-argv-guard expects the update check this run turned off); and three I haven't diagnosed (data-host-clone, git-annex-remote-nemar, update-check);
    • 11 fail only in the full run and pass alone on both (tests sharing one process, and the flaky pr-review-evidence);
    • 1 failed on this branch alone: release-version-flag's help-tree test, because zarr and engine were registered in src/index.ts but not in the test's command tree. Fixed in the last commit; the file passes 30/30.
  • Final head (98d273b): typecheck, lint and Biome on the changed files clean; targeted root tests 794 pass, backend 342 pass, 0 fail; the MCP golden fixtures unchanged; mcp-smoke.sh passes.
  • Manual, read-only smoke:
    • production, public nm000103: a repeated read in a fresh process sent zero range requests; physical values equal digital × scale + offset exactly over 64,500 values;
    • staging xx099901 (MEG, 340 channels): read through the derived zarr-test host, and a repeat read sent zero range requests.

Review

Every commit was reviewed before it merged into this branch: per-phase reviews (up to three reviewers on the larger phases, with each fix commit reviewed again until clean), then one integration-scoped review of the whole diff against dev. That review found no Critical or Important issues and confirmed the MCP server and the engine share one core; its eight Moderate and Minor points were fixed and reviewed.

CI on this PR

Jobs that need secrets (api-test, integration-dev, the docs checkout) will fail here at authentication, as they do on my fork, where the results matched the base commit test by test. Details and options in #2142. The unit tier and the other jobs run normally.

Known limits

  • Public datasets only; private data needs its own cache rules, left to a later decision in ADR 0120.
  • A dataset made private later keeps its cached public bytes on a machine until cleared or evicted.
  • Protocol v0 may still change; only the CLI uses it. Python and MATLAB clients, a recording listing in the CLI, and events are not part of this PR.

Docs

User docs live in nemarOrg/docs. A draft page for nemar zarr read, nemar engine and the settings is below; I can open it there once this is reviewed.

Draft docs page

Reading Zarr recordings from the command line

nemar zarr read reads a time window of one recording of a public dataset straight from its Zarr store. Nothing is downloaded but the chunks the window needs, and they are cached on disk, so reading the same window again sends no data request (the dataset's index and records are read again each time the engine starts, which is how a re-conversion is noticed). The dataset must be public and have a Zarr conversion (a dataset awaiting re-conversion answers "not ready").

nemar zarr read

nemar zarr read <dataset> <recording> --start <seconds> --duration <seconds> [options]

<recording> is the recording's BIDS path (or its zarr path). Channels are numbered from 0.

Option Meaning
--start <seconds> Start of the window from the recording's start. Required.
--duration <seconds> Length of the window. Required.
--channels 0,1,5 Channels to read, in the order given. Default: all.
--units physical|digital physical (default): float64, digital x scale + offset. digital: the stored int16.
--representation <id> The Zarr group to read. Default: the first.
--out <file> Write the window as raw little-endian, channel-major values (one row per channel).
--json Print the window's description (header, representation, --out path) as JSON.

Without --out or --json it prints a summary: dataset, recording, representation, sampling rate, channels, samples, dtype, filled ranges, and the first values of up to four channels. A filled range is a stretch the store does not hold; it reads as each channel's offset in physical units (0 with --units digital) and is always reported.

Loading a file written with --out (the summary prints these lines too):

Python  x = numpy.fromfile("window.bin", "<f8").reshape(nchans, n)     # "<i2" for --units digital
MATLAB  f = fopen('window.bin', 'r', 'l'); x = fread(f, [n nchans], 'float64=>float64')'; fclose(f);

Errors exit with status 1 and a hint: not found (check the id and the path), not ready (awaiting re-conversion), out of range (check --start, --duration, --channels), too large (shorten the window or name fewer channels), upstream error (the data host failed; try again).

nemar engine status

Starts the local Zarr engine and checks that it answers, then shows the engine's version and protocol, the largest window it sends, the Zarr host and data plane in use, and the disk cache (directory, size, entries, limit). --json prints the same as JSON.

The Zarr host follows your account's API host (zarr.nemar.org for api.nemar.org, zarr-test.nemar.org for api-test.nemar.org; any other API host uses production). NEMAR_ZARR_BASE overrides it.

nemar engine cache clear

Removes the disk cache's entries and reports the bytes freed (--json available). The cache holds public data only, at <config dir>/zarr-cache/public/v1/, 2 GiB by default. The cache directory, including one named by NEMAR_ZARR_DISK_CACHE_DIR, is made private (mode 0700), and nemar engine cache clear removes only files the cache wrote and refuses a directory the cache did not create. A dataset that is made private later leaves its already cached public bytes until you clear the cache.

nemar doctor

nemar doctor has a "Zarr engine" section: the engine starts and answers within 10 seconds, and the cache directory is writable. It does not change doctor's exit code.

Environment

Variable Meaning
NEMAR_ZARR_BASE Zarr host (https, or http to 127.0.0.1 or [::1])
NEMAR_ZARR_DATA_PLANE Data plane's /data mount (default: your API host + /data)
NEMAR_ZARR_DISK_CACHE_DIR Cache directory
NEMAR_ZARR_DISK_CACHE_BYTES Cache limit in bytes (default 2 GiB; 0 turns it off)
NEMAR_ZARR_MAX_PAYLOAD_BYTES Largest window the engine sends (default 256 MiB)
NEMAR_ZARR_RANGE_CACHE_BYTES In-memory range cache (default 256 MiB)

🤖 Generated with Claude Code

* Pin read_window output with a golden test

* Move sharding to shared/zarr

* Move blosc decoder to shared/zarr/codecs

* Move isV3Index to shared/zarr

* Point path comments at shared/zarr

* Make golden capture runs fail loudly

* Type isV3Index with ZarrIndexAny

* Rewrap comments that name moved paths
* Rewrap ragged comment in get-events

* Add Fetcher and JsonCache to shared/zarr

* Move shard footer loading to shared/zarr

* Move array metadata loading to shared/zarr

* Move level-0 read loop to shared/zarr

* Move value assembly to shared/zarr

* Check shared/zarr read against zarr-python

* Point moved-function comments at shared/zarr

* Pin read_window error-stage texts

* Test shared/zarr cache contracts

* Soften shared/zarr throw guarantees

* Align shared/zarr loader docs
* Tighten footer cache overwrite check

* Add engine protocol schema and zod mirror

* Check protocol schema against zod mirror

* Add frame codec and cross-language fixtures

* Publish engine protocol schema as v0

* Record read core and protocol ADR

* Allow null id on unreadable requests

* Open responses, keep requests strict

* Add scale, offset and payload limit

* Specify window and range semantics

* Harden frame decoding and its tests

* Compile protocol schema strictly in test

* Fix schema route cache comment

* Align the engine protocol with neuroschema

Squash of the neuroschema alignment fix PR (a2a47e3..d07a787), reviewed in five rounds. Fork CI cannot run on this PR (its base is a feature branch); local gates: targeted root and backend tests, typecheck, Biome, byte-identical fixtures, and a full unit-pure run on d07a787 compared with base 785c172, its new failures rerun in isolation (clean by the pre-registered rule).
Sync the Zarr epic with upstream dev
Sync the Zarr epic with upstream dev
@marordyan
marordyan marked this pull request as ready for review November 14, 2026 07:30
@neuromechanist

Copy link
Copy Markdown
Contributor

Note from the portable compute epic (#2145): whichever of #2144 and the portable compute epic lands second deletes 120 from TOLERATED_GAPS in test/adr-index.unit.test.ts and keeps both rows of the ADR index.

@neuromechanist

Copy link
Copy Markdown
Contributor

Note from the portable compute program (owner decision Q8, option A, time-boxed): the branch feature/issue-2144-epic-reader points at this PR's head (98d273b), with every commit unchanged and nothing rewritten, so the organization's Tests workflow runs with its secrets on these commits (that workflow only runs on main, dev and feature/issue-*-epic-*). This PR's own checks and its base are unchanged. A read-only review of the branch and its CI result will follow here.

@neuromechanist

Copy link
Copy Markdown
Contributor

Review of the reader epic at head 98d273b

This is a review, not a change request.
Your commits are unchanged: the organization's copy (feature/issue-2144-epic-reader) points at the same head so the Tests workflow can run with the organization's secrets.
Everything below was checked against that head, and the fixes proposed here were tried in a throwaway copy, never on your branch.

Short version. The read core, the engine and its client are careful work.
I found no code path that can send a credential anywhere, the reader's physical values reproduce the derivative digest vectors bit for bit, and the tests follow the no-mocks rule (stand-in servers and real children, no replaced logic).
Two things should change before the merge into dev: three test fixes so CI goes green (item 1), and one performance fix in the client's frame reader (item 2).
The rest can follow in later pull requests.

1. What keeps the Tests run red: three tests, all test-side

The run on this head failed in unit-pure shard 6 and integration-dev shard 6, in three tests this pull request adds; every other job passed.
None of the three hides a production problem: the engine is started with a three-word command line and a filtered environment, never with a large argument, and rangeCeiling is a pure function whose test is merely slow.

1a. test/zarr-engine-window.test.ts:30, "every window, at every alignment, fits incompressible chunks and its footer".
It makes about 27 million expect calls and hit Bun's 5-second default (7.1 s on the runner, about 2.5 s on a laptop).
Collect the failures in plain JavaScript and assert once; the test then takes under 0.2 s, and it still fails if the chunk overhead constant is lowered below the blosc2 overhead (I checked by lowering it to 16):

  test("every window, at every alignment, fits incompressible chunks and its footer", () => {
    const failures: string[] = [];
    // ... the same loops, with the two expects replaced by:
    //   if (ceiling < footer) failures.push(`footer ${where}`);
    //   if (ceiling < worst) failures.push(`start ${start} ${where} (shard ${shardSamples})`);
    expect(failures).toEqual([]);
  });

1b and 1c. test/zarr-engine-client.test.ts:449 ("a large reply written just before exiting survives a parent that reads a second late") and :580 ("a reply below the cap is accepted even when it waited unread").
Both fail on Linux with E2BIG: argument list too long, posix_spawn.
Each puts a 320,000-byte payload, base64-encoded (about 430 KB), into the child's -e script, and Linux allows one argument at most 128 KiB; macOS has no such limit, which is why they pass locally.
Keep the reply's size (it must exceed the pipe buffer for the test to mean anything) and pass it through a file in the test's temporary directory instead:

import { mkdtempSync, rmSync, writeFileSync } from "node:fs";

/** A frame written to a file in this test's temporary directory, for a child to read: a large
 *  frame cannot travel in the `-e` argument, which Linux caps at 128 KiB. */
const frameFile = (header: Record<string, unknown>, payload?: Uint8Array): string => {
  const file = join(tmp, "reply.bin");
  writeFileSync(file, encodeFrame(header, payload));
  return JSON.stringify(file);
};

Then, at line 449, build const file = frameFile({...}, new Uint8Array(n * 8).fill(7)) and start the child with
`process.stdout.write(require("fs").readFileSync(${file}), () => process.exit(0))`;
at line 580, build the frame the same way and use
`process.stdout.write(require("fs").readFileSync(${file})); setInterval(() => {}, 1000)`
with maxQueuedBytes: 400_000 as now.
Both tests pass with this change, and the small frames elsewhere in the file can stay as they are.

2. The client's frame reader copies a large reply once per chunk

FrameReader.push (src/lib/zarr-engine/frame-stream.ts:88) joins the whole buffer with each new chunk until the frame is complete (line 93, and line 157 returns until then).
For a window reply that is quadratic in its size: the child's stdout arrives in 64 to 256 KiB pieces, so a large payload is copied thousands of times.
Measured through EngineClient on a laptop: 128 MiB takes 2.5 s, 256 MiB (the default max_payload_bytes) 6.8 s, and 512 MiB 60 s.
At the 1 GiB that NEMAR_ZARR_MAX_PAYLOAD_BYTES allows, the extrapolated time passes the client's 120-second call timeout, so a window the engine is configured to send cannot be received.
The fix is local to the reader: once next() knows a frame's end (frameEnd), record it, keep later chunks in a list without joining, and join once when the buffered total reaches it (and in end(), join whatever is pending before the truncation check).
I tried exactly that in a throwaway copy: 512 MiB then took 0.16 s, and test/zarr-engine-frame-stream.test.ts, test/zarr-engine-frame.test.ts, test/zarr-engine-client.test.ts and test/zarr-engine-runtime.test.ts all passed (183 tests).
A test that reads a reply of a few tens of MiB in small chunks under a time bound would keep it from coming back.

3. The merge into dev

The branch is about 50 commits behind dev.
A trial merge conflicts in two files only, and both resolve by keeping both sides:

  • .context/decisions/README.md: keep the 0120 line, placed before dev's 0121 to 0123.
  • CHANGELOG.md: keep both sets of "Unreleased" bullets; your "### Changed" section lands before "### Known limits", and dev has no other "### Changed" under "Unreleased", so no heading is duplicated.

One line must change in the merge commit itself: test/adr-index.unit.test.ts on dev tolerates the gap at 0120 (TOLERATED_GAPS = new Set([120]), with the comment line above it saying to delete the entry when the ADR lands), and the test fails once 0120 exists while the entry is still there.
Delete 120 from that set (it becomes empty) and the comment line, in the merge commit.
With that, the trial merge passed bun run typecheck, bun run lint, bun test test/adr-index.unit.test.ts, and the targeted zarr, MCP, help-tree and release-flag tests (1166 pass, 0 fail across 41 files).
package.json merges on its own; your only change there is the fzstd line, and the version is untouched.

4. Security: the public and private boundary

  • No credential can leave the engine. Every engine read goes through createPublicFetcher (src/lib/zarr-engine/http.ts:169), which refuses Authorization, Cookie and Proxy-Authorization in any header form, sends credentials: "omit", follows at most three redirects by hand with each target re-checked, and refuses an https to http step.
    The engine child is not given NEMAR_API_KEY or any other secret (the allowlisted environment and its test), and nothing in the engine or in src/commands/zarr.ts and src/commands/engine.ts attaches a key.
    So there is no path that sends a credential to a host the user did not name, across a redirect, or to a presigned or CDN host.
    The Worker side hands the core its plain fetch, as before.
  • Wording, for the record: the Fetcher type in shared/zarr/interfaces.ts:7 refuses nothing; the refusal lives in the public fetcher. The reader ADR proposed in pull request 2181 (ADR 0124) words it that way, and a future authorization-aware fetcher has to carry its own origin check and must not follow a redirect with a credential attached.
  • Index-supplied paths and bases are handled well: the dataset id is checked before any URL is built, zarr paths and group names that could escape the data base are refused, and a loopback http base in an index is accepted only under a loopback zarr base.
  • Suggestions, none blocking:
    • The body caps, the range ceiling and both range caches read init?.headers?.Range with that exact spelling (http.ts:300, :348, :464, disk-cache.ts:533). The core always sends Range, but a caller that wrote range would skip the ceiling and the caps while the header is still sent. Reading it through new Headers(init?.headers).get("range") closes that.
    • A redirect may go to any https origin, including a private-network address. With no credential that is low risk, but refusing a cross-origin redirect (the data host does not need one today) would make the fetcher's reach exactly the configured bases and the index's data_base.
    • The engine's fetches carry no timeout of their own; the CLI client gives up after 120 s, but another client of the protocol would wait as long as the connection does. An AbortSignal.timeout per request in the public fetcher would bound it.
    • decodeBloscZstdInt16 (shared/zarr/codecs/blosc.ts:143 and :160) allocates header.nbytes and lets fzstd size its own output before the decoded length is checked against the geometry. Passing the expected chunk size in, refusing a different nbytes before allocating, and giving fzstd a preallocated output would bound memory against a corrupt or hostile chunk. The code moved unchanged from the backend, so this can be its own small change.

5. Fixture dataset ids

No change is needed in this pull request.
nm099500 to nm099504 were already used by the MCP tests on dev before this branch (the fixture generator and backend/test/mcp-read-window.test.ts); this branch adds two files derived from that same fixture and a few more stand-in ids (nm099505, nm099506, nm099510 to nm099514), all served only by test servers on 127.0.0.1 and never created as datasets.
They do sit in the band the allocator will one day reach rather than in the reserved fixture band (nm099900 to nm099999), so moving the whole family is worth a separate follow-up on dev; please do not move them to nm099995 in the meantime, because that is the next standing fixture id, reserved for a dataset that will really exist.
The sample records copied from nm000103 and the help examples that name it only read public data, which is fine: that dataset is live and is never written to.

6. The conformance suite (ADR 0124)

Much of this pull request can seed the suite every reader is held to:

  • the synthetic sharded store and its zarr-python reference (backend/test/fixtures/mcp/nm099500-*, scripts/zarr/zarr_python_reference.py): the stock-reader guarantee as a test case;
  • the read-window golden (nm099500-read-window-golden.json): sample rule, channel order and filled ranges;
  • the engine frame fixtures with their independent Python codec (test/fixtures/engine-frames/, scripts/zarr/engine_frame_fixtures.py), for any reader that adds the engine transport.

I also fed the four store content vectors of shared/contract/derivatives-vectors.json through assembleChannelMajor with physical units and hashed the result under the contract's header: all four match, including "two roundings, never fused".
So the reader computes physical values exactly as storeContentSha256 defines them; a test that does this would be a good first suite case.
What is still missing for a reader to produce the store content digest itself:

  • channel labels and units: protocol v0 defers channel names (the schema says so at the scale/offset description), and the digest header needs per-channel labels and units with its all-or-none and shared-or-none rules;
  • which field the rate comes from (the index group's rate today), pinned so every reader hashes the same number;
  • the missing-shard case: a shard object that answers 404 is an upstream_error here, while zarr-python reads an absent shard as fill values; the suite should decide which answer is right and hold every reader to it.

7. Smaller points

  • FORCE_COLOR. With FORCE_COLOR set in the shell, 14 tests in test/zarr-cli.test.ts, test/zarr-engine-cli.test.ts, test/zarr-doctor-cli.test.ts and test/zarr-engine-client.test.ts fail, because Bun warns on the child's stderr when both FORCE_COLOR and NO_COLOR are set and the tests expect it empty. CI does not set it, so this is local only. The engine never writes color, so dropping FORCE_COLOR and NO_COLOR from its allowlist (src/lib/zarr-engine/client.ts:61), and dropping FORCE_COLOR from the environment runNemar builds (test/helpers/zarr-engine-standin.ts:139), removes it; with both changes the four files pass with FORCE_COLOR=1 (87 tests).
  • The disk cache and an existing directory. When NEMAR_ZARR_DISK_CACHE_DIR names a directory that already exists and holds other files, the first write sets it to mode 0700 (disk-cache.ts:299) and uses it without a marker. clear already refuses such a directory; refusing to write into it, or at least not changing its mode, would match.
  • fzstd is pinned exactly at 0.1.1 with the same integrity hash as the backend's, has no dependencies, and is bundled into dist/ by the build; nothing to change.

This branch was previously deployed

1 inactive deployment
npm — 98d273bd Deployed Nov 14, 2026 by neuromechanist via publish #4447
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants