Repository navigation
Release: NEMAR MCP server (epic #1065) - #1334
Merged
Merged
Conversation
* chore: bump backend hono to ^4.11.4 Prerequisite for the MCP SDK's @modelcontextprotocol/hono peer (^4.11.4); previously pinned ^4.6.0. typecheck and the backend suite are green at this version. * fix: z.record requires an explicit key schema The single-argument z.record(valueSchema) form is invalid under zod 4 (z.record(keySchema, valueSchema) is required); the two-argument form is valid under zod 3 too, so this is a safe fix regardless of which major version the repo ends up on. Found while gating the phase 1 MCP contracts (issue #1293) against a zod 4 bump. * test: add live zarr v3 index and catalog fixtures Captured with curl -sL -A "nemar-cli/mcp-phase1" (issue #1293): - zarr-index-nm000329-slice.json: first 3 stores of the live index, keeping events_parquet and top-level doi/license/citation. - zarr-index-on003392-meg-sss-slice.json: the one live MEG store carrying an ADR 0028 sss object. - zarr-catalog-slice.json: 3 entries from zarr.nemar.org/catalog.json. - zarr-array-level0.zarr.json / zarr-array-view1.zarr.json: the level-0 and view/1 array metadata documents for nm000329's eeg_250hz group, matching the epic's verified geometry. * feat: zarr index v3 and MCP server contracts Issue #1293, phase 1 of epic #1065. shared/contract/zarr-index.ts is a .passthrough() zod reader mirroring shared/zarr-index.schema.json (the producer's closed contract stays the JSON Schema; this is a client-side lower bound on it, matching dataset.ts's convention). shared/contract/mcp.ts is the MCP wire vocabulary ADR 0049 fixes: the provenance envelope every tool response carries, buildReadRecipe() (recipe-first read_window, per the plan), and one input/output schema pair per tool (search_datasets, describe_dataset, list_recordings, get_events, render_overview, read_window), including the taste cap (60s x 64 channels) read_window's schema enforces. Both contract-*.unit.test.ts files exercise the public schema/function surface against real fixtures (on008083, and the two live slices), including three deliberate mutation tests (dropped source_commit, format_version 1, non-hex commit) proving the zod reader rejects what the Ajv-compiled JSON Schema also rejects. * feat: MCP transport and decode-path spike Issue #1293 step 5. Answers the two things the phase 1 plan left open: does @modelcontextprotocol/server@2.0.0 + hono@2.0.0 serve both protocol eras under workerd (wrangler dev), and which blosc/ zstd decode path (decision 7) actually runs there. Both eras of the transport work: server/discover, tools/list, tools/call in the 2026-07-28 envelope, the legacy initialize handshake, a legacy tools/call with no envelope, the -32020 rejection (disagreeing Mcp-Name, missing Mcp-Method), and GET/ DELETE 405 all behave as documented -- see smoke.sh. numcodecs' WASM Blosc codec does not run under workerd (dynamic WebAssembly.instantiate() on a runtime-fetched buffer is disallowed by the embedder); the pure-JS decoder in decode.ts (blosc2 header parse + fzstd + unshuffle) works and matches a Python-derived ground truth exactly. Path (b) is the chosen decode primitive. fixtures/chunk.bin is one real inner chunk of nm000329's level-0 array, captured via two range requests against the shard's trailing index; README.md documents the byte-level derivation and the full measurement set (bundle size, wall time). Replaced by phase 2 (#1294) and deleted in that PR. * docs: MCP server design doc Issue #1293. Purpose and constraints, the 2026-07-28 protocol revision and dual-era behavior (with the exact SEP-2243 header/ envelope mechanics verified against a real wrangler dev run, not just the spec summary), host and routing (phase 2 scope), the six- tool surface with cost class and cache behavior per tool, the envelope and read-recipe field tables, compute-minimization rules and the cache key scheme, the analytics event shape, the pinned dependency table with the spike's verdicts, the client compatibility table, and open items for phases 2-5. ADR 0050 is omitted from this PR per decision 11: origin/dev does not yet carry ADR 0049 (PR #1292 is still open). The decision text lives in this document instead; phase 5 files the ADR once #1049 has landed. * docs: cache the rendered PNG, reject over-cap taste The design doc drifted from plan decision 4 (cache the rendered overview PNG keyed by width, not the decoded level) and from its own schema (an over-cap taste is rejected by the input schema, not downgraded to a recipe). The zarrita snippet now uses the documented FetchStore form; zarrita's open.v3 takes a store, not a URL string. * fix: guard the spike decoder and assert HTTP status in smoke.sh decodePathB now refuses a short buffer, a non-int16 typesize, a zero blocksize (blosc2 marks a single block with blocksize == nbytes, never 0), a cbytes that disagrees with the input, offsets or lengths past the end of the chunk, and an odd decoded byte length that Int16Array would silently floor. Offsets and lengths are read unsigned. smoke.sh checks the HTTP status on every call, keeps bodies in a mktemp dir it removes on exit, and kills workerd along with wrangler. Smoke run: 10/10 PASS. * fix: contract review findings for the MCP wire schemas - a catalog row's null doi/license is authoritative over the index's stale hoisted copy; the index is consulted only without a row - fillTemplate fills layout templates in one pass and throws on a template that lacks a placeholder the caller needs - buildReadRecipe takes Zarr's own array metadata (data_type), with a zarrArrayMetadataSchema reader for the served arrays' zarr.json - a taste requires channels: the schema cannot know n_channels and a live MEG store has 320 - sss is present exactly when derived is true, enforced on the store reader and the envelope; sss models the converter's known keys - ranges enforce end >= start; flagToBoolean is the one 0|1|null to boolean conversion; min_participants dropped (no server-side filter) - tests: 26 -> 51, covering every tool input default and cap, both read_window output branches, the catalog document, the real zarr.json fixtures, and each fallback branch of the envelope * docs: align the design doc with the reviewed contracts search_datasets loses min_participants and names the two catalog routes it wraps; read_window's taste requires channels; the envelope table states the catalog-over-index rule; the recipe table splits dtype from codecs; section 12 records how phase 2 registers tools across the zod 3 / zod 4 split (zod4 npm alias plus a JSON Schema drift test).
* mcp: dependencies and bindings for the phase 2 host (#1294) Adds @modelcontextprotocol/server (pinned 2.0.0), the zod4 npm alias (npm:zod@^4.2.0, resolved by Bun without touching the repo zod pin), and fzstd (needed by the decoder promotion in the next commit). New ANALYTICS_MCP and MCP_HOSTNAME bindings in Bindings and wrangler-sccn.toml (prod + env.dev), matching the DATA_HOSTNAME/ ZARR_HOSTNAME/ANALYTICS_LLM pattern. * mcp: add the mcp host-route fork and workers.dev path mount (#1294) New HostRoute arm "mcp" in resolveHostRoute, alongside data/zarr/api, keyed off MCP_HOSTNAME (default mcp.nemar.org). index.ts forks to the mcp sub-app on that host and forwards the api host's exact /mcp path to it too, for the workers.dev/dev fallback -- a single-path forward, not a full sub-tree mount, so only the transport endpoint (not the descriptor at "/") is reachable that way. Note: backend/src/routes/mcp.ts (mcpRoutes) lands in a later commit; this commit's index.ts import is completed by that one. * mcp: promote the blosc/zstd decoder out of the phase 1 spike (#1294) Moves the spike's decodePathB (backend/spike/mcp-transport/src/decode.ts, deleted in an earlier commit) to backend/src/services/blosc-decode.ts as decodeBloscZstdInt16, dropping path (a) (numcodecs' WASM Blosc codec, which the spike proved does not run under workerd) and its import entirely. Fixtures moved to backend/test/fixtures/blosc/. Not yet wired into any tool or route -- read_window's taste mode (phase 4) is its first caller. * mcp: add composeCitation and describe_dataset's additive fields (#1294) composeCitation in shared/contract/mcp.ts is a line-for-line TypeScript port of dataset_citation in scripts/zarr/generate_zarr.py; verified byte-identical against the nm000329 index fixture's citation, produced by the Python function from the same public dataset row (captured as test/fixtures/dataset-row-nm000329.json). describeDatasetOutputSchema gains zarr_source_commit and zarr_store_count (additive, optional). provenanceEnvelopeSchema's module doc is corrected: the envelope rides recording-level tool responses (phases 3-4), not search_datasets/describe_dataset, which carry doi/license/citation/zarr_status/zarr_source_commit/ zarr_verify_status directly instead. * mcp: add the ANALYTICS_MCP metrics service and its tests (#1294) buildMcpDataPoint/recordMcpToolCall (backend/src/services/mcp-metrics.ts) mirror buildAccessDataPoint's index+blobs+doubles shape but write to a dedicated Analytics Engine dataset (ANALYTICS_MCP) so blob positions never collide with access-metrics.ts's. tool is the index/blob1; the dataset id (or "-") is blob2; cache_status is blob3; elapsed_ms and upstream_bytes are the two doubles. No-ops without the binding, never throws. The route-level "one point per tools/call, including on isError" assertion in this test file depends on backend/src/routes/mcp.ts, which lands in a later commit. * mcp: add zod4 registration mirrors and the schema parity test (#1294) backend/src/mcp/schemas.ts mirrors search_datasets'/describe_dataset's input/output schemas against the zod4 alias, since registerTool needs a Standard Schema with JSON Schema support (zod 4 has it, the repo's zod 3.25.76 does not). Same field names, defaults, caps and regexes as the zod 3 wire contract (shared/contract/mcp.ts), every field .describe()'d so the published JSON Schema teaches the caller. mcp-schema-parity.test.ts is the drift test: one shared table of valid and invalid inputs (defaults, the cap, cap+1, a malformed id, an unknown passthrough key) run through both copies' .safeParse, asserting identical accept/reject verdicts and parsed values -- behavior parity, not a comparison of two different JSON Schema generators' output. * mcp: add search_datasets, describe_dataset, and the mcp sub-app (#1294) search_datasets (backend/src/mcp/tools/search-datasets.ts) wraps executeDatasetSearch when a query is given, or the public-catalog list query otherwise, filling license/zarr facts SearchResult doesn't carry via one follow-up query keyed by the page's ids. describe_dataset (describe-dataset.ts) is one fresh D1 query on the public predicate, never index.json; unknown/private/sandbox ids answer a tool error naming search_datasets. backend/src/mcp/server.ts builds the McpServer (name "nemar", a 24h public cache hint on tools/list and server/discover) and wraps every tool callback in withToolMetrics, which records exactly one ANALYTICS_MCP point per call on both the success and error paths. backend/src/routes/mcp.ts is the sub-app: its own origin gate (allowedOrigin/corsHeaders reused from zarr-data.ts, since this server does not mount @modelcontextprotocol/hono's standalone origin validation) and rate-limiter bridge, a GET / descriptor, and createMcpHandler built per request on /mcp. mcp-route.test.ts drives the full protocol/tool surface against a real bun:sqlite D1 (both protocol eras, tools/list cache hints, filters, FTS, tool errors, the Mcp-Name -32020 mismatch, the origin gate, 405/ 204). mcp-fork.test.ts drives the real worker entry point (backend/src/index.ts's default export) for the host-fork and path-mount wiring. * mcp: add the workerd smoke script (#1294) backend/scripts/mcp-smoke.sh starts wrangler dev --local against the api-host /mcp path mount (the hostname fork can't be exercised on loopback) and asserts server/discover, tools/list with its cache hint, a tool error naming search_datasets, a malformed dataset_id's SDK input-schema rejection, the legacy initialize + tools/call idiom, the Mcp-Name -32020 mismatch, GET/DELETE /mcp 405, OPTIONS /mcp 204, and the origin gate. Could not be run to completion in this environment: wrangler dev --local fails to start the worker at all (a pre-existing, unrelated export-shape issue in the unmodified index.ts -- see this phase's PR body and .context/mcp-server-design.md section 12 for the full derivation). The script is written and verified equivalent-behavior via mcp-route.test.ts driving the real SDK under Bun. * mcp: update the design doc and systems inventory for phase 2 (#1294) mcp-server-design.md: section 2 corrected (no @modelcontextprotocol/hono -- this server lives inside the existing single worker behind resolveHostRoute, not standalone; origin gate and rate limiting done in the sub-app instead), section 2.1 gains the verified isError-vs-JSON-RPC- error tool-validation shape, section 3 marked implemented, section 5 corrected (the provenance envelope rides recording-level tools only), section 8 resolved (a dedicated ANALYTICS_MCP dataset), section 10 gains the real bundle delta (+1054.40 KiB / +206.36 KiB gzip) and the fixture relocation note, section 12 phase 2 items ticked, including the wrangler-dev-local blocker with full reproduction evidence. systems-inventory.md section 1 gains the mcp.nemar.org/mcp-test.nemar.org hosts next to data/zarr. * mcp: schemas.ts imports the limit constants instead of redefining them (#1323 review) SEARCH_DATASETS_DEFAULT_LIMIT/SEARCH_DATASETS_MAX_LIMIT were redefined locally in the zod4 mirror file; a mirror must never own a number the zod 3 wire contract (shared/contract/mcp.ts) already owns. Imports them from there instead. No other duplicated constant or regex found in the file -- DATASET_ID_RE was already imported from zarr-index.ts, and the enum literals (zarr_status, zarr_verify_status, next_cheapest_tool) have no separately-exported array in the contract to import. * mcp: describe_dataset uses toVersionTag directly, drops the route import (#1323 review) Replaces withCanonicalLatestVersion (routes/datasets/catalog.ts) -- which needed a double cast to apply to DescribeDatasetRow anyway -- with toVersionTag (shared/contract/version.ts) applied inline to row.latest_version when it is a non-empty string, exactly what withCanonicalLatestVersion itself does. A tool no longer imports a route module. No test changes needed; mcp-route.test.ts's citation assertion (composeCitation against the v-tagged version) still passes unchanged. * mcp: unblock the workerd smoke with a minimal entry, not index.ts (#1323 review, issue #1324) wrangler dev --local cannot start backend/src/index.ts in this environment at all (filed as #1324; out of this phase's scope). Adds mcp-smoke-entry.ts (re-exports only mcpRoutes.fetch as the default ExportedHandler) and mcp-smoke.wrangler.toml (same compatibility_date/ compatibility_flags as wrangler-sccn.toml; a local-only DB binding; ENVIRONMENT=development; nothing else) as a throwaway config the smoke script points at instead. Because the entry IS the mcp sub-app, every path -- including "/" -- now reaches it directly, so GET / asserts the real mcp descriptor instead of the (unrelated) API root the prior path-mount workaround was limited to. describe_dataset/search_datasets need the MIGRATED schema even for a row that doesn't exist (an unmigrated D1 has no `datasets` table, which surfaces as a D1_ERROR that doesn't name search_datasets -- correcting the "none should [need migrations]" assumption with evidence). The script now wipes and re-applies every migration to the local D1 first, via `wrangler d1 execute --local --file` per file with full-line comments stripped -- the same technique scripts/d1-migration-check.ts already uses to work around `d1 migrations apply` false-positiving on "BEGIN TRANSACTION"/"COMMIT" inside a `--` comment. Result: all 14 checks PASS on repeated runs, under real workerd, with the unchanged compatibility_date 2024-12-01 (not bumped). Per-call wall times and the full PASS/FAIL table are recorded in .context/mcp-server-design.md section 12 (next commit) and the PR body. * mcp: record the real workerd smoke results in the design doc (#1323 review) Section 12's phase 2 note is updated from "blocked" to the actual result: all 14 checks PASS on real workerd via the mcp-smoke-entry.ts throwaway entry, compatibility_date 2024-12-01 unchanged, with the full PASS/FAIL table and per-call wall times. Also makes explicit (section 10.2) that Bun installing two copies of zod 4 is expected, not a defect -- the "two-copy risk" section 12 named ahead of time. * mcp: fix contract scoping/citation edge cases; de-duplicate mirror literals (#1323 review A/B) shared/contract/mcp.ts: - Module doc corrected: the provenance envelope rides recording-level tool responses only (the section comment already said so; the top-of-file doc still claimed "every tool response"). - composeCitation: each DOI candidate is trimmed BEFORE the concept_doi/doi fallback, so a whitespace-only concept_doi correctly falls through instead of winning empty. Year requires exactly four digits (/^\d{4}$/), stricter than the ported Python's year.isdigit() (which also accepts a short prefix like "20") -- a converter-side follow-up to align dataset_citation itself, not scope here. New tests: whitespace concept_doi with a real doi keeps the DOI segment; created_at: "20" yields no year segment. nm000329 parity retested and still passes. - Exports ZARR_STATUS_VALUES, ZARR_VERIFY_STATUS_VALUES, NEXT_CHEAPEST_TOOL_VALUES (the enums now build from them) and the SearchDatasetsHit type, so a mirror imports the literals instead of hand-duplicating them. searchDatasetsOutputSchema gains optional note/truncated. backend/src/mcp/schemas.ts imports those arrays instead of redefining the enum literals, and mirrors note/truncated. mcp-schema-parity.test.ts extended: an unknown key nested inside results[] and inside cost_hint, plus cases for note/truncated. * mcp: degrade instead of throwing, attribute unresolved hits, drop casts (#1323 review C/D) New backend/src/mcp/tool-types.ts: the shared ToolOutcome = { result, metrics? } type both tools now return, replacing their two ad hoc per-tool outcome interfaces (and the datasetId field on them -- server.ts's withToolMetrics reads the dataset id from the request args instead, in the next commit). search-datasets.ts: - executeDatasetSearch's method: "unavailable" (the FTS table missing) now answers isError: true naming the fallback (browse without query, or retry) instead of an empty page passed through silently, plus one console.error. envelope.warning carries through as `note`; envelope.truncated passes through too. - A hit whose id has no row in the follow-up license/zarr map is no longer presented as license: null, has_zarr: false without comment -- console.warn names the id, and a `note` on the response says which ids are unresolved. Extracted into a pure mergeHitsWithCatalog(results, extraMap), unit-tested (mcp-search-datasets.unit.test.ts) with a real, deliberately incomplete Map for the miss path -- no fake needed. - The `r.has_hed as 0 | 1 | null | undefined` cast is gone: explicit === 1 / === 0 / else-null comparison instead. hits is typed SearchDatasetsHit[], not unknown[]. describe-dataset.ts: buildCostHint now takes the ALREADY-NARROWED zarrStatus, never the raw column, so the cost_hint text and the output's own zarr_status field cannot disagree. console.warn when a raw zarr_status or zarr_verify_status is non-null but outside the known set (zarr_verify_status in particular comes from free-form sweep_stamps JSON with no DB enum behind it). * mcp: type the tool name, add the outcome blob, attribute ids before the call (#1323 review E) mcp-metrics.ts: McpToolCallEvent.tool is McpToolName (the contract's closed set), not string. New outcome: "ok" | "tool_error" | "exception" blob (4th position: tool, dataset_id, cache_status, outcome). buildMcpDataPoint's doc and tests updated for the new field/blob count. Design doc section 8 update lands in the design-doc commit, not here, to keep that file's changes in one place. server.ts: withToolMetrics reworked to withToolMetrics(env, name, getDatasetId, fn) -- getDatasetId(args) runs BEFORE fn, so a thrown D1 error is still attributed to the right dataset instead of recording "-" because the tool never got far enough to report one itself. fn now returns the shared ToolOutcome (previous commit); try/catch records outcome: "exception" and rethrows; result.isError records "tool_error", otherwise "ok". search_datasets registers with `() => undefined` (it names no single dataset); describe_dataset with `(args) => args.dataset_id`. mcp-metrics.test.ts: outcome asserted throughout; new test drives a genuinely unmigrated D1 (freshDb() with no schema) through the real sub-app and asserts exactly one point records outcome: "exception" with the dataset id taken from the request arguments -- PR #1323 review item G.13, landed here (not the later tests commit) since it is the same behavior this commit introduces and splitting one file's cohesive metrics-attribution tests across two commits would reduce review clarity, not improve it; noted as a grouping deviation in the PR body. * mcp: one MCP-specific CORS header set, cite the design doc by content (#1323 review F) New mcpCorsHeaders(origin): corsHeaders(origin) (zarr-data.ts) plus the THREE fields that actually differ for this transport -- Access-Control-Allow-Methods (POST, OPTIONS), Access-Control-Allow-Headers (the four Streamable HTTP request headers), and Access-Control-Expose-Headers (MCP-Protocol-Version, Content-Type). The OPTIONS preflight, the POST response re-wrap, the rate limiter's 429 branch, and app.onError all used to ship zarr's own Range/GET-HEAD-OPTIONS/ ETag-Content-Length-Content-Range-Accept-Ranges set on those four responses; now they share one function and one comment explaining the override, instead of the OPTIONS handler's own comment claiming only two fields differed. Module doc's "(decision 2)" citation replaced with the actual source: .context/mcp-server-design.md section 3, "Host and routing". Grepped the whole diff for "decision \d" and found one more real hit (design doc section 10, from the earlier smoke-results commit) -- fixed there too. The remaining "decision N" text in the design doc is phase 1's own, already-merged prose citing ITS OWN plan by name ("decision 7 of the phase 1 plan") -- a correctly-scoped historical citation, not the dangling plan-number reference this item is about, so left as-is. * mcp: fill the review's remaining test gaps (#1323 review G, items 12/14-20) mcp-route.test.ts: - G.12: a real in-memory CacheStorage stand-in (RateLimitCache, the same technique zarr-data-cache.test.ts's redirect-exemption block uses), ENVIRONMENT: "production", the ip bucket seeded at MAX_REQUESTS -- asserts the 429 carries Access-Control-Allow-Origin for https://nemar.org and the MCP Access-Control-Allow-Methods. - G.14: the converted row's citation contains "(v1.2.0)"; the pending row's (no dataset_versions row) contains no "(v" segment at all. - G.15: explicit has_hed true/false on the two search_datasets hits, not just typeof. - G.16: a self-contained EEG/MEG fixture pair (distinct modality, task, has_hed) proves each filter narrows the no-query path to the matching row, excluding a row that HAS a different value -- not just one with a NULL column. - G.17: limit at the cap (100, echoes in output.limit) and limit: 1 (exactly one row, output.count still the real total) on both the no-query and query paths. - G.18: a non-matching query and has_zarr: true with no converted fixture both answer count: 0 with no isError. - G.19: describe_dataset on a zarr_status: "failed" row (cost_hint names it); search_datasets(query) after DROP TABLE datasets_fts answers isError naming "search index" (the real degradation path, method: "unavailable", end to end). blosc-decode.test.ts (G.20): every remaining guard triggered individually against real, precisely-computed byte offsets in the fixture (memcpyed flag, bit-shuffle flag, nbytes<=0 distinct from blocksize<=0, nbytes % typesize, a block offset past the end, a zero and an oversized compressedLen, and a decompressed-length mismatch -- the last built by declaring a smaller nbytes/blocksize than the real, untouched zstd payload actually decompresses to). G.13 (the thrown-D1-exception path) landed in the earlier metrics commit instead of here -- see that commit's message for why. * mcp: smoke script checks the D1 wipe's exit status, traps before spawning (#1323 review H) The D1 state wipe no longer swallows a real failure with `2>/dev/null || true`: it now only skips the wipe when the directory is genuinely absent (first run), and exits loudly if find -delete itself fails on an existing directory. trap cleanup EXIT is now registered BEFORE wrangler dev is spawned (previously after), so an interruption or failure between spawn and the old trap registration point can no longer leave an orphaned --local wrangler/workerd process holding the port and the D1 state lock. Re-ran end to end twice after these changes: 14/14 PASS both times, compatibility_date 2024-12-01 unchanged and starting the SDK cleanly. Per-call wall times unchanged within noise (server/discover ~7-8ms, tools/list ~3-4ms, tools/call ~2-7ms). * mcp: design doc catches up with the review's implementation changes (#1323 review I) Section 2: the dual-era legacy-serving claim now cites its two actual sources by name -- mcp-route.test.ts's legacy test drives describe_dataset in-process, mcp-smoke.sh's legacy check drives search_datasets under real workerd -- rather than one generic "verified in phase 2" citation. Section 5.1 (search_datasets): documents the three PR #1323 review additions -- note (executeDatasetSearch's own warning, and unresolved follow-up-lookup ids, joined when both fire), truncated (executeDatasetSearch's own, query path only), and the search-index- unavailable case now answering isError instead of a silent count: 0 (the ADR 0005 boundary: partial data still serves, an infra failure does not get to look like "no matches"). Section 8: the outcome blob (4th position: "ok" | "tool_error" | "exception") and withToolMetrics's getDatasetId(args)-before-fn attribution fix, matching the reworked implementation. Also fixes the "decision 9" plan-number citation in section 10 (introduced in an earlier commit of this same review pass) to describe the decision by content instead.
…rview) (#1326) * deps(backend): add hyparquet and fast-png for MCP phase 3 hyparquet reads events.parquet for get_events; fast-png encodes the render_overview PNG. Both pure JS, no WASM, matching the fzstd/blosc precedent from phase 2. hyparquet-compressors is deliberately NOT added here -- see the get_events commit for why. * feat(contract): add phase 3 recording-tool fields and legacy index schema shared/contract/zarr-index.ts: zarrIndexLegacySchema, a deliberately minimal lower bound for format_version 1/2 documents (no source_tree/ derived, source_commit unregexed since v1 predates the 40-hex guarantee). shared/contract/mcp.ts: additive fields on list_recordings (nullable source_commit, index_format_version, discovered/failure/pending_count, excluded_legacy_non_raw_count, note, envelope; group summary gains n_view_levels/n_samples/view_chunk_columns) and get_events (limit/offset, total_count/truncated/note/envelope) and render_overview (columns_read/chunks_read/bytes_read/envelope). All additive and passthrough, per the repo's lower-bound convention. * refactor(mcp): extract the shared public-catalog row read (decision 3) catalog-row.ts pulls describe_dataset's D1 query, row shape, and not-found error text out into loadPublicDatasetRow/PublicDatasetRow/ datasetNotFoundResult, plus a new isZarrReady/zarrNotReadyResult pair for the recording-level tools. describe_dataset now calls it instead of owning the query itself; its existing tests (mcp-route.test.ts) stay green unchanged, proving the extraction is behavior-preserving. * feat(mcp): add the index.json reader and the projection cache index-reader.ts (decision 2): fetches index.json IN PROCESS through the real zarr sub-app (D1 gate, edge cache, purge list unchanged), narrows format_version 3 against zarrIndexSchema and everything else against zarrIndexLegacySchema, reports sourceCommit: null rather than crashing on a non-40-hex source_commit. projection-cache.ts (decision 4): synthetic https://mcp.nemar.org/_cache/<id>/<commit>/<projection> keys, 7-day TTL, the safeCachePut discipline from zarr-data.ts (a throwing cache.match/put is logged once and treated as a miss/no-op). zarr-fidelity-sweep.ts: exports GITHUB_RAW_ORIGIN and rawContentUrl (no behavior change) for get_events' events.tsv fallback to reuse. Test infra: InMemoryCache/keyFor moved to test/helpers/cache.ts (was private to zarr-data-cache.test.ts); a new Bun.serve() fixture-server helper stands in for both S3 and raw.githubusercontent.com; fixtures under test/fixtures/mcp/ are live captures (curl -A "nemar-cli/mcp- phase3") of nm000329 (v3, 112 stores, events.parquet, one view-5 and one view-4 pyramid chunk) and nm000111 (v1), plus a synthetic events.tsv with n/a cells and an HED column. * feat(mcp): add list_recordings tool (decision 6) tool-types.ts: RecordingToolDeps, the shared dep bundle (env, executionCtx, cache, fetch, zarrRoutes, rawGithubBase) the three recording-level tools all take. envelope.ts (decision 5): buildEnvelopeForStore, the one place that infers a legacy (v1) store's source_tree/derived/engine_version and attaches the fixed legacy note, so the three tools cannot diverge. Takes resolved index-level facts as plain fields (not a whole parsed index document) so an envelope can be built on a projection cache HIT without a second index.json fetch. list-recordings.ts: the tool itself, plus loadRecordingsProjection/ resolveRecording/recordingNotFoundResult, exported for get_events and render_overview to share -- a list_recordings call primes the same cache entry those two read, and vice versa. Excludes a legacy store under derivatives/sourcedata/code (ADR 0027) and counts the exclusion. Not yet wired into server.ts; lands with the mirrors/wiring commit. * feat(mcp): add get_events tool (decision 7) Primary path: reads events.parquet once per (dataset_id, source_commit) with hyparquet, groups rows by store_path, writes every store's rows plus a store-list summary to the projection cache in one pass. Converts hyparquet's BigInt sample_index to Number once. No columns filter, so subject/session/task/run and any x_-prefixed extra pass through per eventRowSchema's .passthrough(). Fallback: derives the sibling events.tsv from the recording's path (BIDS naming) and fetches it from raw.githubusercontent.com via zarr-fidelity-sweep.ts's rawContentUrl; computes sample_index locally (Math.floor(onset_s * rate + 0.5), the converter's own formula) and flags estimated: true. A clean 404 answers events: [] with a note. Codec: a hand-rolled ZSTD-only compressors map built on fzstd, NOT the hyparquet-compressors package -- that package's compressors export eagerly WASM-compiles its SNAPPY entry (hysnappy) at module load, which crashes isolate startup under real workerd regardless of whether any dataset's parquet ever uses SNAPPY (none does; every column sampled is ZSTD). Found via the smoke script, not bun test. Not yet wired into server.ts; lands with the mirrors/wiring commit. * feat(mcp): add render_overview tool (decision 8, moved from phase 4) overview.ts: pure helpers, no I/O. pickViewLevel returns the COARSEST view level satisfying width_px (biosigio's Math.floor(n/4) per level, applied iteratively); buildChunkPlan computes chunk keys from view_chunk_columns (default 1024) with no special-casing for "the last level" -- the ceil() formula already produces one chunk when a level's own column count is under the chunk size, verified against nm000329's real level 4 (541 cols) and level 5 (135 cols, one chunk of chunk_columns: 135); reassembleViewChunks stitches decoded [2, n_channels, chunkCols] chunks into one [2, n_channels, totalCols] buffer; renderOverviewPng draws one grayscale band per channel (rowPx = clamp(floor(1200/n_channels), 2, 24)), a min-max bar per width_px column bucket, via fast-png. render-overview.ts: resolves recording/group off the shared recordings projection (never a probe fetch, never zarr.json), fetches only the chosen level's chunks straight from data_base, decodes with decodeBloscZstdInt16, caches the whole PNG per (dataset_id, source_commit, recording, group, width_px) -- a hit skips fetch, decode, and encode entirely. Not yet wired into server.ts; lands with the mirrors/wiring commit. * feat(mcp): wire the three phase 3 tools with zod4 mirrors (decisions 9, 10) schemas.ts: zod4 mirrors for list_recordings/get_events/render_overview input+output, plus a provenanceEnvelopeSchema4 mirror (including the ADR 0028 sss-iff-derived superRefine) since it rides all three outputs. server.ts: BuildMcpServerDeps grows cache/fetch/zarrRoutes/rawGithubBase; a RecordingToolDeps bundle is built once per request and passed to all three new registerTool calls. routes/mcp.ts: McpRoutesDeps grows the same four fields, defaulting to caches.default (lazily), the global fetch, the real zarrDataRoutes instance, and GITHUB_RAW_ORIGIN -- threaded into buildMcpServer. Tests: mcp-recording-tools.test.ts (list_recordings + get_events through the real route: cache-hit-skips-index.json proof, v1 legacy listing, the events.tsv fallback's n/a-as-null and floor(onset*rate+0.5) sample_index, unknown-recording error text) and mcp-overview.test.ts (pickViewLevel/buildChunkPlan/renderOverviewPng unit tests against the real decoded nm000329 fixture chunk, plus the route-level request-log proof that render_overview reads only view/<L>/ keys, never /0/c/ or zarr.json, and that a repeat call is a cache hit with an empty log). mcp-route.test.ts's tools/list assertion now expects all five tools. mcp-schema-parity.test.ts gains the decision-10 case table (defaults, caps, cap+1, malformed id, width_px/limit boundaries, negative offset) for all three new schema pairs. * test(mcp): smoke-test five tools under real workerd tools/list assertion now checks for all five tool names; a new check calls list_recordings(xx000000) against the migrated-but-empty local D1 and expects the shared not-found error, mirroring the existing describe_dataset(xx000000) check. This script is what caught the hyparquet-compressors/hysnappy WASM-under-workerd crash during implementation (bun test alone stayed green -- see the get_events commit for the fix). Two consecutive runs here, all 15 checks PASS, compatibility_date unchanged at 2024-12-01. * docs: update MCP design doc for phase 3 Sections 5.3-5.5: the v1/legacy behavior, get_events pagination and the ZSTD-only codec finding, render_overview's level rule and image geometry, all against the live nm000329/nm000111 measurements. Section 7: the concrete recordings/events/overview cache key shapes and the 7-day TTL. Section 9: hyparquet/hyparquet-compressors/fast-png rows updated with the real workerd verdicts. Section 10.3 (new): the measured phase 3 bundle delta. Section 12: render_overview marked DONE under phase 3 (decision 1's move from phase 4), the zarrita open item closed. * fix(mcp): correctness fixes from phase 3 review (group A) Addresses PR #1326 review items 1-13: - CRITICAL: thread sss/units_report through list_recordings' projection, get_events, and render_overview's envelope call sites -- previously dropped, crashing buildEnvelopeForStore's SSS-iff-derived assertion (ADR 0028) on any MEG SSS derived store (on003392). - CRITICAL: get_events resolves the requested group name FIRST and returns an explicit groupNotFoundResult error (shared with render_overview) instead of silently returning an empty row list for a typo'd group. - Decouple get_events' parquet DATA-SOURCE gate (eventsParquetUrl alone) from its CACHE gate (sourceCommit !== null); the two were wrongly combined, causing a spurious fallback to the strictly-worse TSV path. Wrap the parquet read in try/catch with a dedicated error result. - Write "[]" cache placeholders for every recordings-projection store absent from a dataset's real events.parquet content. - loadEventsFromTsvFallback returns a discriminated union: only a 404 is absence; any other non-2xx or a thrown fetch is a distinct error result (tsvFetchFailedResult), not silently treated as "no events". Track bytesFetched for the TSV path too. - Log tool exceptions (console.error) in withToolMetrics before rethrow. - Move narrowZarrVerifyStatus out of describe-dataset.ts into catalog-row.ts (exported) so envelope.ts can use it instead of an unsafe type cast on the raw D1 column. - render_overview's envelope dtype is "int16" only on a cache MISS (the call that actually decoded a view array); a cache HIT now reports dtype: null instead of a hardcoded, misleading "int16". wired through NO_COMMIT_NOTE, mirroring list_recordings exactly. - Version the projection cache payload shape (PROJECTION_SCHEMA_VERSION in the cache key) and validate a cache hit against a zod schema before trusting it; a schema mismatch is a miss, not a crash. guard with a descriptive throw. - list_recordings: note a zero-match modality filter against the dataset's actual known modalities; updated tool description to call out that a count of 0 is not an error. No mocks: all coverage lands in group B against real D1, a real Bun.serve() fixture upstream, and a real in-memory cache. * test(mcp): coverage for phase 3 review findings (group B) Addresses PR #1326 review items 14-24: - pickViewLevel: add boundary cases (135, 2167 columns) to the level table. - reassembleViewChunks: new pure-function test with three literal synthetic chunks (a short last one), spot-checking column values at every boundary. - A constant channel (min === max) renders without NaN, one stable band. - New MEG SSS derived-store coverage (item 1's crash, now with real tests): real on003392 MEG SSS index data with geometry overridden to a small synthetic size, plus matching synthetic blosc-encoded chunk bytes generated via scripts/zarr/generate_mcp_test_fixture.py (same numcodecs Blosc config biosigio's real exporter uses) -- covers list_recordings, get_events, and render_overview against a real SSS store. - get_events edge cases: offset past end, limit 5000, a nonexistent group (item 2's fix), a CRLF events.tsv with half-sample rounding, a parquet file with zero rows for the requested store. - modality + min_duration_s combined filtering. - zarr_store_count 0 answers the not-ready error, for all three tools. - A legacy v1 index with an empty source_commit: the projection cache is never even consulted (item 17) -- proven with a SpyCache wrapper around InMemoryCache that records every URL passed to match(), since the zarr sub-app's own independent edge cache made a naive upstream-fetch-count assertion give a false pass. - projection-cache: a stored entry that fails schema validation is a miss, not a crash (item 11's schema versioning). - metrics: cache_status ("miss" then "hit") for all three recording-level tools, driven through a real fixture server + zarr routes. - schema parity: provenanceEnvelopeSchema/schema4 (5 cases, including both invalid sss/derived combinations) and the envelope embedded in all three tools' output schemas. - fixture-server: malformed Range ("bytes=abc-def") and out-of-range Range both answer 416 + Content-Range: bytes */<len>. No mocks: real D1 (bun:sqlite), a real Bun.serve() fixture upstream, a real in-memory projection cache. * docs(mcp): cite by content, not plan decision numbers (group C) Addresses PR #1326 review items 25-34: - Remove every "decision N" / "plan decision N" citation from code, tests, helpers, shared/contract/mcp.ts, and .context/mcp-server-design.md, per AGENTS.md's citation-discipline rule. Replaced with ADR numbers, design-doc section references, or plain descriptions. (zarr-fidelity-sweep.ts keeps its OTHER "decision N" comments -- those cite epic #1181's own plan, not this one, and are out of scope here; only the single line this PR's diff touches was fixed.) - Stage ADR 0050's content in the design doc's section 12: the "no WASM in the Worker bundle" rule, covering both numcodecs (phase 1) and hysnappy/hyparquet-compressors (phase 3), plus the real reason the ADR file itself is deferred (ADR 0049 not yet on dev; the ADR index test enforces gapless numbering) rather than "the plan's numbering rule". - Add a phase 3 smoke-results table (15 checks, five tools) to section 12, mirroring phase 2's own table -- both from two consecutive live runs of backend/scripts/mcp-smoke.sh. - Fix overview.ts's computeViewLevelColumns doc comment: the claim that the iterative Math.floor(n/4) loop differs from Math.floor(n0/4**L) in one step was false for non-negative integers (floor-division associativity); reworded to say the loop mirrors biosigio's own implementation rather than claiming a numerical difference that does not exist. - Fix projection-cache.ts's module doc: correct an inverted claim about which documents get the short untokened TTL split (zarr.json / dataset-level docs) versus the flat 24h chunk TTL this file's projections share the rationale with. index-reader.ts: DEFAULT_ZARR_BASE's fallback behavior is NOT shared with zarrPurgeTargets() (empty list when unset) or s3PublicUrl (bucket-URL fallback); clarify it is a new, distinct fallback. - zarr-fidelity-sweep.ts: rawContentUrl's doc pointed at .context/README.md for the "nemarDatasets names a dataset's repo after its own id" fact; it actually lives in .context/systems-inventory.md. - fixture-server.ts: clarify FixtureRequestLogEntry.url differs in NAME (not just shape) from zarr-data-cache.test.ts's LoggedRequest.path. - cache.ts: restore the full 206-refusal comment (with its #1181 review item 13 citation) instead of a bare "unchanged" claim. - Add "(as of 2026-09-08)" to the "about half the catalog is v1" claim, in both index-reader.ts and the design doc, with a cross-reference between the two. * docs: drop dangling plan-decision citations Comments citing "decision N" point at epic plan briefs that do not exist in the repo. Each comment already states the rule it cites, so the parenthetical is removed rather than rewritten. Comment-only. * fix(mcp): bound the get_events per-store cache fan-out A parquet miss wrote one cache entry per store in the dataset. nm000281 publishes 25,253 stores, so a single get_events call could issue 25k Response constructions and JSON.stringify calls inside one request's waitUntil. Cap the fan-out at 2000 stores; the requested store's entry and the _stores summary are always written.
* feat: add read_window taste caps and fields * feat: add read_window's pure decode helpers * test: add synthetic sharded store fixture * feat: implement and register read_window tool * test: add read_window route-level tests * test: extend mcp parity/contract test tables * chore: smoke-test and measure read_window * docs: update MCP design doc for phase 4 * fix(mcp): shard footers carry a constant entry count A shard index always has shard_samples / chunk_samples entries, for every shard including the truncated last one; chunks past the array's extent are absent entries, not omitted ones. nInnerForShard computed the count from the remaining extent instead, so for nm000329's shard 1 it read the last 1028 footer bytes rather than 1204, starting 11 entries into the real footer. Every chunk served from that shard was 11,000 samples (44 s) late, and nothing errored because each misread entry pointed at a real, full-size, cleanly decoding chunk: sample 75,000 returned sample 86,000. A present chunk that straddles n_samples is also stored full size and fill-padded, so a decoded chunk's stride and its valid span are separate quantities; conflating them made the length check reject the tail of every recording whose length is not a multiple of chunk_samples. The synthetic generator had encoded the same wrong rule, which is why the suite was green. It now derives the entry count from the chunk grid, pads the straddling chunk, and emits its expected values as data. The real boundary footer (c/0/1) is committed as a fixture so the rule is pinned against the actual producer rather than against our own generator. Also from the review: report filled ranges instead of returning fill values as if they were signal, refuse a Range request answered with a bare 200, refuse an empty channels array before any byte is fetched, make ShardIndexEntry and ChunkRead discriminated unions, and reject a window that rounds to zero samples. * test(mcp): cover the boundary shard, caps, and refusal paths Every one of these was a gap a mutation proved: the code could be broken and the suite stayed green. - The channel-samples cap, the only taste bound no schema enforces, was tested ~14x past its limit, so > versus >= was indistinguishable. Asserted now at exactly 65,536 and 65,537 through the pure predicate, plus a product-versus-sum case. - An empty channels array with taste: true. - Unknown recording, unknown group, a group with no sharding geometry (which refuses a taste but still serves a recipe), and a group with no usable rate. No v3 fixture in the repo could reach the last two, so both index variants are built by removing fields rather than committing near-duplicates. - A Range-ignoring origin, using the fixture server's new opt-in capability. The assertion names the protocol reason: tolerating a bare 200 still trips the byte-length check further down, so a looser assertion passed with the status check removed. - Expected physical values now come from the generator's emitted nm099500-expected.json instead of a hand-written mirror of its formula, and a sample in the boundary chunk's fill padding is proved to be refused rather than served. * docs(mcp): correct the shard geometry and byte accounting Section 5.6 stated Zarr's "last chunk may be shorter" rule for the shard index and for a straddling inner chunk. Neither holds: the entry count is constant per shard and a straddling chunk is stored full size with fill padding. Both are now recorded with the measured evidence, including what the misread returned from live data. Also: bytes_read was described as Range-read bytes in two places while also counting the array-metadata GET, which is a plain full GET; on003392's shard_samples is named per store (68000 on sub-06, 69000 on sub-01) rather than presented as one figure for the group name; and the memory table gains a boundary-shard case, since the first two cases both start at sample 0 and so never touched the shard the defect lived in. * test(zarr): make the format claims executable, not prose Phase 4's shard-footer defect shipped because a claim about the on-disk format was measured on two stores, written into the design doc, and then trusted by the code, the tests, the fixture and five reviewers alike. Everything agreed with the prose; the prose disagreed with production. Two gates so that class of bug has to announce itself: zarr-geometry-conformance.ts (bun run zarr:geometry-check) checks every invariant the readers depend on against every index the catalog publishes: footer entry-count divisibility, the index agreeing with the array it describes, inner chunks spanning every channel, the codec chain and dtype, per-channel scale/offset lengths, and every events.parquet column being ZSTD with the columns eventRowSchema requires. It also reports the observed distributions, since a small sample is what makes a varying figure look constant. A transient 429/5xx is retried, reported separately, and never counted as a violation; concurrency is 4 after an earlier 12 drew 500s from the host that serves real users. zarr-schema-contract.test.ts now requires the published JSON Schema and the zod contract to agree on required-ness field by field, driven off the shared fixture, with the deliberate asymmetry (the producer's gate refuses an undeclared field, consumers tolerate it) asserted as a difference. First full run: 314 v3 indexes, 90,966 groups, 285 parquet footers, 13,023 columns, zero violations. It also corrected two claims this repo was repeating: chunk_samples takes 33 distinct values and shard_samples 256, not the single figures a two-store sample suggested, and the widest store is 415 channels, not 320.
* docs: ADR 0049 supersedes 0025 on compute locality Compute runs in the browser by default with eegprep as the engine, OSA owns the execution runtime, and only HPC submission is gated. Restates the broker decision from 0025 and marks 0025 superseded. Index test: 7 of 8 pass; the gapless-numbering check fails until epic #1272 lands ADR 0047 and 0048 on dev. * docs: ADR 0049 corrects the picard dependency claim python-picard is pure Python with an optional numexpr fast path; the browser blockers in eegprep's declared dependencies are oct2py, pyedflib, and psutil. Notes the no-BLAS numpy build in Pyodide and the torch-free ICLabel need.
* docs: file ADR 0050 and add a Python client verification ADR 0050 formalizes the rule two phases learned the hard way: no WebAssembly in the Worker bundle, ever, for any package this server depends on. Both failures (numcodecs instantiating WASM from a runtime fetch, and hyparquet-compressors compiling hysnappy's WASM at module load and so crashing isolate startup for every request) were invisible to bun test and caught only under real workerd, which is why the smoke script is the gate rather than an optimization. It could not be filed until ADR 0049 landed on dev, since the index test enforces gapless numbering; the content staged in the design doc is now a pointer to the ADR. scripts/mcp/verify_streamable_http.py drives a target host with the official Python MCP SDK: the negotiated revision, all six tools with their cache hint, then a chain from search through describe and list_recordings into the three recording tools and both read_window modes, against a recording it discovers rather than one hardcoded here. mcp-smoke.sh checks our own framing with curl and says nothing about whether a real SDK can talk to us, and OSA speaks this SDK, so this is the client that has to work. It is also where the docs page will copy its examples from. Its SDK usage was validated against a local MCPServer harness rather than discovered during a cutover, which caught two things worth having: any check failure inside the client's task group came back as a bare "ExceptionGroup: unhandled errors" with none of the message that explains it, and an over-cap refusal can arrive either as a tool error or as a raised protocol error, so the check accepts both. Design doc records the verified SDK version and the two assumptions in #1297 that turned out to be wrong: there is no For-agents guide to extend, and OSA has no MCP runtime at all while its two existing NEMAR tools call an endpoint that now 404s. * ci: run the ADR index test when ADRs change test/adr-index.unit.test.ts validates gapless numbering, README index agreement, and superseded-target existence. Nothing in the path filters matched .context/**, so it never ran when those files changed: #1292 and #1330 both showed a fully green check list with the one test that could have failed skipped, and both were verified by hand instead. Same shape as the shared/** gap this file already documents, same fix. Wired into unit-pure, with a self-reference so editing the filter re-runs it. * fix: harden the MCP verification script Review of scripts/mcp/verify_streamable_http.py found the script could not connect at all and could report a pass it had not earned. Fixed: - The default URL omitted the /mcp transport path. GET / is a descriptor and POST / answers Not Found, which during a cutover reads like an unprovisioned hostname. A bare host is now normalized and reported. - The over-cap probe wrapped its call in `except Exception`, so a 429 or a 502 became a PASS. Removed: both a tool error and a schema rejection arrive as is_error, so an exception there is a transport failure. - The probe sat exactly AT all three caps, which the schema accepts (`>`). Now one second over the duration cap with one channel, so exactly one cap is crossed, and the refusal text must name it. - The envelope check tested 3 of 14 required fields. Now all 14, plus non-null where the contract forbids null, and required only where the contract makes it required: it is .optional() on three tools, so failing hard everywhere reported missing provenance against a valid legacy index. - The subject was `results[0]`, now scanned for a v3 index with a usable group. has_zarr means converted, not v3, so a legacy hit is legitimate. - Read `results` only; `datasets` is catalog.json's field. - Notes are counted and repeated in the closing line, so a degraded run cannot look clean. Adds the wire-level test the script's cap assertion depends on: an over-cap taste answers isError with a text block naming the cap and the recipe remedy. Validated against a local MCPServer stand-in with twelve fault injections; all ten defects fail with the right message and the two honest degradations exit 0 with a note. * fix: correct ADR 0050's platform claim and the docs target An accuracy review of ADR 0050 and the design doc's phase 5 entry found four wrong claims, two of which would misdirect future work. All verified before changing, two by re-running the experiment. ADR 0050 said there is no flag permitting dynamic WASM compilation, so the restriction was the platform's and not configurable. Inverted. Measured under wrangler dev --local: compiling at MODULE LOAD is permitted by allow_eval_during_startup, which is the default from compatibility_date 2025-06-01, so the phase 3 shape is blocked only because this worker pins 2024-12-01. Compiling inside a request stays disallowed under every combination tested. So the decision survives but gains a real consequence: a routine compatibility_date bump silently removes the guard that caught the phase 3 failure, which is why the rule is written down and smoke-gated rather than left to the runtime. The flag's docs never mention WebAssembly, so the ADR now says the behavior is measured, not documented. ADR 0050 also framed both failures as dependencies that broke the server with a green suite. Neither was ever in backend/package.json: numcodecs was the spike's path (a), and hyparquet-compressors was removed before merge. The lesson holds, the history does not, and an ADR outranks the design doc. Design doc: the For-agents guide #1297 names does exist, published at docs platform/for-agents.md since 2026-09-03, so the docs work is a section on that page and #1297's premise was right. develop/zarr-contract.md is gone, split into six pages under platform/zarr/. Both errors came from reading a stale local checkout instead of origin/main. Also: #1324 has three non-handler exports, not two strings; the fixtures are real bytes where a real store exercises the path and synthetic only where the catalog has no such shape; the subpath-import option is named; and the custom-domain zone-edit caveat is moved next to the sequencing it gates.
Two critical findings from the epic-level review, both confirmed by running the code rather than reading it. render_overview threw a RangeError on its DEFAULT path for much of the catalog. reassembleViewChunks derived each chunk's width from its decoded length and documented "the last chunk may be shorter", which is false: Zarr stores a boundary chunk FULL SIZE and fill-padded. Verified against nm000329 eeg_250hz level 3, which width_px 800 selects: all three real objects carry blosc nbytes 258048 = 2 x 63 x 1024 int16, so the third holds 1024 columns of which 119 are real. The old code advanced past the end, wrote each channel over the next channel's region, then overran the buffer. This is the same nominal-stride versus valid-span split sharding.ts already documents for level 0; the trap had not been carried to the view path. chunkColumns is now passed as the stride and clamped to the level's own columns, matching the real chunk grids read from S3 (view/4 and view/5 clamp to 541 and 135). A short chunk is now refused instead of inferred from. The reassembly moved inside the try, so a geometry disagreement is this file's typed tool error rather than an opaque JSON-RPC internal error. The synthetic fixture encoded the same wrong assumption, widths (50, 50, 17), which is why the suite was green; regenerated full-stride with fill padding. Second: /mcp matches none of the rate limiter's path patterns, so a caller sending any 32-character bearer selected the token bucket, on a surface with no authentication at all. That gave a fresh 1000/min allowance per rotated string, bypassing the 500/min IP floor, and drove a D1 tokens-to-users lookup per novel token from one anonymous IP. Adds an explicit anonymousSurface option and passes it from the MCP sub-app, so the whole sub-app including the descriptor keys on IP. Tested through the real sub-app, not the helper: both tests fail when the flag is removed.
The cache layer rested on a claim that is false: "a re-conversion mints a new
source_commit, so the OLD key never again resolves to different bytes".
source_commit is the dataset repo's HEAD, and the documented back-catalog
mechanism re-converts at an UNCHANGED HEAD, an engine bump re-queues a done
row and bumps no dataset version (ADR 0033); a --clean rebuild and a retry
after an infra failure are the same shape.
So a re-conversion left every MCP projection stale for a 7-day TTL while
index.json itself refreshed in 5 minutes, and nothing purges these synthetic
keys. Two of the namespaces make that a correctness bug, not staleness:
- array/ caches scale[] and offset[], which a taste multiplies by. Stale
constants against re-quantized data return silently wrong PHYSICAL values,
with no error and nothing in filled_ranges.
- shardidx/ caches parsed byte offsets and replays them as Range reads
against whatever object is at that key today. Every 206 and length check
passes by construction, because the origin returns exactly the bytes asked
for, just the wrong ones.
The key now carries datasets.zarr_converted_at, set to datetime('now') by
every conversion and readable from the D1 row the tools already load before
any fetch, which is what rules out the index's ETag or updated_utc: the whole
point of the recordings entry is to answer without fetching index.json.
The namespace is also environment-derived now. It was the literal
mcp.nemar.org, but caches.default is zone-scoped and both MCP hostnames are
custom domains on the nemar.org zone, so prod and staging read and wrote each
other's entries while their data_base pointed at different buckets.
Also bounds render_overview, whose cache key was the caller's raw width_px: a
caller walking 1..4000 forced up to 4000 renders and cache writes per
recording and group, each re-fetching the same view chunks from S3 with no
edge cache in front. The key now carries the level and a width rounded UP to
one of nine buckets, floored at the level's own column count so it stays a
pure downsample. Level selection deliberately keeps using the RAW width:
choosing from the rounded value read a FINER level than asked for. And the
chunk plan is capped at 64, since the count comes from the index and a
shallow pyramid on a long recording could fan one call out past Cloudflare's
1000-subrequest ceiling.
Tests: a re-conversion at an unchanged commit no longer serves the old entry,
prod and staging keys differ, and the fixtures carry a conversion stamp.
Both are on a free, anonymous endpoint, so one request could cost arbitrary
Worker memory and S3 bandwidth.
get_events read every events.parquet WHOLE, before applying limit/offset.
Measured against the live bucket: 285 public datasets publish one, 13 over
5 MB, and nm000104 is 99,863,763 bytes with 5,411,570 rows across 1131
stores. So get_events(nm000104, limit: 1) pulled 95 MB through the Worker,
zstd-decompressed it in pure JS, materialized 5.4 M row objects and ran 5.4 M
zod parses. That does not fit a 128 MB isolate, and the failure re-amplified:
the isolate died before waitUntil ran, so nothing cached, so the next call
repeated the whole read, at up to 500 requests a minute from one IP.
Now bounded at 16 MiB / 100,000 rows, checked from the footer this path
already fetches, so a refusal costs one footer read. Refusing beats paging
because a row's store_path is only known after the row is read, and it costs
the caller little: index.json publishes events_parquet as a public URL, so
the refusal hands that over instead. That is the recipe-first posture ADR
0049 sets for signal bytes.
read_window's four taste caps all bound the RESPONSE and none bound the
DECODE, because a stored chunk holds every channel in the store: cost is
n_channels(store) x windowSamples no matter how few channels were asked for.
sharding.ts says so in prose and no guard was added. The worst legal request
in the catalog, found by scanning all 108 meg/ieeg/emg indexes, is on004696's
ieeg_1000hz: 256 channels at 1000 Hz, and {channels: [0], duration_s: 60}
passes every cap then decodes 15.4 M int16 samples, about 31 MB retained, to
return 500 KB. The 415-channel store in the archive makes it about 50 MB, and
128 MB is per ISOLATE, so two concurrent tastes take out every unrelated
request in flight. Now bounded at 4 M decoded samples, refused before any
fetch, with a message naming the STORE's channel count and a duration that
does fit, because "ask for fewer channels" is the obvious next move and the
one that does not work.
Fixtures: a real 100,001-row parquet, 4.5 KB because every column is a
repeated constant, and a v3 index with on004696's geometry. Both guards
mutation-checked, and both route tests assert nothing was fetched.
The zod3/zod4 drift test only compared whether the two copies AGREED, so a case both rejected passed silently. The read_window taste fixture was invalid (no filled_ranges, required on both copies), which made its own case vacuous and the three "field removed, still rejected" cases built from it vacuous too, since the base was already rejected. One was labeled "accepted" while both copies rejected it. Net effect: phase 4's entire taste output contract, filled_ranges' nested shape included, was unguarded by the test the epic relies on for exactly that. The fixture now carries a non-empty filled_ranges, and assertParity asserts the VERDICT, taken from the label by the file's own convention. That turns every "-- rejected" label into a real assertion as well. Adds the missing cases: filled_ranges absent, an entry missing end_s, and an empty array. Separately, envelope.dtype meant three things. The contract says it comes from the array-metadata document, but render_overview never reads zarr.json (its own module doc says so) and emitted "int16" on a cache miss and null on a hit, while read_window emits the real value including on a cache hit. It is now defined as the level-0 array's dtype from that array's own zarr.json, null when the answering tool did not read that document, and it explicitly does NOT mean "read on this call" -- chunks_read and bytes_read are for that. So only read_window populates it and render_overview always sends null.
get_events answered dishonestly in three ways. Its "no rows" note was computed on the PRE-filter rows, so a group that exists and has no events answered events: [], total_count: 0, note: null, indistinguishable from a confident "this group has no events". The note is now computed after filtering and names the group it was empty for. It also did not filter by group at all unless the caller named one, while events.parquet is one row per (event, CHANNEL GROUP). A two-group store therefore returned every event twice and reported total_count as double, with no note, while the envelope described only the first group's rates. It now defaults to the first group like render_overview and read_window already do, and says which group answered and what else the recording has. A new two-group fixture makes the doubling visible: the test reads 4 where it used to read 8. And the dropped-row count lived only in the miss path's return value, so it reached the one caller that happened to populate the cache entry; every later call was a hit reporting 0 dropped rows with a total_count that quietly omitted them. The count now travels in the entry (schema version bumped to 2), with a fixture carrying one row eventRowSchema rejects. The tsv fallback's no-rate note now says the file was NOT FETCHED, rather than leaving an empty list that reads as "this recording has no events". Its byte accounting falls back to the body length when content-length is absent, the same thing array-metadata.ts already did, so the figure that would reveal an oversized read cannot silently report 0. index.json is bounded at 24 MiB for an inline parse. Its size is not bounded by anything this code controls: nm000281 publishes 25,253 stores in 12,846,915 bytes, and describe_dataset's own tool description already cites that number as the reason it never reads the file, while the three tools that do read it were unbounded. The bound clears today's largest with headroom, refuses without reading the body when the header is over it (proven by making the body throw if read), and hands over the public URL. read_window's data-driven throws are typed results now. An index whose shard_samples is not an exact multiple of chunk_samples used to escape as an opaque JSON-RPC internal error; it is a fact about someone else's published geometry, so it is a typed refusal, computed before any I/O since the index alone determines it. Same for planShardReads. And a cached shard footer whose entry count disagrees with the geometry is treated as a miss rather than handed on to throw. Removed the events/_stores cache entry: written once per parquet miss, validated by its own schema, read by nothing but a test.
…rver Epic #1065: NEMAR MCP server
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the NEMAR Model Context Protocol (MCP) server to production. Opened
with the
-devNsuffix intact, per the release pipeline: CI owns the strip, thetag and the npm publish.
What this release is
19 commits, 105 files, +21,118 lines, and almost all of it is one epic. The MCP
server runs inside the existing backend Worker and answers at
mcp.nemar.org,anonymous like the rest of the data plane. Six tools, cheapest first:
search_datasets,describe_dataset,list_recordings,get_events,render_overview,read_window. The posture is ADR 0049's: a stateless,recipe-first broker that hands a client what it needs to fetch bytes itself and
decodes signal data only for an opt-in, hard-capped taste.
Plus ADR 0049 (#1292) and ADR 0050 (no WebAssembly in the Worker bundle).
No migrations, so
migrations:d1-checkdoes not apply.Why the deploy matters beyond the code
mcp.nemar.orgis acustom_domain = trueroute, so the hostname does notexist until this deploys. That is also why the four recording-level tools are
unverified so far:
mcp-test.nemar.orgcame up on the dev deploy and everytransport-level check passed against it, but
zarr-test.nemar.org's catalog iscount: 0-- the dev environment has no converted dataset at all -- solist_recordings,get_events,render_overviewandread_windowhave nothingto read there. Production has 625. Verifying them, and writing the docs section
and OSA's client runtime from those real responses, is what happens immediately
after this merges (issue #1297; #1065 stays open until then).
Verified on staging before promoting
Against the live
mcp-test.nemar.org, from the official Python MCP SDK:2026-07-28negotiated.inputSchema;tools/listcarries itscache hint (
ttlMs=86400000,scope=public).GET /mcp405,DELETE /mcp405,OPTIONS /mcp204, descriptorGET /200,a POST with no
Mcp-Method400, a disallowedOrigin403.describe_datasetend to end on an exemplar, with its sandbox DOI, composedcitation, and a
cost_hintthat says conversion has not started.list_recordingson an unconverted dataset refuses with the actualzarr_statusand points atdescribe_dataset, rather than answering an emptylist.
nm000103, a real private dataset, isrefused with the same wording an unknown id gets. No existence leak.
Reviewed as its own diff
The five phase PRs were each reviewed, and then the integrated diff was reviewed
again on #1332 specifically for the seams, which found five criticals. All are
fixed here; the two worth knowing:
render_overviewthrew aRangeErroron its default path for much of thecatalog. Zarr stores a boundary chunk full size and fill-padded, and the
reassembler inferred each chunk's width from its length. Same nominal-stride
versus valid-span trap phase 4 fixed for level 0, one axis over -- and the
fixture encoded the same wrong assumption, which is why the suite was green.
source_commitalone, on the false premisethat a re-conversion always changes it. An engine bump re-converts at an
unchanged HEAD (ADR 0033), so stale
scale[]/offset[]could be servedagainst re-quantized data -- silently wrong physical values -- and stale shard
byte offsets replayed as Range reads where every 206 and length check passes on
the wrong bytes.
Also:
/mcpfell into the token rate-limit bucket on a surface with no auth, twoanonymous reads were unbounded (a 95 MB parquet; a taste that decoded 31 MB to
return 500 KB), and the zod3/zod4 drift test was passing vacuously on phase 4's
entire taste contract. Full write-up in #1332's review comment.
Dependency note, checked rather than assumed: zod 3 stays on its major
(3.25.76). zod 4 arrives only as a
zod4npm alias for the SDK'sregisterTool,so no existing route's validation changed. The
z.record(z.string(), ...)editsin
admin/doi.tsare the equivalent two-argument form.Gates
Epic-branch CI green with the live tier actually running (
unit-pure,api-test,integration-dev,e2e-sandbox,zenodo-sandbox-test), 342 MCPtests,
backend/scripts/mcp-smoke.shgreen under real workerd (mandatory per ADR0050), typecheck and biome clean.