Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,41 @@ Environments and pre-release checks: [`.context/release-safety-playbook.md`](.co
normally throughout. The cohort stranded before the stamp existed
(directory-format datasets converted before 2026-08-22, #1172) is recovered by
`hallu-zarr.sh --backfill-dir-formats`, dry-run by default.
**Verification is a standing sweep, not a conversion-time gate alone**
(issue #1068, epic #1181 phase 8): the converter's own
`channel_count_mismatch` check withholds an unfaithful store when it is
built, but `has_zarr` KEEPS ITS EXISTING MEANING (converted: `zarr_status
= 'ready'` with at least one store) so no caller of `?has_zarr=1` sees a
silently narrower result set. `has_zarr_verified` is the new, separate,
stricter filter: converted AND the fidelity sweep's last verdict was
`verified`. `backend/src/services/zarr-fidelity-sweep.ts`'s
`runZarrFidelitySweep` re-derives ground truth per sampled recording
(`channels.tsv` row count, sidecar `SamplingFrequency`/`RecordingDuration`)
from the dataset's own git-tracked BIDS metadata via the public,
credential-free `raw.githubusercontent.com` content host (never the
GitHub API/App/PAT, which is what makes it safe on the non-prod cron too
-- `DEV_CRON_ALLOWLIST` in index.ts) -- candidates are restricted to
`status='active' AND visibility='public'` for exactly that reason: a
private repo can't be read anonymously, so it would only ever produce
`unverifiable` noise. It FAILS OPEN ON THE ROW, never on the verdict: a
transient infra error (a non-2xx that isn't 404, a network throw, or
either fetch budget running out mid-dataset -- a sweep-wide 600 and a
per-dataset 90, both index plus sidecar fetches) aborts that one
dataset's verification for the run -- nothing is stamped, it lands in the
run's `errors`, and the row stays a candidate; only a clean 404 at every
nearest-first candidate path is real absence, and only a value that
actually parsed counts as checked. Stamps `zarr_verify_status`
(`verified` / `failed` / `unverifiable`) plus `zarr_verified_at` into
`sweep_stamps` (ADR 0034/0035 -- no new column); a re-conversion (a
changed `zarr_source_commit`) OR a stamped commit that is null (a fixed
fossilisation bug -- the write side now stamps `''`, never JSON `null`,
but the candidate predicate also re-arms on a null stamp so an
already-fossilised row un-sticks too) re-arms verification. A fresh
conversion shows `zarr_verify_status: null` until the daily sweep (plus
`POST /admin/datasets/zarr-fidelity-sweep`, `nemar admin
zarr-fidelity-sweep`) reaches it; the viewer keeps reading index.json
regardless (ADR 0005 -- verification is reported, never a precondition
for serving).

---

Expand Down Expand Up @@ -410,7 +445,7 @@ and what is historical. The entries worth knowing by name:
| `nemar dataset publish` | request, status, resend |
| `nemar dataset` (access) | request-access, access, invite, collaborators |
| `nemar sandbox` | training run, status, reset — required before uploading |
| `nemar admin` | users, approve, revoke, role, notify, s3, repo, ci, doi, publish, revert, make-public, delete-dataset, bulk-delete, reindex, hed-sweep, data-integrity-sweep, doctor, summary, notice, email-preferences, e2e-test |
| `nemar admin` | users, approve, revoke, role, notify, s3, repo, ci, doi, publish, revert, make-public, delete-dataset, bulk-delete, reindex, hed-sweep, data-integrity-sweep, zarr-fidelity-sweep, doctor, summary, notice, email-preferences, e2e-test |
| `nemar admin import*` | OpenNeuro import, status, rollback, retry, verify, recover (issue #754, epic #967) |
| `nemar admin fleet` | drift, enforce, revalidate — governance across dataset repos (epic #713) |
| `nemar admin exemplar` | create, status, remint-dois — the staging exemplar fleet |
Expand Down
36 changes: 35 additions & 1 deletion backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ import {
warningStageForDaysLeft,
} from "./services/staleness";
import { publishZarrCatalog } from "./services/zarr-catalog";
import { runZarrFidelitySweep } from "./services/zarr-fidelity-sweep";
import type { Bindings, Variables } from "./types/bindings";

// Create the API app with all routes
Expand Down Expand Up @@ -293,10 +294,17 @@ export const PROD_SANDBOX_CLEANUP_QUERY =
* against the shared `nemarDatasets` org, or mutate a real DOI or the PROD
* bucket -- `publishZarrCatalog` (services/zarr-catalog.ts) qualifies
* because it only ever reads/writes the CALLING env's own D1 and S3 bucket.
* `runZarrFidelitySweep` (services/zarr-fidelity-sweep.ts, issue #1068,
* epic #1181 phase 8) qualifies for the same reason PLUS one more: its
* dataset-repo reads go through the public, credential-free
* `raw.githubusercontent.com` content host, never the GitHub API/App/PAT --
* see that module's doc comment for why that is a materially different,
* dev-safe exposure from `signal-defaults-sweep.ts`'s GitHub-API-based
* `getBidsTreeStats` (which IS prod-only, in the guarded block below).
* `backend/test/cron-sweep-wiring.test.ts` pins that every name here is
* both actually called and called outside the guarded block.
*/
export const DEV_CRON_ALLOWLIST = ["publishZarrCatalog"] as const;
export const DEV_CRON_ALLOWLIST = ["publishZarrCatalog", "runZarrFidelitySweep"] as const;

async function scheduledCleanup(env: Bindings): Promise<void> {
const db = env.DB;
Expand Down Expand Up @@ -982,5 +990,31 @@ export default {
),
),
);
// Issue #1068 (epic #1181 phase 8): the standing Zarr fidelity
// verification sweep. ALLOWED ON THE NON-PROD CRON -- listed in
// DEV_CRON_ALLOWLIST above: it only reads this env's own S3 bucket plus
// the public, credential-free raw.githubusercontent.com content host,
// and writes only this env's own D1 -- no email, no GitHub App/PAT
// dispatch against the shared nemarDatasets org, no DOI mutation.
ctx.waitUntil(
runZarrFidelitySweep(env)
.then((r) => {
const failedIds = r.results
.filter((d) => d.verdict === "failed")
.map((d) => d.dataset_id);
console.log(
`[zarr-fidelity-sweep] processed=${r.processed} verified=${r.verified} failed=${r.failed} unverifiable=${r.unverifiable} errors=${r.errors.length} remaining=${r.remaining ?? "?"}${failedIds.length > 0 ? ` failedDatasets=${failedIds.join(",")}` : ""}`,
);
for (const e of r.errors) {
console.error(`[zarr-fidelity-sweep] ${e.dataset_id}: ${e.error}`);
}
})
.catch((err) =>
console.error(
"[zarr-fidelity-sweep] sweep failed:",
err instanceof Error ? (err.stack ?? err.message) : err,
),
),
);
},
};
2 changes: 2 additions & 0 deletions backend/src/routes/admin/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import type { AdminRouter } from "./shared";
import { registerUsersRoutes } from "./users";
import { registerWithdrawRoutes } from "./withdraw";
import { registerZarrCatalogRoutes } from "./zarr-catalog";
import { registerZarrFidelitySweepRoutes } from "./zarr-fidelity-sweep";

export const adminRoutes: AdminRouter = new Hono();

Expand All @@ -37,3 +38,4 @@ registerExemplarRoutes(adminRoutes);
registerNoticeRoutes(adminRoutes);
registerWithdrawRoutes(adminRoutes);
registerZarrCatalogRoutes(adminRoutes);
registerZarrFidelitySweepRoutes(adminRoutes);
80 changes: 80 additions & 0 deletions backend/src/routes/admin/zarr-fidelity-sweep.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
/**
* Admin route: on-demand Zarr fidelity verification sweep (issue #1068,
* epic #1181 phase 8).
*
* Same shape as `routes/admin/zarr-catalog.ts` (issue #1062, phase 2): the
* daily cron (backend/src/index.ts) is the regular driver, and this route
* exists for an operator who wants an immediate batch without waiting for
* the next tick. One implementation (`runZarrFidelitySweep`,
* services/zarr-fidelity-sweep.ts) shared by both callers so they cannot
* drift.
*/

import {
ZARR_FIDELITY_SWEEP_DEFAULT,
ZARR_FIDELITY_SWEEP_MAX,
runZarrFidelitySweep,
} from "../../services/zarr-fidelity-sweep";
import type { AdminRouter } from "./shared";

/**
* `deps.sweep` defaults to the real `runZarrFidelitySweep` and exists so
* `backend/test/zarr-fidelity-sweep-route.test.ts` can register this exact
* route against a fresh app with the S3/GitHub-raw boundaries substituted
* (real local `Bun.serve()` receivers) -- the same DI-seam idiom
* `registerZarrCatalogRoutes` uses for `publishZarrCatalog`.
* `registerZarrFidelitySweepRoutes(adminRoutes)` (routes/admin/index.ts)
* omits it, so production always resolves the default.
*/
export function registerZarrFidelitySweepRoutes(
admin: AdminRouter,
deps: { sweep?: typeof runZarrFidelitySweep } = {},
): void {
const sweep = deps.sweep ?? runZarrFidelitySweep;

/**
* POST /admin/datasets/zarr-fidelity-sweep?limit=N
*
* Bounded batch (default {@link ZARR_FIDELITY_SWEEP_DEFAULT}, max
* {@link ZARR_FIDELITY_SWEEP_MAX}) of the fidelity sweep. Fails loud with a
* 500 on a candidate-query error (e.g. migration 0073 not applied);
* per-dataset failures are collected in the response body's `errors`
* instead of throwing (see `runZarrFidelitySweep`'s doc comment).
*
* PR #1203 review, item 6: a candidate QUERY failure (thrown) is a 500 as
* before, but a call that reached D1 fine and processed at least one
* candidate, only to have EVERY one of them error out (zero real
* verdicts), answers 502 with `ok: false` -- a 200 there would read as
* "the sweep ran cleanly and verified nothing", which is not what
* happened. A partial result (some verdicts, some errors) is still a
* successful sweep and stays 200 `ok: true`; so does an empty candidate
* set (`processed: 0`) -- there was nothing to fail.
*/
admin.post("/datasets/zarr-fidelity-sweep", async (c) => {
const limitRaw = Number.parseInt(c.req.query("limit") || "", 10);
const limit = Number.isFinite(limitRaw) ? limitRaw : undefined;

try {
const result = await sweep(c.env, { limit });
const totalFailure = result.processed > 0 && result.errors.length === result.processed;
if (totalFailure) {
return c.json(
{
...result,
ok: false,
error: `All ${result.processed} processed candidate(s) errored; see errors[]`,
},
502,
);
}
return c.json({ ...result, ok: true });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
console.error("[zarr-fidelity-sweep] candidate query failed:", msg);
return c.json(
{ error: `Failed to query sweep candidates (is migration 0073 applied?): ${msg}` },
500,
);
}
});
}
25 changes: 23 additions & 2 deletions backend/src/routes/datasets/catalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,12 @@ const FACET_PROJECTION_COLUMNS = `d.subject_count,
d.zarr_failure_count,
d.zarr_deterministic,
d.zarr_failed_at,
-- Issue #1068 (epic #1181 phase 8): the standing fidelity
-- verification sweep's verdict, derived from the JSON
-- sweep_stamps column (ADR 0034/0035 -- no new column).
-- Null until the sweep reaches this dataset.
json_extract(d.sweep_stamps, '$.zarr_verify_status') AS zarr_verify_status,
json_extract(d.sweep_stamps, '$.zarr_verified_at') AS zarr_verified_at,
d.total_recording_duration,
d.recording_duration_min,
d.recording_duration_max,
Expand Down Expand Up @@ -318,7 +324,8 @@ export function parseMinScore(c: FilterQueryContext): number {

/**
* Parse every filter query param -- the legacy bespoke filters (nine as of
* epic #1144 phase 3, #1147, D6; has_zarr added by issue #1062) plus the
* epic #1144 phase 3, #1147, D6; has_zarr added by issue #1062;
* has_zarr_verified added by issue #1068, epic #1181 phase 8) plus the
* full facet table -- into one
* `DatasetFilterOptions`, shared by both catalog endpoints so they can no
* longer drift apart on which filters they honour. Before this, `GET
Expand Down Expand Up @@ -354,6 +361,13 @@ export function parseFilterQuery(
// review, item 8: documented here because it is easy to assume `=false`
// means "exclude zarr datasets" when it silently means "no filter".
const hasZarr = c.req.query("has_zarr") === "1" || c.req.query("has_zarr") === "true";
// #1068 (epic #1181 phase 8): same `1`/`true` convention as has_zarr, and
// deliberately a SEPARATE param rather than a stricter value for
// `has_zarr` -- see dataset-filters.ts's `hasZarrVerified` doc comment for
// why `has_zarr`'s existing meaning must not narrow under callers already
// relying on it.
const hasZarrVerified =
c.req.query("has_zarr_verified") === "1" || c.req.query("has_zarr_verified") === "true";
// #970: same `1`/`true` convention.
const dataComplete =
c.req.query("data_complete") === "1" || c.req.query("data_complete") === "true";
Expand All @@ -374,6 +388,7 @@ export function parseFilterQuery(
hasDoi,
hasHed,
hasZarr,
hasZarrVerified,
dataComplete,
recent,
licenseTiers,
Expand Down Expand Up @@ -627,7 +642,8 @@ export function registerCatalogRoutes(datasetRoutes: DatasetsRouter): void {
* - Admin: all datasets (including private managed datasets)
*
* Filter params: modality, author, task, has_doi, recent, sort, search, owner,
* license, data_complete, has_hed, has_zarr, include_unknown, and the facet table
* license, data_complete, has_hed, has_zarr, has_zarr_verified, include_unknown,
* and the facet table
* (epic #1144 phase 3, #1147) -- see shared/facets.ts for the full list of
* facet query params (subjects, channels, sessions, size, files, citations,
* duration, recording_length, recordings, unavailable, age, rate,
Expand Down Expand Up @@ -1133,6 +1149,11 @@ export function registerCatalogRoutes(datasetRoutes: DatasetsRouter): void {
ORDER BY created_at DESC
LIMIT 1
) AS latest_version,
-- Issue #1068 (epic #1181 phase 8): same derivation as the list
-- projection above (FACET_PROJECTION_COLUMNS) -- d.* alone does not
-- surface a json_extract expression.
json_extract(d.sweep_stamps, '$.zarr_verify_status') AS zarr_verify_status,
json_extract(d.sweep_stamps, '$.zarr_verified_at') AS zarr_verified_at,
u.username as owner_username,
u.github_username as owner_github
FROM datasets d
Expand Down
19 changes: 19 additions & 0 deletions backend/src/services/channel-montage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,25 @@ export function parsePowerLineFrequency(content: string): number | null {
return v === 50 || v === 60 ? v : null;
}

/**
* Read `RecordingDuration` (seconds) from a `*_eeg.json`/`*_ieeg.json`/
* `*_meg.json`/`*_emg.json` sidecar for the zarr fidelity sweep (issue
* #1068, epic #1181 phase 8) -- ground truth for `duration_mismatch`.
* Same shape as parseSamplingFrequency: null on absent/invalid JSON, a
* non-number, a non-finite value, or <= 0 (a recording cannot have zero or
* negative duration).
*/
export function parseRecordingDuration(content: string): number | null {
let obj: Record<string, unknown>;
try {
obj = JSON.parse(content) as Record<string, unknown>;
} catch {
return null;
}
const v = obj.RecordingDuration;
return typeof v === "number" && Number.isFinite(v) && v > 0 ? v : null;
}

/**
* Reject BIDS's "not applicable" placeholder and blank strings. Shared by
* the two free-text sidecar parsers below; not exported -- an external
Expand Down
19 changes: 19 additions & 0 deletions backend/src/services/dataset-filters.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,15 @@ export interface DatasetFilterOptions {
* verified" rather than redefining this one out from under existing
* callers (PR #1201 review, item 9). */
hasZarr?: boolean;
/** Only datasets whose Zarr copy has PASSED the standing fidelity
* verification sweep (issue #1068, epic #1181 phase 8):
* `has_zarr AND zarr_verify_status = 'verified'`. A strict narrowing of
* `hasZarr` above, not a replacement for it -- a fresh conversion is
* `has_zarr=true` with `zarr_verify_status` still null until the daily
* sweep (services/zarr-fidelity-sweep.ts) reaches it; the viewer keeps
* reading index.json regardless (ADR 0005). Same nullable-safe
* bespoke-filter shape as `hasZarr`, not a facet-table entry. */
hasZarrVerified?: boolean;
dataComplete?: boolean;
recent?: number;
licenseTiers?: LicenseTier[];
Expand Down Expand Up @@ -179,6 +188,16 @@ export function buildDatasetFilterClauses(
// idx_datasets_zarr_status (migration 0035/0071) -- no new index needed.
clauses += " AND d.zarr_status = 'ready' AND COALESCE(d.zarr_store_count, 0) > 0";
}
if (opts.hasZarrVerified) {
// #1068: has_zarr's predicate ANDed with the sweep's stamped verdict.
// json_extract returns SQL NULL for a never-swept row (sweep_stamps NULL
// or the key absent/JSON-null -- ADR 0035), which never equals the
// string literal, so an unswept or non-'verified' row is excluded, same
// as has_hed/has_zarr's own NULL-excludes-by-default convention.
clauses +=
" AND d.zarr_status = 'ready' AND COALESCE(d.zarr_store_count, 0) > 0" +
" AND json_extract(d.sweep_stamps, '$.zarr_verify_status') = 'verified'";
}
if (opts.dataComplete) {
// #970: same nullable-safe idiom as has_hed -- `= 1` excludes both 0
// (checked, incomplete) and NULL (not audited yet). Backed by
Expand Down
1 change: 1 addition & 0 deletions backend/src/services/dataset-search.ts
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,7 @@ export function hasActiveFilters(filters: DatasetFilterOptions): boolean {
filters.hasDoi ||
filters.hasHed ||
filters.hasZarr ||
filters.hasZarrVerified ||
filters.dataComplete ||
(filters.recent && filters.recent > 0) ||
(filters.licenseTiers && filters.licenseTiers.length > 0) ||
Expand Down
19 changes: 18 additions & 1 deletion backend/src/services/zarr-catalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,16 @@ export interface ZarrCatalogSourceRow {
zarr_converted_at: string | null;
zarr_source_commit: string | null;
zarr_errors: number | null;
/** Issue #1068 (epic #1181 phase 8): the standing fidelity verification
* sweep's verdict/timestamp, projected from the JSON sweep_stamps column
* by the candidate SQL below (ADR 0034/0035 -- no new column). Null on a
* freshly-converted dataset the sweep has not reached yet; the catalog
* population itself is UNCHANGED (still every has_zarr row) -- a
* consumer that wants only verified entries filters on
* `zarr_verify_status` client-side or calls the API with
* `has_zarr_verified=1`. */
zarr_verify_status: string | null;
zarr_verified_at: string | null;
}

export interface ZarrCatalogDataset {
Expand All @@ -94,6 +104,9 @@ export interface ZarrCatalogDataset {
zarr_converted_at: string | null;
zarr_source_commit: string | null;
zarr_errors: number | null;
/** Issue #1068 (epic #1181 phase 8). See {@link ZarrCatalogSourceRow}. */
zarr_verify_status: string | null;
zarr_verified_at: string | null;
index_url: string;
}

Expand All @@ -119,7 +132,9 @@ export const ZARR_CATALOG_CANDIDATE_SQL = `SELECT
d.dataset_id, d.name, d.concept_doi, d.license, d.modalities, d.tasks,
d.subject_count, d.has_hed, d.hed_version, d.zarr_status, d.zarr_store_count,
d.recording_count, d.recordings_unavailable, d.total_recording_duration,
d.zarr_converted_at, d.zarr_source_commit, d.zarr_errors
d.zarr_converted_at, d.zarr_source_commit, d.zarr_errors,
json_extract(d.sweep_stamps, '$.zarr_verify_status') AS zarr_verify_status,
json_extract(d.sweep_stamps, '$.zarr_verified_at') AS zarr_verified_at
FROM datasets d
WHERE d.status = 'active' AND d.visibility = 'public'
AND d.zarr_status = 'ready' AND COALESCE(d.zarr_store_count, 0) > 0
Expand Down Expand Up @@ -172,6 +187,8 @@ export function buildZarrCatalog(
zarr_converted_at: row.zarr_converted_at,
zarr_source_commit: row.zarr_source_commit,
zarr_errors: row.zarr_errors,
zarr_verify_status: row.zarr_verify_status,
zarr_verified_at: row.zarr_verified_at,
index_url: `${base}${row.dataset_id}/zarr/index.json`,
}));
return {
Expand Down
Loading
Loading