Repository navigation
Phase 5: exemplar endpoint + CLI clone tool + webhook forwarder (#928) - #948
Conversation
New backend/src/routes/admin/exemplar.ts (registered in the admin barrel):
- POST /admin/datasets/exemplar creates a private nemarDatasets/<xx> repo + a D1
row (is_exemplar=1, is_sandbox=1, visibility='private', source='nemar-exemplar')
with repo-rollback on D1 failure; 403 in production (staging-only fleet).
- POST /admin/datasets/:id/exemplar/remint-dois idempotently re-mints the concept
DOI on the sandbox shoulder; 403 in production.
Corrected the exemplar-band regex to /^xx0999\d{2}$/ (8-char xx099900-xx099999;
the plan draft's /^xx099\d{2}$/ was a digit short). Added a prod-invariant guard
to scheduledCleanup that loudly logs if any is_exemplar=1 row ever exists in
production (the Phase 4 visibility carve-outs have no runtime env check). Updated
the admin route-inventory pin (78 -> 81). New endpoint tests (freshDb): gate/
validation/rollback paths. Backend suite 496 pass; tsc + biome clean.
Forwarder (the one substantive prod deploy, config-gated): add DEV_WEBHOOK_MIRROR_URL binding + prod [vars] value; inside the reserved prod dev-range short-circuit in /webhooks/github, fire-and-forget re-post the raw, still-HMAC-signed delivery (body + signature/event/delivery headers) to the dev worker via executionCtx.waitUntil. Outbound-only; unset => no-op; a dev outage never affects prod. Tests assert forward fires only for prod + dev-range + var set, never on the dev worker. Dispatch env-awareness: thread s3_bucket (env.S3_BUCKET) into the manifest/ archive/zarr repository_dispatch client_payloads, plus callback_base_url (env.API_BASE_URL) for archive/zarr (manifest already builds callback_url from API_BASE_URL). Undefined => omitted from JSON => the central workflows keep their prod default, so this is backward-compatible. Backend 499 pass; tsc + biome clean.
New src/lib/exemplar-clone.ts: single-process prepare -> copy -> finalize clone of a public nm/on dataset into a staging exemplar (xx099900-xx099999), modeled on import-openneuro.ts. prepare clones the source, scrubs dataset_description.json (Name -> "[TEST COPY]", drops DatasetDOI), createExemplar, disables the inherited prod nemar-s3 remote, configures a fresh nemar-s3-dev remote, pushes. copy does resumable cross-bucket server-side S3 copy (nemar/<src>/objects -> nemar-dev/<xx>/objects), --include-derived also copies zarr/archives/records.json. finalize verifies, registers keys against nemar-s3-dev (uuid-match guard), pushes git-annex, addCi, waits for BIDS validation, reindexes, and (--publish) requests + approves with a sandbox DOI. Adds nemar admin exemplar create|status|remint-dois (driven by a starter scripts/exemplar-fleet.json), createExemplar/remintExemplarDois API clients, and pure-helper tests (scrub, key-rewrite, fleet-parse). Reuses two existing helpers by widening their surface (backward-compatible): markInheritedOpenNeuroRemotesIgnored gains an optional remoteNames param; waitForBidsValidationRun + PollOutcome are exported. Root tsc + biome clean; 77 CLI tests pass.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR948Version: 🤖 Updated on each push. |
Silent-failure review (5 findings, all fixed):
- CLI clone tool: replace all 19 process.exit(1) in prepare/copy/finalize with
throw, so `nemar admin exemplar create --all`'s per-entry catch actually
continues past a failing fleet entry instead of the first failure killing the
whole batch (matters for the Phase 7 fleet publish).
- CLI: refuse to publish when the metadata reindex did not fully succeed
(reindexOk gate before requestPublication) — a failed refresh could otherwise
publish an exemplar still carrying the source dataset's real .nemar/metadata.json.
- CLI `exemplar status`: distinguish a 404 ("not created") from 401/429/500
instead of an empty catch masking every error as "not created".
- Webhook forwarder: log a non-2xx response from the dev mirror (fetch().catch
only fires on network errors, so a secret-drift 401 / 404 / 5xx was silently
swallowed with a false forwarded:true).
- Prod-invariant guard: escalate an is_exemplar=1-in-production violation to an
admin email alert (new sendExemplarInvariantAlertEmail) instead of a passive
daily log line.
Backend 499 pass; CLI tests pass; tsc + biome clean (backend + root).
Review (Sonnet: correctness + silent-failure)Correctness reviewer: no bugs. Independently ran tests + tsc + Biome and verified the D1 INSERT columns, the fail-closed 403 gates (no prod path reaches the exemplar INSERT), the triple-gated forwarder ( Silent-failure reviewer: 5 findings, all fixed (
Below-threshold note (no change): the scrub clears the structured Verification: backend 499 pass, CLI tests pass, tsc (backend + root) + Biome clean. |
9932669
into
feature/issue-923-epic-test-staging
Epic #923, Phase 5. Closes #928.
Adds the machinery to populate and serve the staging exemplar fleet. Actually publishing the curated fleet + dev cron + E2E is Phase 7.
Workstreams
routes/admin/exemplar.ts):POST /admin/datasets/exemplar(creates a private nemarDatasets/ repo + D1 rowis_exemplar=1, repo-rollback on D1 failure) andPOST /admin/datasets/:id/exemplar/remint-dois(idempotent sandbox concept-DOI re-mint). Both 403 in production. Corrected the exemplar-band regex to/^xx0999\d{2}$/. Added a prod-invariant guard inscheduledCleanupthat loudly logs anyis_exemplar=1row in prod (closes the Phase 4 review concern). Route-inventory pin 78 -> 81.lib/exemplar-clone.ts+nemar admin exemplar create|status|remint-dois): single-process prepare -> copy -> finalize clone of a public nm/on dataset, modeled onimport-openneuro.ts. Scrubsdataset_description.json, cross-bucket server-side S3 copy (nemar -> nemar-dev), git-annex registration with uuid-match guard, CI + reindex + optional sandbox-DOI publish. Starterscripts/exemplar-fleet.json.DEV_WEBHOOK_MIRROR_URLbinding + prod var; inside the reserved prod dev-range short-circuit, fire-and-forget re-post the raw HMAC-signed delivery to the dev worker viawaitUntil. Outbound-only; unset => no-op.s3_bucket/callback_base_urlinto the manifest/archive/zarrclient_payload(backward-compatible; workflows default to prod). Cross-repo.github/emit_manifest.pychanges documented for owner deploy in.context/.Prod safety
Prod no-op except the two deliberate, config-gated additions (
DEV_WEBHOOK_MIRROR_URL, additive dispatch fields). Creation/remint 403 in prod; the invariant guard only logs.Tested
bun test: 499 pass, 0 fail (new exemplar-endpoint + forwarder tests; route-inventory pin updated)bun test(touched/affected): 77 pass (pure-helper tests; orchestrator validated E2E per no-mocks policy)tsc --noEmit(backend + root) + Biome cleannemar admin exemplar --helpcommand tree wired