Skip to content

Phase 5: exemplar endpoint + CLI clone tool + webhook forwarder (#928) - #948

Merged
neuromechanist merged 5 commits into
feature/issue-923-epic-test-stagingfrom
feature/issue-928-phase5-clone-tool
Jul 8, 2026
Merged

neuromechanist merged 5 commits into
feature/issue-923-epic-test-stagingfrom
feature/issue-928-phase5-clone-tool

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Epic #923, Phase 5. Closes #928.

Adds the machinery to populate and serve the staging exemplar fleet. Actually publishing the curated fleet + dev cron + E2E is Phase 7.

Workstreams

  • A — Backend (routes/admin/exemplar.ts): POST /admin/datasets/exemplar (creates a private nemarDatasets/ repo + D1 row is_exemplar=1, repo-rollback on D1 failure) and POST /admin/datasets/:id/exemplar/remint-dois (idempotent sandbox concept-DOI re-mint). Both 403 in production. Corrected the exemplar-band regex to /^xx0999\d{2}$/. Added a prod-invariant guard in scheduledCleanup that loudly logs any is_exemplar=1 row in prod (closes the Phase 4 review concern). Route-inventory pin 78 -> 81.
  • B — CLI (lib/exemplar-clone.ts + nemar admin exemplar create|status|remint-dois): single-process prepare -> copy -> finalize clone of a public nm/on dataset, modeled on import-openneuro.ts. Scrubs dataset_description.json, cross-bucket server-side S3 copy (nemar -> nemar-dev), git-annex registration with uuid-match guard, CI + reindex + optional sandbox-DOI publish. Starter scripts/exemplar-fleet.json.
  • C — Webhook forwarder (the one config-gated prod deploy): DEV_WEBHOOK_MIRROR_URL binding + prod var; inside the reserved prod dev-range short-circuit, fire-and-forget re-post the raw HMAC-signed delivery to the dev worker via waitUntil. Outbound-only; unset => no-op.
  • D — Dispatch env-awareness: thread s3_bucket/callback_base_url into the manifest/archive/zarr client_payload (backward-compatible; workflows default to prod). Cross-repo .github/emit_manifest.py changes documented for owner deploy in .context/.

Prod safety

Prod no-op except the two deliberate, config-gated additions (DEV_WEBHOOK_MIRROR_URL, additive dispatch fields). Creation/remint 403 in prod; the invariant guard only logs.

Tested

  • Backend bun test: 499 pass, 0 fail (new exemplar-endpoint + forwarder tests; route-inventory pin updated)
  • CLI bun test (touched/affected): 77 pass (pure-helper tests; orchestrator validated E2E per no-mocks policy)
  • tsc --noEmit (backend + root) + Biome clean
  • nemar admin exemplar --help command tree wired

New backend/src/routes/admin/exemplar.ts (registered in the admin barrel):
- POST /admin/datasets/exemplar creates a private nemarDatasets/<xx> repo + a D1
  row (is_exemplar=1, is_sandbox=1, visibility='private', source='nemar-exemplar')
  with repo-rollback on D1 failure; 403 in production (staging-only fleet).
- POST /admin/datasets/:id/exemplar/remint-dois idempotently re-mints the concept
  DOI on the sandbox shoulder; 403 in production.

Corrected the exemplar-band regex to /^xx0999\d{2}$/ (8-char xx099900-xx099999;
the plan draft's /^xx099\d{2}$/ was a digit short). Added a prod-invariant guard
to scheduledCleanup that loudly logs if any is_exemplar=1 row ever exists in
production (the Phase 4 visibility carve-outs have no runtime env check). Updated
the admin route-inventory pin (78 -> 81). New endpoint tests (freshDb): gate/
validation/rollback paths. Backend suite 496 pass; tsc + biome clean.
Forwarder (the one substantive prod deploy, config-gated): add
DEV_WEBHOOK_MIRROR_URL binding + prod [vars] value; inside the reserved prod
dev-range short-circuit in /webhooks/github, fire-and-forget re-post the raw,
still-HMAC-signed delivery (body + signature/event/delivery headers) to the dev
worker via executionCtx.waitUntil. Outbound-only; unset => no-op; a dev outage
never affects prod. Tests assert forward fires only for prod + dev-range + var
set, never on the dev worker.

Dispatch env-awareness: thread s3_bucket (env.S3_BUCKET) into the manifest/
archive/zarr repository_dispatch client_payloads, plus callback_base_url
(env.API_BASE_URL) for archive/zarr (manifest already builds callback_url from
API_BASE_URL). Undefined => omitted from JSON => the central workflows keep their
prod default, so this is backward-compatible. Backend 499 pass; tsc + biome clean.
New src/lib/exemplar-clone.ts: single-process prepare -> copy -> finalize clone
of a public nm/on dataset into a staging exemplar (xx099900-xx099999), modeled
on import-openneuro.ts. prepare clones the source, scrubs
dataset_description.json (Name -> "[TEST COPY]", drops DatasetDOI), createExemplar,
disables the inherited prod nemar-s3 remote, configures a fresh nemar-s3-dev
remote, pushes. copy does resumable cross-bucket server-side S3 copy
(nemar/<src>/objects -> nemar-dev/<xx>/objects), --include-derived also copies
zarr/archives/records.json. finalize verifies, registers keys against
nemar-s3-dev (uuid-match guard), pushes git-annex, addCi, waits for BIDS
validation, reindexes, and (--publish) requests + approves with a sandbox DOI.

Adds nemar admin exemplar create|status|remint-dois (driven by a starter
scripts/exemplar-fleet.json), createExemplar/remintExemplarDois API clients, and
pure-helper tests (scrub, key-rewrite, fleet-parse). Reuses two existing helpers
by widening their surface (backward-compatible): markInheritedOpenNeuroRemotesIgnored
gains an optional remoteNames param; waitForBidsValidationRun + PollOutcome are
exported. Root tsc + biome clean; 77 CLI tests pass.
@github-actions

github-actions Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR948

Version: 0.8.76-PR948.38f4ef0
Commit: 38f4ef0


🤖 Updated on each push.

Silent-failure review (5 findings, all fixed):
- CLI clone tool: replace all 19 process.exit(1) in prepare/copy/finalize with
  throw, so `nemar admin exemplar create --all`'s per-entry catch actually
  continues past a failing fleet entry instead of the first failure killing the
  whole batch (matters for the Phase 7 fleet publish).
- CLI: refuse to publish when the metadata reindex did not fully succeed
  (reindexOk gate before requestPublication) — a failed refresh could otherwise
  publish an exemplar still carrying the source dataset's real .nemar/metadata.json.
- CLI `exemplar status`: distinguish a 404 ("not created") from 401/429/500
  instead of an empty catch masking every error as "not created".
- Webhook forwarder: log a non-2xx response from the dev mirror (fetch().catch
  only fires on network errors, so a secret-drift 401 / 404 / 5xx was silently
  swallowed with a false forwarded:true).
- Prod-invariant guard: escalate an is_exemplar=1-in-production violation to an
  admin email alert (new sendExemplarInvariantAlertEmail) instead of a passive
  daily log line.

Backend 499 pass; CLI tests pass; tsc + biome clean (backend + root).
@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review (Sonnet: correctness + silent-failure)

Correctness reviewer: no bugs. Independently ran tests + tsc + Biome and verified the D1 INSERT columns, the fail-closed 403 gates (no prod path reaches the exemplar INSERT), the triple-gated forwarder (c.executionCtx only touched inside the var-set branch), the byte-identical-on-prod dispatch fields, and the CLI key-naming pipeline + uuid-drift guard.

Silent-failure reviewer: 5 findings, all fixed (38f4ef0):

  • HIGH — --all batch loop was defeated by process.exit() in the clone tool (bypassed the per-entry catch → first failure killed the whole fleet run). Converted all 19 exit sites to throw; the CLI action's existing per-entry catch now correctly continues and reports "N of M failed". Matters for the Phase 7 fleet publish.
  • HIGH — reindex failure was non-fatal but --publish proceeded anyway → could publish an exemplar still carrying the source's real .nemar/metadata.json. Now refuses to publish unless the metadata reindex fully succeeded.
  • HIGH — webhook forward's .catch only caught network errors, not non-2xx → silent forward failure with a false forwarded: true. Added a .then status-code log.
  • MEDIUM — empty catch in exemplar status masked 401/429/500 as "not created". Now distinguishes 404.
  • MEDIUM — prod-invariant guard was a passive daily log. Now escalates to an admin email alert (reusing the staleness escalation path).

Below-threshold note (no change): the scrub clears the structured DatasetDOI/Name in dataset_description.json, but a source README.md mentioning the real DOI in prose persists until the finalize-phase reindex regenerates it — a pre-existing metadata-pipeline tradeoff on a private staging repo, and it doesn't touch the "no resolvable source DOI" invariant on the structured fields.

Verification: backend 499 pass, CLI tests pass, tsc (backend + root) + Biome clean.

@neuromechanist
neuromechanist merged commit 9932669 into feature/issue-923-epic-test-staging Jul 8, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-928-phase5-clone-tool branch July 8, 2026 00:22

This branch was previously deployed

1 inactive deployment
npm — 38f4ef05 Deployed Jul 8, 2026 by neuromechanist via publish #1724
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant