Skip to content

Phase 7: dev cron safety gates + exemplar fleet - #954

Merged
neuromechanist merged 6 commits into
feature/issue-923-epic-test-stagingfrom
feature/issue-930-phase7-fleet-e2e
Jul 18, 2026
Merged

neuromechanist merged 6 commits into
feature/issue-923-epic-test-stagingfrom
feature/issue-930-phase7-fleet-e2e

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Final phase of epic #923. Enables the daily cron on the dev/staging worker, finalizes the curated exemplar fleet, and stands the staging environment up for the first time.

The headline finding: Phases 1-6 were merged as code but nothing had ever been deployed. The dev worker was running code from 2026-07-06, dev D1 was still at migration 0056, none of the four -test hostnames resolved, and the nemar-website-test Pages project did not exist. Phase 7 therefore had to stand the stack up before it could validate anything.

Why the cron needed gating first

Enabling a cron on the dev worker is not a config nicety, because the dev D1 is a partial production mirror: 722 dataset rows (530 ds*, 192 real nm*, 0 xx) and 609 users with real email addresses, with a live RESEND_API_KEY on the worker. [env.dev] had never had a [triggers] block, so scheduled() had never run there. Auditing what the daily branch would have done against those rows:

  • archiveRetrySweep selects on archive_status alone with no dataset-id prefix or environment filter, then dispatches repository_dispatch to the hardcoded nemarDatasets/.github. On mirrored rows that means real Actions runs against real dataset repos.
  • The staleness warn/notify block matches nm* rows directly and sends real Resend mail. applyDevWrap only prefixes [DEV] on the subject; it does not suppress or redirect delivery, so this would have mailed real researchers "your dataset will be deleted in N days".
  • reconcileReservedVersionDois picks sandbox-vs-production EZID auth from the DOI's own shoulder, not from ENVIRONMENT, so real 10.82901 DOIs resolve to production credentials. Mitigating fact verified against the live worker: prod EZID credentials are absent from dev (only EZID_SANDBOX_*), so it errors rather than mutating a DOI. It is still guarded, because that is a configuration accident away from being live.
  • deleteDatasetCascade scopes S3, D1, and Vectorize by environment, but its GitHub half always targets nemarDatasets/<id>, so a stray cleanup match deletes a real repository.

Changes

Fail-safe allowlist in scheduled() splits the daily jobs into a set proven safe against a mirror D1 and a production-only set. The comment states that a new daily job is production-only by default, so the next one added does not silently inherit dev execution.

scheduledCleanup non-production narrowing: sandbox deletion is pinned to the dev ephemeral band xx090001-xx099899 (new half-open constants + isDevEphemeralSandboxId in datasetId.ts), which excludes the curated fleet, prod's sandbox band, and every legacy row; the staleness block and the stuck-import_jobs sweep are production-only. Production behavior is unchanged in all three cases.

Defense-in-depth guards inside archive-retry.ts and doi-reconcile.ts so the guarantee survives a caller that is not this cron.

[env.dev.triggers] crons = ["0 4 * * *"], offset from prod's 0 3. The gates and the trigger ship in the same bundle, so the cron can never exist without its guards.

Fleet spec finalized on live evidence rather than the planned strict smallest-of-modality rule. Checking store_count at zarr.nemar.org/<id>/zarr/index.json showed the smallest MEG (on004398, CTF) and the smallest iEEG (on006392, .tdat) both report 0 stores, meaning biosigIO cannot convert them and they would never render in the viewer. Substituted the smallest sources that actually convert.

Slot Role Source Size Zarr stores
xx099900 EEG baseline on003805 16 MB none upstream
xx099901 MEG on000246 2343 MB 3
xx099902 iEEG on006136 285 MB 14
xx099903 EMG nm000155 447 MB 11
xx099904 HED 8.4.0 nm000189 20 MB 20
xx099905 EEG native nm000135 22 MB 5
xx099906 multi-modal on006394 521 MB 60

Every source was verified public, unarchived, and on main. Total is roughly 3.6 GB.

Infrastructure stood up (all dev-scoped; no production resource touched)

  • Applied 0057_dataset_exemplar.sql to dev D1.
  • Deployed this branch to nemar-api-dev; the deploy auto-provisioned the three Workers custom domains and registered the cron.
  • Created the nemar-website-test Pages project, deployed the staging build, attached test.nemar.org, set a staging-only SESSION_SECRET, and created the staging branch in the website repo.
  • All four hosts verified live (HTTP 200): api-test, data-test, zarr-test, test.nemar.org.
  • test.nemar.org serves X-Robots-Tag: noindex, nofollow and a Disallow: / robots.txt.
  • Confirmed prod cron is still ["0 3 * * *","*/30 * * * *"] and dev is ["0 4 * * *"].
  • Prod D1 does not even have the is_exemplar column yet, which is the strongest form of the Phase 5 prod invariant.

Test plan

  • bun test backend: 523 pass, 0 fail.
  • New backend/test/cron-dev-safety.test.ts (24 tests) runs the actual cleanup SQL against real in-memory SQLite with the full migrated schema, asserting the band spares exemplars, prod-band rows, and real nm/on/ds rows. The environment-gate tests observe whether D1 is reached at all, because both jobs swallow their own query errors and "resolved without throwing" is not evidence the guard fired.
  • Fleet tests assert no nm000000 placeholder survives (the value --all silently skips) and that sources are distinct.
  • tsc --noEmit clean; Biome clean on all changed files.
  • CLI suite compared against an epic-branch baseline under an identical isolated config: 50 pre-existing failures both sides, all from missing TEST_ADMIN_API_KEY. The one extra (datalad ... save commits new files) passes in isolation, so it is a parallel-execution flake.
  • Exemplar endpoint exercised live: xx099900 created on dev with is_exemplar=1, is_sandbox=1, visibility=private, source=nemar-exemplar, source_id=on003805.
  • Fleet data copy and publish are not yet done (see below).

Outstanding

  • Blocked on AWS credentials. exemplar-clone.ts reads AWS_ACCESS_KEY_ID/SECRET from the ambient environment (unlike e2e-test.ts, which fetches per-user S3 credentials from the backend), and the local AWS session has expired. xx099900 is left in a resumable partial state: repo and D1 row exist, data not copied. Re-running create tolerates "already exists" and the copy resumes by S3 listing diff.
  • Prod webhook forwarder deliberately deferred to epic finalization, per the epic plan; it is the one substantive production deploy. Until it ships, pushes to xx09* repos are gated at prod and never reach dev, so the webhook-driven enrichment round-trip is not validated here.
  • Website CI deploys stay blocked until CLOUDFLARE_API_TOKEN/CLOUDFLARE_ACCOUNT_ID repo secrets are added; staging was deployed by hand. Separately, Fix staging deploy: Pages rejects -c config path website#155 fixes a Phase 6 defect found by running it: deploy-test.yml passed -c wrangler.test.toml, which wrangler pages deploy rejects outright, so that workflow would have failed on its first run.

Closes #930
Part of epic #923

@github-actions

github-actions Bot commented Jul 18, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR954

Version: 0.8.76-PR954.de18150
Commit: de18150


🤖 Updated on each push.

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review findings addressed (sonnet code-reviewer)

One real gap found, now fixed. sweepBlockedBidsValidationRequests (section 5 of scheduledCleanup) was the remaining daily-cron path that still reached real production records outside production, and my original audit missed it. Its candidate query filters on request status alone with no dataset-id prefix restriction, so against the prod-mirror dev D1 it would select real datasets' publication requests, read their real repos through the shared nemarDatasets App installation (burning that token's rate limit daily), and rewrite publication_requests.status for them, drifting the mirror from prod's real state. That directly contradicted the "safe outside prod" comment I had written one screen above it.

Fixed the same way as the other production-only jobs: gated at the call site in index.ts, plus a defense-in-depth guard inside publication-sweep.ts so a future caller inherits it, plus gate tests in both directions. My inaccurate scheduled() comment now enumerates exactly which scheduledCleanup sections run outside production.

Test drift trap removed. The reviewer correctly flagged that hand-copying the cleanup SQL into the test could silently drift. The queries are now exported as NON_PROD_SANDBOX_CLEANUP_QUERY / PROD_SANDBOX_CLEANUP_QUERY from index.ts and imported by the test, matching the ARCHIVE_RETRY_SWEEP_QUERY pattern already used elsewhere in this codebase. The test now asserts against the real SQL rather than a copy.

Stale config comments fixed. Two comments in wrangler-sccn.toml asserted "dev has no cron triggers", which this PR made false. They now state that dev's triggers carry only 0 4 * * *, which never matches AUTO_IMPORT_CRON, so auto-import still cannot fire in dev. Documentation drift rather than a safety gap, but a future reader would have been reasoning from a false premise.

Confirmed correct, no change needed: the half-open band ["xx090001", "xx099900") and its string comparison (ids are fixed-width, and nm/on/ds sort below xx09); isNonProductionEnv fail-closed polarity at every new call site; byte-identical production behavior; variable scoping across the brace restructuring; and that the probe-based gate tests genuinely observe the guard rather than passing on the swallowed query error.

Backend suite is now 531 pass / 0 fail. Redeployed to nemar-api-dev (v0.8.76-dev5) so the running staging worker carries the sweep gate, since the cron is live.

@neuromechanist

Copy link
Copy Markdown
Contributor Author

E2E run: first exemplar published, two more bugs found and fixed

xx099900 (from on003805) is published end to end on staging, verified in D1 rather than trusting the CLI's exit status:

dataset_id  visibility  is_exemplar  concept_doi              latest_version_doi              pub_status
xx099900    public      1            10.5072/fk2xx099900      10.5072/fk2xx099900.v1.0.0      published

The DOI is on the sandbox shoulder 10.5072/FK2 as designed, never the production 10.82901. All three staging surfaces serve it: it appears in the api-test catalog (confirming the Phase 4 is_exemplar visibility carve-outs actually work), data-test serves v1.0.0 with its manifest, and test.nemar.org renders the page titled "[TEST COPY] Multisensory Gamma Entrainment" (confirming the dataset_description.json scrub).

Bug 1: annex-uuid marker copied and registered as a git-annex key (fixed here)

The first real run failed finalize with 1 of 3 git-annex key registrations failed. Listing <src>/objects/ returns the S3 special remote's annex-uuid marker alongside the content blobs, and it was being both copied into the dev bucket and handed to batchSetKeysPresent. It is not annexed content, and copying it would also overwrite the freshly-initremoted nemar-s3-dev's own identity. Now excluded via an exported isAnnexContentKey predicate, with a regression test. After the fix the same dataset registered cleanly: copied 2, skipped 0 then Registered 2 files in git-annex.

Bug 2: my own gate would have stranded staging publishes (fixed here)

Gating sweepBlockedBidsValidationRequests to production, as the review suggested and I first implemented, was too blunt. An exemplar requested while its BIDS validation is still running lands in blocked (which is exactly what happened to xx099900), and that sweep is what later unblocks it. Production-only would have left every staging publish stuck forever.

Corrected to scope by dataset range instead of skipping: outside production the candidate query adds AND pr.dataset_id LIKE 'xx09%', the same fence the sandbox cleanup uses. Staging keeps the sweep it needs, and it still cannot touch the real nm/on rows in the mirror. Tests assert both directions against real SQLite.

Separately filed

#955: exemplar creation is not retry-safe after a partial failure. Prepare always re-clones the source into a fresh tmpdir, so a retry builds a new git-annex UUID that cannot claim the existing annex-uuid marker, and then fails to push a diverged git-annex branch. The only reliable recovery today is delete-dataset then recreate, which is what I did. The command's own error messages advise a plain re-run, which does not work.

Backend suite is 525 pass / 0 fail. The remaining six exemplars are running now.

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Fleet complete: all 7 exemplars published, E2E validated

xx099900  public  10.5072/fk2xx099900  hed=0  eeg        <- on003805
xx099901  public  10.5072/fk2xx099901  hed=0  anat,meg   <- on000246
xx099902  public  10.5072/fk2xx099902  hed=0  ieeg       <- on006136
xx099903  public  10.5072/fk2xx099903  hed=0  emg        <- nm000155
xx099904  public  10.5072/fk2xx099904  hed=1  eeg        <- nm000189
xx099905  public  10.5072/fk2xx099905  hed=1  eeg        <- nm000135
xx099906  public  10.5072/fk2xx099906  hed=0  beh,eeg    <- on006394

Every DOI is on the sandbox shoulder 10.5072/FK2, never the production 10.82901.

Verified across every staging surface: all 7 return HTTP 200 on test.nemar.org/dataset/<id> rendering "[TEST COPY] ..." titles, all 7 serve versions and manifests on data-test, all 7 appear in the api-test catalog (so the Phase 4 is_exemplar visibility carve-outs work end to end), and zarr-test serves xx099904's store index (20 stores, 0 failures) plus individual store metadata, matching prod's behavior for the same source.

Cross-cutting features validated on staging, which is the entire point of this environment: ?has_hed=1 returns exactly ["xx099905","xx099904"], the two HED sources and nothing else (#869). License classification is correct across CC0 / CC-BY-ND-4.0 / CC-BY-NC-ND-4.0 (#653). Modality detection produced eeg, ieeg, emg, anat,meg, and beh,eeg from the real BIDS trees.

Production is provably untouched by the whole run: still v0.8.75, still 760 datasets, zero xx ids in the prod catalog, and zero xx0999* objects in the prod bucket. The dev worker's bucket isolation was also confirmed independently: a mirrored row like nm000103 reports versions: [] on data-test while prod serves its real v2.0.0.

Known gaps, both cross-repo rather than in this PR

Enrichment fails for every exemplar because run-enrichment.yml hardcodes https://api.nemar.org/webhooks/llm-enrich, so staging-dispatched runs call production, which answers 404 Dataset not found. That is the Phase 1 prod-safety gate behaving exactly as designed, but it means the central-workflow enrichment path cannot be exercised on staging until callback_base_url ships. The CLI's inline reindex is a separate path and does work, which is why metadata, modalities and has_hed are populated correctly regardless.

run-generate-archive.yml still hardcodes s3://nemar, so archive generation must stay disabled for staging until it reads s3_bucket; data-test reports archive: null for every exemplar. Nothing was dispatched, so nothing was written to the prod bucket.

Both are now recorded with concrete failure evidence in .context/phase5-cross-repo-owner-deploys.md.

Operational note

The exemplar tool reads AWS credentials from the ambient environment, and the local session credentials are short-lived (~15 min) and do not auto-refresh. Three exemplars died mid-run on ExpiredToken. Run them one at a time with a fresh eval "$(aws configure export-credentials --format env)" immediately before each. One further failure was a transient GitHub Connection reset by peer, which per #955 required a delete-and-recreate rather than a retry.

@neuromechanist
neuromechanist merged commit bb815a8 into feature/issue-923-epic-test-staging Jul 18, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-930-phase7-fleet-e2e branch July 18, 2026 09:26

This branch was previously deployed

1 inactive deployment
npm — de181502 Deployed Jul 18, 2026 by neuromechanist via publish #1737
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant