Repository navigation
Phase 7: dev cron safety gates + exemplar fleet - #954
neuromechanist merged 6 commits into
Conversation
…ing' into feature/issue-930-phase7-fleet-e2e
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR954Version: 🤖 Updated on each push. |
Review findings addressed (sonnet code-reviewer)One real gap found, now fixed. Fixed the same way as the other production-only jobs: gated at the call site in Test drift trap removed. The reviewer correctly flagged that hand-copying the cleanup SQL into the test could silently drift. The queries are now exported as Stale config comments fixed. Two comments in Confirmed correct, no change needed: the half-open band Backend suite is now 531 pass / 0 fail. Redeployed to |
E2E run: first exemplar published, two more bugs found and fixed
The DOI is on the sandbox shoulder Bug 1:
|
Fleet complete: all 7 exemplars published, E2E validatedEvery DOI is on the sandbox shoulder Verified across every staging surface: all 7 return HTTP 200 on Cross-cutting features validated on staging, which is the entire point of this environment: Production is provably untouched by the whole run: still v0.8.75, still 760 datasets, zero Known gaps, both cross-repo rather than in this PREnrichment fails for every exemplar because
Both are now recorded with concrete failure evidence in Operational noteThe exemplar tool reads AWS credentials from the ambient environment, and the local session credentials are short-lived (~15 min) and do not auto-refresh. Three exemplars died mid-run on |
bb815a8
into
feature/issue-923-epic-test-staging
Summary
Final phase of epic #923. Enables the daily cron on the dev/staging worker, finalizes the curated exemplar fleet, and stands the staging environment up for the first time.
The headline finding: Phases 1-6 were merged as code but nothing had ever been deployed. The dev worker was running code from 2026-07-06, dev D1 was still at migration 0056, none of the four
-testhostnames resolved, and thenemar-website-testPages project did not exist. Phase 7 therefore had to stand the stack up before it could validate anything.Why the cron needed gating first
Enabling a cron on the dev worker is not a config nicety, because the dev D1 is a partial production mirror: 722 dataset rows (530
ds*, 192 realnm*, 0xx) and 609 users with real email addresses, with a liveRESEND_API_KEYon the worker.[env.dev]had never had a[triggers]block, soscheduled()had never run there. Auditing what the daily branch would have done against those rows:archiveRetrySweepselects onarchive_statusalone with no dataset-id prefix or environment filter, then dispatchesrepository_dispatchto the hardcodednemarDatasets/.github. On mirrored rows that means real Actions runs against real dataset repos.nm*rows directly and sends real Resend mail.applyDevWraponly prefixes[DEV]on the subject; it does not suppress or redirect delivery, so this would have mailed real researchers "your dataset will be deleted in N days".reconcileReservedVersionDoispicks sandbox-vs-production EZID auth from the DOI's own shoulder, not fromENVIRONMENT, so real10.82901DOIs resolve to production credentials. Mitigating fact verified against the live worker: prod EZID credentials are absent from dev (onlyEZID_SANDBOX_*), so it errors rather than mutating a DOI. It is still guarded, because that is a configuration accident away from being live.deleteDatasetCascadescopes S3, D1, and Vectorize by environment, but its GitHub half always targetsnemarDatasets/<id>, so a stray cleanup match deletes a real repository.Changes
Fail-safe allowlist in
scheduled()splits the daily jobs into a set proven safe against a mirror D1 and a production-only set. The comment states that a new daily job is production-only by default, so the next one added does not silently inherit dev execution.scheduledCleanupnon-production narrowing: sandbox deletion is pinned to the dev ephemeral bandxx090001-xx099899(new half-open constants +isDevEphemeralSandboxIdindatasetId.ts), which excludes the curated fleet, prod's sandbox band, and every legacy row; the staleness block and the stuck-import_jobssweep are production-only. Production behavior is unchanged in all three cases.Defense-in-depth guards inside
archive-retry.tsanddoi-reconcile.tsso the guarantee survives a caller that is not this cron.[env.dev.triggers] crons = ["0 4 * * *"], offset from prod's0 3. The gates and the trigger ship in the same bundle, so the cron can never exist without its guards.Fleet spec finalized on live evidence rather than the planned strict smallest-of-modality rule. Checking
store_countatzarr.nemar.org/<id>/zarr/index.jsonshowed the smallest MEG (on004398, CTF) and the smallest iEEG (on006392,.tdat) both report 0 stores, meaning biosigIO cannot convert them and they would never render in the viewer. Substituted the smallest sources that actually convert.on003805on000246on006136nm000155nm000189nm000135on006394Every source was verified public, unarchived, and on
main. Total is roughly 3.6 GB.Infrastructure stood up (all dev-scoped; no production resource touched)
0057_dataset_exemplar.sqlto dev D1.nemar-api-dev; the deploy auto-provisioned the three Workers custom domains and registered the cron.nemar-website-testPages project, deployed the staging build, attachedtest.nemar.org, set a staging-onlySESSION_SECRET, and created thestagingbranch in the website repo.api-test,data-test,zarr-test,test.nemar.org.test.nemar.orgservesX-Robots-Tag: noindex, nofollowand aDisallow: /robots.txt.["0 3 * * *","*/30 * * * *"]and dev is["0 4 * * *"].is_exemplarcolumn yet, which is the strongest form of the Phase 5 prod invariant.Test plan
bun testbackend: 523 pass, 0 fail.backend/test/cron-dev-safety.test.ts(24 tests) runs the actual cleanup SQL against real in-memory SQLite with the full migrated schema, asserting the band spares exemplars, prod-band rows, and realnm/on/dsrows. The environment-gate tests observe whether D1 is reached at all, because both jobs swallow their own query errors and "resolved without throwing" is not evidence the guard fired.nm000000placeholder survives (the value--allsilently skips) and that sources are distinct.tsc --noEmitclean; Biome clean on all changed files.TEST_ADMIN_API_KEY. The one extra (datalad ... save commits new files) passes in isolation, so it is a parallel-execution flake.xx099900created on dev withis_exemplar=1, is_sandbox=1, visibility=private, source=nemar-exemplar, source_id=on003805.Outstanding
exemplar-clone.tsreadsAWS_ACCESS_KEY_ID/SECRETfrom the ambient environment (unlikee2e-test.ts, which fetches per-user S3 credentials from the backend), and the local AWS session has expired.xx099900is left in a resumable partial state: repo and D1 row exist, data not copied. Re-runningcreatetolerates "already exists" and the copy resumes by S3 listing diff.xx09*repos are gated at prod and never reach dev, so the webhook-driven enrichment round-trip is not validated here.CLOUDFLARE_API_TOKEN/CLOUDFLARE_ACCOUNT_IDrepo secrets are added; staging was deployed by hand. Separately, Fix staging deploy: Pages rejects -c config path website#155 fixes a Phase 6 defect found by running it:deploy-test.ymlpassed-c wrangler.test.toml, whichwrangler pages deployrejects outright, so that workflow would have failed on its first run.Closes #930
Part of epic #923