Skip to content

Phase 5: Remediate recoverable corpus (re-copy the 45 + audit) - #978

Merged
neuromechanist merged 6 commits into
feature/issue-967-epic-openneuro-integrityfrom
feature/issue-972-phase5-remediate
Jul 21, 2026
Merged

neuromechanist merged 6 commits into
feature/issue-967-epic-openneuro-integrityfrom
feature/issue-972-phase5-remediate

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Final phase of epic #967. Ships tooling to re-copy the 45 OpenNeuro datasets whose upstream is still accessible (published with 0-byte content in the incident) via the path hardened in Phases 1-3, plus the checked-in target list. Almost no new logic — the hardened copy, batch re-dispatch, idempotent prepare, and audit all already exist.

This PR does NOT re-copy the 45 real datasets. The re-copy is production-only (POST /admin/datasets/import and the retry engine both refuse outside ENVIRONMENT=production), so the actual run is a separate, explicitly-gated operational step after the epic deploys. The .context/recover-runbook.md documents that sequence.

What ships (CLI-only; no backend change, no migration)

  • scripts/recover-datasets.json — the 45 on###### targets, disjoint from Phase 4's 11 (45 + 11 = 56, asserted by test).
  • src/lib/recover-datasets.ts — loader + resolveRecoverTargets (--force guard), mirroring the Phase 4 withdrawn-datasets pattern.
  • nemar admin recover [ids...] [--all] [--execute] [--yes] [--force] [--json] — dry-run by default. On --execute: resolves each target's OpenNeuro ds###### source from import_jobs up front (fails loudly if any target has no row — never a silent skip), verifies/reclassifies each target before dispatch (so the sticky-complete callback doesn't leave import_jobs stale), then one batch gh workflow run onboard-openneuro.yml via the shared dispatch helper (reused from import-openneuro).
  • nemar admin recover status — reports each target's data_complete/bytes_present (the completion oracle, not import_jobs.status).
  • Acceptance audit is the existing nemar admin data-integrity-sweep --older-than 0 (documented, no new audit code).

Closes #972
Part of epic #967

Test plan

  • Root + backend tsc --noEmit: clean. Biome: clean.
  • CI pure tier (exact classifier): 2426 pass / 0 fail across 173 files.
  • test/recover-datasets.test.ts: valid JSON, 45 unique on###### ids, disjoint from the 11, resolveRecoverTargets --force guard.
  • The actual re-copy dispatch/verify is prod-only and not runnable here (same constraint as every admin dispatch path); it's the post-deploy operational step.

45 checked-in on###### targets whose OpenNeuro upstream is accessible,
disjoint from the 11 withdrawn in Phase 4 (45 + 11 = 56). Mirrors
withdrawn-datasets.json/parseWithdrawnDatasets exactly.
Verify (reclassify) + batch-dispatch re-copy for datasets published with
0-byte content whose upstream is accessible; reuses the existing import
verify/status endpoints and the onboard-openneuro.yml dispatch (extracted
into dispatchOpenNeuroImportWorkflow, shared with import-openneuro). Dry-run
by default, mirrors the withdraw/restore command ergonomics.
Documents the new CLI commands in AGENTS.md and the post-deploy operational
sequence (recover --all --execute -> recover status -> data-integrity-sweep
--older-than 0) as a gated operator step, not run in this PR.
@github-actions

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR978

Version: 0.9.1-PR978.ad6f0de
Commit: ad6f0de


🤖 Updated on each push.

Pulls the sourceByTarget/missingSource logic out of the recover command
into a pure, unit-tested function (resolved/missing/duplicate-row cases)
so the non-null assertion downstream rests on tested behavior, not an
inlined filter.
- Dry-run and execute branches now emit pure JSON on stdout under --json
  (every human console.log gated behind !options.json); recover --json
  and recover --all --json now pipe cleanly into jq.
- Execute path exits non-zero when some (not all) targets fail verify,
  matching reindex/hed-sweep's partial-bulk-failure convention, so a
  chained caller doesn't read a partial failure as success.
- Wires the extracted resolveImportSources() in for source-id resolution.
- Corrects the addHelpText's non-prod claim: the proximate reason
  --execute fails outside production is a missing import_jobs row, not
  an environment gate on verify itself (verify is NOT env-gated, so it
  would reclassify a real row if pointed at the wrong backend).
Same fix as the CLI help text: --execute fails outside production
because the target ids have no import_jobs row, not because verify
itself is environment-gated. Adds an explicit DANGER note that verify
would reclassify a real row on any environment that has one, and a
reminder that a quarantined/blocklisted target needs import retry, not
just recover's reclassify.
@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review round complete (4 Sonnet reviewers, parallel)

No critical/HIGH. Four Medium/accuracy items, all fixed (3 follow-up commits 59abd0b..fd6af0b):

  • recover --json is now pure JSON on stdout (human text gated behind !options.json; JSON printed after; matches recover status/withdraw). Verified recover --all --json | jq parses.
  • Partial verify failure exits non-zero (if (failedTargets.length > 0) process.exit(1) after the output, mirroring reindex/hed-sweep) so CI/chained callers don't read a partial batch as success. JSON still emits on partial failure.
  • resolveImportSources extracted + unit-tested (the pure source-id map + missing-source diff the non-null assertion depends on): 6 new tests (present / empty / absent / duplicate-last-wins / non-target / mixed).
  • Corrected the misattributed non-prod claim in the help text + runbook: --execute fails against non-prod because the target on###### ids have no import_jobs row there (dev D1 is exemplars-only) — not because of the two prod-gated endpoints, which aren't in the synchronous path. Added that verify/reclassify is NOT env-gated (so only ever point --execute at production) and that a quarantined/blocklisted target needs import retry, not recover.

Accepted/deferred: pre-existing duplicated mutual-exclusion guard + trivial helpers (unchanged); the stale onboard-openneuro.yml top-of-file comment (out of scope — different repo, not touched; tracked separately).

Independently re-verified: root + backend tsc clean, Biome clean, CI pure tier 2432 pass / 0 fail across 173 files. The actual re-copy run of the 45 is the post-deploy operational step in .context/recover-runbook.md.

This branch was previously deployed

1 inactive deployment
npm — fd6af0bf Deployed Jul 21, 2026 by neuromechanist via publish #1786
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant