Repository navigation
Phase 5: Remediate recoverable corpus (re-copy the 45 + audit) - #978
Merged
neuromechanist merged 6 commits intoJul 21, 2026
Conversation
Verify (reclassify) + batch-dispatch re-copy for datasets published with 0-byte content whose upstream is accessible; reuses the existing import verify/status endpoints and the onboard-openneuro.yml dispatch (extracted into dispatchOpenNeuroImportWorkflow, shared with import-openneuro). Dry-run by default, mirrors the withdraw/restore command ergonomics.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR978Version: 🤖 Updated on each push. |
Pulls the sourceByTarget/missingSource logic out of the recover command into a pure, unit-tested function (resolved/missing/duplicate-row cases) so the non-null assertion downstream rests on tested behavior, not an inlined filter.
- Dry-run and execute branches now emit pure JSON on stdout under --json (every human console.log gated behind !options.json); recover --json and recover --all --json now pipe cleanly into jq. - Execute path exits non-zero when some (not all) targets fail verify, matching reindex/hed-sweep's partial-bulk-failure convention, so a chained caller doesn't read a partial failure as success. - Wires the extracted resolveImportSources() in for source-id resolution. - Corrects the addHelpText's non-prod claim: the proximate reason --execute fails outside production is a missing import_jobs row, not an environment gate on verify itself (verify is NOT env-gated, so it would reclassify a real row if pointed at the wrong backend).
Same fix as the CLI help text: --execute fails outside production because the target ids have no import_jobs row, not because verify itself is environment-gated. Adds an explicit DANGER note that verify would reclassify a real row on any environment that has one, and a reminder that a quarantined/blocklisted target needs import retry, not just recover's reclassify.
Contributor
Author
Review round complete (4 Sonnet reviewers, parallel)No critical/HIGH. Four Medium/accuracy items, all fixed (3 follow-up commits
Accepted/deferred: pre-existing duplicated mutual-exclusion guard + trivial helpers (unchanged); the stale Independently re-verified: root + backend |
neuromechanist
merged commit Jul 21, 2026
6aaa930
into
feature/issue-967-epic-openneuro-integrity
1 check passed
This was referenced Jul 21, 2026
3 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final phase of epic #967. Ships tooling to re-copy the 45 OpenNeuro datasets whose upstream is still accessible (published with 0-byte content in the incident) via the path hardened in Phases 1-3, plus the checked-in target list. Almost no new logic — the hardened copy, batch re-dispatch, idempotent prepare, and audit all already exist.
This PR does NOT re-copy the 45 real datasets. The re-copy is production-only (
POST /admin/datasets/importand the retry engine both refuse outsideENVIRONMENT=production), so the actual run is a separate, explicitly-gated operational step after the epic deploys. The.context/recover-runbook.mddocuments that sequence.What ships (CLI-only; no backend change, no migration)
scripts/recover-datasets.json— the 45on######targets, disjoint from Phase 4's 11 (45 + 11 = 56, asserted by test).src/lib/recover-datasets.ts— loader +resolveRecoverTargets(--forceguard), mirroring the Phase 4 withdrawn-datasets pattern.nemar admin recover [ids...] [--all] [--execute] [--yes] [--force] [--json]— dry-run by default. On--execute: resolves each target's OpenNeurods######source fromimport_jobsup front (fails loudly if any target has no row — never a silent skip), verifies/reclassifies each target before dispatch (so the sticky-completecallback doesn't leaveimport_jobsstale), then one batchgh workflow run onboard-openneuro.ymlvia the shared dispatch helper (reused fromimport-openneuro).nemar admin recover status— reports each target'sdata_complete/bytes_present(the completion oracle, notimport_jobs.status).nemar admin data-integrity-sweep --older-than 0(documented, no new audit code).Closes #972
Part of epic #967
Test plan
tsc --noEmit: clean. Biome: clean.test/recover-datasets.test.ts: valid JSON, 45 uniqueon######ids, disjoint from the 11,resolveRecoverTargets--forceguard.