You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
recover/re-copy: finalize not idempotent for already-published datasets (Dataset is already published) #985
Found during the epic #967 recovery canary (on002814, prod, 0.9.1) — before running the 45-batch.
Symptom:nemar admin recover on002814 --execute dispatched onboard-openneuro.yml, which ran cleanly through parse-ids -> prepare -> copy (all 8 shards) -> finalize, where it FAILED:
✔ Verified 44895 objects present
✖ Failed to request publication: Dataset is already published
##[error]Process completed with exit code 1
The re-copy itself WORKED: after the run, nemar admin import verify on002814 reports verified complete (44909/44909 objects present) and the dataset serves HTTP 200 at data.nemar.org. The 0-byte data is fixed.
Root cause:finalizeImport (the onboard-openneuro finalize step) unconditionally calls requestPublication. For a RE-copy of an already-published dataset (every one of the 45 #967 recover targets), that fails with 'Dataset is already published' and process.exit(1). Phase 2 made prepare idempotent for re-dispatch (Step 4b), but the finalize publication path was never made idempotent.
Impact:
The workflow run reports failure even though the data was re-copied (misleading; noisy across a 45-batch).
Any finalize step AFTER requestPublication is skipped on the exit(1) — so datasets.data_complete stays stale/unaudited, and any manifest/archive/records regeneration that lives past that point does NOT run. So the S3 data is fixed but downstream artifacts + the catalog completeness signal are not refreshed. (on002814's data_complete is still unaudited after the run.)
Affects ALL 45 recover targets (all already-published), so the batch would produce 45 'failed' runs with fixed data but stale bookkeeping.
Fix: in the recover/re-copy finalize path, detect an already-published dataset and SKIP (or treat as success) the publication request, then continue to the per-key verify + completeness/manifest updates. A re-copy of a published dataset should be a data-repair, not a re-publish. Add a real test for the already-published re-copy path.
Interim: after a re-copy lands the data, data-integrity-sweep --older-than 0 sets data_complete correctly (independent of finalize), so the catalog signal can be repaired separately.
Blocks the clean batch recovery of the 45. Part of the #967 follow-up set (with #984).
Fixed and shipped: PR #986 (de9ec36, "fix: skip publish request/approve for already-published re-copy"), released in v0.9.2 on 2026-07-22 and on main since.
finalizeImport used to request publication unconditionally, so a #967 recover re-copy of an already-published dataset failed with "Dataset is already published" after it had verified the data was present. src/lib/import-openneuro.ts now decides via decidePublicationRequestAction(visibility) — public returns skip-already-published, anything else returns request — keyed on the dataset's live visibility rather than on assuming a first-time import. Covered in test/import-openneuro.test.ts.
Closing. (PRs merged to dev never auto-close issues here — the default branch is main.)
Activity
Fix prod bugs #984 (version DOI prefix) + #985 (finalize idempotency) (…
Merge pull request #987 from nemarOrg/dev
neuromechanist commentedon Aug 3, 2026
Fixed and shipped: PR #986 (
de9ec36, "fix: skip publish request/approve for already-published re-copy"), released in v0.9.2 on 2026-07-22 and onmainsince.finalizeImportused to request publication unconditionally, so a #967 recover re-copy of an already-published dataset failed with "Dataset is already published" after it had verified the data was present.src/lib/import-openneuro.tsnow decides viadecidePublicationRequestAction(visibility)—publicreturnsskip-already-published, anything else returnsrequest— keyed on the dataset's live visibility rather than on assuming a first-time import. Covered intest/import-openneuro.test.ts.Closing. (PRs merged to
devnever auto-close issues here — the default branch ismain.)