Repository navigation
Description
Corrected. The original premise of this issue was wrong.
on008014and the rest of the cohort were deliberately withdrawn, and the withdrawal was executed correctly. I verified the concept DOIs are tombstoned at EZID and the repos are private. Rewritten below to the real defect.
Summary
Datasets that have been deliberately withdrawn are not removed from the Hallu Zarr conversion queue. They keep getting re-attempted, fail (correctly, their data is gone by design), and pollute the conversion failure log.
The withdrawal itself is fine
scripts/withdrawn-datasets.json lists 11 datasets, 9 with the note "OpenNeuro source returns 403 for every client, including OpenNeuro's own CLI/datalad" and 2 with "No recoverable source was found". The prod ops worked:
10.82901/nemar.on008014 -> 302 https://ezid.cdlib.org/tombstone/id/doi:10.82901/NEMAR.ON008014
10.82901/nemar.on007987 -> 302 https://ezid.cdlib.org/tombstone/id/doi:10.82901/NEMAR.ON007987
10.82901/nemar.on008092 -> 302 ...tombstone...
10.82901/nemar.on008099 -> 302 ...tombstone...
10.82901/nemar.on008115 -> 302 ...tombstone...
10.82901/nemar.on008065 -> 302 ...tombstone...
Contrast a healthy dataset: 10.82901/nemar.on008082 -> 302 https://nemar.org/dataset/on008082. Repos are private, and withdrawn_at / withdrawn_reason exist as first-class columns (migration 0060). So the 0-byte content in these datasets is expected, not a leak.
The actual defect
10 of the 11 withdrawn datasets are still rows in zarr-queue.db on Hallu, and 9 of them are pending, meaning they are queued to be attempted again:
on004148 pending 0 on008014 pending 0
on005279 pending 0 on008017 pending 0
on005516 failed 5 on008065 pending 0
on007987 pending 0 on008092 pending 0
on008099 pending 0
on008115 pending 0
(only on007816 is absent, which was the one skipped during the withdrawal ops)
The queue's reconcile step enqueues and recovers rows but never drops a dataset that has been withdrawn, so these persist indefinitely. Each attempt clones, tries to stream objects that are 0-byte by design, fails, and writes failure lines that make the conversion log look worse than it is.
Requested work
-
reconcileshould skip and remove queue rows for datasets withwithdrawn_atset - Expose withdrawal state where the converter can see it (the queue reconcile reads
GET /datasets, so the listing needs to carry it, or offer a withdrawn filter) - Purge the 10 stale rows currently in
zarr-queue.db
Secondary: withdrawal is invisible in the CLI
nemar dataset status on008014 prints Status: active and shows the DOI as a live link, with no indication the dataset is withdrawn. This is what led me to misdiagnose it. withdrawn_at is populated in D1 but never surfaced.
- Show withdrawal state and reason in
nemar dataset status -
nemar admin import statuscounts withdrawn datasets in itscomplete 495total; they should be reported separately
Related: #967 (root incident), #984 (version DOIs not tombstoned, still open), nemarDatasets/.github#88 (converter scope).
Activity
[-]Pre-#967 imports remain active with DOIs while 0-100% of their data is 0-byte[/-][+]Withdrawn datasets are never dropped from the Zarr queue and keep being re-attempted[/+]neuromechanist commentedon Aug 1, 2026
Queue cleaned up (2026-08-01)
Purged the stale rows on Hallu.
zarr-queue.db: 767 -> 755 rows, backup at/mnt/local/zarr-state/zarr-queue.db.bak-20260801T060300Z.Rather than trusting the checked-in lists, I resolved the concept DOI for every one of the 767 queue rows (
xargs -P 24, about a minute):The 12 tombstoned rows, all deleted:
on004148on005279on005516on007987on008014on008017on008065on008092on008099on008115withdrawn-datasets.jsonon003645on004212recover-datasets.json(the limbo pair)That is 10 of the 11 withdrawn-list entries;
on007816was never in the queue, consistent with it being the one skipped during the withdrawal ops.They will not come back on their own:
reconcileenqueues fromGET /datasets, which 404s for all of them. The code fix in this issue is still wanted so this cannot recur, but the immediate noise is gone. 755 now matches the catalog count reconcile reports.Resolving the DOI is the reliable "is this pulled?" test. The lists are not:
on003645andon004212are tombstoned yet sit on the recover list, not the withdrawn list.Follow-up: withdrawn zarr stores are orphaned, but not exposed
Checked whether withdrawn content is still served. It is not:
So withdrawal correctly revoked public read. However the stores were never deleted.
on003645alone still holds 40,446 objects, 8.4 GB unders3://nemar/on003645/zarr/. Five of the twelve hadconverted_version=1.0.0(on003645,on004148,on005279,on005516,on008017), so there is more.s3://nemar/<id>/zarr/for withdrawn datasets, and have the withdraw path remove derived artifacts (zarr, archives) rather than only revoking accessTwo caveats for anyone re-testing: bare prefixes return 404 on
data.nemar.organdzarr.nemar.orgeven for live datasets, so only object-key probes mean anything; andon008082returns 403 despite a live DOI because it is private/incomplete, so 403 alone does not imply withdrawn.neuromechanist commentedon Aug 1, 2026
Completed the residual-storage audit across all 12 purged datasets. Presence of
s3://nemar/<id>/zarr/:This lines up exactly with the queue metadata: the five with residual stores are precisely the five whose deleted rows carried
converted_version=1.0.0. The other seven never completed a conversion, so there is nothing to reclaim.So the cleanup scope is bounded and known: five withdrawn datasets hold orphaned zarr stores, all currently returning 403 to the public, of which only
on003645is measured so far at 8.4 GB.neuromechanist commentedon Aug 1, 2026
Sizes for the orphaned zarr stores, now measured:
on004148on005279on003645on008017on005516So reclaiming derived artifacts for withdrawn datasets is worth roughly 64 GB and three quarters of a million objects from four datasets alone, before
on005516is counted.Worth noting the object counts as much as the bytes:
on004148is 628k objects on its own, which matters for listing cost and for theindex.jsonegress concern in #1035.The useful predictor is already in the queue: the five datasets with residual stores are exactly the five whose rows carried
converted_version=1.0.0, so a withdrawal cleanup does not need to probe S3 blindly.neuromechanist commentedon Aug 1, 2026
Full scan finished.
on005516was the outlier still counting, and it is the largest of the set.Complete orphaned-zarr figures for the 12 purged datasets:
on005516on004148on005279on003645on008017The remaining seven (
on004212,on007987,on008014,on008065,on008092,on008099,on008115) have 0 objects and 0 bytes underzarr/, confirmed, so nothing to reclaim there.Revising my earlier estimate upward: this is 124 GB and 1.73 million objects, not the ~64 GB I posted from the partial scan.
The object count is arguably the more important number here. 1.73M objects of derived data for datasets that are tombstoned and unreachable is a standing listing and lifecycle cost, and it compounds the
index.jsonegress problem in #1035.Both of the "no recoverable source" datasets (
on005516,on005279) are in this set, which is the clearest case for deletion rather than retention: there is no path by which they become servable again.neuromechanist commentedon Aug 23, 2026
Both halves are merged to
dev.Queue (#1131,
e9c0c4a) --reconcilenow parks rows the catalog no longer lists asunlisted, soclaim_nextstops handing them out. Three decisions worth recording:unlisted, notwithdrawn: absence from the public catalog is all the converter can observe, and a dataset returned to private looks identical. Verified the premise directly --api.nemar.org/datasets/on008014returnsDataset not foundwhileon008082returns normally.converted_versionsurvives and a dataset that comes back does not reconvert work we already have.This also covers the "purge the 10 stale rows" item without a manual step: the next reconcile parks them. The "expose withdrawal state to the converter" item turned out to be unnecessary, since the catalog already stops listing withdrawn datasets.
Visibility (#1134,
2b14906) --nemar dataset statusshows aWithdrawn:line with the reason, marks the DOI(tombstoned), and drops the download hint.nemar admin import statusreportswithdrawn Nalongsideby_status(not inside it -- the import genuinely completed, so the row is correctlycomplete).The
Data:line mattered more than expected. It readincomplete ... see nemar admin data-integrity-sweep, pointing an operator at an integrity sweep for content removed on purpose -- the exact false lead this issue opened with. It now readsremoved as part of the withdrawal. That one was caught by running the command against real data, not by the tests.Verified end to end on
on008014against the live API, withon008082as an unchanged control.