Skip to content

Withdrawn datasets are never dropped from the Zarr queue and keep being re-attempted #1048

Closed

Description

@neuromechanist

Corrected. The original premise of this issue was wrong. on008014 and the rest of the cohort were deliberately withdrawn, and the withdrawal was executed correctly. I verified the concept DOIs are tombstoned at EZID and the repos are private. Rewritten below to the real defect.

Summary

Datasets that have been deliberately withdrawn are not removed from the Hallu Zarr conversion queue. They keep getting re-attempted, fail (correctly, their data is gone by design), and pollute the conversion failure log.

The withdrawal itself is fine

scripts/withdrawn-datasets.json lists 11 datasets, 9 with the note "OpenNeuro source returns 403 for every client, including OpenNeuro's own CLI/datalad" and 2 with "No recoverable source was found". The prod ops worked:

10.82901/nemar.on008014 -> 302 https://ezid.cdlib.org/tombstone/id/doi:10.82901/NEMAR.ON008014
10.82901/nemar.on007987 -> 302 https://ezid.cdlib.org/tombstone/id/doi:10.82901/NEMAR.ON007987
10.82901/nemar.on008092 -> 302 ...tombstone...
10.82901/nemar.on008099 -> 302 ...tombstone...
10.82901/nemar.on008115 -> 302 ...tombstone...
10.82901/nemar.on008065 -> 302 ...tombstone...

Contrast a healthy dataset: 10.82901/nemar.on008082 -> 302 https://nemar.org/dataset/on008082. Repos are private, and withdrawn_at / withdrawn_reason exist as first-class columns (migration 0060). So the 0-byte content in these datasets is expected, not a leak.

The actual defect

10 of the 11 withdrawn datasets are still rows in zarr-queue.db on Hallu, and 9 of them are pending, meaning they are queued to be attempted again:

on004148  pending  0      on008014  pending  0
on005279  pending  0      on008017  pending  0
on005516  failed   5      on008065  pending  0
on007987  pending  0      on008092  pending  0
                          on008099  pending  0
                          on008115  pending  0

(only on007816 is absent, which was the one skipped during the withdrawal ops)

The queue's reconcile step enqueues and recovers rows but never drops a dataset that has been withdrawn, so these persist indefinitely. Each attempt clones, tries to stream objects that are 0-byte by design, fails, and writes failure lines that make the conversion log look worse than it is.

Requested work

  • reconcile should skip and remove queue rows for datasets with withdrawn_at set
  • Expose withdrawal state where the converter can see it (the queue reconcile reads GET /datasets, so the listing needs to carry it, or offer a withdrawn filter)
  • Purge the 10 stale rows currently in zarr-queue.db

Secondary: withdrawal is invisible in the CLI

nemar dataset status on008014 prints Status: active and shows the DOI as a live link, with no indication the dataset is withdrawn. This is what led me to misdiagnose it. withdrawn_at is populated in D1 but never surfaced.

  • Show withdrawal state and reason in nemar dataset status
  • nemar admin import status counts withdrawn datasets in its complete 495 total; they should be reported separately

Related: #967 (root incident), #984 (version DOIs not tombstoned, still open), nemarDatasets/.github#88 (converter scope).

Activity

  1. changed the title [-]Pre-#967 imports remain active with DOIs while 0-100% of their data is 0-byte[/-] [+]Withdrawn datasets are never dropped from the Zarr queue and keep being re-attempted[/+] on Aug 1, 2026
  2. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    ContributorAuthor

    Queue cleaned up (2026-08-01)

    Purged the stale rows on Hallu. zarr-queue.db: 767 -> 755 rows, backup at /mnt/local/zarr-state/zarr-queue.db.bak-20260801T060300Z.

    Rather than trusting the checked-in lists, I resolved the concept DOI for every one of the 767 queue rows (xargs -P 24, about a minute):

    755 LIVE   12 TOMB   0 ambiguous
    

    The 12 tombstoned rows, all deleted:

    id queue status list
    on004148 on005279 on005516 on007987 on008014 on008017 on008065 on008092 on008099 on008115 8 pending, 2 failed withdrawn-datasets.json
    on003645 on004212 1 failed, 1 pending recover-datasets.json (the limbo pair)

    That is 10 of the 11 withdrawn-list entries; on007816 was never in the queue, consistent with it being the one skipped during the withdrawal ops.

    They will not come back on their own: reconcile enqueues from GET /datasets, which 404s for all of them. The code fix in this issue is still wanted so this cannot recur, but the immediate noise is gone. 755 now matches the catalog count reconcile reports.

    Resolving the DOI is the reliable "is this pulled?" test. The lists are not: on003645 and on004212 are tombstoned yet sit on the recover list, not the withdrawn list.

    Follow-up: withdrawn zarr stores are orphaned, but not exposed

    Checked whether withdrawn content is still served. It is not:

    live      on007990/zarr/index.json -> 200
    withdrawn on003645/zarr/index.json -> 403
    withdrawn on008014/zarr/index.json -> 403
    

    So withdrawal correctly revoked public read. However the stores were never deleted. on003645 alone still holds 40,446 objects, 8.4 GB under s3://nemar/on003645/zarr/. Five of the twelve had converted_version=1.0.0 (on003645, on004148, on005279, on005516, on008017), so there is more.

    • Reclaim s3://nemar/<id>/zarr/ for withdrawn datasets, and have the withdraw path remove derived artifacts (zarr, archives) rather than only revoking access

    Two caveats for anyone re-testing: bare prefixes return 404 on data.nemar.org and zarr.nemar.org even for live datasets, so only object-key probes mean anything; and on008082 returns 403 despite a live DOI because it is private/incomplete, so 403 alone does not imply withdrawn.

  3. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    ContributorAuthor

    Completed the residual-storage audit across all 12 purged datasets. Presence of s3://nemar/<id>/zarr/:

    on003645  present   (40,446 objects, 8.4 GB)
    on004148  present
    on005279  present
    on005516  present
    on008017  present
    on004212  none      on008014  none      on008092  none
    on007987  none      on008065  none      on008099  none
                                            on008115  none
    

    This lines up exactly with the queue metadata: the five with residual stores are precisely the five whose deleted rows carried converted_version=1.0.0. The other seven never completed a conversion, so there is nothing to reclaim.

    So the cleanup scope is bounded and known: five withdrawn datasets hold orphaned zarr stores, all currently returning 403 to the public, of which only on003645 is measured so far at 8.4 GB.

  4. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    ContributorAuthor

    Sizes for the orphaned zarr stores, now measured:

    dataset objects size
    on004148 628,104 38.30 GB
    on005279 37,711 8.85 GB
    on003645 40,446 8.40 GB
    on008017 60,380 8.11 GB
    on005516 (still counting, large HBN dataset)
    subtotal (4 of 5) 766,641 63.66 GB

    So reclaiming derived artifacts for withdrawn datasets is worth roughly 64 GB and three quarters of a million objects from four datasets alone, before on005516 is counted.

    Worth noting the object counts as much as the bytes: on004148 is 628k objects on its own, which matters for listing cost and for the index.json egress concern in #1035.

    The useful predictor is already in the queue: the five datasets with residual stores are exactly the five whose rows carried converted_version=1.0.0, so a withdrawal cleanup does not need to probe S3 blindly.

  5. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    ContributorAuthor

    Full scan finished. on005516 was the outlier still counting, and it is the largest of the set.

    Complete orphaned-zarr figures for the 12 purged datasets:

    dataset objects size withdrawal reason
    on005516 968,241 60.74 GB no recoverable source
    on004148 628,104 38.30 GB upstream 403
    on005279 37,711 8.85 GB no recoverable source
    on003645 40,446 8.40 GB recover-list (limbo)
    on008017 60,380 8.11 GB upstream 403
    TOTAL 1,734,882 124.40 GB

    The remaining seven (on004212, on007987, on008014, on008065, on008092, on008099, on008115) have 0 objects and 0 bytes under zarr/, confirmed, so nothing to reclaim there.

    Revising my earlier estimate upward: this is 124 GB and 1.73 million objects, not the ~64 GB I posted from the partial scan.

    The object count is arguably the more important number here. 1.73M objects of derived data for datasets that are tombstoned and unreachable is a standing listing and lifecycle cost, and it compounds the index.json egress problem in #1035.

    Both of the "no recoverable source" datasets (on005516, on005279) are in this set, which is the clearest case for deletion rather than retention: there is no path by which they become servable again.

  6. neuromechanist commented on Aug 23, 2026

    @neuromechanist
    ContributorAuthor

    Both halves are merged to dev.

    Queue (#1131, e9c0c4a) -- reconcile now parks rows the catalog no longer lists as unlisted, so claim_next stops handing them out. Three decisions worth recording:

    • Named unlisted, not withdrawn: absence from the public catalog is all the converter can observe, and a dataset returned to private looks identical. Verified the premise directly -- api.nemar.org/datasets/on008014 returns Dataset not found while on008082 returns normally.
    • Parked, not deleted, so converted_version survives and a dataset that comes back does not reconvert work we already have.
    • The sweep is gated on a verified-complete catalog read, since a short paginated fetch would otherwise be read as "these datasets are gone" and park the whole queue.

    This also covers the "purge the 10 stale rows" item without a manual step: the next reconcile parks them. The "expose withdrawal state to the converter" item turned out to be unnecessary, since the catalog already stops listing withdrawn datasets.

    Visibility (#1134, 2b14906) -- nemar dataset status shows a Withdrawn: line with the reason, marks the DOI (tombstoned), and drops the download hint. nemar admin import status reports withdrawn N alongside by_status (not inside it -- the import genuinely completed, so the row is correctly complete).

    The Data: line mattered more than expected. It read incomplete ... see nemar admin data-integrity-sweep, pointing an operator at an integrity sweep for content removed on purpose -- the exact false lead this issue opened with. It now reads removed as part of the withdrawal. That one was caught by running the command against real data, not by the tests.

    Verified end to end on on008014 against the live API, with on008082 as an unchanged control.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions