Skip to content

Import failure: on004212 (ds004212) #81

Open

Description

@neuromechanist

Dataset: on004212 | OpenNeuro source: ds004212
Missing objects: 12395/26620 (at declared size, per nemar admin import verify)
Failure class: needs-triage

What happened: part of the epic #967 (nemarOrg/nemar-cli) OpenNeuro 0-byte-content recovery. This dataset did not reach a complete state. The hardened finalize gate correctly did NOT publish incomplete data — the dataset retains its real (partial) state; nothing empty was published.

Triage action: ~47% missing after copy — large. Likely substantial upstream-403; verify. If mostly 403, treat like the upstream-403 class (leave incomplete or withdraw).

Refs: epic nemarOrg/nemar-cli#967; recovery run followed .context/recover-runbook.md. data_complete=0 is the honest catalog state until resolved.

Activity

  1. added
    import-failureOpenNeuro import/recovery failed for a dataset
    needs-triageNo cause could be determined from the reported error; classify by hand from the run
    on Jul 22, 2026
  2. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    MemberAuthor

    Triage data from the Hallu Zarr backfill, since this is still needs-triage.

    The S3 data is not missing, it is genuinely 1 byte upstream. Checking every annex object under s3://nemar/on004212/objects/ and comparing the actual object size against the size encoded in its own git-annex key (SHA256E-s<BYTES>--<hash>):

    • 9,742 annex objects
    • 0 size mismatches, 0 zero-byte objects

    So this dataset passes every integrity check we have. The problem is that 468 recordings resolve to key SHA256E-s1--36a9e7f1..., that is, the canonical content is a single byte, and all 468 point at the same object.

    Zarr conversion consequently fails for 468 recordings with a download/read error, and the dataset contributes 516 unique failing recordings overall (the remaining 16 are No continuous raw recording).

    This is a distinct failure mode from the 0-byte cohort (on008014 and friends, nemarOrg/nemar-cli#1048): there the S3 objects are truncated to 0 bytes and disagree with their keys, so integrity checks catch them. Here the byte-for-byte content is internally consistent and simply hollow, so nothing flags it.

    Worth confirming whether upstream ds004212 actually serves 1-byte files for these paths, or whether the 1-byte content was baked in at import time and then hashed, which would make the key itself wrong.

    Also note this dataset has no .nemar/availability-report.json, so it is invisible to the completeness sweep from epic #999.

  3. neuromechanist commented on Aug 1, 2026

    @neuromechanist
    MemberAuthor

    Follow-up correcting my previous comment: on004212 has already been pulled. I had treated it as a live dataset with mysterious 1-byte content; it is not live.

    Current state:

    on004212 on003645
    concept DOI tombstoned at EZID tombstoned at EZID
    repo visibility private private
    public API 404 404
    availability report none 169/822 files, 51.2% bytes
    on withdrawn-datasets.json no no
    on recover-datasets.json yes yes

    The inconsistency worth resolving: these two are on the recover list, meaning "published with 0-byte content in the epic #967 incident; OpenNeuro source is accessible", so the intent is to re-copy and restore them. But their DOIs are tombstoned, which is the withdrawal end-state.

    That combination is an outlier, not the norm. Checking all 45 entries on the recover list:

    tombstoned: 2   live: 43
    

    Only on004212 and on003645 are tombstoned. The other 43 recover-list datasets kept live DOIs while awaiting re-copy. So these two are in a limbo state: too withdrawn to serve, but not on the withdrawn list, and still nominally queued for recovery.

    Worth deciding explicitly which list each belongs on:

    • if recoverable, restore the DOI after re-copy (admin restore exists)
    • if not, move them to withdrawn-datasets.json so the tombstone matches the bookkeeping

    Also note on004212 has no .nemar/availability-report.json, so it is invisible to the epic #999 completeness sweep and to any decision made from those reports. on003645 does have one.

    Both are still live rows in the Hallu Zarr queue (on003645 as failed after 5 attempts, on004212 still being attempted), which is the same stale-queue problem tracked in nemarOrg/nemar-cli#1048.

    Cross-ref: #69 tracks on003645.

  4. neuromechanist commented on Sep 16, 2026

    @neuromechanist
    MemberAuthor

    Re-measured on the correct basis: 39.0% of data keys missing, not 28.2% (#1396, ADR 0064)

    Re-measured against the bucket on 2026-09-16:

    on004212: 7,495 of 19,220 distinct annexed data keys have no object. 61.0% of its data is available.

    The 28.2% figure recorded for this dataset in ADR 0064 is on the wrong basis
    and understates the gap.
    28.2% is 7,503 of 26,620 tree entries; the ADR's own
    rule is distinct annex keys, which gives 7,495 of 19,220 = 39.0% missing. Two
    identical files share one key, so an entry count runs above a key count and the
    ratio comes out smaller. That is precisely the error ADR 0064 was written to
    forbid, and it appeared in the ADR itself. A correction is going up as its own PR.

    What is missing is derivatives, and the raw recordings are intact. All 7,495
    missing keys are under derivatives/meg_paper/.../permutations/. ADR 0064 names
    this dataset as the sharpest cost of a flat ratio and accepts it rather than
    solving it: withdrawing denies a reader working recordings over missing
    permutation files, but a raw-versus-derivatives weighting is a second rule to
    argue about and the simple one is auditable today. Revisit if the fleet grows
    more datasets of this shape.

    Separately, 11,725 keys are present in the bucket and not recorded in the
    location log. The sweep deliberately does not register them while content is
    missing (ADR 0061): writing the surviving registrations would make a dataset
    whose real problem is missing content look repaired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    import-failureOpenNeuro import/recovery failed for a datasetneeds-triageNo cause could be determined from the reported error; classify by hand from the run

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions