You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What happened: part of the epic #967 (nemarOrg/nemar-cli) OpenNeuro 0-byte-content recovery. This dataset did not reach a complete state. The hardened finalize gate correctly did NOT publish incomplete data — the dataset retains its real (partial) state; nothing empty was published.
Triage action: ~47% missing after copy — large. Likely substantial upstream-403; verify. If mostly 403, treat like the upstream-403 class (leave incomplete or withdraw).
Refs: epic nemarOrg/nemar-cli#967; recovery run followed .context/recover-runbook.md. data_complete=0 is the honest catalog state until resolved.
Triage data from the Hallu Zarr backfill, since this is still needs-triage.
The S3 data is not missing, it is genuinely 1 byte upstream. Checking every annex object under s3://nemar/on004212/objects/ and comparing the actual object size against the size encoded in its own git-annex key (SHA256E-s<BYTES>--<hash>):
9,742 annex objects
0 size mismatches, 0 zero-byte objects
So this dataset passes every integrity check we have. The problem is that 468 recordings resolve to key SHA256E-s1--36a9e7f1..., that is, the canonical content is a single byte, and all 468 point at the same object.
Zarr conversion consequently fails for 468 recordings with a download/read error, and the dataset contributes 516 unique failing recordings overall (the remaining 16 are No continuous raw recording).
This is a distinct failure mode from the 0-byte cohort (on008014 and friends, nemarOrg/nemar-cli#1048): there the S3 objects are truncated to 0 bytes and disagree with their keys, so integrity checks catch them. Here the byte-for-byte content is internally consistent and simply hollow, so nothing flags it.
Worth confirming whether upstream ds004212 actually serves 1-byte files for these paths, or whether the 1-byte content was baked in at import time and then hashed, which would make the key itself wrong.
Also note this dataset has no .nemar/availability-report.json, so it is invisible to the completeness sweep from epic #999.
Follow-up correcting my previous comment: on004212 has already been pulled. I had treated it as a live dataset with mysterious 1-byte content; it is not live.
Current state:
on004212
on003645
concept DOI
tombstoned at EZID
tombstoned at EZID
repo visibility
private
private
public API
404
404
availability report
none
169/822 files, 51.2% bytes
on withdrawn-datasets.json
no
no
on recover-datasets.json
yes
yes
The inconsistency worth resolving: these two are on the recover list, meaning "published with 0-byte content in the epic #967 incident; OpenNeuro source is accessible", so the intent is to re-copy and restore them. But their DOIs are tombstoned, which is the withdrawal end-state.
That combination is an outlier, not the norm. Checking all 45 entries on the recover list:
tombstoned: 2 live: 43
Only on004212 and on003645 are tombstoned. The other 43 recover-list datasets kept live DOIs while awaiting re-copy. So these two are in a limbo state: too withdrawn to serve, but not on the withdrawn list, and still nominally queued for recovery.
Worth deciding explicitly which list each belongs on:
if recoverable, restore the DOI after re-copy (admin restore exists)
if not, move them to withdrawn-datasets.json so the tombstone matches the bookkeeping
Also note on004212 has no .nemar/availability-report.json, so it is invisible to the epic #999 completeness sweep and to any decision made from those reports. on003645 does have one.
Both are still live rows in the Hallu Zarr queue (on003645 as failed after 5 attempts, on004212 still being attempted), which is the same stale-queue problem tracked in nemarOrg/nemar-cli#1048.
Re-measured on the correct basis: 39.0% of data keys missing, not 28.2% (#1396, ADR 0064)
Re-measured against the bucket on 2026-09-16:
on004212: 7,495 of 19,220 distinct annexed data keys have no object. 61.0% of its data is available.
The 28.2% figure recorded for this dataset in ADR 0064 is on the wrong basis
and understates the gap. 28.2% is 7,503 of 26,620 tree entries; the ADR's own
rule is distinct annex keys, which gives 7,495 of 19,220 = 39.0% missing. Two
identical files share one key, so an entry count runs above a key count and the
ratio comes out smaller. That is precisely the error ADR 0064 was written to
forbid, and it appeared in the ADR itself. A correction is going up as its own PR.
What is missing is derivatives, and the raw recordings are intact. All 7,495
missing keys are under derivatives/meg_paper/.../permutations/. ADR 0064 names
this dataset as the sharpest cost of a flat ratio and accepts it rather than
solving it: withdrawing denies a reader working recordings over missing
permutation files, but a raw-versus-derivatives weighting is a second rule to
argue about and the simple one is auditable today. Revisit if the fleet grows
more datasets of this shape.
Separately, 11,725 keys are present in the bucket and not recorded in the
location log. The sweep deliberately does not register them while content is
missing (ADR 0061): writing the surviving registrations would make a dataset
whose real problem is missing content look repaired.
Activity
neuromechanist commentedon Aug 1, 2026
Triage data from the Hallu Zarr backfill, since this is still
needs-triage.The S3 data is not missing, it is genuinely 1 byte upstream. Checking every annex object under
s3://nemar/on004212/objects/and comparing the actual object size against the size encoded in its own git-annex key (SHA256E-s<BYTES>--<hash>):So this dataset passes every integrity check we have. The problem is that 468 recordings resolve to key
SHA256E-s1--36a9e7f1..., that is, the canonical content is a single byte, and all 468 point at the same object.Zarr conversion consequently fails for 468 recordings with a download/read error, and the dataset contributes 516 unique failing recordings overall (the remaining 16 are
No continuous raw recording).This is a distinct failure mode from the 0-byte cohort (
on008014and friends, nemarOrg/nemar-cli#1048): there the S3 objects are truncated to 0 bytes and disagree with their keys, so integrity checks catch them. Here the byte-for-byte content is internally consistent and simply hollow, so nothing flags it.Worth confirming whether upstream
ds004212actually serves 1-byte files for these paths, or whether the 1-byte content was baked in at import time and then hashed, which would make the key itself wrong.Also note this dataset has no
.nemar/availability-report.json, so it is invisible to the completeness sweep from epic #999.neuromechanist commentedon Aug 1, 2026
Follow-up correcting my previous comment:
on004212has already been pulled. I had treated it as a live dataset with mysterious 1-byte content; it is not live.Current state:
on004212on003645withdrawn-datasets.jsonrecover-datasets.jsonThe inconsistency worth resolving: these two are on the recover list, meaning "published with 0-byte content in the epic #967 incident; OpenNeuro source is accessible", so the intent is to re-copy and restore them. But their DOIs are tombstoned, which is the withdrawal end-state.
That combination is an outlier, not the norm. Checking all 45 entries on the recover list:
Only
on004212andon003645are tombstoned. The other 43 recover-list datasets kept live DOIs while awaiting re-copy. So these two are in a limbo state: too withdrawn to serve, but not on the withdrawn list, and still nominally queued for recovery.Worth deciding explicitly which list each belongs on:
admin restoreexists)withdrawn-datasets.jsonso the tombstone matches the bookkeepingAlso note
on004212has no.nemar/availability-report.json, so it is invisible to the epic #999 completeness sweep and to any decision made from those reports.on003645does have one.Both are still live rows in the Hallu Zarr queue (
on003645asfailedafter 5 attempts,on004212still being attempted), which is the same stale-queue problem tracked in nemarOrg/nemar-cli#1048.Cross-ref: #69 tracks
on003645.neuromechanist commentedon Sep 16, 2026
Re-measured on the correct basis: 39.0% of data keys missing, not 28.2% (#1396, ADR 0064)
Re-measured against the bucket on 2026-09-16:
on004212: 7,495 of 19,220 distinct annexed data keys have no object. 61.0% of its data is available.The 28.2% figure recorded for this dataset in ADR 0064 is on the wrong basis
and understates the gap. 28.2% is 7,503 of 26,620 tree entries; the ADR's own
rule is distinct annex keys, which gives 7,495 of 19,220 = 39.0% missing. Two
identical files share one key, so an entry count runs above a key count and the
ratio comes out smaller. That is precisely the error ADR 0064 was written to
forbid, and it appeared in the ADR itself. A correction is going up as its own PR.
What is missing is derivatives, and the raw recordings are intact. All 7,495
missing keys are under
derivatives/meg_paper/.../permutations/. ADR 0064 namesthis dataset as the sharpest cost of a flat ratio and accepts it rather than
solving it: withdrawing denies a reader working recordings over missing
permutation files, but a raw-versus-derivatives weighting is a second rule to
argue about and the simple one is auditable today. Revisit if the fleet grows
more datasets of this shape.
Separately, 11,725 keys are present in the bucket and not recorded in the
location log. The sweep deliberately does not register them while content is
missing (ADR 0061): writing the surviving registrations would make a dataset
whose real problem is missing content look repaired.