Repository navigation
Open
Description
Summary
Large uploads can leave an S3 object that is shorter than its annex key. git-annex still records it as present on nemar-s3, and nemar-cli's resume check passes it. The object is also wrong, not just cut off: the bytes after the gap are shifted.
We found three such objects. In each one, the missing length is an exact multiple of 32,752 bytes:
| Dataset | Key size (bytes) | Size on S3 (bytes) | Short by | = n × 32,752 |
|---|---|---|---|---|
| nm000390 (published) | 3,387,411,027 | 3,380,271,091 | 7,139,936 | 218 |
| nm000573 (private) | 24,738,809,184 | 24,738,219,648 | 589,536 | 18 |
| nm000572 (private) | 8,247,995,200 | 8,235,320,176 | 12,675,024 | 387 |
All three have been re-uploaded. nm000390 and nm000572 now match their keys in size and SHA-256; nm000573's object has the right size and its full SHA-256 check is running. For nm000390, NEMAR's data_complete flag showed 0, which is how the first one was spotted.
What happens
- Where the gap is: a read-only byte comparison of the nm000573 object against the local annex object shows them identical up to offset 11,811,157,248. That's a multiple of 32,752, just before the end of part 11 of 24. After it, the S3 bytes equal the local bytes shifted by exactly 589,536 bytes. So whole 32,752-byte read chunks never reached any part.
- Why 32,752 bytes: that is the default lazy-ByteString chunk size, which git-annex's S3 special remote uses when streaming a part.
- When it happens: every case came from uploads where the source sat on a slow network filesystem (CephFS on an HPC cluster), during a burst of S3
RequestTimeouterrors and retries. S3 then acceptedCompleteMultipartUpload, so git-annex logged the key as present. - Versions: git-annex 10.20260717, S3 special remote with
partsize=1GiB, no chunking or encryption. Seen with nemar-cli 0.10.13. 0.10.23 uploads through the samegit annex copy --to nemar-s3path, so upgrading doesn't change it.
Why it isn't caught
- git-annex marks the key present after the multipart upload completes. It doesn't check the object size.
- The resume presence check (
git annex fsck --fast --numcopies=1 --mincopies=1 --from nemar-s3) only checks presence. It returned success on both bad objects before we repaired them. - A later
nemar dataset uploadtherefore sees nothing to do, and the dataset looks complete on the client side.
Suggested fixes
- Client (cheap): after each upload, and in the resume check,
HEADevery object and compareContentLengthwith the size encoded in the key (SHA256E-s<bytes>--…). On a mismatch, treat the key as not present and upload it again. For us, a HEAD of every key in a dataset takes seconds to minutes. - Server: run the same size comparison when a version is finalised or published, and block or flag it on a mismatch.
data_completealready caught nm000390, but only after publication. - Defence in depth: per-part checksums on multipart uploads (
Content-MD5orx-amz-checksum-*), so S3 itself rejects a short part. This probably needs a git-annex change; we can report it upstream if that helps. - Bucket sweep: a one-off scan of all datasets for objects whose size differs from their key size. Other users who upload from slow storage may be affected without knowing.
Our workaround
- Detection: for every key at the dataset's HEAD, compare the S3 listing size and the
HEADContentLengthwith the key size. - Repair: re-upload only the short keys (parallel multipart from the local annex object), read the whole object back from S3 and check its SHA-256 against the key, then
git annex setpresentkey.
This week's size checks across 49 of our datasets found only the two private keys above. We're happy to share the scripts.
Metadata
Metadata
Assignees
Labels
No labels
Activity