Skip to content

Uploads can leave truncated S3 objects that git-annex records as present; the resume check doesn't catch them #1934

Open

Description

@bruAristimunha

Summary

Large uploads can leave an S3 object that is shorter than its annex key. git-annex still records it as present on nemar-s3, and nemar-cli's resume check passes it. The object is also wrong, not just cut off: the bytes after the gap are shifted.

We found three such objects. In each one, the missing length is an exact multiple of 32,752 bytes:

Dataset Key size (bytes) Size on S3 (bytes) Short by = n × 32,752
nm000390 (published) 3,387,411,027 3,380,271,091 7,139,936 218
nm000573 (private) 24,738,809,184 24,738,219,648 589,536 18
nm000572 (private) 8,247,995,200 8,235,320,176 12,675,024 387

All three have been re-uploaded. nm000390 and nm000572 now match their keys in size and SHA-256; nm000573's object has the right size and its full SHA-256 check is running. For nm000390, NEMAR's data_complete flag showed 0, which is how the first one was spotted.

What happens

  • Where the gap is: a read-only byte comparison of the nm000573 object against the local annex object shows them identical up to offset 11,811,157,248. That's a multiple of 32,752, just before the end of part 11 of 24. After it, the S3 bytes equal the local bytes shifted by exactly 589,536 bytes. So whole 32,752-byte read chunks never reached any part.
  • Why 32,752 bytes: that is the default lazy-ByteString chunk size, which git-annex's S3 special remote uses when streaming a part.
  • When it happens: every case came from uploads where the source sat on a slow network filesystem (CephFS on an HPC cluster), during a burst of S3 RequestTimeout errors and retries. S3 then accepted CompleteMultipartUpload, so git-annex logged the key as present.
  • Versions: git-annex 10.20260717, S3 special remote with partsize=1GiB, no chunking or encryption. Seen with nemar-cli 0.10.13. 0.10.23 uploads through the same git annex copy --to nemar-s3 path, so upgrading doesn't change it.

Why it isn't caught

  • git-annex marks the key present after the multipart upload completes. It doesn't check the object size.
  • The resume presence check (git annex fsck --fast --numcopies=1 --mincopies=1 --from nemar-s3) only checks presence. It returned success on both bad objects before we repaired them.
  • A later nemar dataset upload therefore sees nothing to do, and the dataset looks complete on the client side.

Suggested fixes

  1. Client (cheap): after each upload, and in the resume check, HEAD every object and compare ContentLength with the size encoded in the key (SHA256E-s<bytes>--…). On a mismatch, treat the key as not present and upload it again. For us, a HEAD of every key in a dataset takes seconds to minutes.
  2. Server: run the same size comparison when a version is finalised or published, and block or flag it on a mismatch. data_complete already caught nm000390, but only after publication.
  3. Defence in depth: per-part checksums on multipart uploads (Content-MD5 or x-amz-checksum-*), so S3 itself rejects a short part. This probably needs a git-annex change; we can report it upstream if that helps.
  4. Bucket sweep: a one-off scan of all datasets for objects whose size differs from their key size. Other users who upload from slow storage may be affected without knowing.

Our workaround

  • Detection: for every key at the dataset's HEAD, compare the S3 listing size and the HEAD ContentLength with the key size.
  • Repair: re-upload only the short keys (parallel multipart from the local annex object), read the whole object back from S3 and check its SHA-256 against the key, then git annex setpresentkey.

This week's size checks across 49 of our datasets found only the two private keys above. We're happy to share the scripts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions