You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
S3 upload of files >5 GB fails silently: buildS3RemoteArgs omits partsize (git-annex single-part PUT hits S3 5 GB cap) #886
nemar dataset uploadcannot upload any data file larger than 5 GB. buildS3RemoteArgs configures the git-annex S3 special remote without partsize=, so git-annex uses a single-part PUT per object. S3 caps single-part PUT at 5 GB (multipart is required above that), so every EEG/iEEG recording over 5 GB (very common) fails — and the user only sees a generic ✖ S3 upload failed: Failed to copy to remote, with no hint about the 5 GB limit.
Upload a dataset containing any data file > 5 GB. Tracking succeeds, then:
- Uploading 120 data files to S3...
✖ S3 upload failed: Failed to copy to remote
git annex copy --to nemar-s3 --debug reveals the real cause — a single PUT of a 160 GB object:
(Remote.S3) String to sign: "PUT\n/nm000276/objects/SHA256E-s160808509440--....eeg ...
S3 responds 400 EntityTooLarge (multipart required). filesCopied stays 0; because the error text lands on git-annex stdout while nemar only surfaces stderr.trim(), the user gets the empty generic "Failed to copy to remote".
No partsize=. Per git-annex docs, without it the S3 remote does single-part PUT (5 GB cap); partsize=1GiB enables multipart.
Impact
Any dataset with a > 5 GB file is a hard failure (not slow) — blocks essentially all raw iEEG/large-EEG uploads.
The error gives no diagnosable hint; contributors are stuck.
Fix
Add partsize to buildS3RemoteArgs (e.g. partsize=1GiB, ideally configurable). Verified locally: git annex enableremote nemar-s3 partsize=1GiB immediately makes multipart PUTs work and a 190 GB object uploads at ~166 MB/s.
Surface git-annex stdout on copy failure (the EntityTooLarge error currently goes to stdout, so users see an empty generic error).
Companion to #884 (unlocked-mode re-read). Both block large HPC datasets end-to-end; happy to PR both together.
Thanks for the issue. Yes, I avoided part size implementation for the first implementation to avoid extra tracking.
Will add since this is now needed. 😊
Activity
neuromechanist commentedon Jul 2, 2026
Thanks for the issue. Yes, I avoided part size implementation for the first implementation to avoid extra tracking.
Will add since this is now needed. 😊
Address #886 review: cap copy-error output length
Phase 2: S3 partsize for >5GB uploads (#886) (#935)
neuromechanist commentedon Jul 7, 2026
Shipped in epic #896 (merged to dev via #943).