Skip to content

S3 upload of files >5 GB fails silently: buildS3RemoteArgs omits partsize (git-annex single-part PUT hits S3 5 GB cap) #886

Closed

Description

@bruAristimunha

Summary

nemar dataset upload cannot upload any data file larger than 5 GB. buildS3RemoteArgs configures the git-annex S3 special remote without partsize=, so git-annex uses a single-part PUT per object. S3 caps single-part PUT at 5 GB (multipart is required above that), so every EEG/iEEG recording over 5 GB (very common) fails — and the user only sees a generic ✖ S3 upload failed: Failed to copy to remote, with no hint about the 5 GB limit.

Environment

  • nemar-cli 0.8.69, git-annex 10.20260420
  • Dataset: BIDS iEEG, .eeg files 8–190 GB (BrainVision float32), 3.25 TB total

Reproduction

Upload a dataset containing any data file > 5 GB. Tracking succeeds, then:

- Uploading 120 data files to S3...
✖ S3 upload failed: Failed to copy to remote

git annex copy --to nemar-s3 --debug reveals the real cause — a single PUT of a 160 GB object:

(Remote.S3) String to sign: "PUT\n/nm000276/objects/SHA256E-s160808509440--....eeg ...

S3 responds 400 EntityTooLarge (multipart required). filesCopied stays 0; because the error text lands on git-annex stdout while nemar only surfaces stderr.trim(), the user gets the empty generic "Failed to copy to remote".

Root cause (code)

src/lib/git-annex.ts — buildS3RemoteArgs (~L621):

const params = [
  "type=S3", "encryption=none", `bucket=${config.bucket}`,
  `fileprefix=${config.prefix.replace(/\/$/, "")}/`, `datacenter=${config.region}`,
  "signature=v4", "autoenable=true", "protocol=https",
];

No partsize=. Per git-annex docs, without it the S3 remote does single-part PUT (5 GB cap); partsize=1GiB enables multipart.

Impact

  • Any dataset with a > 5 GB file is a hard failure (not slow) — blocks essentially all raw iEEG/large-EEG uploads.
  • The error gives no diagnosable hint; contributors are stuck.

Fix

  1. Add partsize to buildS3RemoteArgs (e.g. partsize=1GiB, ideally configurable). Verified locally: git annex enableremote nemar-s3 partsize=1GiB immediately makes multipart PUTs work and a 190 GB object uploads at ~166 MB/s.
  2. Surface git-annex stdout on copy failure (the EntityTooLarge error currently goes to stdout, so users see an empty generic error).

Companion to #884 (unlocked-mode re-read). Both block large HPC datasets end-to-end; happy to PR both together.

Activity

  1. neuromechanist commented on Jul 2, 2026

    @neuromechanist
    Contributor

    Thanks for the issue. Yes, I avoided part size implementation for the first implementation to avoid extra tracking.
    Will add since this is now needed. 😊

  2. added
    P1High priority - important features, significant bugs
    P0Critical priority - security, data loss, blocking issues
    and removed
    P1High priority - important features, significant bugs
    on Jul 2, 2026
  3. added 2 commits that reference this issue on Jul 6, 2026
  4. neuromechanist commented on Jul 7, 2026

    @neuromechanist
    Contributor

    Shipped in epic #896 (merged to dev via #943).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P0Critical priority - security, data loss, blocking issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions