Skip to content

Fix S3 initremote failure on upload resume - #337

Merged
neuromechanist merged 3 commits into
mainfrom
336-s3-initremote-fails-on-resume-bucket-annex-uuid-conflict-after-failed-upload
Apr 1, 2026
Merged

neuromechanist merged 3 commits into
mainfrom
336-s3-initremote-fails-on-resume-bucket-annex-uuid-conflict-after-failed-upload

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Fixes #336. nemar dataset upload fails when re-running after a previous failed upload because configureS3Remote() cannot handle an already-existing git-annex special remote.

Three root causes fixed:

  • annex-ignore false failure: git-annex prints "Remote origin not usable by git-annex; setting annex-ignore" to stderr when it encounters the GitHub origin remote. This warning caused non-zero exit codes, making initremote appear to fail even when it might have succeeded. Fix: pre-set remote.origin.annex-ignore = true before calling initremote.

  • enableremote missing S3 params: When initremote failed with "already exists", the fallback enableremote call only passed the remote name, not the bucket/region/prefix/credentials. This meant the remote could not reconnect. Fix: pass full S3 config params to enableremote.

  • No pre-check for existing remote: Always tried initremote first, which fails if the remote exists from a prior attempt. Fix: check git annex info --json for existing remote before choosing initremote vs enableremote.

Changed files

  • src/lib/git-annex.ts - Rewrite configureS3Remote() with three helper functions:
    • filterAnnexInfoMessages() - strips informational stderr lines
    • buildS3RemoteArgs() - shared config params for init/enable
    • annexRemoteExists() - checks git-annex branch for existing remote

Test plan

  • Build passes (bun run build)
  • Type check passes (tsc --noEmit via pre-commit hook)
  • Lint passes (biome via pre-commit hook)
  • Manual test: fresh upload on new dataset (initremote path)
  • Manual test: retry upload after failed attempt (enableremote path)
  • Manual test on affected datasets (nm000349, nm000150, nm000147)

- Pre-set annex-ignore on origin to prevent false failures
- Check if remote exists before initremote to avoid conflicts
- Pass full S3 config params to enableremote fallback
- Filter informational stderr messages from error detection

Fixes #336
- Fix silent success: always fail on non-zero exit code, use
  filtered stderr for message only (not for deciding failure)
- Extract enableS3RemoteWithConfig helper to deduplicate logic
- Make annexRemoteExists log warnings instead of silent catch
- Narrow parenthesized filter to specific safe patterns only
- Check git config return value and warn on failure
- Guard catch with instanceof Error, add context prefix
- Log residual stderr warnings even on initremote success
- Fix misleading comments per review feedback
@github-actions

github-actions Bot commented Apr 1, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR337

Version: 0.7.25-PR337.e6b51b0
Commit: e6b51b0


🤖 Updated on each push.

@neuromechanist

Copy link
Copy Markdown
Contributor Author

PR Review Summary

Ran 4 review agents (code-reviewer, silent-failure-hunter, comment-analyzer, code-simplifier). All findings addressed in e72dd8e.

Critical (2 found, 2 fixed)

  • Silent success on filtered stderr: exitCode !== 0 && realStderr returned success when stderr was fully filtered. Fixed: always fail on non-zero exit code; use filtered stderr for the error message only.
  • annexRemoteExists swallowing errors: Empty catch and unchecked exitCode collapsed "check failed" into "does not exist". Fixed: log warnings for both paths so failures are visible.

High (2 found, 2 fixed)

  • git config return value discarded: Protective annex-ignore config could silently fail. Fixed: check exitCode and warn.
  • Overly broad parenthesized filter: startsWith("(") && endsWith(")") could catch real errors like (S3 error: Access Denied). Fixed: narrowed to specific known safe patterns (merging, recording state, scanning, checking).

Important (1 found, 1 fixed)

  • Duplicated enableremote logic: Same 8-line block repeated twice. Fixed: extracted enableS3RemoteWithConfig() helper.

Comments (4 found, 4 fixed)

  • Fixed misleading claim about annex-ignore causing exit codes
  • Added missing "why" explanation for error-check pattern
  • Changed "race condition" to accurate "fallback" wording
  • Restored lost publicurl/autoenable context in JSDoc

Medium (2 found, 2 fixed)

  • Stderr warnings on success discarded: Fixed: log residual stderr even when initremote exits 0.
  • Broad catch with unsafe cast: Fixed: guard with instanceof Error, add context prefix.

Strengths noted

  • Clean extraction of shared logic into helpers
  • Belt-and-suspenders approach (pre-config + filter backup)
  • Defensive fallback for initremote "already exists" after pre-check

@neuromechanist
neuromechanist merged commit daeccd3 into main Apr 1, 2026
7 of 8 checks passed
neuromechanist added a commit that referenced this pull request Apr 17, 2026
Brings to dev from main:
- Maintenance mode middleware (#342)
- MAINTENANCE_MODE=read-only flip (prod only; dev [env.dev.vars] stays off)
- Upload resume fix (#335)
- S3 initremote fix on resume (#337)
- PR-merge workflow, HED detection, bulk CI deploy (#338)
- npm publish fixes (Node 20, npm@11, OIDC)
- GitHub Release auto-tag
@neuromechanist
neuromechanist deleted the 336-s3-initremote-fails-on-resume-bucket-annex-uuid-conflict-after-failed-upload branch April 28, 2026 01:23

This branch was previously deployed

1 inactive deployment
npm — e6b51b0f Deployed Apr 1, 2026 by neuromechanist via publish #580
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 initremote fails on resume: bucket annex-uuid conflict after failed upload

1 participant