Skip to content

Upload/push: integrate a moved main on adjusted branches (#2140); keep tracking record and renew credentials in dataset push (#2141) - #2173

Open
bruAristimunha wants to merge 3 commits into
nemarOrg:devfrom
bruAristimunha:fix/adjusted-push-and-data-resume-2140-2141
Open

bruAristimunha wants to merge 3 commits into
nemarOrg:devfrom
bruAristimunha:fix/adjusted-push-and-data-resume-2140-2141

Conversation

@bruAristimunha

Copy link
Copy Markdown
Contributor

Fixes #2140. Addresses #2141 (parts 1 and 2; part 3, re-validation on every run, is left as is because validation and the identifier preflight are gates).

Three commits on top of v0.10.33 (25f48b1), one per problem. None of them changes validation, the identifier preflight, the deposit attestation, publication, or the CLI-version (426) check; each commit message says so.

1. b66b789 Integrate moved main on adjusted push (#2140)

On a git-annex adjusted branch (adjusted/main(unlocked), which every upload tree uses), a push refused as non-fast-forward (fetch first) was returned as a failure because the fetch+rebase retry is disabled for adjusted branches. NEMAR's own post-validation commit ("Update NEMAR metadata (pipeline: validated)") moves main after every validated push, so the next upload to the same dataset was almost always refused.

  • src/lib/upload/integrate-remote.ts (new): fetch, then git annex merge <remote>/<base> (the adjusted-branch-aware merge), retry the push at most 3 times, never force.
  • On a conflict it stops with the conflicting paths and leaves the tree clean (no markers, no MERGE_HEAD). git annex merge resolves annexed-file conflicts silently into .variant-* copies; that case is detected beforehand and undone if it slips through.
  • upload integrates origin/main before validation when the dataset exists, so the validated tree is the pushed tree; dataset push uses the same path.
  • Tests: test/local-patch-2140-adjusted-push.test.ts (8 tests: bot commit integrated and pushed; conflicting README stops cleanly; data-file conflict stops cleanly; retry bound), run with git-annex 10.20261026 and 10.20230126.

2. 3ae00e5 Keep tracking record after upload success (#2141, part 1)

printUploadSuccess cleared the whole progress record, so a re-run on an unchanged tree re-scanned every data file in the tracking step (hours on a 10,000-file MEF/BrainVision tree on CephFS). The tracking stamp and data-file fingerprint are now kept after success; a re-run with an unchanged file list goes straight to the location-log reconcile. Any change to the file list still re-tracks.

  • Test: test/local-patch-2141-retain-tracking.test.ts (second run on unchanged files: no "Tracking N data files", no scan).

3. 7ee9115 Renewing data copy before git in push (#2141, part 2)

dataset push requested one credential and ran one git annex copy with no renewal, after the git push. It now uses the upload's copy path (openS3Remote lease + renewLease, copyAnnexedToRemote over listPendingAtRemote), copies data before the git push, and pushes no git if the copy fails.

  • Test: test/local-patch-2141-push-copy.test.ts (4 tests with a local directory special remote: a credential close to expiry is renewed; only the 3 missing of 5 files are copied; a failed copy pushes nothing).

Test results

  • Repository suite (33 files, 673 tests): 654 pass / 13 skip / 6 fail on both unpatched v0.10.33 and this branch, same 6 failures (pre-existing).
  • New tests: 8/8, 1/1, 4/4.

Field context

On a 4.8 TB private batch (10,595 data files) 0.10.13 needed about 20 runs over more than a day (2 h credentials, re-tracking each run), and the final push was refused by the post-validation metadata commit. On 0.10.22 the next batch (2.3 TB) copied in one run with 15 credential renewals; these patches remove the remaining re-tracking and push failure.

Independently of this PR: one object of that batch was stored 22 MB short on S3 while the CLI and the location log reported it present (#1934); a per-key size+sha256 read-back caught it.

bpinto lane nemarcli added 3 commits November 13, 2026 21:48
Local patch, not upstream. A push from a git-annex adjusted branch
(adjusted/main(unlocked), where the upload leaves the tree) that is
refused as non-fast-forward is now integrated and retried instead of
failing at once:

- pushToGitHub: on a non-fast-forward rejection of an adjusted branch,
  integrateRemoteIntoAdjustedBranch runs `git fetch <remote>`, checks
  for conflicts with `git merge-tree --write-tree` (writes no work tree
  or ref), then `git annex merge <remote>/<base>` (fast-forward or
  merge, re-adjusted by git-annex; never a plain `git merge`). The
  remote tip must then be in HEAD and no `.variant-` file may appear
  (git-annex auto-resolves annexed conflicts that way and exits 0).
  The git-annex branch privacy preflight is re-run, and the push is
  retried, bounded by PUSH_REBASE_RETRIES (3).
- Any conflict stops with the paths named; HEAD, the base and basis
  refs and the work tree are put back (no markers, no MERGE_HEAD).
- upload: when .nemar/config.json exists (resume of an existing
  dataset) and the tree is on an adjusted branch, origin/main is
  integrated the same way before the identifier preflight and the BIDS
  validation, so the validated tree is the pushed tree. A fetch failure
  only warns; a conflict stops the upload before anything is sent.
  Skipped on --dry-run.

Never forces. NEMAR policy gates are NOT weakened: validation, the
identifier preflight and acknowledgement, the deposit attestation,
publication and the CLI version (426) check are untouched and still
run, on the integrated tree.

Test: test/local-patch-2140-adjusted-push.test.ts (real git and
git-annex, local bare remote; git-annex 10.20230126 and 10.20261026).
Local patch, not upstream (nemarOrg#2141 part i). printUploadSuccess no longer
deletes .nemar/upload-progress.json. retainTrackingAfterSuccess keeps
the per-file records (path, size, mtime, status) and the "tracking"
stamp, and drops every other step stamp and the remote-check stamp.

A re-run with an unchanged data-file list therefore has nothing to add
(computeFilesToUpload is empty), runs no `git annex add`, and goes
straight to the location-log reconcile (listPendingAtRemote) and the
check of the remote, then saves, pushes and deploys CI as a fresh run
does. A new or changed file (size or mtime) is still tracked again,
alone; --restart still clears everything.

NEMAR policy gates are NOT weakened: validation gating, the identifier
preflight, the deposit attestation, publication and the CLI version
(426) check are untouched.

Test: test/local-patch-2141-retain-tracking.test.ts (runDataSteps +
printUploadSuccess twice on a real repo with a directory remote; the
`git annex add` calls are counted through the test git shim).
Local patch, not upstream (nemarOrg#2141 part ii). `nemar dataset push` with
NEMAR credentials now copies through the upload's renewing path:
copyAnnexedForPush -> copyAnnexedToRemote (unchanged) over the
location log (pending paths only, bounded batches, a CredentialLease
renewed through the same POST /datasets/:id/upload-credentials, at
most DEFAULT_MAX_REFRESHES times, every annexed file required at the
remote at the end). Batch sizes come from the annex keys. Cached
credentials are cleared before and after.

The copy now runs BEFORE the git push, so the pushed git-annex branch
already records the copies; a copy failure exits 1 before anything is
pushed. Without NEMAR credentials (environment AWS credentials) the
original one-shot `git annex copy` is kept. The existing S3 remote is
used as configured (no enableremote); the 403 collaborator handling
and the public/branch-protection refusal are unchanged.

NEMAR policy gates are NOT weakened: validation, the identifier
preflight, attestation, publication and the CLI version (426) check
are untouched.

Test: test/local-patch-2141-push-copy.test.ts (directory remote, a
lease about to expire is renewed, a file with no content anywhere
fails with the fetch hint, copy-before-git order in the command).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant