Repository navigation
Upload/push: integrate a moved main on adjusted branches (#2140); keep tracking record and renew credentials in dataset push (#2141) - #2173
Open
bruAristimunha wants to merge 3 commits into
Conversation
added 3 commits
November 13, 2026 21:48
Local patch, not upstream. A push from a git-annex adjusted branch (adjusted/main(unlocked), where the upload leaves the tree) that is refused as non-fast-forward is now integrated and retried instead of failing at once: - pushToGitHub: on a non-fast-forward rejection of an adjusted branch, integrateRemoteIntoAdjustedBranch runs `git fetch <remote>`, checks for conflicts with `git merge-tree --write-tree` (writes no work tree or ref), then `git annex merge <remote>/<base>` (fast-forward or merge, re-adjusted by git-annex; never a plain `git merge`). The remote tip must then be in HEAD and no `.variant-` file may appear (git-annex auto-resolves annexed conflicts that way and exits 0). The git-annex branch privacy preflight is re-run, and the push is retried, bounded by PUSH_REBASE_RETRIES (3). - Any conflict stops with the paths named; HEAD, the base and basis refs and the work tree are put back (no markers, no MERGE_HEAD). - upload: when .nemar/config.json exists (resume of an existing dataset) and the tree is on an adjusted branch, origin/main is integrated the same way before the identifier preflight and the BIDS validation, so the validated tree is the pushed tree. A fetch failure only warns; a conflict stops the upload before anything is sent. Skipped on --dry-run. Never forces. NEMAR policy gates are NOT weakened: validation, the identifier preflight and acknowledgement, the deposit attestation, publication and the CLI version (426) check are untouched and still run, on the integrated tree. Test: test/local-patch-2140-adjusted-push.test.ts (real git and git-annex, local bare remote; git-annex 10.20230126 and 10.20261026).
Local patch, not upstream (nemarOrg#2141 part i). printUploadSuccess no longer deletes .nemar/upload-progress.json. retainTrackingAfterSuccess keeps the per-file records (path, size, mtime, status) and the "tracking" stamp, and drops every other step stamp and the remote-check stamp. A re-run with an unchanged data-file list therefore has nothing to add (computeFilesToUpload is empty), runs no `git annex add`, and goes straight to the location-log reconcile (listPendingAtRemote) and the check of the remote, then saves, pushes and deploys CI as a fresh run does. A new or changed file (size or mtime) is still tracked again, alone; --restart still clears everything. NEMAR policy gates are NOT weakened: validation gating, the identifier preflight, the deposit attestation, publication and the CLI version (426) check are untouched. Test: test/local-patch-2141-retain-tracking.test.ts (runDataSteps + printUploadSuccess twice on a real repo with a directory remote; the `git annex add` calls are counted through the test git shim).
Local patch, not upstream (nemarOrg#2141 part ii). `nemar dataset push` with NEMAR credentials now copies through the upload's renewing path: copyAnnexedForPush -> copyAnnexedToRemote (unchanged) over the location log (pending paths only, bounded batches, a CredentialLease renewed through the same POST /datasets/:id/upload-credentials, at most DEFAULT_MAX_REFRESHES times, every annexed file required at the remote at the end). Batch sizes come from the annex keys. Cached credentials are cleared before and after. The copy now runs BEFORE the git push, so the pushed git-annex branch already records the copies; a copy failure exits 1 before anything is pushed. Without NEMAR credentials (environment AWS credentials) the original one-shot `git annex copy` is kept. The existing S3 remote is used as configured (no enableremote); the 403 collaborator handling and the public/branch-protection refusal are unchanged. NEMAR policy gates are NOT weakened: validation, the identifier preflight, attestation, publication and the CLI version (426) check are untouched. Test: test/local-patch-2141-push-copy.test.ts (directory remote, a lease about to expire is renewed, a file with no content anywhere fails with the fetch hint, copy-before-git order in the command).
5 of 6 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2140. Addresses #2141 (parts 1 and 2; part 3, re-validation on every run, is left as is because validation and the identifier preflight are gates).
Three commits on top of v0.10.33 (
25f48b1), one per problem. None of them changes validation, the identifier preflight, the deposit attestation, publication, or the CLI-version (426) check; each commit message says so.1.
b66b789Integrate moved main on adjusted push (#2140)On a git-annex adjusted branch (
adjusted/main(unlocked), which every upload tree uses), a push refused as non-fast-forward (fetch first) was returned as a failure because the fetch+rebase retry is disabled for adjusted branches. NEMAR's own post-validation commit ("Update NEMAR metadata (pipeline: validated)") movesmainafter every validated push, so the next upload to the same dataset was almost always refused.src/lib/upload/integrate-remote.ts(new): fetch, thengit annex merge <remote>/<base>(the adjusted-branch-aware merge), retry the push at most 3 times, never force.MERGE_HEAD).git annex mergeresolves annexed-file conflicts silently into.variant-*copies; that case is detected beforehand and undone if it slips through.uploadintegratesorigin/mainbefore validation when the dataset exists, so the validated tree is the pushed tree;dataset pushuses the same path.test/local-patch-2140-adjusted-push.test.ts(8 tests: bot commit integrated and pushed; conflicting README stops cleanly; data-file conflict stops cleanly; retry bound), run with git-annex 10.20261026 and 10.20230126.2.
3ae00e5Keep tracking record after upload success (#2141, part 1)printUploadSuccesscleared the whole progress record, so a re-run on an unchanged tree re-scanned every data file in the tracking step (hours on a 10,000-file MEF/BrainVision tree on CephFS). The tracking stamp and data-file fingerprint are now kept after success; a re-run with an unchanged file list goes straight to the location-log reconcile. Any change to the file list still re-tracks.test/local-patch-2141-retain-tracking.test.ts(second run on unchanged files: no "Tracking N data files", no scan).3.
7ee9115Renewing data copy before git in push (#2141, part 2)dataset pushrequested one credential and ran onegit annex copywith no renewal, after the git push. It now uses the upload's copy path (openS3Remotelease +renewLease,copyAnnexedToRemoteoverlistPendingAtRemote), copies data before the git push, and pushes no git if the copy fails.test/local-patch-2141-push-copy.test.ts(4 tests with a local directory special remote: a credential close to expiry is renewed; only the 3 missing of 5 files are copied; a failed copy pushes nothing).Test results
Field context
On a 4.8 TB private batch (10,595 data files) 0.10.13 needed about 20 runs over more than a day (2 h credentials, re-tracking each run), and the final push was refused by the post-validation metadata commit. On 0.10.22 the next batch (2.3 TB) copied in one run with 15 credential renewals; these patches remove the remaining re-tracking and push failure.
Independently of this PR: one object of that batch was stored 22 MB short on S3 while the CLI and the location log reported it present (#1934); a per-key size+sha256 read-back caught it.