Repository navigation
Description
Found while running the first real exemplar clone in epic #923 Phase 7.
Problem
nemar admin exemplar create <id> runs prepare -> copy -> finalize in one process. If it fails anywhere at or after the S3-remote step, re-running the same command cannot recover, and each retry fails differently:
- First retry fails at
initremote:The bucket already exists, and its annex-uuid file indicates it is used by a different special remote. git-annex: Cannot reuse this bucket.Prepare always clones the SOURCE into a fresh tmpdir, so it builds a brand-new git-annex repo with a brand-new remote UUID, which cannot claim theannex-uuidmarker the previous attempt already wrote unders3://nemar-dev/<xx>/objects/. - Clearing that S3 prefix gets past
initremote, but the run then fails pushing:! [rejected] git-annex -> git-annex (non-fast-forward) ... Aborting — the exemplar would be uncloneable.The exemplar repo already carries the git-annex branch from the first attempt, and the fresh clone's branch has diverged.
The only reliable recovery today is a full reset: nemar admin delete-dataset <xx-id> (repo + S3 + D1), then create again.
This is masked by the fact that the D1 insert tolerates "already exists" and the copy phase resumes via an S3 listing diff, which together make the command look resumable. Its own error messages also advise re-running ("Re-run finalize to retry", "Re-run to resume"), but there is no CLI entry point that re-enters finalize alone; create always re-runs prepare.
Impact
Any transient failure (network, GitHub, a BIDS-validation hiccup) during a fleet run leaves that entry unrecoverable without a manual delete. --all continues past failures, so a multi-entry run can leave several exemplars in this state.
Suggested fix
Make prepare idempotent against an exemplar that already exists, rather than assuming a virgin target. Options, roughly in order of preference:
- If the exemplar repo already exists, clone it (not the source) so the git-annex branch and remote config are inherited, and use
enableremoteinstead ofinitremotewhennemar-s3-devis already configured in the git-annex branch. - Or add an explicit
--resetflag that performs the delete-and-recreate cycle in one step, so the documented recovery path is a single command. - Or split the phases into separate subcommands (
exemplar prepare|copy|finalize) so the advice in the existing error messages is actually actionable.
At minimum, the error messages should stop advising a plain re-run and instead point at the delete-and-recreate path.
Related
- The same run surfaced a separate bug, fixed in Phase 7: dev cron safety gates + exemplar fleet #954:
objects/listings include the S3 remote'sannex-uuidmarker, which was being copied and then handed tobatchSetKeysPresent, failing with "1 of 3 git-annex key registrations failed".
Activity