Skip to content

Exemplar creation is not retry-safe after a partial failure #955

Open

Description

@neuromechanist

Found while running the first real exemplar clone in epic #923 Phase 7.

Problem

nemar admin exemplar create <id> runs prepare -> copy -> finalize in one process. If it fails anywhere at or after the S3-remote step, re-running the same command cannot recover, and each retry fails differently:

  1. First retry fails at initremote: The bucket already exists, and its annex-uuid file indicates it is used by a different special remote. git-annex: Cannot reuse this bucket. Prepare always clones the SOURCE into a fresh tmpdir, so it builds a brand-new git-annex repo with a brand-new remote UUID, which cannot claim the annex-uuid marker the previous attempt already wrote under s3://nemar-dev/<xx>/objects/.
  2. Clearing that S3 prefix gets past initremote, but the run then fails pushing: ! [rejected] git-annex -> git-annex (non-fast-forward) ... Aborting — the exemplar would be uncloneable. The exemplar repo already carries the git-annex branch from the first attempt, and the fresh clone's branch has diverged.

The only reliable recovery today is a full reset: nemar admin delete-dataset <xx-id> (repo + S3 + D1), then create again.

This is masked by the fact that the D1 insert tolerates "already exists" and the copy phase resumes via an S3 listing diff, which together make the command look resumable. Its own error messages also advise re-running ("Re-run finalize to retry", "Re-run to resume"), but there is no CLI entry point that re-enters finalize alone; create always re-runs prepare.

Impact

Any transient failure (network, GitHub, a BIDS-validation hiccup) during a fleet run leaves that entry unrecoverable without a manual delete. --all continues past failures, so a multi-entry run can leave several exemplars in this state.

Suggested fix

Make prepare idempotent against an exemplar that already exists, rather than assuming a virgin target. Options, roughly in order of preference:

  • If the exemplar repo already exists, clone it (not the source) so the git-annex branch and remote config are inherited, and use enableremote instead of initremote when nemar-s3-dev is already configured in the git-annex branch.
  • Or add an explicit --reset flag that performs the delete-and-recreate cycle in one step, so the documented recovery path is a single command.
  • Or split the phases into separate subcommands (exemplar prepare|copy|finalize) so the advice in the existing error messages is actually actionable.

At minimum, the error messages should stop advising a plain re-run and instead point at the delete-and-recreate path.

Related

  • The same run surfaced a separate bug, fixed in Phase 7: dev cron safety gates + exemplar fleet #954: objects/ listings include the S3 remote's annex-uuid marker, which was being copied and then handed to batchSetKeysPresent, failing with "1 of 3 git-annex key registrations failed".

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions