Skip to content

feat: detect and repair lagging recovery escrows - #11200

Merged
huggingbot merged 42 commits into
mainfrom
feat/mfa-recovery-reconciliation
Nov 16, 2026
Merged

huggingbot merged 42 commits into
mainfrom
feat/mfa-recovery-reconciliation

Conversation

@huggingbot

@huggingbot huggingbot commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Explanation

  • Before a mutation, read every escrow's version with an AAL1 approval. Differing versions fail with replica_lagging.
  • Add repairReplicas(identifier). It copies the highest record to escrows one version behind, as a mutation addressed only to them (register, updateIdentifiers or updateRecoverySecret).
  • Clear a pending mutation when every remaining escrow rejects it, even if others have receipts: the mutation is obsolete or was already repaired.
  • Allow abort while writing once the approval has expired, there are no receipts, and every target escrow is still at expectedVersion.
  • Compare only the secret on reads; a repaired replica holds the same secret under a different mutation ID.

References

huggingbot and others added 9 commits October 29, 2026 15:39
Add getIdentifiers to the controller and CubistEscrowProvider (escrow #8),
and pass the operation's AuthController token to generateChallenge (escrow #10).

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Send identifierAuthorizations as a list sharing one challenge per escrow
(escrow #7): one proof per SIWE identifier the escrow has not enrolled,
plus the enrolled-identifier proof for updates.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Acquire or renew the registration-service lease before every approval.
The 6-minute lease outlives the 5-minute approval, so another operation
can only take the lease after this one's approvals have expired.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@huggingbot
huggingbot requested review from a team as code owners October 29, 2026 12:58
Co-authored-by: Cursor <cursoragent@cursor.com>
huggingbot and others added 14 commits November 2, 2026 19:55
Fold repairReplicas into resume(identifier): with no pending mutation, it copies the highest escrow record to escrows one version behind. Drop abort while writing: local approval expiry does not prove a delayed write cannot land, since the device clock may run ahead of the escrow's.

Co-authored-by: Cursor <cursoragent@cursor.com>
- Require every escrow and one version for getIdentifiers (replica_lagging)
- Probe escrows moved off expectedVersion without identifier proofs, even
  with receipts, so resumed writes never fork and clear without the
  original identifier or SIWE wallet
- Clear with mutation_superseded, unsent, when a renewed approval no
  longer covers the payload and every remaining escrow moved
- Reject updates whose identifier is not enrolled on every escrow
  (identifier_not_enrolled)
- Let resume(identifier) authorize a pending update with another
  enrolled identifier
- Reuse the pre-write escrow reads to find new SIWE identifiers
- Move receipt fetching, approval renewal and enrollment checks into
  #prepareWrite, so #replicateMutation only does the write
- Name the 1FA/2FA approvals #authorizeRead/#authorizeWrite; renew an
  expired write approval in #reauthorizeWrite
- Rename getMutationReceipts to getAppliedMutationReceipts, records to
  escrowIdentifiers, and unanimousMutationRejection to
  getRejectionFromEveryEscrow (moved to replica-utils)
- planRepair returns the operation to replay and the lagging escrows
- Test that an unapplied mutation stays pending when renewing its
  approval fails
huggingbot and others added 8 commits November 10, 2026 15:48
…e-enrollment

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	packages/mfa-recovery-controller/CHANGELOG.md
#	packages/mfa-recovery-controller/src/MfaRecoveryController.ts
#	packages/mfa-recovery-controller/src/escrow-providers/cubist-escrow-provider.test.ts
#	packages/mfa-recovery-controller/src/escrow-providers/cubist-escrow-provider.ts
#	packages/mfa-recovery-controller/src/identifier-auth.ts
…t' into feat/mfa-recovery-locks

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	packages/mfa-recovery-controller/CHANGELOG.md
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	packages/mfa-recovery-controller/CHANGELOG.md
#	packages/mfa-recovery-controller/src/MfaRecoveryController.test.ts
#	packages/mfa-recovery-controller/src/MfaRecoveryController.ts
#	packages/mfa-recovery-controller/src/identifier-auth.ts
…at/mfa-recovery-reconciliation

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	packages/mfa-recovery-controller/README.md
#	packages/mfa-recovery-controller/src/MfaRecoveryController-method-action-types.ts
#	packages/mfa-recovery-controller/src/MfaRecoveryController.ts
#	packages/mfa-recovery-controller/src/state-machine.ts
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Base automatically changed from feat/mfa-recovery-locks to main November 12, 2026 06:47
…at/mfa-recovery-reconciliation

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	packages/mfa-recovery-controller/src/MfaRecoveryController.test.ts
…onciliation

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	packages/mfa-recovery-controller/CHANGELOG.md
#	packages/mfa-recovery-controller/src/MfaRecoveryController.test.ts
#	packages/mfa-recovery-controller/src/MfaRecoveryController.ts
@ieow

ieow commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

A half-applied mutation gets cleared when the remaining escrows reject (functional). The old guard "no receipts yet" is gone. So if escrow A already returned a receipt and escrow B now throws MutationRejectedError, the pending mutation is cleared:

MfaRecoveryController.ts
Ln 748–752

const rejection = getRejectionFromEveryEscrow(results);
if (rejection) {
  await this.#clearPending();
  throw rejection;
}

The PR's reasoning is that such a mutation "is obsolete or was already repaired". But #prepareWrite has just confirmed that every remaining escrow is still at expectedVersion. If one had moved, the code would have taken the receipt or mutation_superseded path. So a rejection here can't mean "already repaired". It means some other refusal: the Cubist provider turns any policy Deny into MutationRejectedError, such as a bad identifier proof or approval timing. The result is that A ends up at newVersion and B at expectedVersion. The error tells the caller to start a new mutation, but that will fail with replica_lagging, so the user is pushed into repair, with two 2FA prompts. Before this PR, the mutation stayed writing and could be resumed.

@ieow

ieow commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Receipts fetched from escrows are never saved before the flow continues (functional, and the docs say otherwise). #withAppliedMutationReceipts (lines 838–853) returns the receipts only in memory. They aren't written to state until line 729, which comes after assertEnrolledInEvery, a possible 2FA re-approval in #authorizeWrite, and identifier authorization. If any of those throw (2FA declined, identifier_not_enrolled, the identifier provider fails), the receipts are dropped. Escrows "keep only recent receipts", so a later resume() may find the receipt gone and clear with mutation_superseded, even though this mutation was applied. That leaves the replicas out of sync. ARCHITECTURE.md says "persist the receipts, then reread the rest", which the code doesn't do.

@ieow

ieow commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

An account can get stuck in an unrepairable state (design). getIdentifiers and every mutation now fail hard on replica_lagging, and the only way out is repair. But repair can be impossible to finish:

  • register and updateIdentifiers repairs need fresh SIWE proofs for every SIWE identifier the lagging escrows don't have (findNewSiweIdentifiers). Those wallets may not be on this device.
  • An updateIdentifiers repair requires identifier to be enrolled on the lagging escrows.
  • If versions are more than one apart, the code throws replica_corruption and offers no recovery path at all.

@ieow

ieow commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

skips the profile check that resume() has (security, low). #prepareWrite rejects a read approval for a different profile (lines 790–795). #isUnapplied (lines 599–615) doesn't. If the user switched profiles, abort() reads the other profile's versions (for example 0, which is below newVersion) and clears a mutation that may have been applied on the original profile. The fix is to add the same sub !== mutation.profileId check.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🛑 Files requiring codeowner review

To merge this PR, you'll need to get approvals from the following owners.

@MetaMask/web3auth (23 files)
  • packages/mfa-recovery-controller/ARCHITECTURE.md
  • packages/mfa-recovery-controller/CHANGELOG.md
  • packages/mfa-recovery-controller/CUBIST_TEST_SEQUENCE.md
  • packages/mfa-recovery-controller/README.md
  • packages/mfa-recovery-controller/src/errors.ts
  • packages/mfa-recovery-controller/src/escrow-providers/cubist-escrow-provider.test.ts
  • packages/mfa-recovery-controller/src/escrow-providers/cubist-escrow-provider.ts
  • packages/mfa-recovery-controller/src/escrow-utils.test.ts
  • packages/mfa-recovery-controller/src/escrow-utils.ts
  • packages/mfa-recovery-controller/src/identifier-auth.ts
  • packages/mfa-recovery-controller/src/index.ts
  • packages/mfa-recovery-controller/src/MfaRecoveryController-method-action-types.ts
  • packages/mfa-recovery-controller/src/MfaRecoveryController.test.ts
  • packages/mfa-recovery-controller/src/MfaRecoveryController.ts
  • packages/mfa-recovery-controller/src/pending-operation-validation.ts
  • packages/mfa-recovery-controller/src/receipt-utils.test.ts
  • packages/mfa-recovery-controller/src/receipt-utils.ts
  • packages/mfa-recovery-controller/src/recovery-registration-lock-provider.test.ts
  • packages/mfa-recovery-controller/src/recovery-registration-lock-provider.ts
  • packages/mfa-recovery-controller/src/replica-utils.ts
  • packages/mfa-recovery-controller/src/state-machine.ts
  • packages/mfa-recovery-controller/src/types.ts
  • packages/mfa-recovery-controller/tests/stubs.ts

ℹ️ Files co-owned by @MetaMask/core-platform (expand for more)

[!NOTE]
For convenience in maintaining this repo, @MetaMask/core-platform co-owns
the following files. It is not required to seek approval from
@MetaMask/core-platform to change these files. This section is purely
informational.

  • packages/mfa-recovery-controller/CHANGELOG.md

@huggingbot

Copy link
Copy Markdown
Member Author

A half-applied mutation gets cleared when the remaining escrows reject (functional). The old guard "no receipts yet" is gone. So if escrow A already returned a receipt and escrow B now throws MutationRejectedError, the pending mutation is cleared:

MfaRecoveryController.ts Ln 748–752

const rejection = getRejectionFromEveryEscrow(results);
if (rejection) {
  await this.#clearPending();
  throw rejection;
}

The PR's reasoning is that such a mutation "is obsolete or was already repaired". But #prepareWrite has just confirmed that every remaining escrow is still at expectedVersion. If one had moved, the code would have taken the receipt or mutation_superseded path. So a rejection here can't mean "already repaired". It means some other refusal: the Cubist provider turns any policy Deny into MutationRejectedError, such as a bad identifier proof or approval timing. The result is that A ends up at newVersion and B at expectedVersion. The error tells the caller to start a new mutation, but that will fail with replica_lagging, so the user is pushed into repair, with two 2FA prompts. Before this PR, the mutation stayed writing and could be resumed.

Escrow apply only returns Deny for: a receipt conflict, already registered, an expectedVersion mismatch, or an identifier taken by another profile. Since #prepareWrite just read every remaining escrow at expectedVersion, the first three mean another client wrote in between (superseded), and the last is permanent. Retrying never succeeds in any of them, and keeping the update pending would leave the user stuck (resume rejected forever, abort refuses). Bad proofs or approvals return Error, not Deny, so they surface as escrow_request_failed and the update stays resumable.

@huggingbot

Copy link
Copy Markdown
Member Author

Receipts fetched from escrows are never saved before the flow continues (functional, and the docs say otherwise). #withAppliedMutationReceipts (lines 838–853) returns the receipts only in memory. They aren't written to state until line 729, which comes after assertEnrolledInEvery, a possible 2FA re-approval in #authorizeWrite, and identifier authorization. If any of those throw (2FA declined, identifier_not_enrolled, the identifier provider fails), the receipts are dropped. Escrows "keep only recent receipts", so a later resume() may find the receipt gone and clear with mutation_superseded, even though this mutation was applied. That leaves the replicas out of sync. ARCHITECTURE.md says "persist the receipts, then reread the rest", which the code doesn't do.

That's harmless, though. The next resume() fetches them again, and the escrow keeps the last 64 receipts per profile, so they'll still be there. Updated the architecture doc: e1bbe6d

@huggingbot

Copy link
Copy Markdown
Member Author

An account can get stuck in an unrepairable state (design). getIdentifiers and every mutation now fail hard on replica_lagging, and the only way out is repair. But repair can be impossible to finish:

  • register and updateIdentifiers repairs need fresh SIWE proofs for every SIWE identifier the lagging escrows don't have (findNewSiweIdentifiers). Those wallets may not be on this device.
  • An updateIdentifiers repair requires identifier to be enrolled on the lagging escrows.
  • If versions are more than one apart, the code throws replica_corruption and offers no recovery path at all.
  1. SIWE proofs: the clients (mobile and extension) should only use the primary SRP's SIWE address. A user without the primary SRP can't log in to MetaMask, so the wallet is always available to sign during repair.
  2. Enrolled identifier: the clients should limit identifier changes. Also, the identifier that approved the original update is still in the user's hands; removing it from the escrows doesn't revoke the passkey or account. So repair always has an identifier from the old set.
  3. More than one version apart: this can't happen with honest clients. Every write requires all escrows to be at the same version, so they're at most one version apart until repaired. A larger gap means a dishonest client corrupted its own state, so replica_corruption with no recovery path is intended.

@huggingbot

Copy link
Copy Markdown
Member Author

skips the profile check that resume() has (security, low). #prepareWrite rejects a read approval for a different profile (lines 790–795). #isUnapplied (lines 599–615) doesn't. If the user switched profiles, abort() reads the other profile's versions (for example 0, which is below newVersion) and clears a mutation that may have been applied on the original profile. The fix is to add the same sub !== mutation.profileId check.

fixed: e14884b

The registration service rejects a JSON content type with an empty body, so every DELETE /recovery/lock/:lock_id returned 400 and the lease was never released.

Co-authored-by: Cursor <cursoragent@cursor.com>

@ieow ieow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@huggingbot
huggingbot added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 22d111e Nov 16, 2026
45 checks passed
@huggingbot
huggingbot deleted the feat/mfa-recovery-reconciliation branch November 16, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants