Skip to content

Record locks: a drain can count a renewal's recall as confirmed early, and skips a recall inside the retry backoff #3268

Open

Description

@kriszyp

Problem

Two narrow gaps in how a record-lock drain (quiesce / quiesceDelegations, resources/recordLockCoordinator.ts) decides what is quiesced:

  1. A recall can confirm a renewal before the old token has stopped admitting. The home renews a delegation in place to token T2, then recalls T2 before the renewal reply reaches the delegate. The delegate still holds T1, takes the token-mismatch branch of onDelegationRecall, records recalledToken, and resolves. The home counts the grant as recalled while T1 admissions can still commit until the renewal reply is processed. A drain proof built on that count can let an operator skip the lease wait.
  2. A drain inside the recall backoff sends nothing. If a contention recall failed less than RECALL_RETRY_MS (1 s) earlier, #beginRecall returns without sending, and the drain reports the grant outstanding without contacting the delegate. A delegate that has recovered still fails the drain proof, so the operator must fence it or wait out the lease. This is a liveness cost, not a safety one.

Status

  • Both found by pre-push review on the fenced-transition work (harper-pro#973), classified pre-existing. Neither is reproduced.
  • Gap 1 is narrow in practice. The delegate's own drain surrenders T1, harper-pro's activation requires every surviving node to prove its drain, and a delegation request is refused while its surrender is in flight.

Possible fix

  1. Make a recall that hits a pending renewal resolve only once that renewal path has drained the inherited admissions. This means threading a settle signal through the acquisition loop's exits.
  2. Let a drain bypass the contention backoff once.

Review-Deferral: 5f78fffa79c8bea3 severity=minor category=concurrency
Review-Deferral: 401782fcd8a36fd7 severity=minor category=reliability

🤖 Generated by Claude Code (Claude Opus); posted via @kriszyp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P3

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions