setResumingThread assertion fires during iroh exam scenario-2 retries (guest-panic recovery path)

[lannbot]

Consumer-observed under v0.2.0 while upgrading polymorph-iroh's deltic pin (exam gate just exam-deltic, host-deltic endpoint exam, scenario 2 "relay echo between two endpoint instances").

Observation

Scenario 2 passed on attempt 3 of its retry budget; attempts 1–2 failed with deltic's own invariant assertion rather than the documented guest failure:

AssertionError: two activations claim the resumed ambient at once — the resolve-one-per-turn discipline was violated

(runtime/src/task/scheduler.ts setResumingThread, the M3-A ambient slot.) The run also recorded 2 guest panics from the KNOWN, documented guest-side RefCell borrow hazard (polymorph-iroh host-deltic/src/run-endpoint.ts header: State::drain's borrow held across a block_on yield point — their bug, latent on every host, and the reason the exam has a retry budget at all).

Why it's worth a look

• The RefCell panic path is the guest aborting with a trap mid-handshake, with other activations of the same instance in flight. The assertion firing there suggests the trap-unwind/poisoning path under jspi can leave/deliver two settled activations racing the ambient claim — a host invariant violation even when triggered by a guest fault.
• Unknown whether this is new in 0.2.0 or pre-existing but unobserved: scheduler: tick skips threads whose instance is not host-enterable #155 (tick skips non-host-enterable threads) changed scheduler selection immediately before the v0.2.0 cut, and poisoned-instance retirement (host streams: a guest trap strands parked host read/write promises #66-era resumeWith early-return) is exactly the code neighborhood.
• The exam's retry budget masks it operationally (their gate is green; the 0.2.0 family upgrade landed), so this is not release-blocking — but an assert_ that can fire on a reachable path wants either a repro + fix or a documented downgrade to a defined recovery.

Repro pointer

polymorph-iroh @ the deltic-0.2.0 branch (or main once merged): just setup && just exam-deltic, scenario 2; flaky (attempt-dependent — interleaving of the signing window with parked pollers). A deltic-local repro would be: instance with a parked callback-ABI activation + a second activation that traps mid-block_on-style async import; inspect whether two resumeWith/engine resumptions can both claim the ambient during the unwind.

[lannbot]

Investigated 2026-08-21. Verdict: real invariant violation, two confirmed
mechanisms, both reproduced deterministically at the scheduler level (no
wasm needed). Neither was introduced by #155 — the defect predates v0.1.0.

Repro

Probes use the park_state_settle_test.ts scaffolding (real Store /
Task / Thread / SuspensionPoint, jspi-mode waitable-set.wait parks,
task.implicitThread set as a real lifted call would). Probe file is
uncommitted scratch (runtime/tests/issue158_probe_test.ts in the
investigation worktree); its A2/B2 cases assert the buggy behavior, so they
land inverted as regressions with the fix.

Mechanism A (confirmed): trap-arm asymmetry in SuspensionPoint.#resumeInner

runtime/src/jspi/bridge.ts — the success arm consumes a live driver claim
before taking its own (consumeClaimIfRunning() at :540, then
setResumingThread at :550, with a comment naming exactly the
running-guest-settles-another-activation shape). The produce-throw arm
(:524-526) skips the consume and calls setResumingThread directly.

Consequence: a running activation (its own engine-resume claim still live
in the resumingThread slot) that synchronously delivers a resume whose
produce throws trips the assert:

• probe A1 (control): same delivery, produce succeeds → claim consumed, no
  assert;
• probe A2: produce throws → AssertionError: two activations claim the resumed ambient at once — the resolve-one-per-turn discipline was violated.

Guest-reachable delivery-trap shapes: a cancellation delivery whose
resumption traps (the class f7abf96 fixed the bracket discipline for —
note its regression test covers only the generator-Thread arm; under jspi
the same cancellable block point is a parked SuspensionPoint, i.e. this
arm); a stream/future rendezvous where the parked peer's copy produce
traps (lowering fault); the #66 retirement walk's peer notifications during
a poisoning unwind ("can run peer code", task/streams.ts:865) — which is the
guest-panic recovery path named in this issue's title.

When it fires, the assert preempts #fail(e): the parked guest never
receives its trap, and the AssertionError propagates instead — e.g. through
Task.requestCancellation's catch, poisoning the callee with the assertion
as the recorded cause rather than the real trap — surfacing to the embedder
as our invariant failure in place of the documented guest failure. That is
exactly the exam's observed presentation.

Mechanism B (confirmed): the one-slot claim cannot represent two pending resumptions

Even on the success arm, consumeClaimIfRunning() only releases when
activationOf() === resumingThread. If guest code of activation X runs
while a different activation Y's claim is pending and synchronously
delivers a resume to Z, nothing consumes Y's claim → assert (probe B2;
probe B shows the empty-bracket variant self-consumes via the claims-top
fallback, which is why ordinary driver traffic doesn't trip it).

X-runs-during-Y's-window is reachable three ways:

1. engine chunk ordering: X's fast-path-hop continuation (jspi pin (j)) was
   queued before Y's suspension was settled, so X's guest chunk runs first,
   under X's claimingFn bracket, with Y's claim live;
2. driveAsync services settled tails before its claim check
   (exec/boundary.ts:796 vs :807, deliberately — the phantom-state gate), so
   a tail's guest chunk can run inside a live claim window;
3. the ambient slots are module-global, not per-Store (resumingThread,
   activationClaims, threadStack — task/scheduler.ts:526/428/291). Two
   independent instantiations interleave claims through one slot. The
   tick/driver refusal discipline is also global, so it mostly serializes —
   but (1) and (2) are not gated. The exam's scenario 2 runs exactly two
   endpoint instances, and its flakiness is interleaving-dependent, which
   fits.

Dating

• v0.2.0 == 59b4d34, the merge of scheduler: tick skips threads whose instance is not host-enterable #155 itself — so the exam build did
  include scheduler: tick skips threads whose instance is not host-enterable #155, but the asymmetry long predates it: present in v0.2.0's
  bridge.ts byte-identically, introduced with the explicit-ambient work
  (M3A-1, 097ae1a, 2026-08-09), before v0.1.0. scheduler: tick skips threads whose instance is not host-enterable #155 is not the cause.
• scheduler: resumeWith has the same not-host-enterable window as tick, and the tick fix cannot transfer #156 (a458161) and f7abf96 landed after the v0.2.0 cut, same day. Both
  fixed adjacent poison-bookkeeping bugs (tick's unrecorded poison marker;
  tails dispatched into locked instances) that made the panic-recovery
  window messier in the exam build — plausibly why the assert stopped
  reproducing on attempt 3 — but both probes fire on current main, so the
  underlying defect is still live.

Recommendation

1. Fix mechanism A now: mirror the success arm — consumeClaimIfRunning()
   before the trap-arm setResumingThread. Symmetric, small, and the arm's
   own comment already argues it is "a RESUMPTION too … exactly like the
   value path". Scheduler-core review discipline applies (single-owner
   territory). Extend the f7abf96 regression to the SuspensionPoint arm at
   the same time, and land inverted A1/A2 probes as tests.
2. Mechanism B needs a design decision, not a hotfix: the single
   resumingThread slot genuinely cannot attribute two legitimately-pending
   engine resumptions, and the assert is correctly detecting lost
   attribution capacity (the Guest callback invoked with null context slot (wit-bindgen async_support.rs:578) — polymorph-tls webcrypto-composed target #24 failure class). Options: promote the slot
   to a small queue with the existing per-entry release edges (matching
   activationClaims' shape), and/or make the ambient state per-Store so
   independent instantiations stop sharing it. A bare downgrade of the
   assert to a defined recovery without one of those risks silent Guest callback invoked with null context slot (wit-bindgen async_support.rs:578) — polymorph-tls webcrypto-composed target #24-class
   misattribution — recommend against.

[lannbot]

Mechanism A is fixed by #207 (trap arm of SuspensionPoint.#resumeInner now mirrors the success arm's consumeClaimIfRunning(); regression tests in runtime/tests/resume_claim_discipline_test.ts, including a pin that mechanism B still asserts so a future change flips loudly).

This issue stays open for what remains:

1. Mechanism B — the single-slot resumingThread claim cannot represent two legitimately-pending engine resumptions (and the ambient slots are module-global across instantiations). Needs a design decision: claim queue with per-entry release edges, and/or per-Store ambient state. A bare assert downgrade is not recommended (the assert correctly detects Guest callback invoked with null context slot (wit-bindgen async_support.rs:578) — polymorph-tls webcrypto-composed target #24-class attribution loss).
2. SP-arm poisoning route (found during the fix): under jspi, a trapping cancellation delivery reaches the guest as a rejected import promise, so requestCancellation's synchronous poisoning branch does not run on this arm — poisoning lands later via the settled-tail failure path instead. Observable end state likely converges with definitions.py's synchronous model, but nobody has verified the intermediate-state parity (e.g. what a sibling activation observes in the window). The new guest-shaped test pins the current behavior explicitly.

[lannbot]

Mechanism-B design evidence, as proposed: tier-3 measurement + the
cross-store gate probe. Both decisive.

Measurement 1: tier 3 of ambient resolution is dead

Question: does resolveAmbient() (runtime/src/task/scheduler.ts:657) ever
actually decide via tier 3 (resumingThread), or do the #24 sentinels +
settle-time tier-2 claims always answer first?

Instrumented run (temporary patch: count + stack-trace every read where
tiers 1-2 are empty and resumingThread !== null, behavior unchanged), on
main @ 1ded75c (includes the mechanism-A fix, #207):

lane	result	tier-3-deciding reads
conformance, FIFO	1257/0 (totals unchanged)	0
sched-seeds (SEED=1, SEED=4242)	green	0
test-runtime (561 tests, incl. all jspi pins)	green	0
smoke-tls (the three-async-component #24 corpus)	PASS	0

Removal run (temporary patch: tier 3 removed from resolveAmbient
entirely — threadStack ?? activationClaims ?? undefined), covering the
engine lanes the instrumented run couldn't:

test-runtime, test-protocol, conformance (1257/0), sched-seeds, shells
(sm + node + jsc + bun — all "OK, matches expectation"), browsers
(chromium + firefox), smoke-tls, smoke-c0 — all green, no expectation
changes.

Conclusion: the slot's attribution role (tier 3) is vestigial post-#24 —
the sentinel discipline always answers first, on every engine lane we have.
Its only live role is the scheduling gate ("a resumption is pending; don't
schedule"). Both temporary patches were reverted; nothing committed.

Measurement 2: the global gate kills concurrent stores on its own (#210)

Probed the suspected claimHops hazard: the driver's speculative claim
(boundary.ts ~:1048) is held across the await of a guest's host-import wait,
the slot is global, and every driver yields at its loop top while ANY claim
is live. Confirmed: an idle second store's driveStoreAsync dies with
AssertionError: driveAsync: a resumed-activation claim was never released
after ~311ms (10,000 hops) while the first store dwells on a slow import.
Filed as #210 — distinct failure mode (no guest fault, no second
resumption), same root: module-global single-slot claim state.

Firmed-up design recommendation

The measurement removes the hard part. Since tier 3 never decides
attribution:

1. Retire tier 3 from resolveAmbient (pinned by the differential
   evidence above; the M3A-1 methodology, re-run).
2. Make the gate per-Store and multi-entry: Store.pendingResumptions
   — a set of resumed-but-not-yet-run activations. Claim = add (assert
   no-duplicate only); release edges unchanged per entry (park consumes via
   consumeClaimIfRunning, finish releases via noteAwaiting →
   releaseClaimOf, the driver clears its own speculative entry).
   Store.tick / the drivers consult only their own store's set.
   • Same-store: strictly more conservative than today (where a second
     legitimate claimant crashes) — tick keeps refusing until all pending
     entries die. One-per-turn stays as per-store policy.
   • Cross-store: de-serializes, safe by disjointness (activations never
     cross stores). Fixes driveAsync's global speculative claim starves concurrent stores: an idle instantiation's driver dies at 10k hops while another dwells on a slow import #210 and setResumingThread assertion fires during iroh exam scenario-2 retries (guest-panic recovery path) #158-mechanism-B's cross-store shape.
   • The "two activations claim the resumed ambient" assert disappears with
     the slot; the invariant it protected (tier-3 attribution unambiguity)
     no longer exists to protect.
3. All claim/release sites already have the store in hand
   (SuspensionPoint.#store, blockCurrentActivation's input,
   noteAwaiting is a Store method, the driver's loop owns its store), so
   no read-site plumbing.
4. Regression coverage to add with the change: the mechanism-B pin in
   resume_claim_discipline_test.ts flips (two same-store pending
   resumptions no longer assert); a two-store concurrent-driver test (the
   driveAsync's global speculative claim starves concurrent stores: an idle instantiation's driver dies at 10k hops while another dwells on a slow import #210 probe, inverted); and the tier-3 retirement is pinned by the
   corpus + shells + browsers lanes as usual.

Remaining open question from the mechanism-A fix, unchanged: whether the
SP arm of a trapping cancellation delivery should also poison synchronously
(currently poisoning arrives via the settled-tail failure path) — separate
decision, still owed.