Harden 07_signals against signal-subscription race - #272
Merged
Conversation
Tests 1, 3, 4, and 5 sent a single fire-once signal after a fixed
pg_sleep. duroxide drops any external event raised before the
wait_for_signal subscription is registered ("no pending subscription
slot", duroxide #154), so under CI/load the signal could land before
the subscription exists and the workflow would wait out its timeout,
causing df.await_instance to time out (status: running).
Replace the fixed-sleep + fire-once pattern with a retry loop that
re-raises the signal while the instance is still 'running' and stops
once it leaves that state, guaranteeing at least one signal lands after
the subscription is registered. Checking status before each send avoids
signaling an already-completed instance.
Verified with 10 consecutive repeats:
./scripts/test-e2e-local.sh 07_signals 10 -> 10 passed, 0 failed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
07_signalsis flaky in CI. Observed failure (PR #263):Reproduced locally under load — the flake is not unique to one sub-test. Tests 1, 3, 4, and 5 all used the same fragile pattern: a fixed
pg_sleepfollowed by a single fire-oncedf.signal.Root cause
The workflows reach their
df.wait_for_signalsubscription only after the runtime starts them (and, for Tests 1 and 5, after a leading activity runs). duroxide drops any external event raised before that subscription exists:PG logs confirmed the leading
INSERTcommitted ~555ms before the subscription was registered, so even gating on observable side effects isn't reliable. When the signal is dropped, the workflow waits out its (longer) timeout anddf.await_instance(_, 10)times out withstatus: running. This is the duroxide unmatched-event-forwarding limitation (#154).Fix (test-only)
Replace the fixed-sleep + fire-once pattern in Tests 1, 3, 4, and 5 with a retry loop that re-raises the signal while the instance is still
runningand stops once it leaves that state. This guarantees at least one signal lands after the subscription is registered; earlier sends are harmlessly dropped. Checking status before each send avoids signaling an already-completed instance.No production code changes — the underlying duroxide behavior is tracked in #154.
Verification