Skip to content

Harden 07_signals against signal-subscription race - #272

Merged
Pino de Candia (pinodeca) merged 1 commit into
mainfrom
fix/harden-07-signals-flaky
Jun 29, 2026
Merged

Harden 07_signals against signal-subscription race#272
Pino de Candia (pinodeca) merged 1 commit into
mainfrom
fix/harden-07-signals-flaky

Conversation

@pinodeca

Copy link
Copy Markdown
Contributor

Problem

07_signals is flaky in CI. Observed failure (PR #263):

07_signals.sql:278: ERROR:  Timeout after 10s waiting for instance 16d0826e (status: running).

Reproduced locally under load — the flake is not unique to one sub-test. Tests 1, 3, 4, and 5 all used the same fragile pattern: a fixed pg_sleep followed by a single fire-once df.signal.

Root cause

The workflows reach their df.wait_for_signal subscription only after the runtime starts them (and, for Tests 1 and 5, after a leading activity runs). duroxide drops any external event raised before that subscription exists:

WARN duroxide::runtime::replay_engine: skipping ExternalEvent delivery:
  no pending subscription slot (event remains in history for audit)
  instance=16d0826e event_name=test_approval_then

PG logs confirmed the leading INSERT committed ~555ms before the subscription was registered, so even gating on observable side effects isn't reliable. When the signal is dropped, the workflow waits out its (longer) timeout and df.await_instance(_, 10) times out with status: running. This is the duroxide unmatched-event-forwarding limitation (#154).

Fix (test-only)

Replace the fixed-sleep + fire-once pattern in Tests 1, 3, 4, and 5 with a retry loop that re-raises the signal while the instance is still running and stops once it leaves that state. This guarantees at least one signal lands after the subscription is registered; earlier sends are harmlessly dropped. Checking status before each send avoids signaling an already-completed instance.

No production code changes — the underlying duroxide behavior is tracked in #154.

Verification

./scripts/test-e2e-local.sh 07_signals 10
=> Results: 10 passed, 0 failed

Tests 1, 3, 4, and 5 sent a single fire-once signal after a fixed
pg_sleep. duroxide drops any external event raised before the
wait_for_signal subscription is registered ("no pending subscription
slot", duroxide #154), so under CI/load the signal could land before
the subscription exists and the workflow would wait out its timeout,
causing df.await_instance to time out (status: running).

Replace the fixed-sleep + fire-once pattern with a retry loop that
re-raises the signal while the instance is still 'running' and stops
once it leaves that state, guaranteeing at least one signal lands after
the subscription is registered. Checking status before each send avoids
signaling an already-completed instance.

Verified with 10 consecutive repeats:
  ./scripts/test-e2e-local.sh 07_signals 10  -> 10 passed, 0 failed
@pinodeca
Pino de Candia (pinodeca) merged commit a7de36a into main Jun 29, 2026
5 checks passed
@pinodeca
Pino de Candia (pinodeca) deleted the fix/harden-07-signals-flaky branch June 29, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant