test(delivery): fix bus-lag test deadlock via non-blocking observation send (RIG-2514)

[rigel-mintaka]

Problem

TestBusLagResubscribesAndKeepsDelivering, TestBusLagResubscribeDeliversWindowMessageLive, and TestBusLagTriggersSweepNotLoss in go/internal/delivery/consumer_test.go flake under cross-test load — intermittently hanging the whole internal/delivery package to the -timeout, which reddens the required CI check and the local jj-hp pre-push moon ci gate. Reproduced 7 of 12 consecutive go test -count=1 -timeout 60s ./internal/delivery/ runs on pristine main; each test passes in isolation.

Root cause

The test fakes signal each observed dispatch/wake on a buffered recorded channel (cap 1024), and the send is blocking (d.recorded <- struct{}{}). The three bus-lag tests deliberately flood 1100 > 1024 messages to force a live-buffer overrun. The recorded fact is appended to a mutex-guarded slice before the token send, and every waiter (waitForMessage / waitForDispatches / waitForWakes / waitFor) re-checks that slice each loop — so once a waiter finds its target it stops draining recorded. With the buffer full, the consumer's Run goroutine then wedges on the next blocking send inside DispatchControl, and:

• in ...Resubscribes..., afterResubscribe never fires → t.Fatal at the 10s resubscribe assertion;
• in ...DeliversWindowMessageLive / ...TriggersSweepNotLoss, cleanup's cancel()+<-done never returns because Run is blocked on the send, not selecting on ctx.

The stalled goroutine hangs the package to the outer -timeout. In isolation the waiter drains fast enough that the buffer never saturates; under parallel-package load it does — a classic observation-channel-as-backpressure deadlock.

Goroutine dumps confirm the wedge at helpers_test.go d.recorded <- struct{}{}, reached from sweepSession / live fanOut, with the test goroutine already past its assertion.

Fix

Make the observation send non-blocking at all four identical sites via one documented helper, signalObserved(ch) (select { case ch <- struct{}{}: default: }). The token is only a wakeup hint — the recorded fact already lives in the fake's calls slice before the send — so dropping a token is safe and starvation-free: a drop happens only when the buffer is full (non-empty), so a blocked waiter always has a token to drain and loop back to re-check the snapshot; when the buffer is empty the send always lands.

Per the house no-retries rule this is a determinism fix, not a retry: the observation channel simply stops exerting backpressure on the code under test.

Verification

• Repro: 7/12 failures on pristine main f97e0438 (each a panic: test timed out with testing.(*common).Fatal on the stack).
• After fix: 20/20 go test -tags unix -count=1 -timeout 60s ./internal/delivery/ runs green; total wall time for 20 runs dropped from 441s (hangs) to 26s.
• -race full delivery suite green; gofmt/go vet/golangci-lint clean.

Closes RIG-2514.

[linear-code]

RIG-2514

[github-actions]

Compass engineering docs preview: https://mintaka-rig-2514-flaky-bus-l.compass-eng-docs.pages.dev

Deployed from mintaka/rig-2514-flaky-bus-lag-tests at 4d4fbc0.