You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: deflake the two nightly macOS timing tests (resolver permit-wait, decode-hang detection) #2809
Nightly verify run 41227995596 on main 0a0b0557 failed on the macOS Apple Silicon runner: 11213 passed, 2 failed, in cargo test --workspace --profile test-fast --features metal,accelerate --no-fail-fast -- --test-threads=1:
server::media_net::resolver_tests::permit_wait_does_not_eat_resolver_time (panicked at src/server/media_net_resolver_tests.rs:344)
server::model_provider::tests::drain_generation_events_impl_detects_phase2_decode_hang (panicked at src/server/model_provider_tests.rs:703)
The nightly was green the two previous nights (bebdd62c, 5c8ef80a). No merge between them touched server::media_net or server::model_provider (the window contains #2799 Windows PTX paths, #2802 a tests-only TF32 pin, #2801 Gemma3n gelu numerics), so these are load-timing failures on the shared runner, not regressions. Both tests assert wall-clock or scheduling-sensitive invariants that the #1057 precedent (tp_e2e flake, fixed with one-sided deterministic bounds) already addresses elsewhere.
Proposed Solution
Reproduce both failures under load on a Linux host first (the tests are server code and run on CUDA/Linux builds too; if either is macOS-only, say so in the PR). Then apply the #1057 pattern: one-sided deterministic bounds or a measured margin so the test fails only when the invariant is genuinely violated, never on scheduler noise. Do not widen a bound past the point where a real violation would still fail; state in the PR what a real violation looks like and why the new bound still catches it.
Acceptance Criteria
Both tests pass 20 consecutive local runs under a concurrent build or test load
The PR states, per test, the invariant it guards and why the new bound still catches a real violation
No other test in the two modules is loosened
The next nightly run on main is green
Verification
cargo test --profile test-fast --features cuda --lib permit_wait_does_not_eat_resolver_time -- --test-threads=1
cargo test --profile test-fast --features cuda --lib drain_generation_events_impl_detects_phase2_decode_hang -- --test-threads=1
Repeat 20x under load (e.g. cargo build --workspace in parallel) before and after the change: the before-loop should reproduce at least one failure to prove the flake, and the after-loop must be 20/20.
Activity
Nightly verify 41227995596 on main 0a0b055 failed on the macOS Apple …