Skip to content

fix: deflake the two nightly macOS timing tests (resolver permit-wait, decode-hang detection) #2809

Closed

Description

@inureyes

Problem / Background

Nightly verify run 41227995596 on main 0a0b0557 failed on the macOS Apple Silicon runner: 11213 passed, 2 failed, in cargo test --workspace --profile test-fast --features metal,accelerate --no-fail-fast -- --test-threads=1:

  • server::media_net::resolver_tests::permit_wait_does_not_eat_resolver_time (panicked at src/server/media_net_resolver_tests.rs:344)
  • server::model_provider::tests::drain_generation_events_impl_detects_phase2_decode_hang (panicked at src/server/model_provider_tests.rs:703)

The nightly was green the two previous nights (bebdd62c, 5c8ef80a). No merge between them touched server::media_net or server::model_provider (the window contains #2799 Windows PTX paths, #2802 a tests-only TF32 pin, #2801 Gemma3n gelu numerics), so these are load-timing failures on the shared runner, not regressions. Both tests assert wall-clock or scheduling-sensitive invariants that the #1057 precedent (tp_e2e flake, fixed with one-sided deterministic bounds) already addresses elsewhere.

Proposed Solution

Reproduce both failures under load on a Linux host first (the tests are server code and run on CUDA/Linux builds too; if either is macOS-only, say so in the PR). Then apply the #1057 pattern: one-sided deterministic bounds or a measured margin so the test fails only when the invariant is genuinely violated, never on scheduler noise. Do not widen a bound past the point where a real violation would still fail; state in the PR what a real violation looks like and why the new bound still catches it.

Acceptance Criteria

  • Both tests pass 20 consecutive local runs under a concurrent build or test load
  • The PR states, per test, the invariant it guards and why the new bound still catches a real violation
  • No other test in the two modules is loosened
  • The next nightly run on main is green

Verification

cargo test --profile test-fast --features cuda --lib permit_wait_does_not_eat_resolver_time -- --test-threads=1
cargo test --profile test-fast --features cuda --lib drain_generation_events_impl_detects_phase2_decode_hang -- --test-threads=1

Repeat 20x under load (e.g. cargo build --workspace in parallel) before and after the change: the before-loop should reproduce at least one failure to prove the flake, and the after-loop must be 20/20.

Related: #1057.

Activity

  1. added
    type:bugBug fixes, error corrections, or issue resolutions
    status:backlogIn the backlog, not yet ready
    and removed
    status:backlogIn the backlog, not yet ready
    on Nov 1, 2026
  2. added a commit that references this issue on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:lowLow prioritystatus:doneCompletedtype:bugBug fixes, error corrections, or issue resolutions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions