Skip to content

fix(http1): avoid hanging a pipelined connection after a backpressured response - #2

Open
fotinakis wants to merge 1 commit into
masterfrom
fix/http1-pipelined-response-wakeup
Open

fix(http1): avoid hanging a pipelined connection after a backpressured response#2
fotinakis wants to merge 1 commit into
masterfrom
fix/http1-pipelined-response-wakeup

Conversation

@fotinakis

@fotinakis fotinakis commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

An HTTP/1 server connection can hang permanently when a pipelined request is parsed while the previous response is still flushing under write backpressure. The pipelined request's response gets buffered but never written, and the task parks with no pending IO to wake it, so the connection sits open and idle until the peer times out.

Sequence

  1. A client pipelines request B behind request A on one connection.
  2. A's response is large and the socket is slow, so it flushes across many polls under backpressure.
  3. While A's response is draining, request B is read and dispatched to the service, so a response for B is ready by the time A finishes.
  4. When A's response finally flushes, the write side idles back to Writing::Init — but poll_loop yields without writing B's response, and parks with nothing pending on the IO. Nothing ever wakes the task, so B's response is never sent.

Final observed state is State { reading: KeepAlive, writing: Init, keep_alive: Disabled } with a dispatched-but-unwritten response in hand.

Root cause

Two gaps in the dispatcher's write path combine:

1. can_write_again doesn't count a freshly dispatched response. It only reported "more to write" for an in-progress body (body_rx):

fn can_write_again(&mut self) -> bool {
    !self.is_closing && self.body_rx.is_some() && self.conn.can_write_body()
}

When A's flush completes and idles the write side, body_rx is None (A is done) even though B has been dispatched and its head hasn't been written. So wants_write_again is false and poll_loop heads for its exit.

2. The write re-check buffers B's response but doesn't flush it. Once can_write_again reports B as writable, the loop's write re-check runs poll_write, which encodes and buffers B's response — but returns Pending without flushing, because the write buffer isn't full yet. The re-check then returned on that Pending:

if self.poll_write(cx)?.is_pending() {
    return Poll::Ready(Ok(()));
}

So B's response ends up committed in write_buf, the task yields, and nothing registered a wake-up — a lost wakeup. B is never flushed to the socket.

Fix

Both changes are in dispatch.rs:

  • can_write_again also returns true when a response has been dispatched (should_poll()) and the connection can write its head (can_write_head()), so the loop writes it instead of yielding.
  • After a Pending write in the loop's re-check, if there's no in-progress body, flush before yielding, so a just-buffered dispatched response actually goes out and — under backpressure — registers a wake-up. Gating on the no-body case leaves the streaming flush cadence untouched (covered by tests/ready_on_poll_stream.rs).

Testing

Found and reproduced with a deterministic simulation harness around the HTTP/1 server: a single-threaded, seeded scheduler driving the real serve_connection future over an in-memory transport with bounded send buffers (so poll_write returns Pending like a full socket), randomized request pipelining, and a liveness check that the connection must terminate.

  • The reproducing seed hangs on master and completes cleanly with this fix.
  • A 5,000-seed sweep shows no remaining hangs and no busy-looping (the harness caps steps and would flag a spin).
  • With this fix plus fix(http1): flush buffered data on shutdown even when pipeline-flushing #1, a 10,000-seed sweep of the same harness is completely clean.
  • The full cargo test --features full suite passes (288 tests).

Notes

I didn't add an in-tree regression test here: reliably reproducing the parse-B-while-A-flushes-under-backpressure interleaving needs a controllable-backpressure IO and step-level scheduling, which is what the external simulation harness provides. Happy to port a minimal deterministic reproducer into tests/ if you'd like one. The maintainer note on hyperium#4018 that poll_loop is fragile and would benefit from an explicit Flushing state is consistent with what this bug looks like from the inside — this change is a targeted fix rather than that larger refactor.

@fotinakis
fotinakis force-pushed the fix/http1-pipelined-response-wakeup branch from 93a2647 to 71208a5 Compare July 13, 2026 02:23
…esponse

An HTTP/1 server connection can wedge permanently when a pipelined request is
parsed while the previous response is still flushing under write backpressure.

Sequence:
- The client pipelines request B behind request A.
- Request A's response is large and the socket is slow, so it flushes across
  many polls under backpressure.
- While A's response drains, request B is parsed and dispatched, so a response
  is already waiting once A finishes.
- When A's response finally flushes, the connection idles the write side back
  to `Init`, but `poll_loop` yields without writing B's response and parks with
  no pending IO. Nothing ever wakes it, so B's response is never sent and the
  connection hangs until the peer times out.

Two gaps combine to cause it, both in the dispatcher's write path:

1. `can_write_again` only reported "more to write" for an in-progress response
   body (`body_rx`). It didn't account for a freshly dispatched response whose
   head hasn't been written yet, so `poll_loop` yielded instead of writing it.

2. Even once `can_write_again` reports the pending response, the loop's write
   re-check writes and buffers it but `poll_write` returns `Pending` without
   flushing (the write buffer isn't full yet). The re-check then returned
   without flushing, leaving the committed response buffered with no IO pending
   to wake the task.

Fix both: `can_write_again` also returns true when a response has been
dispatched and the connection can write its head; and the write re-check
flushes after a pending write so the buffered response actually goes out and,
under backpressure, registers a wake-up.

Found via randomized, deterministic simulation testing of the HTTP/1 server
under write backpressure; the fix clears the reproducing case and a sweep of
thousands of seeds shows no remaining hangs.
@fotinakis
fotinakis force-pushed the fix/http1-pipelined-response-wakeup branch from 71208a5 to b2005cb Compare July 14, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant