Skip to content

serve: the launch summary says the service is ready even when the post-launch smoke test failed #523

Closed

Description

@rominf

Current behavior

After launching a managed engine in an interactive terminal, rocm serve runs a one-request smoke test (serve_summary::run_smoke_test) to measure time-to-first-token and throughput, then prints a deployment summary.

The smoke test is documented as best-effort:

Best-effort: any error (server not OpenAI-compatible, refused, timed out) yields empty metrics rather than failing the launch.

So every failure collapses into SmokeMetrics::default(), and the summary still prints status: ready followed by usage instructions. The only trace of the failure is that the two metrics read n/a, with no reason given.

Why it matters

"The server could not report token timings" and "the engine died while answering its first request" produce the same summary. In the second case the user is told the service is ready and shown how to use it, although it is not serving.

This is the "claim of an outcome" case AGENTS.md §3 describes: a line the CLI prints about what it just did should be asserted together with the resulting state.

Expected behavior

The launch itself can stay best-effort — a smoke test that fails need not fail the launch — but the summary should not present a service as ready when its first request failed. At minimum, say why the metrics are missing.

Possible solution

DeploymentSummary already carries notes, which the summary prints. Recording the smoke test's error there (e.g. smoke test failed: the engine closed the connection mid-response) would make the two cases distinguishable at negligible cost. Whether a failed smoke test should also change the reported status is a product decision.

How this surfaced

While fixing #514 — the smoke test's stream reader spinning forever when the engine closes mid-response. With that fixed, the same situation now completes quickly, and it becomes visible that the summary it then prints still reports the service as ready.

Environment

  • Built from main. Only reached when stdout is a terminal, since summary mode requires one.

Activity

  1. added 3 commits that reference this issue on Nov 16, 2026
    82a6754
    b20182e
    e5bcd69
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions