You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
serve: the launch summary says the service is ready even when the post-launch smoke test failed #523
After launching a managed engine in an interactive terminal, rocm serve runs a one-request smoke test (serve_summary::run_smoke_test) to measure time-to-first-token and throughput, then prints a deployment summary.
The smoke test is documented as best-effort:
Best-effort: any error (server not OpenAI-compatible, refused, timed out) yields empty metrics rather than failing the launch.
So every failure collapses into SmokeMetrics::default(), and the summary still prints status: ready followed by usage instructions. The only trace of the failure is that the two metrics read n/a, with no reason given.
Why it matters
"The server could not report token timings" and "the engine died while answering its first request" produce the same summary. In the second case the user is told the service is ready and shown how to use it, although it is not serving.
This is the "claim of an outcome" case AGENTS.md §3 describes: a line the CLI prints about what it just did should be asserted together with the resulting state.
Expected behavior
The launch itself can stay best-effort — a smoke test that fails need not fail the launch — but the summary should not present a service as ready when its first request failed. At minimum, say why the metrics are missing.
Possible solution
DeploymentSummary already carries notes, which the summary prints. Recording the smoke test's error there (e.g. smoke test failed: the engine closed the connection mid-response) would make the two cases distinguishable at negligible cost. Whether a failed smoke test should also change the reported status is a product decision.
How this surfaced
While fixing #514 — the smoke test's stream reader spinning forever when the engine closes mid-response. With that fixed, the same situation now completes quickly, and it becomes visible that the summary it then prints still reports the service as ready.
Environment
Built from main. Only reached when stdout is a terminal, since summary mode requires one.
Activity
fix(serve): say so in the summary when the post-launch smoke test fai…
fix(serve): say so in the summary when the post-launch smoke test fai…
fix(serve): say so in the summary when the post-launch smoke test fai…