Skip to content

test(e2e): pin public-bind endpoint authentication (EAI-7409) - #722

Open
fredespi wants to merge 10 commits into
mainfrom
test-e2e-serve-auth-and-selectors
Open

fredespi wants to merge 10 commits into
mainfrom
test-e2e-serve-auth-and-selectors

Conversation

@fredespi

Copy link
Copy Markdown
Collaborator

Summary

Pins the public-bind endpoint authentication contract (EAI-7409) end to end: with --allow-public-bind, the CLI issues a key once with Bearer guidance, and the live endpoint answers 401 without a key, 401 with a wrong key, and 2xx with the issued key. This was the highest-value item left from EAI-8072; the pre-flight refusal was already pinned in the earlier PR.

The scenario is @requires-gpu @requires-engine:vllm, so it skips on Windows/lemonade hosts (where the CLI deliberately refuses a public bind) and runs only on MI300X. Readiness probing retries while the engine binds, and a timeout reports serve output, service log tail and the last endpoint response.

Test plan

  • Linux container gate: 224 scenarios, 222 passed, 2 xfail as expected, 0 unexpected failures; exact CI clippy commands clean.
  • MI300X dispatch earlier on this scenario: green (1 scenario, 6/6 steps).
  • MI300X re-run on the current head is queued behind the serial runner; later dispatches on 2026-08-25 failed with a runner-side vLLM "Failed to infer device type", not a test failure.

Completes the public-bind contract begun in #293. That change pinned the
pre-flight refusal (a non-loopback bind without --allow-public-bind); this pins
the other half, and the security property the coverage audit ranked first: after
opting in, the endpoint must not be open.

`rocm serve --host 0.0.0.0 --allow-public-bind` generates an API key, prints it
exactly once with the `Authorization: Bearer` instructions, and hands the serving
engine a key file to enforce. The scenario drives a real managed serve and then
talks to the endpoint directly: a request with no key and a request with the
wrong key must both be refused as unauthorized, and the key that was just issued
must be accepted.

GPU lane, because the ENGINE enforces the key (the CLI only issues and hands it
over) and `serve` has no plan-only mode, so enforcement cannot be observed on the
mock lane. The pure key-resolution rules — notably that a loopback bind stays
credential-free even when --api-key is passed — are already unit-tested in the
CLI, so this covers only what a unit test cannot: that a live public endpoint
actually authenticates.

Adds an endpoint_api_key field to the test World and exposes three existing serve
helpers to the new step module.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The scenario served straight from a fresh isolated data directory, which has no
ROCm runtime configured, so `rocm serve` refused before it could issue a key:
"device_policy: gpu_required; no active ROCm runtime is configured". Every other
GPU serve scenario opens with `Given a managed runtime is active`, which points
the scenario at the runner's shared managed runtime; this one was missing it.

Found by a scoped GPU dispatch of the scenario, which is also why the run needed
reading rather than trusting: the self-hosted E2E lane is non-blocking, so the
workflow run reports success even when the suite inside it fails.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
A managed serve returns once the supervisor is up, so for the first seconds the
engine has not bound the port and the readiness probe's connection is refused.
The probe treated any transport error as fatal, so the scenario died on that
first refusal before it ever reached its assertions.

Split the request helper: the readiness wait now retries until the endpoint
answers (or the timeout expires), while the assertions keep treating a refused
connection as fatal — by then the endpoint has already answered, so a transport
failure there is a real fault rather than a startup race.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The scenario was tagged @requires-gpu only, so it also selected the native-Windows
GPU lane. There the effective serve engine is lemonade, and the CLI deliberately
refuses a public bind on Windows lemonade because the endpoint key cannot be
enforced through it — a designed-in refusal, not a defect. The scenario would have
driven that refusal and failed deterministically on that lane; the MI300X dispatch
could not surface it because that host resolves to vLLM.

Add @requires-engine:vllm, the same precedent serve-vllm-default-on-instinct uses.
Because vLLM cannot start on Windows or on families that do not prefer it, the
scenario now resolves to a skip on the Strix lanes rather than needing an
expectations row for behaviour that is working as designed. It also keeps this cold
serve off the time-capped Strix lanes.

Also correct the show-once comment: the key does legitimately appear a second time
inside the curl example line, so the assertion counts the dedicated `api key:` line
rather than occurrences of the key.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
A run on MI300X timed out after the full readiness budget with nothing but "did
not answer an authenticated request within 300s", which cannot distinguish the
three failures that look identical from outside: the engine never started, it is
still loading weights, or it is up and rejecting the key it just issued — the
last of which would be a real defect in the contract under test rather than a
slow start.

Carry the engine's own log tail, the serve stdout, and the last observed endpoint
response into the timeout panic, the same diagnostics the other serve steps
already collect.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…nd-selectors

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/tests/e2e.rs
@fredespi
fredespi requested a review from a team as a code owner October 23, 2026 15:05
@fredespi
fredespi requested a review from rominf October 23, 2026 15:05
rominf
rominf previously requested changes Oct 23, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · f75e4ee

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Pins the "missing/invalid credentials are rejected" criterion of EAI-7409, which now resolves to ROCMAI-318. It does not cover the ticket's other criteria (secrets not logged or persisted insecurely, status/log protection), and it does not claim to.

Blocking

  • (new) The new scenario's title breaks the required feature_naming test, so CI is red on Linux and Windows — tests/e2e-cucumber/features/networking.feature:64
    • The scenario is titled Scenario: 4 - A public endpoint ….
    • tests/e2e-cucumber/tests/feature_naming.rs:148 requires per-feature sequential prefixes, and the three scenarios above it use networking-01 to networking-03.
    • I ran cargo test -p e2e-cucumber --test feature_naming in the checkout. It exited 101: scenario_names_are_indexed_sequentially_per_feature panicked with networking.feature: scenario 4 is named "4 - A public endpoint issues a key once and then enforces it" but must start with "networking-04 - ".
    • The same panic is the only failure in both "Test (affected crates)" (job 120489475310) and "windows-build-and-test" (job 120489475455). The last three main runs of that workflow are green.
    • Confidence 100 · mechanical · Fix: rename it to Scenario: networking-04 - A public endpoint issues a key once and then enforces it.

Non-blocking

  • (new) The endpoint URL hardcodes the shared serve port instead of using the existing constant — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:25
    • ENDPOINT_URL = "http://127.0.0.1:11435/v1" copies the value owned by SERVE_PORT (serving_steps.rs:172) and SERVE_BASE_URL (serving_steps.rs:78).
    • If the port moves, ensure_serve_port_free frees the new port while these steps keep probing 11435.
    • Separately, the step sets world.endpoint and world.model_name (lines 119 and 121), but the Then-steps read the constant, not world.endpoint.
    • Confidence 75 · mechanical · Fix: make SERVE_BASE_URL pub(crate) or build the URL from SERVE_PORT, and have the Then-steps read world.endpoint.
  • (new) The new serve step lacks the relaunch and failure evidence its sibling has — endpoint_auth_steps.rs:99-122, compared with serving_steps.rs serve_and_wait / serve_gpu_model
    • It launches through run_rocm_ok with a single attempt.
    • The sibling path reports a non-zero exit together with the engine log, and it has a budgeted relaunch. Per its own comment, that relaunch exists because this model on MI300X has both timed out and served fine within the same run.
    • Here, a non-zero exit panics without the service-log tail. Only the timeout path attaches it.
    • Reusing serve_and_wait as-is would not work, because its readiness probe is unauthenticated and would get 401. So this is a matter of extending the shared helper, not swapping it in.
    • Confidence 70 · logic.
  • (new) The feature's header comment is now inaccurate — tests/e2e-cucumber/features/networking.feature:3-9
    • It says these scenarios "run on the mock (no GPU) on every platform the suite runs on".
    • The new scenario is @requires-gpu @requires-engine:vllm and runs only on vLLM-capable dcgpu hosts.
    • The scenario-level comment is correct; the header is what is now stale.
    • Confidence 75 · mechanical · Fix: qualify the header, e.g. "all but networking-04".

Decisions for the author

  • A cold vLLM serve on the per-PR MI300X lane, versus pinning enforcement at all — tradeoff
    • Enforcement lives in the engine (ROCM_SERVE_API_KEY_FILE → VLLM_API_KEY at engines/vllm/src/process.rs:256), and nothing else tests that path: there is no unit test touching VLLM_API_KEY. So a GPU scenario is the only place this can be proven.
    • The cost is roughly 65 s of serial MI300X time per run (observed: 16:39:36 to 16:40:41). The scenario comment argues the choice, and this note is only to confirm it was deliberate.

Positive signals

  • The readiness wait polls with the issued key and treats a refused connection as "not yet". That means an open endpoint (200 for any bearer) cannot satisfy the 401 assertions, and a stray loopback engine on 11435 produces a red result, never a false green.
  • The timeout panic carries serve stdout, the service-log tail and the last endpoint status. That separates "never started" from "still loading" from "rejecting its own key".

Deployment notes

None

What this covered

  • What was read: every changed file in prw-base...f75e4ee6, read completely. Also: the CLI code that prints and hands over the key (service_launch.rs, serve_cmd.rs), the vLLM adapter's key handling, E2eWorld's Drop teardown, the expectation and tag resolution for @requires-engine, and runtime_steps.rs for the Given step.
  • Verdicts on the added tests:
    1. Would it pass with the behaviour reverted? No. The PR changes no production code; it pins existing behaviour. By reading, if the key stopped reaching vLLM, the readiness wait would pass and the unauthenticated step would get 200 instead of 401, so the scenario fails.
    2. What stays green if one branch breaks? Nothing found. A wrong-key-accepted engine fails the wrong-key step. A key printed but not passed to the engine times out in readiness. A second dedicated api key: line fails the show-once step.
    • I did not run either mutation: the scenario needs a GPU and none was available here. vLLM's guarding of /v1/* is from knowledge of vLLM, not from source in the checkout.
  • Lane gating, cleanup and CI:
    • Gating checked: the scenario skips on Strix Windows, Strix Ubuntu and W7900, and runs on MI300X.
    • Cleanup checked: E2eWorld Drop runs rocm services stop even when a step panics, so the 0.0.0.0 bind lasts no longer than any other serve. No expectations.toml rows are affected.
    • On "E2E tests (MI300X, current)" the new scenario ran and passed (6/6 steps). The job reports 246 scenarios (240 passed, 6 failed). The failures are at chat.feature:70, model_serving.feature:11/16/68, runtime_setup.feature:32 and therock_next_generation.feature:166. Main's MI300X run on the PR's base 1c33248 (run 39975715090) fails at exactly the same six locations with the same count. Five of the six are declared xfail; the one unexpected failure, therock-next-09 (a vLLM/amd-aiter dependency conflict), is the same on main. None of it is attributable to this PR.
  • Local runs: cargo check and cargo clippy -D warnings on the e2e target are clean (exit 0).
  • Passes that did not run:
    • Prior-changes pass: earlier reviews are out of bounds for this run.
    • History: only a light check (recent git log on the feature file).
  • Fan-out: two independent workers, one for the code pass and one for CI attribution. I verified their findings myself.
  • Discussion: not reconciled against the PR's comments or reviews.

…ve URL

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi requested a review from rominf October 23, 2026 18:16
@fredespi
fredespi dismissed rominf’s stale review October 23, 2026 19:32

Requested changes addressed: commits were pushed after this review and all review threads are resolved. Re-review requested.

@fredespi
fredespi requested a review from a team October 23, 2026 19:32

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 02726d1

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Round 2. No change request filed. The earlier change request's blocker was the networking-04 title failing feature_naming. It was standing, not re-found blind, and was settled by putting it to the reviewer after this review: the title now reads networking-04 - …, feature_naming passes 4/4 locally, and "Test (affected crates)" and "windows-build-and-test" are green at this head. The one new blocking-class finding below (a stale sentence in docs/testing.md) understates coverage rather than overstating it, so the automation reports it without gating on it. The only red check, "E2E tests (MI300X, current)", has the same 5 xfail + 1 unexpected failure (therock-next-09) as main's run at the base 1c33248; networking-04 passed all 6 steps there.

Review — needs work

Full review of the whole change.
This PR adds end-to-end evidence for one acceptance criterion of EAI-7409 (now ROCMAI-318): "Missing/invalid credentials are rejected". It does not claim to close the ticket, which is still In Progress.

Blocking

  • (new) docs/testing.md still says this assertion is not exercised anywhere — docs/testing.md:955-958
    The doc says: "Verifying that the running server actually rejects an unauthenticated request requires a live engine; that end-to-end assertion is a deferred follow-up and is not yet exercised by the GPU acceptance scripts." networking-04 is exactly that assertion, so the doc became false when this PR landed the scenario. AGENTS.md §5 requires docs/testing.md to be updated in the same change and the claim to be checked wherever it repeats. The code is the side that governs here: the scenario exists and passed on MI300X at this head.
    Confidence 85 · mechanical · Fix: rewrite the sentence to name networking-04 (features/networking.feature) and say it runs on the MI300X vLLM lane only, which is non-required. Native Windows and lemonade-default hosts skip it.

Non-blocking

  • (standing, expanded) serve_public_bind brings back the serve-then-poll pattern that serve_and_wait was written to replace — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:84-102, :108-129

    • What the in-file guidance says: the doc comment on serve_and_wait in serving_steps.rs says it "Replaces the run_rocm_ok + readiness-poll pair", and adds "A non-zero exit is reported the same way rather than through run_rocm_ok, since the engine log explains those launches too."
    • Lost failure evidence: the new step calls crate::run_rocm_ok and discards the device-state string that ensure_serve_port_free() returns. Its timeout panic carries no device state, stderr, archived log or stop status (serve_failure_evidence).
    • Lost readiness guard: it returns on any 2xx and drops host_serve_target().2 as _. model_is_ready's doc warns that a plain 200 on the shared port 11435 can come from a leaked serve. In that case the "without the key is refused" step would fail with a message that looks like a security defect, when the real cause is a harness leak. This is a misleading red, not a false green.
    • Commit message: 039d0e4 claims "the same diagnostics the other serve steps already collect", which is not accurate. The repo squash-merges, so the squash message should not repeat that claim.

    Confidence 85 · architectural · Fix: add an optional bearer token to model_is_ready/serve_and_wait (and have it return stdout), or make serve_failure_evidence pub(crate). Then route both the non-zero-exit and timeout paths through it, and require the ready-substring in the authenticated /models body.

  • (standing) The feature header comment is now partly stale and not re-wrapped — tests/e2e-cucumber/features/networking.feature:6-9
    Line 6 still says the scenarios drive the binary "against the mock server", and the feature's stated purpose is the CLI's native HTTP stack. networking-04 runs on a real engine, and its HTTP checks go through the harness's reqwest, not the CLI. The edited line 8 is 105 characters while its neighbours wrap at about 80.
    Confidence 90 · mechanical · Fix: re-wrap the header, and say networking-04 is a GPU serve scenario that checks enforcement from the harness.

Decisions for the author

  • (standing) The scenario only runs on a non-required lane — tradeoff
    @requires-engine:vllm limits it to MI300X. .github/workflows/e2e-selfhosted.yml says that check "isn't in the required list" and skips merge_group, so a regression in key enforcement would not block a merge. The lemonade-on-Linux enforcement path (--api-key-file) has no scenario. The tag is well argued (Windows lemonade refuses a public bind by design) and the PR text names the lane. Confirm you accept that this contract is guarded only in an advisory way.
  • (new) What "issues a key once" proves — non-blocking-improvement
    shows_key_once counts lines that start with api key:. It would stay green if the key leaked onto any other line. It also never checks the documented promise that the key does not appear in rocm services or rocm logs. Possible fixes:
    • assert output.matches(key).count() == 2 (the api key: line plus the curl example), or
    • add a step that lists services and checks the key is absent.

Positive signals

  • try_models_status and get_models_status keep two cases apart: "no listener yet" is retried during readiness, and "answered with a status" is asserted. The timeout panic names the three cases that look identical from outside, including the one where the engine rejects the key it was issued, which would be a real contract defect.
  • The @requires-engine:vllm comment explains why this is a tag rather than an xfail row, and the cited facts check out: ensure_public_bind_engine_supported and resolve_endpoint_auth_loopback_stays_credential_free exist as described.

Deployment notes

None

What this covered

  • Read: all four changed files in full, against prw-base 1c33248…02726d1a. Also the surrounding harness (serving_steps.rs serve/readiness helpers, run_rocm_ok, capability tags, the World cleanup), the CLI's public-bind and key-printing code, and how the vLLM and lemonade engines receive the key. The work was split across two independent read-only workers (code pass; agent-instruction/history/comments/commits pass), plus my own verification.
  • Run locally:
    • cargo test -p e2e-cucumber --test feature_naming: 4/4 passed.
    • cargo clippy -p e2e-cucumber --all-targets -- -D warnings: clean.
    • Sign-off check: all 6 commits signed off.
  • CI at the head:
    • Every check passes except "E2E tests (MI300X, current)".
    • In that job, networking-04 passed all 6 steps.
    • The job's 6 failures are the same 6 scenarios that fail on main's run at this base SHA. They are pre-existing, not caused by this PR.
  • Step verdicts:
    • The "without the key" and "wrong key" steps both go red if the engine stops enforcing the key.
    • The "issued key is accepted" step adds nothing beyond the readiness wait.
  • Could not run: GPU scenarios (no GPU here), so the GPU evidence above is CI's, not mine. Prior-changes pass: no earlier review comments were read, by design.
  • Not reconciled: this report was not checked against the PR's comment discussion.

…k, fix feature header

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi requested a review from rominf October 26, 2026 09:28
@fredespi

Copy link
Copy Markdown
Collaborator Author

#722 (review)

Fixed in 9008999. docs/testing.md now names networking-04 and says it runs on the MI300X vLLM lane only (non-required; Windows/lemonade-default hosts skip it). The key-once step now fails if the key appears on any line other than the api key line and the curl example. The networking.feature header is re-wrapped and says networking-04 is a GPU serve scenario checked from the harness. Not done, left for a follow-up: routing serve_public_bind through serve_and_wait/serve_failure_evidence needs a shared-helper change (bearer token in model_is_ready), so it stays out of this PR. The non-required-lane tradeoff is accepted as documented.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9008999

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Round 3. No change request filed. Last round's blocking-rated item (the docs/testing.md sentence calling the end-to-end auth check a deferred follow-up) was standing, not re-found blind; it was settled by putting it to the reviewer after this review: the sentence is gone, the new text names networking-04 and matches the scenario's tags, and no other surface repeats the old claim. The one blocking-class finding below (a bare backtick path where AGENTS.md §5 asks for a relative link, to a file that exists) is reported without gating, consistent with earlier rounds in this repo. The only red check, "E2E tests (MI300X, current)", has the same result as main's run at base 1c33248 (5 xfail + 1 unexpected failure, therock-next-09); networking-04 passed all 6 steps there, so it is not attributed to this PR.

Review — needs work

Full review of the whole change.
Adds end-to-end coverage for the "missing/invalid credentials are rejected" acceptance criterion of EAI-7409. It does not claim to close the ticket, and the ticket stays open (In Progress), which is consistent with that.

Blocking

  • (new) The new docs/testing.md citation uses a bare path instead of a relative link — docs/testing.md:956
    The added text cites the feature file as (`tests/e2e-cucumber/features/networking.feature`). AGENTS.md §5 says a citation you add or edit should be a link relative to the citing file, not a bare backtick path. The reason given is that the docs-links CI job and the lychee hook can only catch a moved or deleted target when it is a link. The same document already does this for sibling feature files, for example [runtime_setup.feature](../tests/e2e-cucumber/features/runtime_setup.feature) at line 527 and [comfyui.feature](...) at line 700.
    Confidence 85 · mechanical · Fix: ([networking.feature](../tests/e2e-cucumber/features/networking.feature)).

Non-blocking

  • (standing, re-found blind in a new form) The new serve step skips the stall relaunch that the other vLLM serve on this lane relies on — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs (serve_public_bind); for comparison, tests/e2e-cucumber/tests/e2e/serving_steps.rs (serve_gpu_model)
    serve_public_bind serves the same target, Qwen/Qwen3.5-0.8B on vLLM via host_serve_target(). It makes one launch and one wait. serve_gpu_model deliberately allows one relaunch for that same model, with the comment "Qwen3.5 has both timed out and served successfully in the same MI300X run". Its retry also calls serve_failure_evidence / rocm services stop first, so a stalled attempt does not keep holding device memory. The new scenario therefore keeps a flake source that the established serve path was written to absorb. It also panics through run_rocm_ok on a non-zero launch, so the service-log evidence that serve_and_wait attaches is missing. The lane is not required, which limits the impact.
    Confidence 70 · logic · Fix: route the launch through the existing serve-and-wait/relaunch machinery, passing the extra --host 0.0.0.0 --allow-public-bind args and a key-bearing readiness probe, instead of a parallel copy.
  • (new) A key the engine rejects only shows up after the full serve timeout — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs (wait_for_authenticated_endpoint)
    A 401 to the issued key is an answer from a live, bound engine. The doc comment itself calls it "a real defect in the contract under test, not a slow start". Even so, the loop keeps polling until the deadline, which spends serial GPU-lane time before reporting a definite failure. Failing fast on 401 would report that defect immediately. The loop also does not check the model id, as model_is_ready does, but the later 401 assertions would catch a wrong server loudly, so this is efficiency, not correctness.
    Confidence 70 · logic · Fix: return a failure as soon as the issued key gets 401 Unauthorized.

Decisions for the author

None

Positive signals

  • The readiness probe uses the issued key and tells a transport failure (None) apart from an HTTP status. Only the assertions treat a transport failure as fatal, so an engine that is still binding is not mistaken for a refusal.
  • The scenario has three independent signals: no key gives 401, a wrong key gives 401, and the issued key is accepted. Removing key enforcement from the engine hand-off would fail it, not pass it.
  • @requires-engine:vllm is the right gate. The harness skips on hosts where vLLM cannot start, and effective_serve_engine uses the same family predicate. So wherever the scenario runs, host_serve_target() resolves to vLLM, and the documented Windows/lemonade refusal is never driven.

Deployment notes

None. The change is test- and docs-only.

What this covered

Read completely: every file in the change, plus the code around it. That includes the serving_steps.rs serve helpers (serve_gpu_model, serve_and_wait, model_is_ready, ensure_serve_port_free), E2eWorld's Drop (which stops managed services on panic, so a timed-out serve does not leak), the @requires-engine skip logic in expectation.rs and capability.rs, and the CLI's key-disclosure printers in apps/rocm/src/service_launch.rs and serve_summary.rs. Diff prw-base (1c33248)…90089998.

Checks run:

  • CI rollup on the head: one check failed, "E2E tests (MI300X, current)". Its log shows networking-04 ran and passed all 6 steps. The only unexpected failure is therock-next-09, where rocm engines install vllm --reinstall hit an unsatisfiable vLLM/amd-aiter pin. This change does not touch that path, so I did not count it against the PR.
  • "Sphinx docs build (-W)" and "Skill checks (skillscope)" were skipped. The diff touches neither Sphinx sources nor skills.
  • The sign-off check returned exit 0: 7 commits checked, none missing.

What did not run, or ran differently:

  • I did not fan out. The change is small enough for one context to read completely, so I used a single pass. The agent-instruction, history and code-comment passes ran in that same context, so they were not independent.
  • Prior-changes pass: not run, because earlier reviews are out of scope for this run.
  • Nothing was built or executed locally. The scenario needs a vLLM-capable GPU, and CI's MI300X run on this head stands in as the run evidence.

The PR discussion was not reconciled here.

…issued key gets 401

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi requested a review from rominf October 26, 2026 11:45
@fredespi

Copy link
Copy Markdown
Collaborator Author

#722 (review)

Fixed in 9567603. The docs/testing.md citation is now a relative link to networking.feature, like the sibling feature links. wait_for_authenticated_endpoint now fails immediately when the issued key gets 401, with the serve stdout and service-log tail, instead of polling to the deadline. Not done: routing serve_public_bind through the serve_and_wait relaunch machinery needs a shared-helper change (key-bearing readiness probe plus extra serve args), so it stays out of this PR as a follow-up; the lane is non-required.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9567603

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — no blocking findings

Full review of the whole change.
Adds end-to-end proof for one acceptance criterion of EAI-7409 (now tracked as ROCMAI-318): a public bind rejects a missing or wrong credential. It does not cover the "secrets are not logged or persisted insecurely" criterion, and it does not claim to close the ticket.

Since the last round (9008999): both items it raised against the previous head are fixed here:

  • docs/testing.md now links networking.feature instead of citing it as a bare path.
  • The readiness wait fails immediately when the issued key gets a 401.

Each finding below is marked new or standing.

Blocking

None

Non-blocking

  • [standing] The new serve step brings back the launch-then-poll pattern that the shared serve helper says it replaced — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:84-136, against tests/e2e-cucumber/tests/e2e/serving_steps.rs:133-160 and :555-590
    The doc comment on serve_and_wait says it "Replaces the run_rocm_ok + readiness-poll pair", because that pair loses the evidence needed to explain a serve that stalls. serve_public_bind is exactly that pair. Compared with the shared path it is missing four things:

    • Non-zero exits: it goes through run_rocm_ok, so a non-zero exit panics without the engine log.
    • Failure evidence: it does not call serve_failure_evidence, archive_service_log or stop-before-report, and it throws away the device state that ensure_serve_port_free() returns.
    • Relaunch: it has no one-relaunch rescue. The shared helper's comment says the same model on the same engine ("has both timed out and served successfully in the same MI300X run") needs that rescue.
    • Model check: its readiness wait accepts any 2xx and never checks the model id. model_is_ready checks it to guard against a leftover server from an earlier scenario. Without the check, a leftover unauthenticated vLLM would make the next step fail with a misleading "must be refused, got 200". That gives a false red, never a false green.

    Confidence 85 · architectural · Fix: add an optional bearer token to model_is_ready/serve_and_wait and let the caller get stdout back to extract the key, rather than keeping a parallel launcher.

  • [new] The docs say the scenario runs on "the MI300X vLLM lane only", but more lanes run it — docs/testing.md:958
    .github/workflows/nightly.yml also runs the full suite, including this scenario, on "E2E tests (MI300X, incl. nightly-only)" (line 365) and "E2E tests (MI350P, incl. nightly-only)" (line 546). gfx950 passes engine_available("vllm"), so @requires-engine:vllm does not skip it there. The workflow file is what decides where the scenario runs, so the docs sentence is the side to correct. The PR description repeats the claim ("runs only on MI300X").

    Confidence 85 · mechanical · Fix: name the vLLM/Instinct lanes, for example "the per-PR MI300X lane, plus the MI300X and MI350P nightly lanes".

  • [new] The step checks the "shown only now" promise as wording only — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:175-178
    AGENTS.md §3 says a promise that something will not happen is asserted together with the resulting state. Where the text is pinned on its own, the assertion should carry a comment naming the test that proves the behavior. This step checks the phrase and that the key does not leak elsewhere in launch stdout. It does not show the key is absent from rocm services or rocm logs, which the docs paragraph it edits also promises, and it carries no such comment. The engine is already up at that point, so a rocm services / rocm logs check would be cheap.

    Confidence 70 · logic · Fix: add those two checks to this scenario, or add a comment pointing at the test that proves the promise.

Decisions for the author

  • Pinning the scenario to vLLM leaves the Linux lemonade and llama-server key paths with no end-to-end coverage — tradeoff

    • What the pin buys: @requires-engine:vllm stops the scenario from driving the Windows-lemonade refusal, which is intended behavior, and keeps a cold serve off the time-capped Strix lanes.
    • What it costs: on Linux the key also reaches lemonade (LEMONADE_API_KEY, engines/lemonade/src/process.rs) and llama-server (--api-key-file). Those are separate enforcement paths that run on the Strix Ubuntu, WSL2 and W7900 lanes, and no scenario checks them.
    • The alternative: gating on OS instead of engine would cover them, at the cost of slower lanes.

    Worth stating in the PR that this choice was deliberate.

  • The public bind on a shared runner, and the issued key printed in panic text — tradeoff
    The scenario binds 0.0.0.0 on a self-hosted runner, so while it runs the endpoint is protected only by the behavior under test. Several failure messages print the full serve stdout, key included, into CI logs and report artifacts of a public repository. The key is random per run, and teardown (the world's Drop) stops the service, so the exposure is short. Redacting the key in panic text would be cheap.

Positive signals

  • The readiness wait fails immediately on a 401 to the issued key, rather than waiting out the deadline. That separates "the endpoint rejects the key it issued" from "the engine is still loading".
  • The leak check counts only the dedicated api key: line and allows only the example: curl line, so it would catch the key appearing anywhere else in launch output. That matches the real output from render_endpoint_client_config in apps/rocm/src/service_launch.rs.

Deployment notes

None

What this covered

Every changed file read completely: docs/testing.md, networking.feature, e2e.rs, endpoint_auth_steps.rs and serving_steps.rs. Diffed prw-base...HEAD, merge-base d510680…95676038. Also traced:

  • the CLI's key printing (apps/rocm/src/service_launch.rs, serve_cmd.rs);
  • the vLLM and lemonade key hand-off (engines/*/src/process.rs);
  • the @requires-engine: tag handling (tests/e2e-cucumber/src/expectation.rs, capability.rs);
  • teardown, and the nightly and self-hosted workflow lane definitions.

Two workers did the code and design passes. Runs in the checkout's own target dir:

  • cargo test --test feature_naming: 4 of 4 passed.
  • cargo clippy --test e2e -- -D warnings: clean.

CI at 9567603:

  • Hosted and required checks: green.
  • Strix Halo, Windows lane: failed in pre-warm, before any scenario ran. A third-party download returned HTTP 403 while installing the lemonade backend, so this is not attributable to this PR. The same lane passed on main tip 63ecab2.
  • MI300X and W7900 lanes: queued. MI300X is the only per-PR lane that runs the new scenario, so there is no CI evidence for it at this head. On the previous head 9008999, MI300X ran networking-04 and all six steps passed. That lane failed only on an unrelated vLLM reinstall scenario (therock-next-09). The difference from that head to this one is a docs link plus the 401 fail-fast branch.

Sign-off check: all 8 commits carry a sign-off trailer (exit 0).

Did not run:

  • The live GPU scenario itself.
  • vLLM's status codes on the pinned version. That vLLM answers 401 on /v1/* is the workers' knowledge of upstream vLLM, not something traced in this repo.
  • Live branch-protection required checks, so "not a required check" in the docs is unverified.
  • The history and prior-changes passes ran with git history only; no earlier review comments were read.

Not reconciled against any discussion here.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 9567603

Scope: this PR is test-only, adding/expanding tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs (~223 new lines) to pin the public-bind endpoint authentication contract, plus small visibility/doc changes elsewhere. I reviewed the diff against main and cross-checked the step assertions against the actual CLI source (apps/rocm/src/service_launch.rs::render_endpoint_client_config, apps/rocm/src/serve_cmd.rs summary-mode branch) to confirm the matched strings ("api key:", "shown only now", "Authorization: Bearer", the "example:" line) are accurate for a piped/non-TTY invocation. I also checked that vLLM's API key is loaded before the listener starts, to rule out a possible 401-race false positive in the readiness wait.

What I could not verify: there is no Rust toolchain in this sandbox, so I did not build the crate or run the e2e/cucumber suite — this is a static reading of the diff only, not an executed verification.

Findings (non-blocking; test-quality/coverage items, no correctness bugs found):

  1. (low-medium confidence, non-blocking) endpoint_auth_steps.rs ~line 137: the no-key-leak assertion (shows_key_once / similar) only scans captured stdout. If serve_public_bind only stores stdout into world.cli_output and discards stderr, a future regression that logs the API key to stderr (e.g. via eprintln! or a tracing/log warning) would not be caught, even though the test's stated purpose is pinning that the key must "never appear ... in the audit log" more broadly. Consider also capturing/asserting over stderr if the harness can combine both streams.

  2. (low confidence, non-blocking/cleanup) ~line 47: a new reqwest::Client appears to be constructed on every call to the status-check helper (try_models_status), rather than being built once and reused (e.g. via LazyLock, mirroring the existing SERVE_BASE_URL/MODELS_URL pattern). Since this is invoked in a polling loop (up to ~600s at 5s intervals) plus from each Then step, reusing a client would avoid repeated connection-pool/TLS setup overhead. Minor, since this is test code on a presumably already-slow serial e2e lane.

  3. (low confidence, non-blocking/cleanup) ~line 164: the models URL appears to be rebuilt as format!("{}/models", endpoint_url()) from the newly-exported SERVE_BASE_URL, duplicating the existing MODELS_URL static already defined in serving_steps.rs (format!("http://127.0.0.1:{SERVE_PORT}/v1/models")). Two independent constructions of the same URL risk silently diverging if the path changes in only one place. Consider reusing the existing static instead.

Overall: no blocking findings — the new assertions appear to correctly pin the intended auth-output contract based on static reading of the diff and cross-reference against the CLI source producing that output.

🤖 Generated with Claude Code

@nowycondro

Copy link
Copy Markdown
Collaborator

🤖 Automated review (Claude Code) · 9567603

I reviewed the new e2e step definitions added for the public-bind endpoint authentication scenario, focusing on tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs, and compared them against the existing GPU-serve step patterns elsewhere in the suite (e.g. serving_steps.rs).

Two non-blocking diagnostics gaps stood out, both consistency issues rather than correctness bugs: (1) wait_for_authenticated_endpoint (line ~84) panics with only a log tail and never calls archive_service_log, unlike every other GPU-serve scenario that persists full evidence via serve_and_wait/serve_failure_evidence — if this scenario stalls on something outside the tail window, CI artifacts won't have the full log to debug. (2) The device_state string returned by ensure_serve_port_free() (line ~116) is discarded here, whereas other scenarios surface it in failure output to flag "a previous engine still holding the GPU" — losing that hint specifically for this scenario's failure mode. Neither blocks merge; both are inconsistencies with established conventions in this test file that could make a future flaky failure harder to triage. A minor efficiency nit: try_models_status (line ~47) constructs a new reqwest::Client on every poll iteration (up to ~120 times over 600s) instead of reusing one client, which is wasteful but harmless against a local endpoint.

I did not have a GPU/vLLM toolchain available to actually run the new scenario, so I could not verify the live 401/401/2xx behavior end-to-end; this review is based on static reading of the diff and surrounding test infrastructure only.

🤖 Generated with Claude Code

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9567603

Review — no blocking findings

Full review of the whole change (merge-base d5106802…95676038), with two independent fan-out passes (code-correctness, design/approach) plus a direct cross-check of the CLI/engine source the new test asserts against.
Implements acceptance criterion 2 ("missing/invalid credentials are rejected") of ROCMAI-318 (née EAI-7409) end to end; criteria 1 (anonymous 0.0.0.0 bind refused) and 3 (secrets not logged/persisted insecurely) are out of scope here and the change does not claim to close the ticket.

Blocking

None

Non-blocking

  • CI rollup shows two self-hosted E2E lanes red, which the PR description does not reflect — PR description test-plan, vs. live rollup
    The description's test-plan says "MI300X re-run on the current head is queued behind the serial runner" and only discusses a historical MI300X infra failure from 2026-08-25. The live rollup for the current head (gh pr view 722 --json statusCheckRollup) shows E2E tests (MI300X, current) and E2E tests (Strix Halo, Windows, current) both FAILURE/COMPLETED (completed 2026-10-26), plus the derived E2E consolidated report (self-hosted) also FAILURE — i.e. already-resolved results the description doesn't mention, not a pending queue. I could not fetch job logs (Actions API returned 403 for this token), so I can't confirm root cause. Note for calibration: .github/workflows/e2e-selfhosted.yml's own top-of-file comment states none of this workfow's check names are in branch protection's required-status-check list, and none use continue-on-error, so these are "real hardware regression shows red" signals by design, not merge gates — which is why this is non-blocking rather than blocking.
    Confidence 100 · architectural · Fix: before merge, re-check live CI state (per this repo's own AGENTS.md §4, "Live State Verification Before Any External Claim") and either update the test-plan checklist to match, or get/paste the actual MI300X and Windows job logs and confirm both failures are the known infra flake rather than something this change triggers.
  • shows_key_once only scans captured stdout for key leakage, not stderr — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:137-160
    world.cli_output is populated by run_rocm_ok, which (per tests/e2e-cucumber/tests/e2e.rs:838-848) discards the child's stderr entirely. The test's own stated purpose is pinning that the key is shown exactly once and never leaks elsewhere, mirroring docs/testing.md's broader "must never appear in rocm services, rocm logs, or the audit log" contract — but a future regression that prints or logs the key via eprintln!/a tracing warning to stderr would pass this check undetected.
    Confidence 85 · logic · Fix: capture/merge stderr into the same assertion (or add a parallel "stderr contains no secret" check) if the harness can combine both streams for this invocation.
  • endpoint_url()'s /models URL is built independently of the existing MODELS_URL static — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:29-31,47-50 vs. tests/e2e-cucumber/tests/e2e/serving_steps.rs:80
    Both ultimately derive from the same SERVE_PORT, so practical divergence risk is low, but the /models suffix is now spelled out in two independent places (format!("{}/models", endpoint_url()) here vs. the pre-existing MODELS_URL static) rather than reusing the latter.
    Confidence 70 · mechanical · Fix: reuse MODELS_URL (now pub(crate)-visible, like SERVE_BASE_URL) instead of re-deriving the path.

Decisions for the author

  • Driving the auth contract with a real HTTP client against a live GPU-backed vLLM engine, rather than a mock/stub standing in for "the engine" — tradeoff
    This is the only way to prove the engine (not the CLI) actually enforces the key, which is the documented point of this scenario, and the pure key-resolution rules are already unit-tested. The cost is that the only end-to-end proof of acceptance criterion 2 runs on a single non-required GPU lane. Both the design and the gating follow established precedent elsewhere in this suite (e.g. serve-vllm-default-on-instinct, serve-vllm-inference), so this isn't a new gap this change introduces — flagging only so the tradeoff is explicit rather than assumed.

Positive signals

  • wait_for_authenticated_endpoint's failure diagnostics (serve stdout + service log tail on both timeout and an unexpected 401) directly address the class of bug this suite's own doc comments say previously made a similar scenario fail "before it ever reached its assertions" — good use of the existing service_log_tail helper rather than a bare panic message.
  • The @requires-engine:vllm tag's rationale is spelled out in the feature file with the exact mechanism it's guarding against (Windows/lemonade can't enforce the key), rather than being a bare tag — makes the gating auditable instead of folklore.
  • Traced the key-issuance and enforcement path end to end (service_launch.rs::render_endpoint_client_config → engines/vllm/src/process.rs setting VLLM_API_KEY on the command before spawn()) and confirmed there is no startup race where the engine could answer before auth is configured, which is exactly the kind of flake risk a scenario like this could otherwise have.

Deployment notes

None — test-only change, no new runtime dependency (reqwest was already a tests/e2e-cucumber dependency), no schema/migration impact.

What this covered

Read the full diff (docs/testing.md, tests/e2e-cucumber/features/networking.feature, tests/e2e-cucumber/tests/e2e.rs, the new tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs in full, tests/e2e-cucumber/tests/e2e/serving_steps.rs) plus the surrounding/cross-referenced source: tests/e2e-cucumber/tests/feature_naming.rs, tests/e2e-cucumber/src/serve_log.rs, apps/rocm/src/service_launch.rs and serve_cmd.rs (output source for the asserted strings and summary-vs-plain dispatch), engines/vllm/src/process.rs/state.rs (key wiring), and .github/workflows/e2e-selfhosted.yml (required-check status). Ran two independent fan-out passes in fresh contexts (blind code-correctness; design/approach via targeted questions) in addition to this direct reading; both converged with my own findings and found no additional defects at or above the confidence bar. Cross-checked EAI-7409/ROCMAI-318's acceptance criteria against the change via the issue tracker.
Did not run: no Rust toolchain available in this environment, so nothing was executed directly — correctness here rests on static reading plus the repo's own passing Test (affected crates) / windows-build-and-test / clippy CI checks (taken at face value) rather than a local run. Could not fetch Actions job logs (403) to independently diagnose the two failing self-hosted E2E lanes beyond their rollup conclusion and the workflow file's own required-check documentation. Prior-review-comments pass not run (this review was reached blind, without reading this PR's existing review threads, per the methodology's independence requirement). Not reconciled against any discussion on this PR.

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this one and it looks good to me — approving, with two small inline nits that I'd be happy to see folded in or waved off.

It's a test-only change, and the scenario it adds (networking-04) passed 6/6 in about 61 seconds on the MI300X lane, so the new coverage genuinely runs. The lane shows red, but the failure is therock-next-09: a uv resolver conflict between amd-aiter==0.1.22 and 0.1.24 in the ROCm10 wheel index, which has nothing to do with this diff. That lane is green on main as of 2026-11-07 and isn't a required context, so I don't read it as a signal against this PR.

One thing I'd ask for rather than require: the head is now 12 days stale. A rebase plus a re-dispatched MI300X run would make the evidence current, and it's cheap insurance given how much of the value here is "the scenario actually passes on real hardware." Related — the PR body's test plan has drifted: it says the MI300X re-run "is queued" and cites failures from 2026-08-25, but a run on this head completed on 2026-10-28. Worth refreshing so the next reader isn't chasing stale results.

On the substance: the mid-readiness 401 fast-fail is the right call — it cleanly separates a genuine contract defect from a model that's merely slow to come up, instead of burning the whole timeout and reporting an ambiguous failure. Using the @requires-engine:vllm tag rather than an expectations.toml row also matches that file's own documented guidance, and the Gherkin itself reads as clean behavioural language rather than a transcript of the mechanics.

Some(status) if status.is_success() => return,
// A bound engine rejecting the key it was issued is a contract defect,
// not a slow start, so report it now instead of at the deadline.
Some(reqwest::StatusCode::UNAUTHORIZED) => panic!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small one, non-blocking: both failure paths here skip the repo's own serve-failure evidence bundle.

This applies to the 401 fast-fail on line 91 and to the deadline assert! on line 100. Both panics do quote serve stdout plus service_log_tail, which is better than nothing — but unlike every other serve step, which routes through serve_failure_evidence (tests/e2e-cucumber/tests/e2e/serving_steps.rs:106), neither calls archive_service_log nor stop_scenario_services.

Two consequences. The engine's startup banner dies with the scenario's TempDir, so the most useful part of the diagnostic is gone by the time anyone reads the failure. And more importantly, a failing run leaves a 0.0.0.0-bound vLLM still holding the GPU — that's the part I'd actually fix, since a leaked GPU-holding process can poison whatever runs next on the lane.

"expected the CLI to say how to send the key, got:\n{output}"
);
assert!(
key.len() >= 16,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this assertion can't fail. ENDPOINT_API_KEY_LEN is a constant 48, so key.len() >= 16 is true by construction, and the properties that could vary — randomness and alphabet — are already covered by generate_endpoint_api_key_is_random_and_alphanumeric.

Beyond being dead weight, it reads as a key-strength guarantee that the Gherkin never promises, which is a little misleading about what this scenario is pinning. I'd just drop it.

…nd-selectors

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/networking.feature
@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 2e739a5b

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. Implements the live-endpoint half of the EAI-7409 public-bind authentication contract (the pre-flight refusal was pinned in an earlier PR); not the rest of EAI-8072.

Blocking

  • networking-05's new serve_public_bind step is rejected by the CLI's own pre-flight validation and can never reach its assertions — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs (the serve_public_bind #[when], invoking run_rocm_ok(world, &["serve", model, "--engine", engine, "--managed", "--host", "0.0.0.0", "--allow-public-bind"])), apps/rocm/src/serve_cmd.rs:527-530, apps/rocm/src/serve_port.rs:110-120, crates/rocm-core/src/process.rs:713-715.
    The invocation omits --port. serve_cmd.rs unconditionally calls validate_port_request(&host, PortRequest::from_flag(port)) before any engine work; validate_port_request bails with "needs an explicit --port" whenever PortRequest::is_auto() and the host is not is_canonical_loopback_host, which is true only for the literal string 127.0.0.1 (process.rs:713-715, doc comment: "localhost, ::1, 0.0.0.0, and other IPv4 addresses are 'custom hosts': they require an explicit port"). 0.0.0.0 is therefore non-canonical, so this step hits the exact refusal path the sibling scenario networking-04 ("A non-canonical host without an explicit port is refused up front") deliberately exercises for a different host. run_rocm_ok panics on non-zero exit, so the scenario fails at the When step, before any key is issued and before any of the four Then assertions run. I confirmed serve_port.rs, process.rs and the relevant serve_cmd.rs lines are byte-identical between this PR's head and current main, and that the step's args have never included --port in any commit since the scenario was introduced (a6559658), so this is not an artifact of an older/different revision.
    This also means the PR's own description ("MI300X dispatch earlier on this scenario: green (1 scenario, 6/6 steps)") and docs/testing.md's new paragraph (claiming the scenario "checks that a request without the key and one with a wrong key get 401, while the issued key is accepted") describe a result the current code cannot produce — the self-hosted "E2E tests (MI300X, current)" check, which is the only one that would actually run this scenario, is still queued/pending against this exact head at review time, so there is no green run of this commit to reconcile this against. The "deferred follow-up" language docs/testing.md replaced would still be accurate today.
    Confidence 92 · logic · Fix: pass an explicit --port in serve_public_bind (the shared SERVE_PORT/SERVE_BASE_URL already used by every other GPU serve scenario in serving_steps.rs is the natural choice, since try_models_status/get_models_status already assume that port), then re-verify this exact commit on the MI300X lane before merging.

Non-blocking

None

Decisions for the author

None

Positive signals

  • wait_for_authenticated_endpoint carefully distinguishes "connection refused while the engine is still binding" (retry) from a definitive 401 (fail fast, since a bound engine rejecting its own issued key is a contract defect, not a slow start) from any other status — a real, previously-hit failure mode is called out in the comments and handled correctly.
  • The "shown exactly once / no leak" assertions in shows_key_once correctly special-case the intentional example: curl ... line so it isn't double-counted as a second disclosure or a leak.
  • docs/architecture.md's new table row is accurate and correctly alphabetized among its neighbors.
  • The @requires-engine:vllm tag rationale in the feature-file comment (Windows/lemonade deliberately refuses a public bind) checks out against apps/rocm/src/service_launch/endpoint_auth.rs's ensure_public_bind_engine_supported.

Deployment notes

This scenario only runs on the self-hosted MI300X lane and is not a required check, so a regression here (including the one above) can sit unnoticed for a while if that lane is backed up — as it is on this very PR, per its own test-plan note about a prior runner-side failure and the currently-queued re-run.

What this covered

Read every changed file in full (docs/architecture.md, docs/testing.md, tests/e2e-cucumber/features/networking.feature, tests/e2e-cucumber/tests/e2e.rs, tests/e2e-cucumber/tests/e2e/serving_steps.rs, and the new tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs), plus the production code the new test drives and asserts against (apps/rocm/src/serve_cmd.rs, apps/rocm/src/serve_port.rs, crates/rocm-core/src/process.rs, apps/rocm/src/service_launch/endpoint_auth.rs, apps/rocm/src/service_launch/start.rs, apps/rocm/src/serve_summary.rs), the full networking.feature file for context, and the CI status roll-up for this head. Diffed be9ede5d1…2e739a5b.

A second independent pass over the same scope was run in parallel and reported the same root-cause defect by an independent trace; its exact wording was not used as evidence, only as corroboration, consistent with reaching this finding blind. Did not run: "prior changes" pass (no prior review threads were read, per method). Agent-instruction pass found no AGENTS.md rule this change violates. History pass: confirmed via git log -p that the missing --port has been present since the scenario's first commit. Not reconciled against any PR discussion thread.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change.
Pins the second half of the public-bind endpoint authentication contract (EAI-7409) — key issuance plus live 401/401/2xx enforcement from a real endpoint — where an earlier PR already covered the pre-flight opt-in refusal; the ticket itself lives on a tracker not reachable from here, so this is taken only from the PR description.

Blocking

  • The new GPU scenario fails on the only CI lane that can run it, on this PR's exact current head, because the new step invocation never passes --port — tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:118-130 (inline comment added)
    • serve_public_bind() runs rocm serve <model> --engine <engine> --managed --host 0.0.0.0 --allow-public-bind with no --port. apps/rocm/src/serve_port.rs's own policy — and the sibling networking-04 scenario added earlier on this same branch — establish that automatic port selection is accepted only on the canonical loopback host 127.0.0.1; any other host, including 0.0.0.0, requires an explicit --port.
    • I read the actual E2E tests (MI300X, current) job log for head 2e739a5b (run 43422777319, job 131876765826): networking-05 panics immediately with Error: \rocm serve --host 0.0.0.0` needs an explicit `--port <1-65535>`` — before the CLI ever issues a key. None of the behavior this PR exists to pin (key issuance, 401 without a key, 401 with the wrong key, 2xx with the issued key) is currently exercised on real hardware.
    • That job's GitHub conclusion reads CANCELLED, not FAILED, only because it was killed roughly 90 minutes later during an unrelated, later scenario in the same long-running suite — so this does not surface as a red required check, but the scenario itself unambiguously failed on this head.
    • Confidence 100 (direct evidence: read the job's own log for this exact head commit) · logic · Fix: pass an explicit --port (e.g. the shared SERVE_PORT / 11435 that ensure_serve_port_free and the readiness helpers already assume) in the args list.

Non-blocking

None

Decisions for the author

None

Positive signals

  • wait_for_authenticated_endpoint cleanly separates "connection refused while the engine is still binding" (tolerated, retried) from "the issued key itself got 401 from a bound endpoint" (fatal, reported immediately) — avoids treating every transport error the same way during a slow GPU serve.
  • shows_key_once's leak check counts the dedicated api key: line rather than raw occurrences of the key string, correctly tolerating the key's deliberate second appearance inside the printed curl example rather than false-flagging it as a leak.
  • The feature-file comments explain why @requires-engine:vllm is load-bearing (Windows lemonade can't enforce the key, so an untagged scenario would fail there) rather than just applying the tag silently.

Deployment notes

None — test-only change. The three helpers widened to pub(crate) in serving_steps.rs carry no behavior change.

What this covered

Read the full diff (be9ede5d1…2e739a5b, all 6 changed files) completely, including the new endpoint_auth_steps.rs in full, plus the surrounding code it depends on: serving_steps.rs (SERVE_BASE_URL, host_serve_target, serve_timeout_for, ensure_serve_port_free, run_rocm_ok), serve_port.rs's port-selection policy, service_launch/start.rs's key-printing format (confirmed the test's string assertions match the real output), feature_naming.rs's per-feature sequential-id check, and the full networking.feature. Checked the PR's 10-commit history, which already fixed several earlier review rounds' findings (scenario naming that broke feature_naming, the key-leak check's false positive on the curl example). Read CI directly via the GitHub API and job logs rather than trusting the green rollup: every required check is green, but the one self-hosted GPU job that actually exercises networking-05 ran against this exact head and failed the new step, a failure the rollup doesn't surface because that lane isn't a required status check and the job's later cancellation (unrelated scenario, ~90 min on) reports as CANCELLED rather than FAILED. Did not run anything locally — no GPU/engine available here — so the CI log is the verification. Ticket EAI-7409 is on a tracker not reachable from here; its criteria are taken only from the PR description.

Single consolidated pass rather than a multi-worker fan-out, given the diff's modest size (276 additions across 6 files, one new ~220-line file) was within what one reviewer could read completely end to end. The "prior changes" pass (separate historical PRs' review comments on these files) surfaced nothing beyond what this PR's own commit history already shows being iterated on. Not reconciled against this PR's existing review-comment discussion — none of it was read for this pass.


🤖 Automated by an AI agent

Comment on lines +118 to +130
let stdout = crate::run_rocm_ok(
world,
&[
"serve",
model,
"--engine",
engine,
"--managed",
"--host",
"0.0.0.0",
"--allow-public-bind",
],
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this invocation fails before the CLI ever issues a key, and the GPU CI log for this exact head confirms it.

No --port is passed, but --host 0.0.0.0 is not the canonical loopback host, so serve_port.rs's own policy (and the sibling networking-04 scenario added on this branch) requires an explicit --port on any non-canonical host — automatic selection is refused there.

I read the E2E tests (MI300X, current) job for this PR's current head 2e739a5b (run 43422777319, job 131876765826). networking-05 fails immediately:

Step panicked. Captured output: `rocm serve Qwen/Qwen3.5-0.8B --engine vllm --managed --host 0.0.0.0 --allow-public-bind` failed (rc=1)
--- stderr ---
Error: `rocm serve --host 0.0.0.0` needs an explicit `--port <1-65535>`. Automatic port selection is supported only on the default host 127.0.0.1, ...

So on real hardware right now, none of the behavior this PR exists to pin — key issuance, 401 without a key, 401 with the wrong key, 2xx with the issued key — is actually exercised. The job's GitHub conclusion shows CANCELLED rather than FAILED only because it was killed ~90 minutes later during an unrelated later scenario, so this failure doesn't surface as a red required check, but the scenario itself unambiguously failed.

Fix: pass an explicit --port in this args list (e.g. the shared SERVE_PORT / 11435 that ensure_serve_port_free/the readiness helpers already assume).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants