Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -842,6 +842,7 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr
| [`tests/e2e/dependency_guard_steps.rs`](../tests/e2e-cucumber/tests/e2e/dependency_guard_steps.rs) | Steps for [`dependency_guard.feature`](../tests/e2e-cucumber/features/dependency_guard.feature): a stand-in package manager proposes removing the ROCm stack, and the CLI must refuse. | flat |
| [`tests/e2e/diagnose_steps.rs`](../tests/e2e-cucumber/tests/e2e/diagnose_steps.rs) | Steps for [`diagnose.feature`](../tests/e2e-cucumber/features/diagnose.feature): `rocm diagnose` matches and `rocm fix` listings, previews, printed verify commands and failing fixes. | flat |
| [`tests/e2e/driver_steps.rs`](../tests/e2e-cucumber/tests/e2e/driver_steps.rs) | Steps for [`driver_install.feature`](../tests/e2e-cucumber/features/driver_install.feature): previewing and approving GPU driver installation plans on WSL, Linux and Windows hosts. | flat |
| [`tests/e2e/endpoint_auth_steps.rs`](../tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs) | Steps for the public-bind scenario in [`networking.feature`](../tests/e2e-cucumber/features/networking.feature): a real serve on a public interface, then direct HTTP requests without, with a wrong, and with the issued endpoint key. GPU lane only, since the engine enforces the key. | flat |
| [`tests/e2e/engines_steps.rs`](../tests/e2e-cucumber/tests/e2e/engines_steps.rs) | Steps for [`engine_shell.feature`](../tests/e2e-cucumber/features/engine_shell.feature): `rocm engines shell` under a pseudo-terminal, since the defect shows only in what the terminal renders. | flat |
| [`tests/e2e/examine_steps.rs`](../tests/e2e-cucumber/tests/e2e/examine_steps.rs) | Steps for [`examine.feature`](../tests/e2e-cucumber/features/examine.feature): `rocm examine`, version and help output, engine listing and the driver plan preview. | flat |
| [`tests/e2e/host_guard.rs`](../tests/e2e-cucumber/tests/e2e/host_guard.rs) | The host check the Windows driver-install steps run before any CLI call, kept free of the cucumber world so a separate test can include it. | flat |
Expand Down
11 changes: 7 additions & 4 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -2068,10 +2068,13 @@ key file (never argv), persisted as a 0600 per-service file (not the OS keychain
which is unavailable on headless serving hosts), and printed once as client
configuration — it must never appear in `rocm services`, `rocm logs`, or the
audit log. Verifying that the running server actually *rejects* an
unauthenticated request requires a live engine; that end-to-end assertion is a
deferred follow-up and is **not yet** exercised by the GPU acceptance scripts
(`scripts/vllm_therock_gpu_test.py`). The unit tests cover key policy,
generation, the 0600 key file, and key-file resolution.
unauthenticated request requires a live engine: the `networking-05` scenario
([networking.feature](../tests/e2e-cucumber/features/networking.feature)) serves a model on a public
bind and checks that a request without the key and one with a wrong key get
401, while the issued key is accepted. It runs on the MI300X vLLM lane only, which
is not a required check; native Windows and lemonade-default hosts skip it. The
unit tests cover key policy, generation, the 0600 key file, and key-file
resolution.

The same requirement holds across respawns, because the invariant belongs to the
recorded *host*, not to the key file: a service recorded on a public host refuses
Expand Down
38 changes: 35 additions & 3 deletions tests/e2e-cucumber/features/networking.feature
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,12 @@ Feature: Native HTTP networking
# certificate store), replacing the generated PowerShell script that used to
# run under `powershell.exe -ExecutionPolicy Bypass` on Windows. These
# scenarios drive the real `rocm` binary end-to-end against the mock server so a
# GET/round-trip reaches a local endpoint over the native stack. They run on the
# mock (no GPU) on every platform the suite runs on, so a regression back to a
# shell-out networking backend surfaces here rather than only in the field.
# GET/round-trip reaches a local endpoint over the native stack. All but
# networking-05 run on the mock (no GPU) on every platform the suite runs on, so
# a regression back to a shell-out networking backend surfaces here rather than
# only in the field. networking-05 is a GPU serve scenario: it starts a real
# engine on a public bind and checks key enforcement from the harness's own
# HTTP client, not through the CLI.

# `rocm services list` extracts host:port from the service record and issues a
# native HTTP GET to `/v1/models` as its readiness probe (served by the mock).
Expand Down Expand Up @@ -52,3 +55,32 @@ Feature: Native HTTP networking
When the user serves a model bound to a non-canonical host without an explicit port
Then serving is refused before any engine starts
And the user is told to pass an explicit port for that host

# The other half of the EAI-7409 contract, and the security property the ticket
# calls out first: opting in to a public bind must not produce an open endpoint.
# The CLI issues an API key, shows it exactly once, and hands the serving engine
# a key file to enforce — so the endpoint must reject callers without the key and
# with the wrong key, while accepting the key it just issued.
#
# GPU lane: the ENGINE enforces the key (the CLI only issues and hands it over),
# so proving enforcement needs a real engine serving a real model. `serve` has no
# plan-only mode that would let the mock lane observe it. The pure key-resolution
# rules (loopback stays credential-free even when a key is passed) are already
# unit-tested in the CLI, so this pins only what a unit test cannot: that a live
# public endpoint actually authenticates.
#
# `@requires-engine:vllm` (same precedent as `serve-vllm-default-on-instinct`) is
# load-bearing, not incidental: on a native-Windows host the effective serve engine
# is lemonade, and the CLI deliberately REFUSES a public bind there because the
# endpoint key cannot be enforced through Windows lemonade. Without this tag the
# scenario would drive that refusal and fail on the Strix-Windows lane — a
# designed-in behaviour, not a bug, so it wants a tag rather than an xfail row. It
# also keeps this cold serve off the time-capped Strix lanes entirely.
@id:networking-public-bind-endpoint-enforces-key @requires-gpu @requires-engine:vllm
Scenario: networking-05 - A public endpoint issues a key once and then enforces it
Given a managed runtime is active
When the user serves a model on a public interface with public binding allowed
Then the CLI shows the endpoint key once and how to send it
And a request without the key is refused as unauthorized
And a request with the wrong key is refused as unauthorized
And a request carrying the issued key is accepted
7 changes: 7 additions & 0 deletions tests/e2e-cucumber/tests/e2e.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ mod e2e {
pub mod dependency_guard_steps;
pub mod diagnose_steps;
pub mod driver_steps;
pub mod endpoint_auth_steps;
pub mod engines_steps;
pub mod examine_steps;
pub mod host_guard;
Expand Down Expand Up @@ -78,6 +79,11 @@ pub struct E2eWorld {
/// Cache-marker destination discovered from `rocmd`'s own JSON report.
pub artifact_marker_path: Option<PathBuf>,
pub endpoint: Option<String>,
/// The endpoint API key `rocm serve` issued for a public bind, captured from
/// the launch output. `None` for a loopback serve, which is credential-free.
/// Used by the public-bind auth scenario to prove the endpoint really enforces
/// the key it printed (see `e2e::endpoint_auth_steps`).
pub endpoint_api_key: Option<String>,
pub model_name: Option<String>,
pub chat_response: Option<serde_json::Value>,
/// The reply to a request sent to the served model's Anthropic-compatible
Expand Down Expand Up @@ -361,6 +367,7 @@ impl Default for E2eWorld {
hub_requests: None,
artifact_marker_path: None,
endpoint: None,
endpoint_api_key: None,
model_name: None,
chat_response: None,
anthropic_response: None,
Expand Down
223 changes: 223 additions & 0 deletions tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
// Copyright © Advanced Micro Devices, Inc., or its affiliates.
//
// SPDX-License-Identifier: MIT

//! Steps for the public-bind endpoint authentication contract (EAI-7409).
//!
//! `rocm serve --host 0.0.0.0 --allow-public-bind` issues an endpoint API key,
//! prints it exactly once, and hands the serving engine a key file to enforce.
//! These steps drive a real managed serve and then talk to the endpoint directly
//! over HTTP with and without credentials, because the ENGINE is what enforces the
//! key — the CLI only issues it. That is also why these steps are GPU-lane only:
//! there is no plan-only serve mode that would surface enforcement without a real
//! engine and model. Contracts read from the CLI's own output (EAI-8072).

use std::time::{Duration, Instant};

use cucumber::{then, when};

use crate::E2eWorld;
use e2e_cucumber::serve_log::service_log_tail;

use crate::e2e::serving_steps::{
SERVE_BASE_URL, ensure_serve_port_free, host_serve_target, serve_timeout_for,
};

/// A public bind listens on all interfaces, so the test still reaches the shared
/// serve port over loopback.
fn endpoint_url() -> &'static str {
SERVE_BASE_URL.as_str()
}

/// Pull the issued endpoint key out of the launch output. `serve` prints it as
/// ` api key: <key>` exactly once, alongside the note and curl example.
fn issued_api_key(output: &str) -> Option<String> {
output
.lines()
.find_map(|line| line.trim().strip_prefix("api key:"))
.map(|key| key.trim().to_owned())
.filter(|key| !key.is_empty())
}

/// GET `<endpoint>/models`, optionally presenting a bearer token. `None` means the
/// request never reached a listener (connection refused / timed out), which is
/// distinct from "answered with a status" — the readiness wait tolerates that while
/// the engine is still binding, but an assertion must not.
async fn try_models_status(bearer: Option<&str>) -> Option<reqwest::StatusCode> {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.build()
.expect("failed to build HTTP client");
let mut request = client.get(format!("{}/models", endpoint_url()));
if let Some(token) = bearer {
request = request.bearer_auth(token);
}
request.send().await.ok().map(|response| response.status())
}

/// As [`try_models_status`], but a transport failure is fatal. Used by the
/// assertions, which only run once the endpoint has already answered — so at that
/// point a refused connection is a genuine failure, not a startup race.
async fn get_models_status(bearer: Option<&str>) -> reqwest::StatusCode {
try_models_status(bearer).await.unwrap_or_else(|| {
panic!(
"GET {}/models did not reach the served endpoint",
endpoint_url()
)
})
}

/// Wait until the authenticated endpoint answers, so the assertions run against a
/// loaded model rather than a still-starting one. Polls WITH the key, because an
/// unauthenticated probe is exactly what the endpoint is meant to reject.
///
/// A managed serve returns as soon as the supervisor is up, so for the first while
/// the engine has not bound the port yet and the connection is simply refused.
/// That is expected here and must be retried — panicking on the first refusal is
/// what made this scenario fail before it ever reached its assertions.
///
/// On timeout the panic carries the engine's own log tail and the last thing the
/// endpoint did, because "did not answer in Ns" alone cannot distinguish the three
/// failures that look identical from outside: the engine never started, it started
/// but is still loading weights, or it is up and REJECTING the key we were issued
/// (which would be a real defect in the contract under test, not a slow start).
async fn wait_for_authenticated_endpoint(key: &str, timeout_secs: u64, serve_stdout: &str) {
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
loop {
let last_seen = match try_models_status(Some(key)).await {
Some(status) if status.is_success() => return,
// A bound engine rejecting the key it was issued is a contract defect,
// not a slow start, so report it now instead of at the deadline.
Some(reqwest::StatusCode::UNAUTHORIZED) => panic!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small one, non-blocking: both failure paths here skip the repo's own serve-failure evidence bundle.

This applies to the 401 fast-fail on line 91 and to the deadline assert! on line 100. Both panics do quote serve stdout plus service_log_tail, which is better than nothing — but unlike every other serve step, which routes through serve_failure_evidence (tests/e2e-cucumber/tests/e2e/serving_steps.rs:106), neither calls archive_service_log nor stop_scenario_services.

Two consequences. The engine's startup banner dies with the scenario's TempDir, so the most useful part of the diagnostic is gone by the time anyone reads the failure. And more importantly, a failing run leaves a 0.0.0.0-bound vLLM still holding the GPU — that's the part I'd actually fix, since a leaked GPU-holding process can poison whatever runs next on the lane.

"public endpoint {} rejected the issued key with 401\n--- SERVE STDOUT ---\n{serve_stdout}\n--- SERVICE LOG (tail) ---\n{}",
endpoint_url(),
service_log_tail(serve_stdout)
),
Some(status) => format!("endpoint answered {status} to the issued key"),
None => "connection refused / timed out".to_owned(),
};
let endpoint = endpoint_url();
assert!(
Instant::now() < deadline,
"public endpoint {endpoint} did not answer an authenticated request within \
{timeout_secs}s\nlast observed: {last_seen}\n--- SERVE STDOUT ---\n{serve_stdout}\
\n--- SERVICE LOG (tail) ---\n{}",
service_log_tail(serve_stdout)
);
tokio::time::sleep(Duration::from_secs(5)).await;
}
}

// ── When ───────────────────────────────────────────────────────────

#[when("the user serves a model on a public interface with public binding allowed")]
async fn serve_public_bind(world: &mut E2eWorld) {
let (model, engine, _) = host_serve_target();
ensure_serve_port_free().await;
// No `--api-key`: the CLI must GENERATE one, which is the behaviour under test.
let stdout = crate::run_rocm_ok(
world,
&[
"serve",
model,
"--engine",
engine,
"--managed",
"--host",
"0.0.0.0",
"--allow-public-bind",
],
);
Comment on lines +118 to +130

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this invocation fails before the CLI ever issues a key, and the GPU CI log for this exact head confirms it.

No --port is passed, but --host 0.0.0.0 is not the canonical loopback host, so serve_port.rs's own policy (and the sibling networking-04 scenario added on this branch) requires an explicit --port on any non-canonical host — automatic selection is refused there.

I read the E2E tests (MI300X, current) job for this PR's current head 2e739a5b (run 43422777319, job 131876765826). networking-05 fails immediately:

Step panicked. Captured output: `rocm serve Qwen/Qwen3.5-0.8B --engine vllm --managed --host 0.0.0.0 --allow-public-bind` failed (rc=1)
--- stderr ---
Error: `rocm serve --host 0.0.0.0` needs an explicit `--port <1-65535>`. Automatic port selection is supported only on the default host 127.0.0.1, ...

So on real hardware right now, none of the behavior this PR exists to pin — key issuance, 401 without a key, 401 with the wrong key, 2xx with the issued key — is actually exercised. The job's GitHub conclusion shows CANCELLED rather than FAILED only because it was killed ~90 minutes later during an unrelated later scenario, so this failure doesn't surface as a red required check, but the scenario itself unambiguously failed.

Fix: pass an explicit --port in this args list (e.g. the shared SERVE_PORT / 11435 that ensure_serve_port_free/the readiness helpers already assume).

let key = issued_api_key(&stdout)
.unwrap_or_else(|| panic!("serve did not print an endpoint api key:\n{stdout}"));
wait_for_authenticated_endpoint(&key, serve_timeout_for(world), &stdout).await;
world.endpoint = Some(endpoint_url().to_owned());
world.endpoint_api_key = Some(key);
world.model_name = Some(model.to_owned());
world.cli_output = Some(stdout);
}

// ── Then ───────────────────────────────────────────────────────────

#[then("the CLI shows the endpoint key once and how to send it")]
async fn shows_key_once(world: &mut E2eWorld) {
let output = world.cli_output.as_deref().expect("no serve output");
let key = world
.endpoint_api_key
.as_deref()
.expect("no endpoint api key captured");
// Exactly one dedicated `api key:` line — this is the launch output's single
// disclosure of the secret, matching the "shown only now" promise beneath it.
// The key also appears inside the adjacent `example: curl -H "Authorization:
// Bearer <key>"` usage line, which is deliberate guidance rather than a second
// disclosure, so count only the dedicated line rather than occurrences of the
// key itself.
let shown = output
.lines()
.filter(|line| line.trim().starts_with("api key:"))
.count();
assert_eq!(
shown, 1,
"expected exactly one dedicated `api key:` line, got {shown}:\n{output}"
);
// The key may appear only on the `api key:` line and in the curl example; any
// other line carrying it means it leaked elsewhere in the launch output.
let leaked: Vec<&str> = output
.lines()
.filter(|line| line.contains(key))
.filter(|line| {
let line = line.trim();
!line.starts_with("api key:") && !line.starts_with("example:")
})
.collect();
assert!(
leaked.is_empty(),
"the key leaked onto lines other than the api key line and curl example: {leaked:?}"
);
assert!(
output.contains("shown only now"),
"expected the show-once note, got:\n{output}"
);
assert!(
output.contains("Authorization: Bearer"),
"expected the CLI to say how to send the key, got:\n{output}"
);
assert!(
key.len() >= 16,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this assertion can't fail. ENDPOINT_API_KEY_LEN is a constant 48, so key.len() >= 16 is true by construction, and the properties that could vary — randomness and alphabet — are already covered by generate_endpoint_api_key_is_random_and_alphanumeric.

Beyond being dead weight, it reads as a key-strength guarantee that the Gherkin never promises, which is a little misleading about what this scenario is pinning. I'd just drop it.

"expected a strong generated key, got one of {} chars",
key.len()
);
}

#[then("a request without the key is refused as unauthorized")]
async fn unauthenticated_refused(_world: &mut E2eWorld) {
let status = get_models_status(None).await;
assert_eq!(
status,
reqwest::StatusCode::UNAUTHORIZED,
"an unauthenticated request to a public endpoint must be refused, got {status}"
);
}

#[then("a request with the wrong key is refused as unauthorized")]
async fn wrong_key_refused(_world: &mut E2eWorld) {
let status = get_models_status(Some("not-the-issued-key")).await;
assert_eq!(
status,
reqwest::StatusCode::UNAUTHORIZED,
"a request with the wrong key must be refused, got {status}"
);
}

#[then("a request carrying the issued key is accepted")]
async fn issued_key_accepted(world: &mut E2eWorld) {
let key = world
.endpoint_api_key
.clone()
.expect("no endpoint api key captured");
let status = get_models_status(Some(&key)).await;
assert!(
status.is_success(),
"the issued key must be accepted, got {status}"
);
}
6 changes: 3 additions & 3 deletions tests/e2e-cucumber/tests/e2e/serving_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ pub(crate) async fn model_is_ready(

/// The OpenAI-compatible base URL every GPU serve scenario serves on (see
/// [`SERVE_PORT`]), and the model listing readiness is judged by.
static SERVE_BASE_URL: LazyLock<String> =
pub(crate) static SERVE_BASE_URL: LazyLock<String> =
LazyLock::new(|| format!("http://127.0.0.1:{SERVE_PORT}/v1"));
static MODELS_URL: LazyLock<String> =
LazyLock::new(|| format!("http://127.0.0.1:{SERVE_PORT}/v1/models"));
Expand Down Expand Up @@ -266,7 +266,7 @@ const ASSISTANT_PORT: u16 = 8001;
/// (see [`wait_for_free_vram`]). Callers that only need the reset can ignore it;
/// a serve that then fails to become ready reports it, because "the previous
/// engine had not released the GPU yet" is otherwise invisible in the log.
async fn ensure_serve_port_free() -> String {
pub(crate) async fn ensure_serve_port_free() -> String {
// Always kill any listener on the shared port — NOT just one that already
// answers /v1/models. A prior scenario's vLLM that is still LOADING holds the
// port and GPU memory without yet serving /v1/models; if we only checked HTTP
Expand Down Expand Up @@ -527,7 +527,7 @@ async fn reserve_default_serve_port(world: &mut E2eWorld) {
/// exercised exactly once, in the `@nightly` `serve-large-model-inference`
/// scenario (Qwen3.6-27B) — never on the per-PR path. Do not raise a serve
/// scenario's model unless a smaller one genuinely cannot prove the assertion.
fn host_serve_target() -> (&'static str, &'static str, &'static str) {
pub(crate) fn host_serve_target() -> (&'static str, &'static str, &'static str) {
if e2e_cucumber::capability::host_capability().effective_serve_engine == "lemonade" {
// GGUF via lemonade's llama.cpp backend; endpoint reports e.g.
// Qwen3-0.6B-Q4_0.gguf, so "Qwen3-0.6B" is the distinctive substring.
Expand Down
Loading