Skip to content

fix(install): never let apt remove ROCm packages during dependency setup (EAI-7957) - #238

Merged
fredespi merged 7 commits into
mainfrom
vllm-openmpi-auto-setup-removes-rocm-system
Aug 21, 2026
Merged

fredespi merged 7 commits into
mainfrom
vllm-openmpi-auto-setup-removes-rocm-system

Conversation

@fredespi

Copy link
Copy Markdown
Collaborator

Summary

rocm install sdk could remove a working ROCm installation without asking.

Installing vLLM needs the OpenMPI runtime, which the CLI installs through the host package manager with apt-get install -y. That -y assumes yes for package removals as well as installs, so when apt decided to satisfy OpenMPI by evicting the ROCm stack, it went ahead unattended — the SDK install runs this path automatically whenever it has root or passwordless sudo. The field report saw rocm, rocm-hip, rocm-hip-runtime-dev, mivisionx-dev and rpp-dev removed while an older toolchain was pulled in, breaking a working ROCm 7.14 install.

This makes that transaction fail loudly instead of silently destroying the installation.

Changes

  • Before running an apt-get install step, simulate it first with apt-get -s and parse the removals apt reports.
  • If the transaction would remove any ROCm/AMDGPU package, refuse: print every affected package and tell the user to resolve the conflict manually. This applies on the --yes path and the unattended path alike, since the reported breakage happened unattended.
  • Packages are matched by component family rather than an exhaustive list, because ROCm ships hundreds of packages and adds more each release. Many core names are dashless (hsakmt-roct, llvm-amdgpu, rocminfo, hipblas), so matching only on a <family>- shape would miss them.
  • The matcher deliberately favours recall: a false positive only asks the user to install by hand, while a false negative breaks their ROCm install. It still leaves the packages these plans legitimately install or swap (openmpi-bin, mpich, libnuma1, libatomic1) removable, so the guard cannot block the very install it protects.
  • Other package managers are untouched — dnf/zypper/pacman plans are additive and have no assume-yes removal path.
  • The guard fails open if the simulation itself cannot run, so it never blocks an otherwise healthy install.

Test plan

  • Unit tests in crates/rocm-core/src/openmpi.rs cover removal parsing (including apt's trailing *), the simulate argv (adds -s, drops -y, tolerates a sudo prefix, ignores non-apt commands), and the package classifier — with the dashless core package names as explicit regression cases.
  • New scenario @id:deps-guard-refuses-rocm-removal (tests/e2e-cucumber/features/dependency_guard.feature) covers the user-visible behaviour: it asserts the CLI refuses, refuses before installing anything, and names every ROCm package at risk. It plants a runtime registry entry and stands in for the package manager, so it needs no GPU, no engine install and no network, and runs on the default mock lane.
  • Verified as a real regression test: with the guard removed the scenario fails, and passes with it restored.
  • Linux container run: clippy clean, 434 unit tests pass, and the e2e suite runs the new scenario green.

The scenario is Linux-only (@requires-os:linux) and assumes an apt host, which every Linux CI lane is; the guard is apt-specific by design. That precondition is recorded in the feature file.

Fixes EAI-7957.

@fredespi
fredespi requested a review from a team as a code owner August 12, 2026 16:55
…957)

`apt-get install -y` assumes yes for removals as well as installs, so the
automatic OpenMPI/libatomic/libnuma setup could apply a dependency solution
that removes the ROCm stack. This runs unattended under `rocm install sdk`
whenever root or passwordless sudo is available, so a pre-existing ROCm
install could be broken with no prompt.

Simulate each apt install with `apt-get -s` first and abort when the
transaction would remove a ROCm/AMDGPU package, naming every affected
package. Other package managers are unaffected: their plans are additive
and have no assume-yes removal path.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
… to end

Pre-PR review found the protected-package matcher too narrow and the new
user-visible refusal untested at the behaviour level.

Match ROCm packages by component-family prefix instead of requiring a
`<family>-` shape. Core packages such as hsakmt-roct, llvm-amdgpu, rocminfo
and hipblas are dashless, so the previous rule did not match them and apt
could still remove them. The matcher now deliberately favours recall: a false
positive only asks the user to install by hand, while a false negative breaks
their ROCm install.

Add the Gherkin scenario root AGENTS.md requires for user-observable
behaviour. It plants a runtime registry entry and stands in for the package
manager, so it needs no GPU, no engine install and no network, and asserts the
CLI refuses, refuses before installing anything, and names every ROCm package
at risk. Verified as a regression test: it fails with the guard removed and
passes with it.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…scenario

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi force-pushed the vllm-openmpi-auto-setup-removes-rocm-system branch from f884097 to 681b91a Compare August 12, 2026 17:18
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…up-removes-rocm-system

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…up-removes-rocm-system

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi force-pushed the vllm-openmpi-auto-setup-removes-rocm-system branch from 1ca5e3d to b2035c7 Compare August 20, 2026 08:02
@fredespi

Copy link
Copy Markdown
Collaborator Author

Note on the red E2E tests (Strix Halo, Windows) check: this is a pre-existing failure on main, not something introduced by this PR.

  • The only failing scenario is serve-hf-checkpoint-inference (0 XPASS; none of this PR's new dependency_guard scenarios failed). This PR does not modify that scenario or its step.
  • The same scenario fails on main's own E2E self-hosted runs (for example run 32258257550 on 8cf6ec9b), reconciling to the same single unexpected failure. It passes on Strix-Halo Ubuntu but has failed on Strix-Halo Windows on every main run since it was introduced in fix(lemonade): find llama-server under nested backend dirs (EAI-8026) #242 (EAI-8026).
  • The Strix-Windows self-hosted job is non-blocking (continue-on-error) and is not one of the required status checks; all required checks here are green.

Fixing it — either the Windows Lemonade serve, or an expectations.toml xfail scoped to the Windows/Lemonade/gfx1151 condition and tied to EAI-8026 — belongs on main with EAI-8026, outside the scope of this dependency-safety fix.

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: apt ROCm-removal guard (EAI-7957)

Verdict: Approve. Well-scoped fix at the correct layer, with a genuine regression test.

What it does

Before any apt-get install step in the OpenMPI/vLLM dependency setup, the CLI now simulates the transaction with apt-get -s, parses the Remv records, and refuses (naming every affected package) if any ROCm/AMDGPU package would be evicted. This closes the reported breakage where apt-get install -y silently removed a working ROCm stack unattended.

Strengths

  • Root cause, correct layer. The -y-assumes-yes-for-removals behavior is handled exactly where the install argv is executed, with pure, unit-testable helpers factored into crates/rocm-core/src/openmpi.rs.
  • Simulation is safe. apt_simulate_argv drops the sudo prefix and the assume-yes flags, so apt-get -s needs neither root nor approval and stays visibly non-destructive.
  • Recall-favoring matcher is the right call. Prefix-family matching (with dashless cores like hsakmt-roct, llvm-amdgpu, rocminfo, hipblas called out) beats an exhaustive list that silently rots each ROCm release. The false-positive/false-negative asymmetry is argued well, and the legitimately-swapped packages (openmpi-bin, mpich, libnuma1, libatomic1) are proven removable.
  • Genuine regression test. The new @id:deps-guard-refuses-rocm-removal scenario runs on the mock lane (no GPU/network) and was verified to fail with the guard removed. Asserting refusal before any install, and that every at-risk package is named, is exactly right.
  • Fail-open is documented and scoped to apt only (dnf/zypper/pacman plans are additive).

CI note

The two red checks (E2E tests (GPU), E2E tests (Strix Halo, Windows)) fail in unrelated live-vLLM/self-hosted scenarios — chat tool-choice (expect_xfail: true), subcommand alphabetical ordering, and short-model-name resolution. None exercise the Linux apt dependency-guard path this PR touches, and the new guard scenario passed on the default lane. Worth a maintainer rerun, but not blocking on the change itself.

Minor / optional (non-blocking)

  1. Residual fail-open window. If apt-get -s errors for a benign reason while the real apt-get install -y still computes and applies a destructive solution, the guard fails open. In practice both use the same solver, so -s failing implies the real install failing — low risk — but a one-line note in the guard doc-comment could make that explicit.
  2. "roc" / "hip" prefixes are broad (e.g. a hypothetical rocksdb removal would be falsely protected). Consistent with the stated recall-first design and the blast radius of a false positive is just "install that dep by hand," so fine as-is — flagging for awareness.

…up-removes-rocm-system

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi added this pull request to the merge queue Aug 21, 2026
Merged via the queue into main with commit f57c95f Aug 21, 2026
25 checks passed
@fredespi
fredespi deleted the vllm-openmpi-auto-setup-removes-rocm-system branch August 21, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants