Repository navigation
fix(install): never let apt remove ROCm packages during dependency setup (EAI-7957) - #238
Conversation
…957) `apt-get install -y` assumes yes for removals as well as installs, so the automatic OpenMPI/libatomic/libnuma setup could apply a dependency solution that removes the ROCm stack. This runs unattended under `rocm install sdk` whenever root or passwordless sudo is available, so a pre-existing ROCm install could be broken with no prompt. Simulate each apt install with `apt-get -s` first and abort when the transaction would remove a ROCm/AMDGPU package, naming every affected package. Other package managers are unaffected: their plans are additive and have no assume-yes removal path. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
… to end Pre-PR review found the protected-package matcher too narrow and the new user-visible refusal untested at the behaviour level. Match ROCm packages by component-family prefix instead of requiring a `<family>-` shape. Core packages such as hsakmt-roct, llvm-amdgpu, rocminfo and hipblas are dashless, so the previous rule did not match them and apt could still remove them. The matcher now deliberately favours recall: a false positive only asks the user to install by hand, while a false negative breaks their ROCm install. Add the Gherkin scenario root AGENTS.md requires for user-observable behaviour. It plants a runtime registry entry and stands in for the package manager, so it needs no GPU, no engine install and no network, and asserts the CLI refuses, refuses before installing anything, and names every ROCm package at risk. Verified as a regression test: it fails with the guard removed and passes with it. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…scenario Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
f884097 to
681b91a
Compare
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…up-removes-rocm-system Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…up-removes-rocm-system Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
1ca5e3d to
b2035c7
Compare
|
Note on the red E2E tests (Strix Halo, Windows) check: this is a pre-existing failure on
Fixing it — either the Windows Lemonade serve, or an |
r0x0r
left a comment
There was a problem hiding this comment.
Review: apt ROCm-removal guard (EAI-7957)
Verdict: Approve. Well-scoped fix at the correct layer, with a genuine regression test.
What it does
Before any apt-get install step in the OpenMPI/vLLM dependency setup, the CLI now simulates the transaction with apt-get -s, parses the Remv records, and refuses (naming every affected package) if any ROCm/AMDGPU package would be evicted. This closes the reported breakage where apt-get install -y silently removed a working ROCm stack unattended.
Strengths
- Root cause, correct layer. The
-y-assumes-yes-for-removals behavior is handled exactly where the install argv is executed, with pure, unit-testable helpers factored intocrates/rocm-core/src/openmpi.rs. - Simulation is safe.
apt_simulate_argvdrops thesudoprefix and the assume-yes flags, soapt-get -sneeds neither root nor approval and stays visibly non-destructive. - Recall-favoring matcher is the right call. Prefix-family matching (with dashless cores like
hsakmt-roct,llvm-amdgpu,rocminfo,hipblascalled out) beats an exhaustive list that silently rots each ROCm release. The false-positive/false-negative asymmetry is argued well, and the legitimately-swapped packages (openmpi-bin,mpich,libnuma1,libatomic1) are proven removable. - Genuine regression test. The new
@id:deps-guard-refuses-rocm-removalscenario runs on the mock lane (no GPU/network) and was verified to fail with the guard removed. Asserting refusal before any install, and that every at-risk package is named, is exactly right. - Fail-open is documented and scoped to apt only (dnf/zypper/pacman plans are additive).
CI note
The two red checks (E2E tests (GPU), E2E tests (Strix Halo, Windows)) fail in unrelated live-vLLM/self-hosted scenarios — chat tool-choice (expect_xfail: true), subcommand alphabetical ordering, and short-model-name resolution. None exercise the Linux apt dependency-guard path this PR touches, and the new guard scenario passed on the default lane. Worth a maintainer rerun, but not blocking on the change itself.
Minor / optional (non-blocking)
- Residual fail-open window. If
apt-get -serrors for a benign reason while the realapt-get install -ystill computes and applies a destructive solution, the guard fails open. In practice both use the same solver, so-sfailing implies the real install failing — low risk — but a one-line note in the guard doc-comment could make that explicit. "roc"/"hip"prefixes are broad (e.g. a hypotheticalrocksdbremoval would be falsely protected). Consistent with the stated recall-first design and the blast radius of a false positive is just "install that dep by hand," so fine as-is — flagging for awareness.
…up-removes-rocm-system Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Summary
rocm install sdkcould remove a working ROCm installation without asking.Installing vLLM needs the OpenMPI runtime, which the CLI installs through the host package manager with
apt-get install -y. That-yassumes yes for package removals as well as installs, so when apt decided to satisfy OpenMPI by evicting the ROCm stack, it went ahead unattended — the SDK install runs this path automatically whenever it has root or passwordless sudo. The field report sawrocm,rocm-hip,rocm-hip-runtime-dev,mivisionx-devandrpp-devremoved while an older toolchain was pulled in, breaking a working ROCm 7.14 install.This makes that transaction fail loudly instead of silently destroying the installation.
Changes
apt-get installstep, simulate it first withapt-get -sand parse the removals apt reports.--yespath and the unattended path alike, since the reported breakage happened unattended.hsakmt-roct,llvm-amdgpu,rocminfo,hipblas), so matching only on a<family>-shape would miss them.openmpi-bin,mpich,libnuma1,libatomic1) removable, so the guard cannot block the very install it protects.Test plan
crates/rocm-core/src/openmpi.rscover removal parsing (including apt's trailing*), the simulate argv (adds-s, drops-y, tolerates asudoprefix, ignores non-apt commands), and the package classifier — with the dashless core package names as explicit regression cases.@id:deps-guard-refuses-rocm-removal(tests/e2e-cucumber/features/dependency_guard.feature) covers the user-visible behaviour: it asserts the CLI refuses, refuses before installing anything, and names every ROCm package at risk. It plants a runtime registry entry and stands in for the package manager, so it needs no GPU, no engine install and no network, and runs on the default mock lane.The scenario is Linux-only (
@requires-os:linux) and assumes an apt host, which every Linux CI lane is; the guard is apt-specific by design. That precondition is recorded in the feature file.Fixes EAI-7957.