Skip to content

driver install: plan Ubuntu 26.04 (resolute) DKMS flow, warn on previous amdgpu-dkms - #778

Merged
juhovainio merged 2 commits into
mainfrom
jira-ROCMAI-747-ubuntu-resolute-driver-plan
Oct 29, 2026
Merged

juhovainio merged 2 commits into
mainfrom
jira-ROCMAI-747-ubuntu-resolute-driver-plan

Conversation

@juhovainio

Copy link
Copy Markdown
Collaborator

Implements ROCMAI-747

What

  • Ubuntu 26.04 now plans the same apt DKMS flow as 22.04/24.04 with the resolute suite. I checked that repo.radeon.com/amdgpu/latest/ubuntu/dists/resolute/Release and the 31.60 equivalent both return 200. The TODO and the "not yet published" native-archive arm are removed.
  • Detects amdgpu-dkms left from a previous installation. The first prepare command prints a warning naming apt-get autoremove -y amdgpu-dkms and the preflight list mentions it. Nothing is removed automatically.
  • rocm-core diagnose drops its 26.04 native-archive special-casing and adds 26.04 to the DKMS support matrix, which a test keeps in step with the plan builder.

Tests

  • Unit: driver_plan_ubuntu_2604_* rewritten for the DKMS flow, plus a test that runs the warning command against a stubbed dpkg-query (warns when installed, silent otherwise).
  • e2e: driver-install-06 now asserts the host's own suite codename and the previous-driver warning on whichever Ubuntu release the lane runs. It is gated @requires-bare-metal, so it runs on the Linux e2e lanes.
  • README, docs/testing.md and docs/manual-testing.md updated.

Not run locally

  • The e2e scenario itself (not run locally); cargo clippy --workspace --all-targets -D warnings is clean and the touched unit tests pass.
  • The pre-push hook failed three times on unrelated parallel-run flakes (therock tar unpack, comfyui status tests; they pass in isolation), so the branch was pushed with --no-verify at the operator's direction. CI is the full gate.

@juhovainio
juhovainio requested a review from a team as a code owner October 28, 2026 11:02
@juhovainio
juhovainio requested a review from tomastola October 28, 2026 11:02

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on — getting Ubuntu 26.04 and the old-driver warning sorted out is genuinely useful, and the core of this change is solid. I went through it closely and want to share a few things, mixed in with what I really liked.

A few things worth fixing before this merges

  1. rocm fix still tells people the old story for Ubuntu 26.04.
    This PR teaches the installer that 26.04 now gets the full driver-install flow, same as 22.04/24.04 — great. But there's a separate file, crates/rocm-core/src/fix.rs, that still has the old wording baked in (around lines 325, 530-531, and 561). It still says things like "Ubuntu 26.04 ships the driver in-tree, just run apt install rocm" — which isn't true anymore after this change. So someone running rocm fix on 26.04 would get advice that contradicts what the installer itself now does. Could you update those three spots to match? It'd also be worth adding a quick test that keeps this file and the installer logic in sync going forward, so this can't quietly happen again.

  2. The "no previous driver installed" line reads like a hard requirement, but it isn't one.
    In apps/rocm/src/driver_install.rs around line 1394, the new line in the preflight-checks list says "no previous amdgpu-dkms install (the first command warns if one is found; nothing is removed automatically)." Since nothing actually blocks or changes based on this, it feels a little misleading sitting in a list of real preconditions. Maybe just move it into the explanatory notes instead, so it doesn't read as something the plan depends on?

  3. One test doesn't quite prove what it looks like it proves.
    In tests/e2e-cucumber/tests/e2e/driver_steps.rs (lines 95-99), the dry-run test checks the warning text, but dry-run mode means the actual warning logic never runs here — so this test can't really confirm the warning only shows up when it should. The good news is you do have a test that proves that properly: previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed. Could you just add a short comment pointing to that test from this one, so it's clear where the real proof lives?

Small, optional polish

  • driver_steps.rs:57 still has a comment mentioning "native-archive," but that code path was removed in this PR — just a leftover word, easy one-line cleanup.
  • The dnf/SLES install paths don't get the same "warn about a previous driver" treatment — totally fine for this PR since it's scoped to the Ubuntu apt docs, just flagging it in case it's useful as a follow-up someday.

A couple of things to double check with you (not blocking, just curious)

  • The old code's reasoning for skipping 26.04 mentioned two things: the package repo not yet having a "resolute" suite, and 26.04 shipping ROCm in-tree already. This PR clearly fixes the first one — has the second one also been confirmed as a non-issue? Just want to make sure that wasn't an open question that got carried forward by accident.
  • The PR description mentions manually checking that the resolute package URLs return 200 — nice diligence! I wasn't able to re-check that myself from here (no outbound network access in my sandbox), so I'm trusting your note on it. No action needed, just flagging that I couldn't independently confirm it.
  • The description also mentions pushing with --no-verify a few times because of flaky pre-push hooks, "at the operator's direction." Just flagging this so it's on record and confirmed as intentional, per the repo's usual signing/verification expectations.

Things I really liked

  • previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed is a genuinely good test — it actually runs the real shell command against a fake dpkg-query and checks both outcomes, rather than just checking a string. That's exactly the right way to test this kind of behavior.
  • Good catch keeping the two cross-crate "do these two plans still agree with each other" tests in sync — easy thing to forget, and you didn't.
  • Really appreciated that the README, the manual-testing doc, the testing doc, and the Gherkin scenario were all updated together with the code, instead of as a follow-up. That makes this much easier to trust.
  • Commit is signed off, and a scan for internal/confidential wording came back clean — nice and tidy.

One heads-up for after this merges

A few of the hardware-backed test lanes (MI300X, Strix Halo, W7900) hadn't finished running yet as of this review, so it's worth a quick look once they report in, especially for the new 26.04 scenario.

Everything else here is in really good shape — nice work, and happy to take another look once the above is addressed!

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 6966a43

Review — needs work

Full review of the whole change.
Implements ROCMAI-747 (named in the description); ticket not retrievable from here.

Blocking

  • crates/rocm-core/src/fix.rs still tells rocm fix the old Ubuntu 26.04 story, contradicting the plan this PR now generates — crates/rocm-core/src/fix.rs:325, crates/rocm-core/src/fix.rs:530-531, crates/rocm-core/src/fix.rs:561
    This PR changes build_driver_install_plan_at (apps/rocm/src/driver_install.rs) and dkms_driver_plan_is_supported (crates/rocm-core/src/diagnose.rs) so Ubuntu 26.04 now gets the identical apt/DKMS flow as 22.04/24.04 — the old ubuntu_native_archive arm and its "ships in-tree, run apt install rocm" guidance are removed from both files, and a cross-file test (driver_plan_support_matches_rocm_core_diagnose) keeps those two in sync, as the PR description says. fix.rs is a third place encoding the same knowledge (the static FixRecipe catalog consumed by rocm fix), and it was not touched: fix-3-rocm-kernel's rationale still reads "...(documented distros only; Ubuntu 26.04 ships the driver in-tree, use apt install rocm)", and fix-12-installer's commands/notes still say "On Ubuntu 26.04 the driver is in-tree: skip the purge/remove, the DKMS reinstall and the DKMS verify, and run sudo apt install rocm instead" and "...or dpkg-query -W rocm after sudo apt install rocm on Ubuntu 26.04." After this PR ships, rocm fix gives a 26.04 user advice that directly contradicts what rocm install driver --dkms and rocm diagnose now do on that same host, and no test guards fix.rs against this drift (only the two already-synced files have one).
    Confidence 90 · architectural · Fix: update the three spots in fix.rs to match the new DKMS-flow behavior, and extend (or add) a consistency test that fails if fix.rs's Ubuntu 26.04 wording diverges from driver_install.rs/diagnose.rs again.

  • Two self-hosted Linux E2E lanes fail at the reviewed commit — CI (gh pr checks 778)
    E2E tests (Strix Halo, Ubuntu, release, current) and E2E tests (Strix Halo, Ubuntu, nightly, current) both report fail at commit 6966a434. These are real bare-metal Linux lanes, exactly the kind the PR's new driver-install-06 scenario (@requires-bare-metal) targets, and the matching Windows and WSL2 Strix Halo lanes on the same run pass — consistent with the failure being specific to this PR's Ubuntu/apt changes rather than infrastructure-wide. The PR description itself says the e2e scenario "was not run locally." Job logs were not reachable from here (403 on the Actions API), so the exact assertion that fails is not established, but a failing check is blocking regardless of cause.
    Confidence 100 · logic · Fix: open the two failing job logs and resolve whatever the new driver-install-06 assertions (codename format, previous-driver warning text) are hitting on real Strix Halo Ubuntu hosts before merge.

Non-blocking

  • Leftover "native-archive" wording in a comment that no longer has a native-archive arm — tests/e2e-cucumber/tests/e2e/driver_steps.rs:57
    The comment reads "...this scenario's premise is Ubuntu specifically (every assertion below hard-codes Ubuntu's apt/native-archive command text)..." but the native-archive arm and its assertions were removed by this PR; every assertion below is apt-only now.
    Confidence 85 · mechanical · Fix: drop "/native-archive" from the comment.

  • The e2e dry-run assertion only proves the warning text is planned, not that it fires conditionally — tests/e2e-cucumber/tests/e2e/driver_steps.rs:97-100
    assert_driver_plan_matches_host_release runs the CLI with --dry-run and asserts output.contains("from a previous installation is installed and was not removed"). Because the plan always emits this command text (the conditional is evaluated by the shell at execution time, not at plan-build time), this assertion passes regardless of whether amdgpu-dkms is actually installed on the runner — it cannot show the warning only appears when expected. The real proof is the unit test previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed in apps/rocm/src/driver_install.rs, which stubs dpkg-query and checks both outcomes, but nothing here points a future reader at it.
    Confidence 75 · test-coverage · Fix: add a short comment next to this assertion pointing at previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed as the test that actually proves the conditional behavior.

Decisions for the author

  • "no previous amdgpu-dkms install" sits in preflight_checks, a list otherwise read as hard preconditions — non-blocking-improvement
    apps/rocm/src/driver_install.rs:1394 adds this line to preflight_checks, but nothing in the plan actually blocks or changes based on it — the self-disclaimer "(the first command warns if one is found; nothing is removed automatically)" is right there in the same string. Every other entry in that list is a real precondition the plan depends on. Moving it into reason/explanatory notes instead would stop it reading as a requirement it isn't; leaving it where it is isn't wrong, just arguably misplaced.

  • Only the apt/DKMS flow gets the previous-driver warning; dnf (RHEL-family) and zypper (SLES) do not — tradeoff
    AMD's stated guidance to remove a previous AMDGPU driver install presumably isn't apt-specific, but this PR scopes the warning to apt_driver_plan only, matching its stated Ubuntu-26.04 focus. Reasonable to keep this PR scoped and do the other package managers as a follow-up, but worth confirming that's the intent rather than an oversight.

Positive signals

  • previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed (apps/rocm/src/driver_install.rs) runs the generated shell command against a stubbed dpkg-query binary and checks both the "installed" and "not installed" outcomes — real behavioral proof rather than a string match, for a case (shell conditional logic baked into a plan string) that's easy to under-test.
  • dkms_driver_plan_support_matrix_covers_every_arm (crates/rocm-core/src/diagnose.rs) and driver_plan_support_matches_rocm_core_diagnose (apps/rocm/src/driver_install.rs) are kept in sync by a dedicated cross-file test, so the one pairing that is guarded against drift (diagnose.rs vs. driver_install.rs) stayed correct through this change even as the Ubuntu 26.04 behavior flipped.
  • README, docs/testing.md, docs/manual-testing.md and the Gherkin feature file were all updated in the same commit as the code change, consistent with the new behavior (the one exception being the untouched fix.rs, noted above).

Deployment notes

  • Two hardware-backed E2E lanes (MI300X, W7900) were still QUEUED as of the reviewed commit and had not reported a result; confirm they pass before merge, per the author's own description.
  • rocm fix's advice for Ubuntu 26.04 will be actively wrong the moment this merges (see Blocking) — this is a behavior regression for any 26.04 user who runs rocm fix rather than rocm install driver --dkms directly.

What this covered

Every file in the diff read in full (README.md, apps/rocm/src/driver_install.rs, crates/rocm-core/src/diagnose.rs, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/driver_install.feature, tests/e2e-cucumber/tests/e2e/driver_steps.rs), plus the full unchanged crates/rocm-core/src/fix.rs and surrounding context in driver_install.rs/diagnose.rs (match arms, apt_driver_plan, driver_command/DriverCommandPhase) needed to judge the change in place. Diff taken between d59f596f6e08297c207f008021eecb37defc3558 (merge-base with origin/main) and 6966a434332da6d76685775974f0955618421336 (PR head). Single-pass review — the diff (7 files, ~400 lines) was small enough for one pass to read completely.

Did not run: history and prior-changes passes were not run as separate workers given the diff size, but git log/the existing cross-file consistency tests were read directly during the main pass. CI job logs for the two failing Strix Halo Ubuntu lanes could not be fetched (403 on the Actions API from here); the failing-check finding above is reported from gh pr checks/statusCheckRollup at face value, without re-deriving what exactly failed. The ROCMAI-747 ticket is named in the description but not retrievable from here; its contents are not assumed.
Not reconciled against the existing review thread on this PR (a prior human reviewer's comments were not read before or during this pass).

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. repo.radeon.com/amdgpu/latest/ubuntu/dists/resolute/Release returns 200, so planning the resolute suite is justified. Two things have to be fixed first.

Blocking

1. fix.rs still says Ubuntu 26.04 plans nothing and tells users to run apt install rocm. — crates/rocm-core/src/fix.rs:325, :530-531, :561

The PR removes the in-tree premise everywhere else (diagnose.rs, the plan builder, README, docs), but three strings in fix.rs were missed:

  • Line 325: "Ubuntu 26.04 ships the driver in-tree, use apt install rocm".
  • Lines 530-531: "On Ubuntu 26.04 the driver is in-tree: skip the purge/remove, the DKMS reinstall and the DKMS verify".
  • Line 561: "rocm install driver --dkms --yes exits non-zero where it plans nothing (Ubuntu 26.04, undocumented distros)".

After this PR, --dkms --yes plans and runs the install on 26.04, so all three are false. Lines 530-531 tell the user to skip the flow this PR enables.

Fix: update or remove the three strings, and add a test that keeps fix.rs aligned with dkms_driver_plan_is_supported.

2. The warning says "stop", but nothing stops, so the install continues over the old driver. — apps/rocm/src/driver_install.rs:1308-1316

The new first Prepare command only echos "…AMD advises removing it first: stop and run … then re-run this install" and exits 0. The remaining commands then run, including apt-get install -y amdgpu-dkms. Under --yes the user gets no chance to stop. The new tests prove the echo fires and that removal is not planned, but nothing checks that the install is held back. The new preflight line ("no previous amdgpu-dkms install (the first command warns…)") reads as a gate and isn't one.

Fix: make code and wording agree. Either say "continuing anyway", or make the command exit non-zero so the plan aborts. If you abort, define how the plan runner reports that. Reword the preflight line to match.

Non-blocking

  • previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed removes its temp dir as the last line, so a failed assert leaks it. The path is also temp_dir()/rocm-prevdrv-<pid>. A unique tempdir with drop cleanup would be sturdier.
  • The warning is in apt_driver_plan(dkms=true), which the Debian 12/13 arm also uses. The README says only "On Ubuntu". Confirm that is intended and say so in the docs.

Decisions for the author

  • Planning a mutating DKMS install on 26.04 — tradeoff. The suite exists, but 26.04 ships a newer in-tree amdgpu. I could not verify that amdgpu-dkms builds against the 26.04 kernel. A plan that runs and then fails the DKMS build is worse than the old refusal. Please confirm this was tried on a real 26.04 host, since the e2e only previews with --dry-run.
  • Hand-mirrored plan matrix — non-blocking improvement. rocm-core still mirrors the plan matrix by hand in dkms_driver_plan_is_supported, which is how fix.rs drifted. A shared table or a parity test would stop that.

@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed in 576d811.

Blocking (tomastola, also raised by jussielo-amd and nowycondro)

  • fix.rs: removed the "26.04 ships the driver in-tree / apt install rocm" text from fix-3-rocm-kernel and fix-12-installer (rationale, commands, notes). Added recipes_do_not_describe_ubuntu_26_04_as_unplanned, which fails if any recipe says "in-tree" or "apt install rocm" again. It is a text guard, not a derived table, so the hand-mirrored plan matrix stays as a possible follow-up.
  • Warning vs. behavior: made the wording match the code. The warning now says "continuing anyway. If the install fails, remove it and re-run", and the plan does not abort. The unit test asserts that wording. I did not add a gate, since aborting under --yes would also need a plan-runner failure path; say if you want that instead.
  • The "no previous amdgpu-dkms" line is gone from preflight_checks and now lives in the plan reason (the test asserts it is no longer a preflight check).

Non-blocking, done

  • The previous-driver unit test now uses tempfile::tempdir() with drop cleanup.
  • The e2e dry-run assertion carries a comment pointing at previous_driver_warning_fires_only_when_amdgpu_dkms_is_installed. Dropped the stale "native-archive" comment.
  • README states the warning applies to every apt DKMS target (Ubuntu and Debian 12/13), as intended.

Left as is / for the author

  • dnf/zypper previous-driver warning: deliberately a follow-up, to keep this PR scoped to apt.
  • Whether amdgpu-dkms builds on a real 26.04 host: not verified here. The e2e only dry-runs the plan.
  • CI: the red E2E lanes fail on a model is being served on GPU (Qwen3-0.6B endpoint never served), not a driver-install scenario. Not touched here; will check on the re-run.
  • The commit was pushed with --no-verify after the pre-push suite hit an unrelated tar launch flake (sdk_tarball_unpack_extracts_a_benign_tarball_into_the_install_root) three times. Touched tests and clippy pass locally.

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both blocking findings are fixed in 576d811.

  • fix.rs: the three stale 26.04 strings are gone, and recipes_do_not_describe_ubuntu_26_04_as_unplanned now keeps recipe text from drifting back.
  • Warning: the message now says "continuing anyway", and the plan's reason says the first command only warns. The misleading preflight line is removed, and the README says the install continues and that Debian 12/13 are covered. The unit test asserts all of this.

Approving. Remaining non-blocking items:

  • The fix.rs guard test only looks for the substrings in-tree and apt install rocm. A differently worded stale claim would pass it. The dkms_driver_plan_is_supported parity idea from before still stands.
  • I still have no evidence that amdgpu-dkms builds on a real 26.04 kernel. The e2e only previews with --dry-run.

…ous amdgpu-dkms

AMD's repo.radeon.com/amdgpu tree now publishes a resolute suite (verified for
both latest and 31.60), so 26.04 takes the same apt DKMS plan as 22.04/24.04
instead of the native-archive refusal. The plan also detects an installed
amdgpu-dkms from a previous installation and warns before any change; nothing
is removed automatically. diagnose drops its 26.04 special-casing to match.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
- fix.rs: drop the stale "26.04 ships the driver in-tree" advice from
  fix-3 and fix-12; add a test that no recipe still says so
- previous-driver warning now says the install continues anyway, and the
  claim moves from preflight_checks into the plan reason
- previous-driver unit test uses a tempdir with drop cleanup
- e2e step: point the dry-run assertion at the test that proves the
  conditional; drop a stale native-archive comment
- README: note the warning also covers Debian 12/13

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio
juhovainio force-pushed the jira-ROCMAI-747-ubuntu-resolute-driver-plan branch from 576d811 to dc814be Compare October 29, 2026 12:33
@juhovainio
juhovainio added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 5c8fdd5 Oct 29, 2026
24 checks passed
@juhovainio
juhovainio deleted the jira-ROCMAI-747-ubuntu-resolute-driver-plan branch October 29, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants