Skip to content

[Issue]: amdgpu-dkms 7.1.3 builds fail non-deterministically on Ubuntu 24.04, abort GNU make, and hard-lock the host during kernel upgrade #240

Open

Description

@Silicontoy

Description

amdgpu-dkms from the AMD 31.50 repository fails non-deterministically while building against Ubuntu 24.04 HWE kernels. The failures have included:

  1. a compile-time raw_atomic_fetch_add error while compiling mmhub_v3_3.o;
  2. /usr/bin/make itself aborting with SIGABRT and double free or corruption (out);
  3. an Ubuntu DKMS Apport hook crash while trying to classify the failed build; and
  4. a complete host hard-lock during a later unattended DKMS retry.

The hard-lock happened while the system was running normally, not after a reboot. The BMC console image froze and the host stopped responding. The persistent journal stops abruptly during apt-daily-upgrade; the unattended-upgrade dpkg log ends in the middle of the DKMS make progress line. No orderly shutdown, kernel panic, OOM, MCE, uncorrectable EDAC event, watchdog report, or pstore record was found.

After power cycling, the newly installed 6.8.0-146-generic kernel was incomplete: its kernel image existed, but /boot/initrd.img-6.8.0-146-generic did not. The host is currently recovered and running the previous 6.8.0-142-generic kernel. I have not deliberately re-run the build because it has already hard-locked this production host.

Environment

  • OS: Ubuntu Server 24.04.4 LTS (noble), x86_64
  • Current working kernel: 6.8.0-142-generic (6.8.0-142.142)
  • Failing target kernel: 6.8.0-146-generic (6.8.0-146.146)
  • amdgpu-dkms: 1:7.1.3.31500000-2390945.24.04
  • DKMS module version: 7.1.3-2390945.24.04
  • amdgpu-dkms-firmware: 1:31.50.0.0.31500000-2390945.24.04
  • AMD repository: https://repo.radeon.com/amdgpu/31.50/ubuntu noble main
  • ROCm Core packages: 10.0.0-4, target gfx90a
  • AMD SMI: ROCm 10.0.0, amdgpu 7.1.3.31500000
  • GPU: 4 x AMD Instinct MI250 cards / 8 x Aldebaran PCI functions (1002:740c)
  • Server: Supermicro AS-4124GQ-TNMI
  • CPU: 2 x AMD EPYC 7443, 96 logical CPUs
  • Memory: 1 TiB
  • Compiler: GCC 13.3.0
  • GNU Make: 4.3-4.1build2
  • DKMS: 3.0.11-1ubuntu13

The AMD DKMS top-level Makefile expands its kernel build to make -j$(nproc), so this host runs the build with -j96.

The currently running stack is healthy on 6.8.0-142-generic:

amdgpu/7.1.3-2390945.24.04, 6.8.0-142-generic, x86_64: installed

filename: /lib/modules/6.8.0-142-generic/updates/dkms/amdgpu.ko.zst
version:  7.1.3.31500000
vermagic: 6.8.0-142-generic SMP preempt mod_unload modversions

How it was triggered

Ubuntu unattended-upgrades installed the following kernel packages on 2026-10-10:

linux-image-6.8.0-146-generic        6.8.0-146.146
linux-headers-6.8.0-146              6.8.0-146.146
linux-headers-6.8.0-146-generic      6.8.0-146.146
linux-modules-6.8.0-146-generic      6.8.0-146.146
linux-modules-extra-6.8.0-146-generic 6.8.0-146.146

The normal kernel post-install hooks then invoked:

dkms autoinstall -k 6.8.0-146-generic

which invokes the package's configured build:

'make' KERNELVER=6.8.0-146-generic

No custom kernel, DKMS patch, compiler override, or manual source modification was involved.

Failure signature 1: raw_atomic_fetch_add

One preserved DKMS report failed consistently at mmhub_v3_3.o:

CC [M]  /tmp/amd.REDACTED/amd/amdgpu/mmhub_v3_3.o
In file included from ./include/linux/atomic.h:80,
                 ...
./include/linux/atomic/atomic-arch-fallback.h: In function 'raw_atomic_fetch_add':
./include/linux/atomic/atomic-arch-fallback.h:675:2: error: #error "Unable to define raw_atomic_fetch_add"
  675 | #error "Unable to define raw_atomic_fetch_add"
      |  ^~~~~
./include/linux/atomic/atomic-arch-fallback.h:677:1: error: no return statement in function returning non-void [-Werror=return-type]
...
./include/linux/atomic/atomic-arch-fallback.h: In function 'raw_atomic_fetch_add_acquire':
./include/linux/atomic/atomic-arch-fallback.h:702:2: error: #error "Unable to define raw_atomic_fetch_add_acquire"
...
./include/linux/atomic/atomic-arch-fallback.h: In function 'raw_atomic_fetch_add_release':
./include/linux/atomic/atomic-arch-fallback.h:728:2: error: #error "Unable to define raw_atomic_fetch_add_release"
...
./include/linux/atomic/atomic-arch-fallback.h: In function 'raw_atomic_fetch_add_relaxed':
./include/linux/atomic/atomic-arch-fallback.h:751:2: error: #error "Unable to define raw_atomic_fetch_add_relaxed"
...
cc1: some warnings being treated as errors
make[4]: *** [scripts/Makefile.build:243: /tmp/amd.REDACTED/amd/amdgpu/mmhub_v3_3.o] Error 1
make[3]: *** [scripts/Makefile.build:481: /tmp/amd.REDACTED/amd/amdgpu] Error 2
make[2]: *** [/usr/src/linux-headers-6.8.0-146-generic/Makefile:1946: /tmp/amd.REDACTED] Error 2
make[1]: *** [Makefile:240: __sub-make] Error 2
make: *** [Makefile:54: modules] Error 2

The following header is byte-identical between the working and failing kernel header packages:

c5bed3e24d0ca84e7548a2e3943ceab666337a72944ac82377bd99878f5dbdd9  6.8.0-142/include/linux/atomic/atomic-arch-fallback.h
c5bed3e24d0ca84e7548a2e3943ceab666337a72944ac82377bd99878f5dbdd9  6.8.0-146/include/linux/atomic/atomic-arch-fallback.h

Failure signature 2: GNU make heap corruption

On another retry, Ubuntu generated /var/crash/_usr_bin_make.0.crash:

ProblemType: Crash
Date: Mon Oct 12 06:21:54 2026 UTC
Package: make 4.3-4.1build2
ExecutablePath: /usr/bin/make
Signal: 6 (SIGABRT)
Uname: Linux 6.8.0-142-generic x86_64
ProcCmdline: make -f ./scripts/Makefile.build obj=/tmp/amd.REDACTED/amd/amdgpu need-builtin= need-modorder=1

Strings recovered from its core dump contain:

double free or corruption (out)
kernelver=6.8.0-146-generic
module=amdgpu
HOSTCC=gcc-13
KERNELRELEASE=6.8.0-146-generic
module_version=7.1.3-2390945.24.04

This is the actual make process aborting; the generic DKMS message about a segmentation fault is not sufficient to describe this failure.

Failure signature 3: DKMS Apport hook crash

The DKMS Apport hook also crashes while handling some failed attempts:

Traceback (most recent call last):
  File "/usr/share/apport/package-hooks/dkms_packages.py", line 87, in <module>
    if 'Segmentation fault' in report['DKMSBuildLog']:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: a bytes-like object is required, not 'str'

This appears to be a separate Ubuntu reporting bug, but it obscures the underlying AMD DKMS failure.

Hard-lock evidence

The final boot's journal shows apt-daily-upgrade starting at 06:01:06 UTC. The dpkg log reaches the DKMS build and then stops mid-line:

2026-10-13 06:01:54 UTC: unattended-upgrade starts the next dpkg transaction
Setting up linux-headers-6.8.0-146-generic (6.8.0-146.146) ...
 * dkms: running auto installation service for kernel 6.8.0-146-generic
Building module:
Cleaning build area...
'make' KERNELVER=6.8.0-146-generic.........

There are no later entries in that boot after 06:02:03 UTC, no shutdown record, and the next boot begins at 19:08:15 UTC after a power cycle.

Additional checks after recovery:

EDAC mc0 ce_count: 0
EDAC mc0 ue_count: 0
EDAC mc1 ce_count: 0
EDAC mc1 ue_count: 0
pstore: empty
swap in use after recovery: 0

The persistent kernel journal contains no OOM, MCE, uncorrectable EDAC error, panic, watchdog lockup, or amdgpu reset/timeout immediately preceding the freeze.

Resulting package/boot state

linux-headers-6.8.0-146-generic  6.8.0-146.146  iF
linux-image-6.8.0-146-generic    6.8.0-146.146  iF

/boot/vmlinuz-6.8.0-146-generic       exists
/boot/initrd.img-6.8.0-146-generic    missing

The previous kernel remains bootable and has a valid initramfs:

/boot/vmlinuz-6.8.0-142-generic
/boot/initrd.img-6.8.0-142-generic

Non-deterministic history

This package also failed several automatic builds against 6.8.0-142-generic on 2026-09-25 through 2026-09-27, sometimes producing the generic DKMS segmentation-fault classification. A later retry on 2026-09-28 completed successfully without changing the amdgpu-dkms package, and that module is currently running.

Against 6.8.0-146-generic, retries from 2026-10-10 through 2026-10-13 produced different observable failure signatures, culminating in the host hard-lock. This variability is why I suspect a parallel-build race, memory corruption in the build process, or another non-deterministic defect rather than only a deterministic kernel API incompatibility. This is an inference, not a confirmed root cause.

Expected behavior

  • amdgpu-dkms should build reproducibly against a supported Ubuntu kernel; or
  • if the kernel is unsupported/incompatible, the build should fail deterministically and cleanly without aborting make, hard-locking the host, or leaving a new default kernel without an initramfs.

Questions

  1. Is amdgpu-dkms 1:7.1.3.31500000-2390945.24.04 expected to support Ubuntu kernel 6.8.0-146.146?
  2. Are there known races or excessive-parallelism issues in this DKMS source when the top-level build uses make -j96?
  3. Is the raw_atomic_fetch_add error known to occur non-deterministically during parallel builds?
  4. Would AMD like a controlled comparison using capped job counts (-j1, -j8, etc.) on a disposable test host? I will not run that experiment on this production host because the build has already caused a hard lock.
  5. Which additional artifacts would be most useful? I can provide the redacted Apport reports and core metadata, but I have intentionally not uploaded the full core dump because it may contain host data.

Related reports

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions