Repository navigation
[Issue]: Memory access fault by GPU node-1: amdgpu: [gfxhub] page fault #204
Description
Problem Description
On Linux kernel 6.18.3 & 6.18.4 ComfyUI crashes after a little while when doing stuff, e.g. wan VAE encoding.
No failures occur after rolling back to kernel 6.17.9.
Memory access fault by GPU node-1 (Agent handle: 0x55f8b54eec30) on address 0x7f2a53193000. Reason: Page not present or supervisor privilege.
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Related: https://github.com/ROCm/ROCm/issues/5051#issuecomment-3730224118
Operating System
Arch Linux
CPU
AMD Ryzen 7 5800X
GPU
RX 7900 GRE
ROCm Version
ROCm 7.1, 7.2
Latest version tested with issue: kernel 6.18.6, pytorch 2.11.0a0+rocm7.11.0a20260117
ROCm Component
No response
Steps to Reproduce
Run several wan workflows
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
journal logs
Jan 09 18:55:45 alexpc kernel: gmc_v11_0_process_interrupt: 632 callbacks suppressed
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc systemd-coredump[134090]: Process 91762 (python) of user 60105 terminated abnormally with signal 6/ABRT, processing...
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc systemd[1]: Started Process Core Dump (PID 134090/UID 0).
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00801A31
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Faulty UTCL2 client ID: SDMA0 (0xd)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: MORE_FAULTS: 0x1
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: WALKER_ERROR: 0x0
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: PERMISSION_FAULTS: 0x3
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: MAPPING_ERROR: 0x0
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: RW: 0x0
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318d000 from client 10
Jan 09 18:57:12 alexpc systemd-coredump[134091]: [🡕] Process 91762 (python) of user 60105 dumped core.
Module /dev/zero (deleted) without build-id.
...
Module /dev/zero (deleted) without build-id.
Module libsharpyuv.817afaaa.so.0 without build-id.
Module libz.db8a765a.so.1 without build-id.
Module libwebp.efd97cc6.so.7 without build-id.
Module libjpeg.2394f5ee.so.8 without build-id.
Module libpng16.c2d532b4.so.16 without build-id.
Module librocroller.so without build-id.
Module libmagma.so without build-id.
Stack trace of thread 91764:
#0 0x00007f2ed709890c n/a (libc.so.6 + 0x9890c)
ROCm/legacy-rocm-build#1 0x00007f2ed703e3a0 raise (libc.so.6 + 0x3e3a0)
ROCm/legacy-rocm-build#2 0x00007f2ed702557a abort (libc.so.6 + 0x2557a)
ROCm/legacy-rocm-build#3 0x00007f2db948d33f _ZN4rocr4core7Runtime14VMFaultHandlerElPv (libhsa-runtime64.so + 0x8d33f)
ROCm/legacy-rocm-build#4 0x00007f2db948c0b3 _ZN4rocr4core7Runtime15AsyncEventsLoopEPv (libhsa-runtime64.so + 0x8c0b3)
ROCm/legacy-rocm-build#5 0x00007f2db94332cd _ZN4rocr2os16ThreadTrampolineEPv (libhsa-runtime64.so + 0x332cd)
ROCm/legacy-rocm-build#6 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#7 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91762:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70931b4 n/a (libc.so.6 + 0x931b4)
ROCm/legacy-rocm-build#3 0x00007f2ed711acb5 epoll_wait (libc.so.6 + 0x11acb5)
ROCm/legacy-rocm-build#4 0x00007f2ed6621981 n/a (select.cpython-313-x86_64-linux-gnu.so + 0x2981)
ROCm/legacy-rocm-build#5 0x00007f2ed757483d _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17483d)
ROCm/legacy-rocm-build#6 0x00007f2ed7648b59 PyEval_EvalCode (libpython3.13.so.1.0 + 0x248b59)
ROCm/legacy-rocm-build#7 0x00007f2ed768956c n/a (libpython3.13.so.1.0 + 0x28956c)
ROCm/legacy-rocm-build#8 0x00007f2ed768618d n/a (libpython3.13.so.1.0 + 0x28618d)
ROCm/legacy-rocm-build#9 0x00007f2ed76828f8 n/a (libpython3.13.so.1.0 + 0x2828f8)
ROCm/legacy-rocm-build#10 0x00007f2ed7682522 n/a (libpython3.13.so.1.0 + 0x282522)
ROCm/legacy-rocm-build#11 0x00007f2ed7682323 n/a (libpython3.13.so.1.0 + 0x282323)
ROCm/legacy-rocm-build#12 0x00007f2ed7680991 Py_RunMain (libpython3.13.so.1.0 + 0x280991)
ROCm/legacy-rocm-build#13 0x00007f2ed7635beb Py_BytesMain (libpython3.13.so.1.0 + 0x235beb)
ROCm/legacy-rocm-build#14 0x00007f2ed7027635 n/a (libc.so.6 + 0x27635)
ROCm/legacy-rocm-build#15 0x00007f2ed70276e9 __libc_start_main (libc.so.6 + 0x276e9)
ROCm/legacy-rocm-build#16 0x000055f8a72ba045 _start (/usr/bin/python3.13 + 0x1045)
Stack trace of thread 91779:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
ROCm/legacy-rocm-build#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
ROCm/legacy-rocm-build#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91812:
#0 0x00007f2db94620f6 _ZN4rocr4core14BusyWaitSignal11WaitRelaxedE22hsa_signal_condition_tlm16hsa_wait_state_t (libhsa-runtime64.so + 0x620f6)
ROCm/legacy-rocm-build#1 0x00007f2db9461dea _ZN4rocr4core14BusyWaitSignal11WaitAcquireE22hsa_signal_condition_tlm16hsa_wait_state_t (libhsa-runtime64.so + 0x61dea)
ROCm/legacy-rocm-build#2 0x00007f2db9465931 _ZN4rocr3HSA25hsa_signal_wait_scacquireE12hsa_signal_s22hsa_signal_condition_tlm16hsa_wait_state_t (libhsa-runtime64.so + 0x65931)
ROCm/legacy-rocm-build#3 0x00007f2ed177c61c _ZN9roctracer11hsa_support6detailL34hsa_signal_wait_scacquire_callbackE12hsa_signal_s22hsa_signal_condition_tlm16hsa_wait_state_t (libroctracer64.so + 0x1161c)
ROCm/legacy-rocm-build#4 0x00007f2e367d73ea _ZN3amd3roc13WaitForSignalE12hsa_signal_sbb (libamdhip64.so + 0x3d73ea)
ROCm/legacy-rocm-build#5 0x00007f2e367e177a _ZN3amd3roc10VirtualGPU14HwQueueTracker16CpuWaitForSignalEPNS0_15ProfilingSignalE (libamdhip64.so + 0x3e177a)
ROCm/legacy-rocm-build#6 0x00007f2e367e44de _ZN3amd3roc10VirtualGPU21releaseGpuMemoryFenceEb (libamdhip64.so + 0x3e44de)
ROCm/legacy-rocm-build#7 0x00007f2e367eaa0c _ZN3amd3roc10VirtualGPU12submitMarkerERNS_6MarkerE (libamdhip64.so + 0x3eaa0c)
ROCm/legacy-rocm-build#8 0x00007f2e367b4946 _ZN3amd7Command7enqueueEv (libamdhip64.so + 0x3b4946)
ROCm/legacy-rocm-build#9 0x00007f2e367b7640 _ZN3amd9HostQueue6finishEb (libamdhip64.so + 0x3b7640)
ROCm/legacy-rocm-build#10 0x00007f2e36482cbe _ZN3hip6Device14SyncAllStreamsEbb (libamdhip64.so + 0x82cbe)
ROCm/legacy-rocm-build#11 0x00007f2e3658afbc _ZN3hip8ihipFreeEPv (libamdhip64.so + 0x18afbc)
ROCm/legacy-rocm-build#12 0x00007f2e36592940 _ZN3hip7hipFreeEPv (libamdhip64.so + 0x192940)
ROCm/legacy-rocm-build#13 0x00007f2ed34b2a24 _ZN3c103hip19HIPCachingAllocator6NativeL15uncached_deleteEPv (libc10_hip.so + 0x19a24)
ROCm/legacy-rocm-build#14 0x00007f2ea3bb3e76 _ZN2at6native40raw_miopen_convolution_forward_out_32bitERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_lbbb (libtorch_hip.so + 0x2fb3e76)
ROCm/legacy-rocm-build#15 0x00007f2ea3baeef3 _ZN2at6native30miopen_convolution_forward_outERNS_9TensorArgEPKcRKS1_S6_N3c108ArrayRefIlEES9_S9_lbbb (libtorch_hip.so + 0x2faeef3)
ROCm/legacy-rocm-build#16 0x00007f2ea3baf287 _ZN2at6native18miopen_convolutionERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefIlEESA_SA_lbb (libtorch_hip.so + 0x2faf287)
ROCm/legacy-rocm-build#17 0x00007f2ea391690a _ZN2at12_GLOBAL__N_112_GLOBAL__N_132wrapper_CUDA__miopen_convolutionERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_SB_bb (libtorch_hip.so + 0x2d1690a)
ROCm/legacy-rocm-build#18 0x00007f2ea394077e _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_SE_bbEXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_132wrapper_CUDA__miopen_convolutionES8_S8_>
ROCm/legacy-rocm-build#19 0x00007f2ebe8c85c7 _ZN2at4_ops18miopen_convolution4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_SB_bb (libtorch_cpu.so + 0x2cc85c7)
ROCm/legacy-rocm-build#20 0x00007f2ebdb074cf _ZN2at6native12_convolutionERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefIlEESA_SA_bSA_lbbbb (libtorch_cpu.so + 0x1f074cf)
ROCm/legacy-rocm-build#21 0x00007f2ebeec0d60 _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_bSF_SE_bbbbEXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_147wrapper_CompositeExplicitAutograd>
ROCm/legacy-rocm-build#22 0x00007f2ebe516c64 _ZN2at4_ops12_convolution4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_bSC_SB_bbbb (libtorch_cpu.so + 0x2916c64)
ROCm/legacy-rocm-build#23 0x00007f2ebdb00846 _ZN2at6native11convolutionERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefIlEESA_SA_bSA_l (libtorch_cpu.so + 0x1f00846)
ROCm/legacy-rocm-build#24 0x00007f2ebeec08c5 _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_bSF_SE_EXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_146wrapper_CompositeExplicitAutograd__co>
ROCm/legacy-rocm-build#25 0x00007f2ebe514049 _ZN2at4_ops11convolution4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_bSC_SB_ (libtorch_cpu.so + 0x2914049)
ROCm/legacy-rocm-build#26 0x00007f2ebdb0bb60 _ZN2at6native13conv3d_symintERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefINS8_6SymIntEEESB_SB_SA_ (libtorch_cpu.so + 0x1f0bb60)
ROCm/legacy-rocm-build#27 0x00007f2ebeff659e _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_SE_EXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_141wrapper_CompositeImplicitAutograd__conv3d>
ROCm/legacy-rocm-build#28 0x00007f2ebea63d4d _ZN2at4_ops6conv3d4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_SB_ (libtorch_cpu.so + 0x2e63d4d)
ROCm/legacy-rocm-build#29 0x00007f2ed1e80f9d _ZN5torch8autogradL18THPVariable_conv3dEP7_objectS2_S2_ (libtorch_python.so + 0x680f9d)
ROCm/legacy-rocm-build#30 0x00007f2ed7594c9d n/a (libpython3.13.so.1.0 + 0x194c9d)
ROCm/legacy-rocm-build#31 0x00007f2ed75611db _PyObject_MakeTpCall (libpython3.13.so.1.0 + 0x1611db)
ROCm/legacy-rocm-build#32 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
ROCm/legacy-rocm-build#33 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
ROCm/legacy-rocm-build#34 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#35 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
ROCm/legacy-rocm-build#36 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#37 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
ROCm/legacy-rocm-build#38 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#39 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
ROCm/legacy-rocm-build#40 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#41 0x00007f2ed7646b01 n/a (libpython3.13.so.1.0 + 0x246b01)
ROCm/legacy-rocm-build#42 0x00007f2ed769920f n/a (libpython3.13.so.1.0 + 0x29920f)
ROCm/legacy-rocm-build#43 0x00007f2ed75611db _PyObject_MakeTpCall (libpython3.13.so.1.0 + 0x1611db)
ROCm/legacy-rocm-build#44 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
ROCm/legacy-rocm-build#45 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
ROCm/legacy-rocm-build#46 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
ROCm/legacy-rocm-build#47 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#48 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
ROCm/legacy-rocm-build#49 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
ROCm/legacy-rocm-build#50 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#51 0x00007f2ed7646b79 n/a (libpython3.13.so.1.0 + 0x246b79)
ROCm/legacy-rocm-build#52 0x00007f2ed769920f n/a (libpython3.13.so.1.0 + 0x29920f)
ROCm/legacy-rocm-build#53 0x00007f2ed75611db _PyObject_MakeTpCall (libpython3.13.so.1.0 + 0x1611db)
ROCm/legacy-rocm-build#54 0x00007f2ed757b769 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17b769)
ROCm/legacy-rocm-build#55 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
ROCm/legacy-rocm-build#56 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
ROCm/legacy-rocm-build#57 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#58 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
ROCm/legacy-rocm-build#59 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
ROCm/legacy-rocm-build#60 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
ROCm/legacy-rocm-build#61 0x00007f2ed765d80d n/a (libpython3.13.so.1.0 + 0x25d80d)
ROCm/legacy-rocm-build#62 0x00007f2ed65c77d5 n/a (_asyncio.cpython-313-x86_64-linux-gnu.so + 0x57d5)
ROCm/legacy-rocm-build#63 0x00007f2ed65c8eb5 n/a (_asyncio.cpython-313-x86_64-linux-gnu.so + 0x6eb5)
Stack trace of thread 91962:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
ROCm/legacy-rocm-build#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
ROCm/legacy-rocm-build#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91964:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
ROCm/legacy-rocm-build#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
ROCm/legacy-rocm-build#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91965:
#0 0x00007f2ed321de6d gomp_barrier_wait_end (libgomp.so + 0x1de6d)
ROCm/legacy-rocm-build#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
ROCm/legacy-rocm-build#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91778:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
ROCm/legacy-rocm-build#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
ROCm/legacy-rocm-build#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91968:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
ROCm/legacy-rocm-build#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
ROCm/legacy-rocm-build#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91963:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
ROCm/legacy-rocm-build#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
ROCm/legacy-rocm-build#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91780:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
ROCm/legacy-rocm-build#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
ROCm/legacy-rocm-build#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91967:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
ROCm/legacy-rocm-build#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
ROCm/legacy-rocm-build#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91777:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
ROCm/legacy-rocm-build#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
ROCm/legacy-rocm-build#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91977:
#0 0x00007f2ed711670d ioctl (libc.so.6 + 0x11670d)
ROCm/legacy-rocm-build#1 0x00007f2db9528730 hsakmt_ioctl (libhsa-runtime64.so + 0x128730)
ROCm/legacy-rocm-build#2 0x00007f2db951f4e3 hsaKmtWaitOnMultipleEvents_Ext (libhsa-runtime64.so + 0x11f4e3)
ROCm/legacy-rocm-build#3 0x00007f2db948bc73 _ZN4rocr4core7Runtime15AsyncEventsLoopEPv (libhsa-runtime64.so + 0x8bc73)
ROCm/legacy-rocm-build#4 0x00007f2db94332cd _ZN4rocr2os16ThreadTrampolineEPv (libhsa-runtime64.so + 0x332cd)
ROCm/legacy-rocm-build#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91994:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed709e1f6 n/a (libc.so.6 + 0x9e1f6)
ROCm/legacy-rocm-build#4 0x00007f2ed76cb115 _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb115)
ROCm/legacy-rocm-build#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
ROCm/legacy-rocm-build#6 0x00007f2ed76a8ec7 n/a (libpython3.13.so.1.0 + 0x2a8ec7)
ROCm/legacy-rocm-build#7 0x00007f2ed76a8d77 n/a (libpython3.13.so.1.0 + 0x2a8d77)
ROCm/legacy-rocm-build#8 0x00007f2ed759a3f2 n/a (libpython3.13.so.1.0 + 0x19a3f2)
ROCm/legacy-rocm-build#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
ROCm/legacy-rocm-build#10 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
ROCm/legacy-rocm-build#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
ROCm/legacy-rocm-build#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
ROCm/legacy-rocm-build#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
ROCm/legacy-rocm-build#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91996:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed709ef08 n/a (libc.so.6 + 0x9ef08)
ROCm/legacy-rocm-build#4 0x00007f2ed76cb0ad _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb0ad)
ROCm/legacy-rocm-build#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
ROCm/legacy-rocm-build#6 0x00007f2ed38084b8 n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x14b8)
ROCm/legacy-rocm-build#7 0x00007f2ed380861a n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x161a)
ROCm/legacy-rocm-build#8 0x00007f2ed75e6573 n/a (libpython3.13.so.1.0 + 0x1e6573)
ROCm/legacy-rocm-build#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
ROCm/legacy-rocm-build#10 0x00007f2ed757b769 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17b769)
ROCm/legacy-rocm-build#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
ROCm/legacy-rocm-build#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
ROCm/legacy-rocm-build#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
ROCm/legacy-rocm-build#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 92160:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed709e1f6 n/a (libc.so.6 + 0x9e1f6)
ROCm/legacy-rocm-build#4 0x00007f2ed76cb115 _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb115)
ROCm/legacy-rocm-build#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
ROCm/legacy-rocm-build#6 0x00007f2ed76a8ec7 n/a (libpython3.13.so.1.0 + 0x2a8ec7)
ROCm/legacy-rocm-build#7 0x00007f2ed76a8d77 n/a (libpython3.13.so.1.0 + 0x2a8d77)
ROCm/legacy-rocm-build#8 0x00007f2ed759a3f2 n/a (libpython3.13.so.1.0 + 0x19a3f2)
ROCm/legacy-rocm-build#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
ROCm/legacy-rocm-build#10 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
ROCm/legacy-rocm-build#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
ROCm/legacy-rocm-build#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
ROCm/legacy-rocm-build#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
ROCm/legacy-rocm-build#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 130827:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
ROCm/legacy-rocm-build#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
ROCm/legacy-rocm-build#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
ROCm/legacy-rocm-build#3 0x00007f2ed709ef08 n/a (libc.so.6 + 0x9ef08)
ROCm/legacy-rocm-build#4 0x00007f2ed76cb0ad _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb0ad)
ROCm/legacy-rocm-build#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
ROCm/legacy-rocm-build#6 0x00007f2ed38084b8 n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x14b8)
ROCm/legacy-rocm-build#7 0x00007f2ed380861a n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x161a)
ROCm/legacy-rocm-build#8 0x00007f2ed75e6573 n/a (libpython3.13.so.1.0 + 0x1e6573)
ROCm/legacy-rocm-build#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
ROCm/legacy-rocm-build#10 0x00007f2ed757b769 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17b769)
ROCm/legacy-rocm-build#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
ROCm/legacy-rocm-build#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
ROCm/legacy-rocm-build#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
ROCm/legacy-rocm-build#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
ROCm/legacy-rocm-build#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
ELF object binary architecture: AMD x86-64
Reacted by Cuzkey
Activity
alexheretic commentedon Jan 16, 2026
Reproduced again with linux
6.18.5.arch1-1, linux-firmware-amdgpu20260110-1, again by running wan 2.2 workflows using ComfyUI with torch2.11.0.dev20260110+rocm7.1.And to repeat myself this isn't happening on kernel
6.17.9.Similar journal to above:
alexheretic commentedon Jan 17, 2026
Also tested latest pytorch rocm nightly https://github.com/ROCm/ROCm/issues/5051#issuecomment-3764110671 and same issue.
alexheretic commentedon Jan 26, 2026
Strangely the issue goes away on latest kernel is using ComfyUI arg
--disable-pinned-memory. Comfy-Org/ComfyUI#11781 (comment)alanina commentedon Jan 27, 2026
I see you have ROCm 7.1 installed when this issue reproduces, did you also try with the latest ROCm 7.2?
alexheretic commentedon Jan 27, 2026
My last repro was with kernel
6.18.6-arch1-1& pytorch2.11.0a0+rocm7.11.0a20260117which does identify itself as 7.2.I could reproduce fairly quickly without
--disable-pinned-memorybut can't reproduce now I'm using--disable-pinned-memory.feffy380 commentedon Jan 27, 2026
@alanina 7.2 did not fix the issue for me on a 7900xtx (100% reproducible).
--disable-pinned-memoryworks for me. Seems like the fix that was supposed to be shipped in 7.2 is specific to rdna4alanina commentedon Jan 27, 2026
Interesting as 7900 is obviously an older generation RDNA3. I wonder if that's a limitation.
I'm sure someone at AMD will check out this bug report and provide guidance.
kentrussell commentedon Mar 18, 2026
Just to confirm, is it happening on 6.18 with multiple versions of ROCm (amdgpu-dkms), but not on the 6.17 base kernel? I just want to make sure that it's the base kernel that's changing, not the ROCm release. And also that you're using amdgpu-dkms and not the amdgpu that's coming from the 6.17/6.18 kernels.
MaoxinYee commentedon Apr 10, 2026
same issue on 6.17.0-20
net/sched: act_mirred: fix loop detection
mmc: sdhci-of-dwcmshc: Prevent illegal clock reduction in HS200/HS400…
bluemoehre commentedon May 23, 2026
Same here with 6.17.0-29 on Framework Desktop - AMD Strix Halo, 128GB.
LLM starts outputting jibberish like e.g. duplicate words and cut-off sentences.
Issue kicks in after using larger context many times.
Either I end up facing an OOM or GPU hangs at 100% load but doing nothing - temps still like idle.
Similarlog entries:
Doenst matter if I am using ROCm 7.2.1 or 7.3.2.