Skip to content

vulkan full offload of a 27B TQ1_0 model crashes the Adreno 840 driver inside vkGetDeviceFaultInfoEXT #30649

Open

Description

@figkcoci-debug

device Redmi K100 Pro Max, Snapdragon 8 Elite Gen 5 (SM8850), Adreno 840, Android 16
build fingerprint Redmi/songyuan/songyuan:16/BP2A.250605.031.A3/OS3.0.310.0.WGNCNXM
vulkan driver build id 30e0772cdc77186f332f82a477169cb7

llama-bench built for arm64 android with GGML_VULKAN=ON, NDK 27, vulkan sdk 1.4.363 glslc
tree is llama.cpp master (4103857 in my fork, two unrelated rpc commits on top of 23deec0)
model is a 27B ternary gguf in the stock TQ1_0 format, 5.33 GiB, 64 blocks

repro
./llama-bench -m Ternary-Bonsai-2-27B-TQ1_0-folded.gguf -ngl 64 -p 64 -n 8 -r 1
-ngl 63 (one block on the cpu) runs fine, tg8 1.10 t/s
-ngl 64 (all blocks on the gpu) dies during the first test, 16 seconds after start
the same command with -p 16 -n 4 passed once, so the failure depends on prompt batch size or timing
the 6.51 GiB TQ2_0 variant of the same model dies the same way, ngl 1 and ngl 32 are fine for both

crash log from logcat -b crash
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0000000000000000
Cause null pointer dereference
#00 pc 00000000002a44a0 /vendor/lib64/hw/vulkan.adreno.so +192 (BuildId 30e0772cdc77186f332f82a477169cb7)
#1 pc 000000000029e13c /vendor/lib64/hw/vulkan.adreno.so qglinternal::vkGetDeviceFaultInfoEXT(VkDevice_T*, VkDeviceFaultCountsEXT*, VkDeviceFaultInfoEXT*)+108
#2 pc 0000000001fd6f78 /data/local/tmp/vkbench/libggml-vulkan.so
#3 pc 0000000001fd7398 /data/local/tmp/vkbench/libggml-vulkan.so
#4 pc 0000000002094814 /data/local/tmp/vkbench/libggml-vulkan.so
#5 pc 00000000020278f0 /data/local/tmp/vkbench/libggml-vulkan.so
#6 pc 00000000020857cc /data/local/tmp/vkbench/libggml-vulkan.so
16 total frames

#2 is ggml_vk_print_device_fault_info which runs after the device_fault flag is set, so the gpu faults first and the driver crashes while the fault is being reported

device props from the ggml log
Adreno (TM) 840 | uma 1 | fp16 1 | bf16 0 | fp4 0 | warp size 64 | shared memory 32768 | int dot 1 | matrix cores none

expected either full offload works or llama fails cleanly. the null deref inside the driver while it reports the fault itself looks like a Qualcomm bug
0cc4m ggerganov the full debuggerd dump and the model file details are available on request

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions