Repository navigation
Rocm slow model weight loading past ~64GB mark on Strix Halo (all fine with Vulkan) #15018
Description
Name and Version
$ llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
version: 6040 (66625a59)
built with cc (GCC) 15.1.1 20250521 (Red Hat 15.1.1-2) for x86_64-redhat-linux```Operating systems
Linux
GGML backends
HIP (Rocm 6.4.2 / 7-beta)
Hardware
| Component | Specification |
|---|---|
| Test Machine | HP Z2 Mini G1a |
| CPU | Ryzen AI MAX+ 395 "Strix Halo" |
| System Memory | 128 GB RAM |
| GPU Memory | 512 MB allocated in BIOS |
| Host OS | Fedora 42, kernel 6.15.6-200.fc42.x86_86_64 |
Models
Any more, at any quantization.
Problem description & steps to reproduce
Whenever I try to load any model which requires more than 64GB of vRAM, the weight loading into memory starts normally and then gets extremely slow past the 64GB mark... The same model with a Vulkan backend loads and works just fine.
Rocm 6.4.2 and 7-beta have exactly the same issue for me.
I made a video on the same machine comparing the weight loading between rocm and vulkan, hopefully this helps as I am not sure what debug output I could actually provide:
Please let me know what other information I can provide or additional test I can do.
First Bad Commit
No response
Relevant log output
$ llama-cli -ngl 99 -m models/llama-4-scout-17b-16e/Q6_K/Llama-4-Scout-17B-16E-Instruct-Q6_K-00001-of-00002.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
build: 6040 (66625a59) with cc (GCC) 15.1.1 20250521 (Red Hat 15.1.1-2) for x86_64-redhat-linux
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device ROCm0 (Radeon 8060S Graphics) - 124522 MiB free
llama_model_loader: additional 1 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 51 key-value pairs and 628 tensors from models/llama-4-scout-17b-16e/Q6_K/Llama-4-Scout-17B-16E-Instruct-Q6_K-00001-of-00002.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama4
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Llama-4-Scout-17B-16E-Instruct
llama_model_loader: - kv 3: general.finetune str = 16E-Instruct
llama_model_loader: - kv 4: general.basename str = Llama-4-Scout-17B-16E-Instruct
llama_model_loader: - kv 5: general.quantized_by str = Unsloth
llama_model_loader: - kv 6: general.size_label str = 17B
llama_model_loader: - kv 7: general.license str = other
llama_model_loader: - kv 8: general.license.name str = llama4
llama_model_loader: - kv 9: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 10: general.base_model.count u32 = 1
llama_model_loader: - kv 11: general.base_model.0.name str = Llama 4 Scout 17B 16E Instruct
llama_model_loader: - kv 12: general.base_model.0.organization str = Meta Llama
llama_model_loader: - kv 13: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Lla...
llama_model_loader: - kv 14: general.tags arr[str,5] = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv 15: general.languages arr[str,12] = ["ar", "de", "en", "es", "fr", "hi", ...
llama_model_loader: - kv 16: llama4.block_count u32 = 48
llama_model_loader: - kv 17: llama4.context_length u32 = 10485760
llama_model_loader: - kv 18: llama4.embedding_length u32 = 5120
llama_model_loader: - kv 19: llama4.feed_forward_length u32 = 16384
llama_model_loader: - kv 20: llama4.attention.head_count u32 = 40
llama_model_loader: - kv 21: llama4.attention.head_count_kv u32 = 8
llama_model_loader: - kv 22: llama4.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 23: llama4.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 24: llama4.expert_count u32 = 16
llama_model_loader: - kv 25: llama4.expert_used_count u32 = 1
llama_model_loader: - kv 26: llama4.attention.key_length u32 = 128
llama_model_loader: - kv 27: llama4.attention.value_length u32 = 128
llama_model_loader: - kv 28: llama4.vocab_size u32 = 202048
llama_model_loader: - kv 29: llama4.rope.dimension_count u32 = 128
llama_model_loader: - kv 30: llama4.interleave_moe_layer_step u32 = 1
llama_model_loader: - kv 31: llama4.expert_feed_forward_length u32 = 8192
llama_model_loader: - kv 32: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 33: tokenizer.ggml.pre str = llama4
llama_model_loader: - kv 34: tokenizer.ggml.tokens arr[str,202048] = ["À", "Á", "õ", "ö", "÷", "ø", ...
llama_model_loader: - kv 35: tokenizer.ggml.token_type arr[i32,202048] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 36: tokenizer.ggml.merges arr[str,439802] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 37: tokenizer.ggml.bos_token_id u32 = 200000
llama_model_loader: - kv 38: tokenizer.ggml.eos_token_id u32 = 200008
llama_model_loader: - kv 39: tokenizer.ggml.padding_token_id u32 = 200018
llama_model_loader: - kv 40: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 41: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv 42: general.quantization_version u32 = 2
llama_model_loader: - kv 43: general.file_type u32 = 18
llama_model_loader: - kv 44: quantize.imatrix.file str = Llama-4-Scout-17B-16E-Instruct-GGUF/i...
llama_model_loader: - kv 45: quantize.imatrix.dataset str = unsloth_calibration_Llama-4-Scout-17B...
llama_model_loader: - kv 46: quantize.imatrix.entries_count u32 = 528
llama_model_loader: - kv 47: quantize.imatrix.chunks_count u32 = 729
llama_model_loader: - kv 48: split.no u16 = 0
llama_model_loader: - kv 49: split.tensors.count i32 = 628
llama_model_loader: - kv 50: split.count u16 = 2
llama_model_loader: - type f32: 146 tensors
llama_model_loader: - type q6_K: 482 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q6_K
print_info: file size = 82.35 GiB (6.56 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 1135
load: token to piece cache size = 1.3873 MB
print_info: arch = llama4
print_info: vocab_only = 0
print_info: n_ctx_train = 10485760
print_info: n_embd = 5120
print_info: n_layer = 48
print_info: n_head = 40
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 8192
print_info: is_swa_any = 1
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 5
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 16384
print_info: n_expert = 16
print_info: n_expert_used = 1
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 10485760
print_info: rope_finetuned = unknown
print_info: model type = 17Bx16E (Scout)
print_info: model params = 107.77 B
print_info: general.name = Llama-4-Scout-17B-16E-Instruct
print_info: vocab type = BPE
print_info: n_vocab = 202048
print_info: n_merges = 439802
print_info: BOS token = 200000 '<|begin_of_text|>'
print_info: EOS token = 200008 '<|eot|>'
print_info: PAD token = 200018 '<|finetune_right_pad|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 200002 '<|fim_prefix|>'
print_info: FIM SUF token = 200004 '<|fim_suffix|>'
print_info: FIM MID token = 200003 '<|fim_middle|>'
print_info: EOG token = 200001 '<|end_of_text|>'
print_info: EOG token = 200008 '<|eot|>'
print_info: max token length = 192
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors: CPU_Mapped model buffer size = 809.29 MiB
load_tensors: ROCm0 model buffer size = 83513.68 MiB
..............................................
Activity
lhl commentedon Aug 1, 2025
I believe your issue is you need disable mmap. Give
--no-mmapfor llama-cli or--mmap 0for llama-bench a try.kyuz0 commentedon Aug 1, 2025
Thank you @lhl, I confirm this fixes the loading issue and the model now loads fine with rocm. It is still not clear to me why this is the case and whether it's expected behaviour or a bug (i.e.: should I close this ticket or does this need work?).
lhl commentedon Aug 2, 2025
I think this is something that should be left opened/triage although obviously will leave it to the llama.cpp devs to decide.
My understanding is that when hipMemcpy() is called directly from the mapped, pageable address, HIP needs to lock/unlock every page. Now, is this expected behavior that a default flag will cause a hugely negative performance regression and that it's in a flag that a normal end-user has no way to figure out? Seems like it's a legit bug/problem.
It seems that mmap should be disabled by some heuristic (eg, by default if running Vulkan/HIP backend if all layers are loaded to GPU memory?) Or maybe via a 2X available memory rule of some sort (you know it'll thrash in that case)?
You might not want to disable mmpa completely depending on if you load all the layers to GPU or not, and maybe this is only applicable for APU vs dGPUs...
samteezy commentedon Aug 5, 2025
I'm dealing with this issue too, as of in the last few days. Larger models like
oss-gpt-120bandcogito-v2-preview-llama-109B-MoErefuse to finish loading withllama-serverunless--no-mmapis specified. Running a Radeon V620 32GB with ROCm 6.4.2.Odd because only recently, I was running into an issue with models refused to load if
--no-mmapwas specified. #14999github-actions commentedon Sep 19, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale.
Add strix-halo-optimizer skill for AMD Ryzen AI Max+ 395
Add strix-halo-optimizer skill for AMD Ryzen AI Max+ 395
liminfei-amd commentedon Jul 23, 2026
I reproduced this on gfx1151 with the same 82.36 GiB Llama-4 Scout Q6_K model.
With default mmap, the clean build timed out after 600 seconds at 34%;
--no-mmapcompleted in 56 seconds.The proposed fix in #26023 stages eligible full-offload IGPU uploads through
bounded pinned buffers. It completed in about 41 seconds; a separate
deterministic small-model control produced unchanged post-load logits.