Misc. bug: [CUDA/ROCm] VRAM leak/fragmentation in ggml_cuda_pool_leg when using Flash Attention #22107

Name and Version

./llama-cli --version
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 32736 MiB):
Device 0: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
Device 1: AMD Radeon Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
version: 8840 (9e5647a)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m '/data/models/gemma-4-31B-it-IQ4_NL.gguf' -ngl 999 -ctk q8_0 -ctv q8_0 -np 1 -b 4096 -ub 256 -fa on -cram 32768 -c 131072

Problem description & steps to reproduce

Environment:

• GPU: AMD Radeon Pro VII (gfx906) / Also applies to any GPU using the Legacy Pool (VMM: no)
• Backend: ROCm / HIP
• Issue: Severe VRAM bloating leading to OOM (HSA exception: MemoryRegion::BlockAllocator::alloc failed)

The Bug:
When using a GPU without VMM support, llama.cpp falls back to ggml_cuda_pool_leg (MAX_BUFFERS = 256).
When Flash Attention is enabled, it constantly requests temporary buffers. Because the context length grows with each generated token, the requested buffer size increases slightly on every step.

Because the new requested size is always slightly larger than the cached buffers, alloc() fails to find a suitable match in buffer_pool. Instead of reusing or freeing, it keeps the old buffers in the cache and allocates a new one via ggml_cuda_device_malloc. This quickly fills up all 256 slots with slightly undersized buffers, effectively hoarding VRAM until an OOM occurs.

The Vulkan backend does not suffer from this issue due to its different memory allocator).

The Workaround / Proposed Fix:
We need to proactively flush the legacy pool cache when we detect this "buffer size climbing" behavior, or at least flush it when cudaMalloc fails due to fragmentation.

I modified the end of the alloc function in ggml_cuda_pool_leg (ggml/src/ggml-cuda/ggml-cuda.cu) to proactively clear the cache if no suitable buffer is found (ibest < 0). This completely stopped the VRAM growth and fixed the OOM crashes on my machine without disabling Flash Attention:

    if (ibest >= 0) {
        ggml_cuda_buffer& b = buffer_pool[ibest];
        void * ptr = b.ptr;
        *actual_size = b.size;
        b.ptr = nullptr;
        b.size = 0;
        return ptr;
    }

    // PROPOSED FIX: Flush the hoard before allocating a new, larger block
    for (int i = 0; i < MAX_BUFFERS; ++i) {
        if (buffer_pool[i].ptr != nullptr) {
            CUDA_CHECK(cudaFree(buffer_pool[i].ptr));
            pool_size -= buffer_pool[i].size;
            buffer_pool[i].ptr = nullptr;
            buffer_pool[i].size = 0;
        }
    }

    void * ptr;
    size_t look_ahead_size = (size_t) (1.05 * size);
    look_ahead_size = 256 * ((look_ahead_size + 255)/256);

I'm opening this issue so the maintainers can decide whether to implement this proactive flush, or perhaps add a defragmentation/flush trigger inside the pool when cudaMalloc returns an OOM error.

First Bad Commit

No response

Relevant log output

HSA exception: MemoryRegion::BlockAllocator::alloc failed.
/home/user/git/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: ROCm error
ROCm error: out of memory
current device: 0, in function alloc at /home/user/git/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:424
ggml_cuda_device_malloc(&ptr, look_ahead_size, device)
/home/user/git/llama.cpp/build/bin/libggml-base.so.0(+0x171a5) [0x7fcd0d5591a5]
/home/user/git/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1eb) [0x7fcd0d55957b]
/home/user/git/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x11f) [0x7fcd0d5596ff]
/home/user/git/llama.cpp/build/bin/libggml-hip.so.0(+0x11c842) [0x7fcd08b1c842]
/home/user/git/llama.cpp/build/bin/libggml-hip.so.0(+0x130777) [0x7fcd08b30777]
/home/user/git/llama.cpp/build/bin/libggml-hip.so.0(_Z12launch_fattnILi512ELi4ELi8EEvR25ggml_backend_cuda_contextP11ggml_tensorPFvPKcS5_S5_S5_S5_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS9_IjLj3EEiiiiiiiiiiiliiliiiiilEimibbbi+0x325) [0x7fcd08d13745]
/home/user/git/llama.cpp/build/bin/libggml-hip.so.0(_Z34ggml_cuda_flash_attn_ext_tile_caseILi512ELi512EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x235) [0x7fcd08d12fb5]
/home/user/git/llama.cpp/build/bin/libggml-hip.so.0(+0x123f88) [0x7fcd08b23f88]
/home/user/git/llama.cpp/build/bin/libggml-hip.so.0(+0x1222f1) [0x7fcd08b222f1]
/home/user/git/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x7f3) [0x7fcd0d575313]
/home/user/git/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa0) [0x7fcd0d2abcf0]
/home/user/git/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xe5) [0x7fcd0d2ae2c5]
/home/user/git/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x35f) [0x7fcd0d2b4a2f]
/home/user/git/llama.cpp/build/bin/libllama.so.0(llama_decode+0xe) [0x7fcd0d2b63ae]
/home/user/git/llama.cpp/build/bin/llama-server(+0x15ba4a) [0x560837b02a4a]
/home/user/git/llama.cpp/build/bin/llama-server(+0x1e2d3f) [0x560837b89d3f]
/home/user/git/llama.cpp/build/bin/llama-server(+0xb6c0a) [0x560837a5dc0a]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29f75) [0x7fcd0cc33f75]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x87) [0x7fcd0cc34027]
/home/user/git/llama.cpp/build/bin/llama-server(+0xbd851) [0x560837a64851]