Misc. bug: CUDA memory leak with MTMD requests loop with Gemma3

Hi.

Thanks a lot for sharing this so great ggml work 💌.

Name and Version

./build/bin/llama-cli --version

ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
version: 7971 (5fa1c19)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux NS5x-NS7xAU.localdomain 6.17.0-14-generic #14~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 15 15:52:10 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Which llama.cpp modules do you know to be affected?

llama-server

$ git log -1
commit 5fa1c190d9fc86c02698b730a2cb933195e19d96 (HEAD -> master, origin/master, origin/HEAD)
Author: Adrien Gallouët <angt@huggingface.co>
Date:   Sun Feb 8 09:06:45 2026 +0100
    rpc : update from common.cpp (#19400)    
    Signed-off-by: Adrien Gallouët <angt@huggingface.co>

$ export CUDA_VERSION=12.9 && cmake -B build -DGGML_CUDA=ON \
 -DCMAKE_CUDA_ARCHITECTURES="86;120" \
 -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-${CUDA_VERSION}/bin/nvcc \
 -DCMAKE_INSTALL_RPATH="/usr/local/cuda-${CUDA_VERSION}/lib64;\$ORIGIN"

$ cmake --build build --config Release -j 10

Command line

./llama.cpp/build/bin/llama-server --port 8012 \
 -m ~/Data/AI_ModelsVision/gemma-3-4b-it-Q4_K_M.gguf \
 --mmproj ~/Data/AI_ModelsVision/mmproj-model-f16.gguf \
 -c 0 --parallel 1

--parallel 1 does not change the count of requests before fail.

Problem description & steps to reproduce

The use case is object detection (MTMD) in small jpeg images (1.7Ko to 19ko) and llama-server fails with CUDA out of memory after ~700 requests.

I do iterations on many different images (thousands) so it should not be image that throws the fail but is it all the time around 700 / 780 requests.

Perhaps it should not be a RTX/CUDA/Driver error because I can relaunch the script and it works fine for another ~700 requests and no message in the system journal (unlike for rtx/cuda/driver errors).

My bug report is poor of precision, but ask me for operations, I'll try to apply them for report completion.

First Bad Commit

No response

Relevant log output

Logs

ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
build: 7971 (5fa1c190d) with GNU 13.3.0 for Linux x86_64
system info: n_threads = 4, n_threads_batch = 4, total_threads = 16

system_info: n_threads = 4 (n_threads_batch = 4) / 16 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

Running without SSL
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '~/Data/AI_ModelsVision/gemma-3-4b-it-Q4_K_M.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on

...
... running well for 700 request ...
...

./llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
CUDA error: out of memory
  current device: 0, in function ggml_cuda_graph_evaluate_and_capture at ./llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:3910
  cudaGraphInstantiate(&graph->instance, graph->graph, __null, __null, 0)
[New LWP 7622]
[New LWP 7621]
[New LWP 7620]
[New LWP 7619]
[New LWP 7618]
[New LWP 7617]
[New LWP 7616]
[New LWP 7615]
[New LWP 7614]
[New LWP 7613]
[New LWP 7612]
[New LWP 7611]
[New LWP 7610]
[New LWP 7609]
[New LWP 7608]
[New LWP 7607]
[New LWP 7606]
[New LWP 7605]
[New LWP 7604]
[New LWP 7603]
[New LWP 7598]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000071e034310813 in __GI___wait4 (pid=9109, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: Aucun fichier ou dossier de ce nom
#0  0x000071e034310813 in __GI___wait4 (pid=9109, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x000071e03496e703 in ggml_print_backtrace () from ./llama.cpp/build/bin/libggml-base.so.0
#2  0x000071e03496e8ab in ggml_abort () from ./llama.cpp/build/bin/libggml-base.so.0
#3  0x000071e028963607 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from ./llama.cpp/build/bin/libggml-cuda.so.0
#4  0x000071e02897bbbb in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from ./llama.cpp/build/bin/libggml-cuda.so.0
#5  0x000071e02897ca7e in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from ./llama.cpp/build/bin/libggml-cuda.so.0
#6  0x000071e03498ae37 in ggml_backend_sched_graph_compute_async () from ./llama.cpp/build/bin/libggml-base.so.0
#7  0x000071e034abde01 in llama_context::graph_compute(ggml_cgraph*, bool) () from ./llama.cpp/build/bin/libllama.so.0
#8  0x000071e034abf8c4 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from ./llama.cpp/build/bin/libllama.so.0
#9  0x000071e034ac6daa in llama_context::decode(llama_batch const&) () from ./llama.cpp/build/bin/libllama.so.0
#10 0x000071e034ac881f in llama_decode () from ./llama.cpp/build/bin/libllama.so.0
#11 0x00005db023ec4a48 in server_context_impl::update_slots() ()
#12 0x00005db023f0c7ae in server_queue::start_loop(long) ()
#13 0x00005db023e25770 in main ()
[Inferior 1 (process 7597) detached]
Abandon (core dumped)

I have a similar issue on Windows 11 using precompiled bins (llama-b8069-bin-win-cuda-13.1-x64), building from source seems to resolve. Note that git pull is ahead b8069, but I don't see resolutions for this in recent commits.
BTW, it seems that cache access (prompt or kv) triggers a new CUDA alloc of variable size depending on model, from 30 to 100MB.

git log -1
commit d612901 (HEAD -> master, origin/master, origin/HEAD)
Author: AesSedai 7980540+AesSedai@users.noreply.github.com
Date: Mon Feb 16 08:44:44 2026 -0800

perplexity: add proper batching (#19661)

Build with:

-G "Visual Studio 17 2022" ^
-DLLAMA_CURL=OFF ^
-DLLAMA_SERVER_SSL=OFF ^
-DGGML_NATIVE=ON ^
-DGGML_RPC=ON ^
-DLLAMA_BUILD_SERVER=ON ^
-DGGML_BACKEND_DL=OFF ^
-DGGML_AVX=ON ^
-DGGML_AVX2=ON ^
-DGGML_AVX512=ON ^
-DGGML_AVX512_VBMI=ON ^
-DGGML_AVX512_VNNI=ON ^
-DGGML_AVX512_BF16=OFF ^
-DGGML_MATMUL_INT8=ON ^
-DGGML_CUDA=ON ^
-DGGML_CUDA_GRAPH=ON ^
-DGGML_CUDA_USE_CUBLASLT=ON ^
-DGGML_CUDA_FORCE_CUBLAS=ON ^
-DGGML_LTO=ON ^
-DGGML_OPENMP=ON ^
-DGGML_FLASH_ATTN=ON ^
-DGGML_CUDA_FA_ALL_QUANTS=true

I have the same issue as OP. I'm using Gemma3-4b-it model (Unsloth UD Q5_K_XL quantization) with mmproj-BF16.gguf and I'm analyzing images one at a time via API with a simple prompt. VRAM usage starts at ~4.7GB but slowly grows as llama-server processes API requests and eventually hits CUDA OOM at 6GB.
I have a RTX 3060 Laptop GPU. Using llama.cpp version 8057 on Linux Mint.

I think it has to do with processing images because I successfully processed 30k+ text prompts with the same Gemma model without loading mmproj.

I tried to narrow this down and come up with a minimal example.

First, I recompiled latest llama.cpp with CUDA.

$ build/bin/llama-server --version
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes
version: 8100 (c0d043034)
built with GNU 13.3.0 for Linux x86_64

Then I started up llama-server with Unsloth Gemma3-4b-it and mmproj directly from HuggingFace using -hf option. With -c 50000 initial VRAM usage according to nvidia-smi is 5479MiB / 6144MiB .

build/bin/llama-server -hf unsloth/gemma-3-4b-it-GGUF -c 50000

Then I ran this test script that generates simple images with random text and sends them to the API endpoint with a prompt to OCR the text.

Test script

Requires pillow and openai libraries.

#!/usr/bin/env python

import base64
import io
import sys
from PIL import Image, ImageDraw
from openai import OpenAI
import random

# --------------------------------------------------------------------------- #
# Helper: Generate a random string
# --------------------------------------------------------------------------- #
def generate_random_string(length=10):
    """Generates a random string of alphanumeric characters."""
    characters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
    return ''.join(random.choice(characters) for _ in range(length))

# --------------------------------------------------------------------------- #
# Core: Generate and save the image
# --------------------------------------------------------------------------- #
def generate_image(text=None):
    """Generates a white background image with black text."""
    img = Image.new("RGB", (768, 512), "white")
    d = ImageDraw.Draw(img)

    # Add the text
    if text:
        x = 768 / 2
        y = 512 / 2
        d.text((x, y), text, fill="black", anchor="ms")

    return img

# --------------------------------------------------------------------------- #
# Core: Ask the VLM to classify the image
# --------------------------------------------------------------------------- #
def get_image_description(img):
    """Sends the image to the local vision model and returns the description."""
    client = OpenAI(
        base_url="http://localhost:8080/v1",
        api_key="foobar",
    )

    # Encode image to base64 in memory
    buffered = io.BytesIO()
    img.save(buffered, format="PNG")
    base64_image = base64.b64encode(buffered.getvalue()).decode("utf-8")

    # Prompt that includes the full descriptions of the palettes
    prompt = (
        "Read the text in this image. Return only the text."
    )

    response = client.chat.completions.create(
        model="gemma-3-4b-it",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{base64_image}",
                        },
                    },
                    {"type": "text", "text": prompt},
                ],
            }
        ],
    )

    # The model should return just the description
    return response.choices[0].message.content.strip()

# --------------------------------------------------------------------------- #
# CLI entry point
# --------------------------------------------------------------------------- #
def main():
    rounds = 1
    
    if len(sys.argv) > 1:
        rounds = int(sys.argv[1])

    for round in range(rounds):
        try:
            text = generate_random_string()

            # Generate the image in memory
            img = generate_image(text)

            # Get the image description
            description = get_image_description(img)

            print(f"Round {round+1}: {description}")

        except Exception as e:
            print(f"Error processing image: {e}", file=sys.stderr)

if __name__ == "__main__":
    main()

Here is the output when tried to run it for 100 rounds:

$ ./desc-dummy-img.py 100
Round 1: hbagZqj3TG
Round 2: xrx9vNuqKF
Round 3: wCtIdqbxPrV
Round 4: tuekEgH8VW
Round 5: DaWJdgs3aR
Round 6: IqWWxqqHtfY
Round 7: Bp43PANIBT
Round 8: gN2wgejs8u
Round 9: b3lvbtbrW8
Round 10: slIMc36tr
Round 11: KkFUb3Xpbg
Round 12: OuHJinWk28
Round 13: gyrmZ9 TFN8
Round 14: D36Z S7M94h
Round 15: d88c2JbDYG
Round 16: IYO Sg40kJY
Error processing image: Connection error.
Error processing image: Connection error.
Error processing image: Connection error.
[...]

So llama.cpp is able to process 16 requests before crashing due to CUDA OOM when VRAM usage hits 6GB, which is what I have on my RTX 3060 Laptop GPU.

Final output from llama.cpp

slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = 497599004465
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 155 | processing task, is_child = 0
slot update_slots: id  3 | task 155 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 280
slot update_slots: id  3 | task 155 | n_tokens = 5, memory_seq_rm [5, end)
srv  process_chun: processing image...
encoding image slice...
image slice encoded in 944 ms
decoding image batch 1/1, n_tokens_batch = 256
sched_reserve: reserving ...
sched_reserve:      CUDA0 compute buffer size =   517.12 MiB
sched_reserve:  CUDA_Host compute buffer size =   117.02 MiB
sched_reserve: graph nodes  = 1369
sched_reserve: graph splits = 2
sched_reserve: reserve took 93.95 ms, sched copies = 1
/home/ozone/git/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
CUDA error: out of memory
  current device: 0, in function ggml_cuda_graph_evaluate_and_capture at /home/ozone/git/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:3891
  cudaGraphInstantiate(&graph->instance, graph->graph, __null, __null, 0)
[New LWP 1610011]
[New LWP 1610010]
[New LWP 1610009]
[New LWP 1610008]
[New LWP 1610007]
[New LWP 1610006]
[New LWP 1610005]
[New LWP 1610004]
[New LWP 1610003]
[New LWP 1610002]
[New LWP 1610001]
[New LWP 1610000]
[New LWP 1609999]
[New LWP 1609998]
[New LWP 1609997]
[New LWP 1609996]
[New LWP 1609995]
[New LWP 1609994]
[New LWP 1609985]
[New LWP 1609980]
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007c8b5ed10813 in __GI___wait4 (pid=1610805, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: Tiedostoa tai hakemistoa ei ole
#0  0x00007c8b5ed10813 in __GI___wait4 (pid=1610805, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x00007c8b5f3b1703 in ggml_print_backtrace () from /home/ozone/git/llama.cpp/build/bin/libggml-base.so.0
#2  0x00007c8b5f3b18ab in ggml_abort () from /home/ozone/git/llama.cpp/build/bin/libggml-base.so.0
#3  0x00007c8b57165ad7 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/ozone/git/llama.cpp/build/bin/libggml-cuda.so.0
#4  0x00007c8b5717b7dd in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /home/ozone/git/llama.cpp/build/bin/libggml-cuda.so.0
#5  0x00007c8b5717c0ee in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/ozone/git/llama.cpp/build/bin/libggml-cuda.so.0
#6  0x00007c8b5f3cde47 in ggml_backend_sched_graph_compute_async () from /home/ozone/git/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007c8b5f502741 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#8  0x00007c8b5f504244 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#9  0x00007c8b5f50b816 in llama_context::decode(llama_batch const&) () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#10 0x00007c8b5f50d2af in llama_decode () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#11 0x00007c8b5f7b6cf8 in mtmd_helper_decode_image_chunk () from /home/ozone/git/llama.cpp/build/bin/libmtmd.so.0
#12 0x00007c8b5f7b8280 in mtmd_helper_eval_chunk_single () from /home/ozone/git/llama.cpp/build/bin/libmtmd.so.0
#13 0x00005e2feeed746b in server_tokens::process_chunk(llama_context*, mtmd_context*, unsigned long, int, int, unsigned long&) const ()
#14 0x00005e2feef3269c in server_context_impl::update_slots() ()
#15 0x00005e2feef79bde in server_queue::start_loop(long) ()
#16 0x00005e2feee92bf0 in main ()
[Inferior 1 (process 1609979) detached]

If I start with a smaller context (-c 30000) then initial VRAM usage is 5089MiB / 6144MiB and I can do 41 requests before hitting OOM. Calculating from this, I think that llama.cpp leaks around 25MB VRAM on every request.

What I have tried as well:

Testing with the same image/prompt over and over. llama.cpp will just return a cached response. There is no memory leak. Therefore the test script has to generate a different image for every request.
Text-only prompts without an image will not cause VRAM usage to grow.
Loading the model without mmproj will cause text+image requests to fail with 500 errors.

What I have not tried:

Other VLMs than Gemma3-4b-it

Nice report, thanks 👌

@Cyrille37

Some more observations:

I tried SmolVLM-2B with the command build/bin/llama-server -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF. VRAM usage is stable at 3445 MiB even after 100 requests with the test script. Result quality is pretty bad: some of the responses are long strings of numbers or gibberish. But that's a model quality issue, not a VRAM issue.
I also tried Qwen3-VL-4B-Instruct with a 10000 token context using the command build/bin/llama-server -hf unsloth/Qwen3-VL-4B-Instruct-GGUF -c 10000. VRAM usage is stable at 5425MiB after 100 requests. It's also amazingly fast - around six times faster than the similarly sized Gemma3 4B!
Finally, I tested Gemma3 again with a smaller test image (256x128 instead of 768x512). The result did not change; with a 30000 context I can still make 41 requests before OOM crashing, so the VRAM leak is apparently not affected by image size.

Conclusion: This VRAM leaking does not apply to all VLMs equally; it might be specific to Gemma.

@Cyrille37 Perhaps you could update the issue title to reflect these findings? E.g. "CUDA memory leak on Gemma3 processing images" or something along those lines?

Does #19754 or alternatively disabling cuda graphs via GGML_CUDA_DISABLE_GRAPHS fix this?

@ORippler

@ORippler Thank you for the suggestions.

I tried again with the test case in #19639 (comment) (anyone can try this BTW). I'm using llama.cpp version: 8179 (ecbcb7e) which already contains #19754.

First I confirmed that the bug is still present (although now I can generate 40 images before crashing, not just 16)
Setting GGML_CUDA_DISABLE_GRAPHS=1 prevents the issue: VRAM usage is stable at 5531MiB according to nvidia-smi

@ORippler

@ORippler Thanks!

Indeed GGML_CUDA_DISABLE_GRAPHS=1 prevents the issue for me as well.

llama.cpp: Docker ghcr.io/ggml-org/llama.cpp:server-cuda12-b8667 (today's latest)
Model: unsloth/gemma-4-E4B-it-GGUF UD-Q8_K_XL + mmproj-BF16
GPU: RTX 5060 Ti
Task: image analysis

Gemma 4 VLM VRAM/Host Memory Leak — Full Investigation Report

Summary

Repeated image inference with Gemma 4 31B + mmproj on llama-server causes two separate memory leaks:

Host memory (RSS): ~46 MiB/request — root-caused to cuBLAS internal workspace caching, fixed by periodic handle reset
VRAM (device memory): ~6.4 MiB/request — unresolved, persists across all tracked CUDA allocation APIs

Both leaks are Gemma 4 + image specific. Qwen3.5-27B with identical VLM setup shows zero leak on both metrics.

Environment

GPU: NVIDIA GeForce RTX 5090 (32 GB VRAM)
OS: Windows 11 + WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2)
CUDA: 12.8
Driver: 595.71
llama.cpp: turboquant fork (TheTom/llama-cpp-turboquant, based on ggml-org HEAD ~2026-04-07)
Also tested: standard ggml-org/llama.cpp (same VRAM leak confirmed)
Model: unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL (17 GB)
mmproj: extracted from Qwen/Qwen3.5-27B and google/gemma-4-31b-it using convert_hf_to_gguf.py --mmproj

Reproduction

llama-server \
  -m gemma-4-31B-it-UD-Q4_K_XL.gguf \
  --mmproj gemma-4-31b-it-mmproj.gguf \
  -c 4096 -ngl 999 --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --ctx-checkpoints 0 --cache-ram 0

# Send 30 identical image requests via /v1/chat/completions with max_tokens=16
# Monitor: nvidia-smi memory.used + /proc/<pid>/status VmRSS

Controlled Experiments

Leak is Gemma 4 + Image Specific

Test	50 requests	VRAM	RSS	Stabilizes?
Gemma 4 + mmproj + images	✅	+322 MiB (6.4/req)	+925 MiB (46/req)	❌ Never
Gemma 4 text-only (no mmproj)	✅	+19 MiB	0	✅ Yes
Qwen3.5-27B + mmproj + images	✅	+56 MiB then stable	0	✅ Yes
Gemma 4 + mmproj + max_tokens=1	✅	+82 MiB then stable	—	✅ Yes (5 req)

Key observations:

Leak rate is proportional to decode token count (more generated tokens = more leak)
Same image repeated → same leak rate (not image-size dependent)
Text-only Gemma 4 → no leak (SWA architecture alone is fine)
Qwen3.5 with identical VLM pipeline → no leak

Leak is NOT in vision encoder

Instrumented clip_image_batch_encode() with ggml_backend_dev_memory() probes at each stage:

Request #3 START: 22232 MiB
  After warmup check:  22232 MiB (delta +0)
  After sched_reset:   22232 MiB (delta +0)
  After build_graph:   22232 MiB (delta +0)
  After alloc_graph:   22232 MiB (delta +0)
  After graph_compute: 22232 MiB (delta +0)
Request #3 END:   22232 MiB (delta +0)

Vision encoder is clean from request 3 onwards. Leak occurs during llama_decode() when generating text with image tokens in KV cache.

Graph allocator buffers are constant

Instrumented process_ubatch() → ggml_backend_sched_alloc_graph():

alloc #1:  backend 0 buf_size=522.50 MiB, graph nodes=2582
alloc #2:  backend 0 buf_size=522.50 MiB, graph nodes=2582
...
alloc #50: backend 0 buf_size=522.50 MiB, graph nodes=2582

Buffer size never changes. ggml_gallocr_needs_realloc() returns false after initial warmup.

ALL tracked CUDA allocations are balanced

LD_PRELOAD interceptor tracking cudaMalloc/cudaFree/cuMemCreate/cuMemMap/cuMemUnmap:

cudaMalloc:  count=67, live at exit = 0.00 MiB  (perfectly balanced)
cudaFree:    count=61, difference = 6 permanent buffers (model weights, KV cache)
cuMemCreate: count=9, total=64.00 MiB
cuMemMap:    count=9
cuMemUnmap:  count=2 (at exit only)
VMM net mapped at exit: 0.00 MiB

No cudaMalloc or cuMemCreate leak. VRAM growth is from allocations NOT visible to these APIs.

Root Cause Analysis

Host Memory Leak (FIXED)

Cause: cuBLAS internal host-side workspace caching. cuBLAS allocates host memory for each unique matmul configuration. Gemma 4's hybrid attention (SWA head_dim=256 + full head_dim=512) combined with image prefill batches (300+ tokens) vs decode batches (1 token) creates many unique configurations. The workspace is only freed on cublasDestroy().

Fix: Periodic cublasDestroy() + lazy recreate every 100 graph computes:

// in ggml_backend_cuda_graph_compute(), after graph evaluation:
static thread_local int compute_count = 0;
if (++compute_count % 100 == 0) {
    cuda_ctx->cublas_reset(); // cublasDestroy + nullptr (lazy recreate)
}

Result: RSS delta went from +925 MiB → 0 MiB over 20 requests.

Note: Resetting every compute (instead of every 100) causes worse RSS leak from create/destroy overhead.

VRAM Leak (UNRESOLVED)

After fixing the host memory leak, VRAM still grows at ~6.4 MiB/request.

What we ruled out:

cudaMalloc/cudaFree balance: perfect
cuMemCreate/cuMemMap: balanced, stops growing after 9 events
Graph allocator buffers: constant (522.50 MiB every alloc)
cuBLAS device workspace (cublasSetWorkspace): no effect
CUDA memory pool (cudaMemPoolAttrReleaseThreshold=0): no effect
CUDA_LAUNCH_BLOCKING=1: made it worse (16.3/req)
turbo3 KV cache: same leak without it (f16 KV = 7.5/req)
Context checkpoints (--ctx-checkpoints 0): no effect
Prompt cache (--cache-ram 0): no effect
Legacy CUDA pool (ggml_cuda_pool_leg vs VMM): same leak
VMM pool shrink (cuMemUnmap in free): crashes

What remains: VRAM growth is not from any tracked allocation API (cudaMalloc, cuMemCreate, cublasSetWorkspace, cudaMemPool). It appears to be CUDA runtime/driver internal state (kernel launch metadata, context bookkeeping) that accumulates when processing varying compute patterns specific to Gemma 4 + image tokens.

Cross-Framework Validation

Framework	Gemma 4 Vision Memory Issue	Status
llama.cpp	VRAM grows ~6.4 MiB/req	This report
vLLM	#28230 Qwen3-VL VRAM grows	OPEN
SGLang	SWAKVPool double-free with dual KV cache	Fixed in fork

The VRAM leak appears to be a systemic issue across frameworks when handling Gemma 4's hybrid SWA architecture with vision inputs.

Suggested Investigation Directions

CUDA runtime internal allocation tracking: cudaMemGetInfo reports decreasing free memory, but no tracked allocation API accounts for it. NVIDIA's compute-sanitizer or internal tooling may reveal which CUDA runtime subsystem is accumulating device memory.
Gemma 4 Shared KV Cache interaction: Gemma 4 has num_kv_shared_layers=20 where later layers reuse K/V from earlier layers. This shared reference pattern during sequence processing (especially with varying-length image token sequences) may cause the KV cache management to retain more state than expected.
Flash attention kernel workspace: The leak occurs during llama_decode() which uses ggml_flash_attn_ext (not cuBLAS) for attention. The flash attention kernel may allocate per-launch device memory that isn't properly reclaimed between varying batch sizes.
Profile with nsys/ncu: Per-kernel device memory delta analysis would pinpoint exactly which kernel launch causes the VRAM increase.

Files & Tools

LD_PRELOAD CUDA tracker: intercepts cudaMalloc/cudaFree/cuMemCreate/cuMemMap/cuMemUnmap
LD_PRELOAD malloc tracker: intercepts malloc/free/realloc with caller address logging
RSS tracking test script: measures both nvidia-smi VRAM and /proc/<pid>/status VmRSS per request
cuBLAS reset patch: periodic cublasDestroy()/recreate (fixes host memory leak)

All available on request.

Related Issues

llama.cpp #19639: CUDA memory leak with MTMD requests loop (Gemma 3)
llama.cpp #21449: System RAM memory leak gemma4-31b
llama.cpp #21468: cache reuse not supported for Gemma 4
vLLM #28230: GPU VRAM continuously increase during Qwen3-VL usage
vLLM #15294: Critical Memory Leak in V1 Engine from image inference
NVIDIA Forum: cublasSetWorkspace does not free auto-allocated workspace

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

Misc. bug: CUDA memory leak with MTMD requests loop with Gemma3 #19639

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Gemma 4 VLM VRAM/Host Memory Leak — Full Investigation Report

Summary

Environment

Reproduction

Controlled Experiments

Leak is Gemma 4 + Image Specific

Leak is NOT in vision encoder

Graph allocator buffers are constant

ALL tracked CUDA allocations are balanced

Root Cause Analysis

Host Memory Leak (FIXED)

VRAM Leak (UNRESOLVED)

Cross-Framework Validation

Suggested Investigation Directions

Files & Tools

Related Issues

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Misc. bug: CUDA memory leak with MTMD requests loop with Gemma3 #19639

Description

Name and Version

Operating systems

Which llama.cpp modules do you know to be affected?

Command line

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

LucaAmigoni commented on Feb 16, 2026

osma commented on Feb 17, 2026

osma commented on Feb 19, 2026

Cyrille37 commented on Feb 19, 2026

osma commented on Feb 19, 2026

ORippler commented on Mar 2, 2026

osma commented on Mar 2, 2026

jstrot commented on Apr 6, 2026

2264K commented on Apr 10, 2026

Gemma 4 VLM VRAM/Host Memory Leak — Full Investigation Report

Summary

Environment

Reproduction

Controlled Experiments

Leak is Gemma 4 + Image Specific

Leak is NOT in vision encoder

Graph allocator buffers are constant

ALL tracked CUDA allocations are balanced

Root Cause Analysis

Host Memory Leak (FIXED)

VRAM Leak (UNRESOLVED)

Cross-Framework Validation

Suggested Investigation Directions

Files & Tools

Related Issues

github-actions commented on May 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions