Repository navigation
Unload model from memory/GPU when idle for set period #4141
Description
This issue was also reported by someone else here: openvinotoolkit/openvino#33665
With the current implementation it is not possible to use the GPU for other workloads while OVMS is running, as it will run out of free vRAM. Other LLM servers approach this by setting a user-defined idle timeout on a model. If no API requests are made to query a specific model for a specific amount of time, that model gets removed from memory. When a new request comes in for the model, it automatically gets loaded into memory again. It would be nice if OVMS had a similar feature.
An explanation of how llama.cpp does it can be found at https://github.com/ggml-org/llama.cpp/tree/master/tools/server#sleeping-on-idle
Reacted by Farshid Tavakolizadeh, Tullio Loffredo, ProudNerd, Peter Hostačný, Nolan Johnson and Rudolf K
Activity
proud-nerd commentedon Jun 3, 2026
Would be insanely helpful. I could utilize my GPU way more for different workloads. Right now I need to decide what models I need and make sure they fit into VRAM. Swapping models by hand if necessary is cumbersome and could be easily prevented with this approach. Especially helpful for more VRAM limited, consumer-oriented GPUs.
exzile commentedon Jun 26, 2026
Picking this up. Here's the approach I'm implementing, scoped to the VRAM-heavy LLM continuous-batching path first. It follows the model llama.cpp established with
--sleep-idle-seconds(PR ggml-org/llama.cpp#18228) — unload weights + KV cache on idle, lazy reload on the next request — adapted to the OVMS model-manager lifecycle.Config (opt-in, no default behavior change)
idle_unload_timeout_secondson the mediapipe/model config entry.0= disabled (default).Idle tracking
/v3/chat/completions,/v3/completions, KFS infer). Status/health/readiness/metrics and model-status endpoints are exempt — they neither reset the idle timer nor trigger a reload, so health probes can't keep a model pinned in VRAM. (Same exemption llama.cpp applies to/healthand/props.)Unload (driven by the existing model-manager watcher)
watcherthread (already running ~1s) checks each graph; whennow - lastActivity > timeoutand there are no in-flight requests, it transitions the graph to a newUNLOADEDstate inPipelineDefinitionStatusand releases the heavy LLM node resource (theGenAiServable/ContinuousBatchingPipelineheld viaLLM_NODE_RESOURCES), freeing both the compiled model and the KV cache from VRAM while keeping the validated graph definition intact.Lazy reload on demand
UNLOADEDgraph re-runsinitializeNodes()to recreate the pipeline, then serves; the first request after idle pays the reload cost. This composes with--cache_diron the CB path (--cache_dir not propagated to LLM continuous batching pipeline (regression vs 2025.4 promise) #4230) so the wake-up is a cache import rather than a full recompile.Concurrency
graphqueue/state-machine handling already used for reload/retire: unload only when in-flight count is 0; requests arriving during unload wait for reload via the same mechanism.Observability
is_sleeping).I'll keep the existing
AVAILABLE/RETIREDsemantics untouched and addUNLOADEDas a distinct state so nothing changes when the feature is off. PR to follow againstmain, with tests covering unload-after-idle and reload-on-request.exzile commentedon Jun 26, 2026
Opened #4332 implementing this, verified end-to-end on GPU (idle unload frees resources after the timeout; the next request transparently wakes the model and serves).
Add idle model unload for mediapipe LLM graphs
Add idle model unload for mediapipe LLM graphs
Add idle model unload for mediapipe LLM graphs