You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Feature Request: Auto-sleep / idle unload for flm serve #562
Please, add an option to flm serve that automatically unloads the model from memory after a configurable period of inactivity, and transparently reloads it on the next incoming request.
Same idea and semantics as the recently merged llama.cpp feature: ggml-org/llama.cpp#18228 (--sleep-idle-seconds).
Activity