Repository navigation
Provide a means to unload the model from GPU memory like OLLAMA_KEEP_ALIVE #204
Description
I have the container running on a home server with nvidia GPU. Most of the time it is idle and when so its consuming a decent amount of GPU memory.
Can we introduce logic to unload from GPU memory after a set timeout period.
Similar to how Ollama unloads itself - which can be configured through the OLLAMA_KEEP_ALIVE variable.
See https://github.com/ollama/ollama/blob/main/docs/faq.md for more details.
Describe alternatives you've considered
I can stop it but I'd prefer to have it there always - with a slight warm up time if idle for a period.
Additional context
Seems to be a well known approach ollama and openwebui use as well to be kind and free up memory when idle.
Reacted by Rick Emerson, Fireblade2534, tentakles, rfjohnso, ChuckNovice, fakezeta, Richard R, recrudesce, Karan Deo Burnwal, Transrian and 9 more
Activity
[-]Provide a means to unload the model from GPU memory like ollama `KEEP_ALIVE`[/-][+]Provide a means to unload the model from GPU memory like `OLLAMA_KEEP_ALIVE`[/+]Ali-Flt commentedon Apr 2, 2025
This feature is crucial for such always-running tasks that take GPU memory when idle.
I usually do this by creating a separate child process for each model run so that the GPU memory is released after the child process is finished.
Anyone has any idea how to implement this idea in this project?
RBEmerson970 commentedon Apr 2, 2025
Working with 16 Gb of VRAM for a 4090, I've never had a problem with keeping Fast Koko awake while using Open WebUI running Llama3.3:70b under Docker under Win 11. If this "improvement" doesn't break anything, sobeit, but "if it ain't broke, don't fix it".
Ali-Flt commentedon Apr 2, 2025
@RBEmerson970 some people myself included run these services on machines that are used for multiple purposes and by multiple people. The same machine might be used for other GPU intensive tasks that require as much memory as available (fine tuning, etc.). In such cases idle services like this become important.
It's definitely a cruical missing feature imo and why would it break anything anyway?
Ali-Flt commentedon Apr 2, 2025
I decided to write my own Kokoro TTS server (which is far simpler than this project obviously) that runs each TTS request in another process so that the model is unloaded immediately.
If it might help anyone here is the code.
The dockerfile and docker-compose can also be found in the same repository.
Mikec78660 commentedon Jun 25, 2025
Would be nice to have this. I want it not to only free up memory but also to reduce power usage. On a P4 that idols at 6-7 watts it goes to 24 watts just having the model in memory. On a P40 that idols at 9watts it goes to 50 watts having this model loaded. So would be great if after a set time it could unload. I would rather not have it unload immediately but after 5 minutes would be great. I have 5 gpus in my server but have about a dozen applications that use them. Even Immich now unloads it's models used for face recognition and item detection in your photos.
jagg2 commentedon Mar 24, 2026
This would be a good feature to have.
I'm running on my old GTX 1080 and it's the difference between having some other models run fully on the gpu and having to off load a layer to the cpu. which tanks performance. Having to wait 30 seconds for TTS to kick in isn't a big issue.
Jakeshadow commentedon Jun 13, 2026
TheRealMaN commentedon Jun 23, 2026
Make this unload itself from RAM too, if only CPU is used
remsky commentedon Sep 2, 2026
Merged into master on #506 as MODEL_AUTO_UNLOAD_TIMEOUT_SECONDS, and will be part of the next release for CPU/GPU, for any who still need it. Cheers