Feature proposal: model unload endpoint to release GPU VRAM without stopping container #473

Background

I'm building a homelab diary app (edelmore-diary) that uses kokoro-fastapi for read-aloud narration. On a shared homelab GPU, the container holds model weights in VRAM indefinitely. When the diary isn't in use, that VRAM is unavailable to other services (Ollama, Frigate, etc.).

My current workaround is stopping/starting the container on demand via the Docker remote API, but that means a 25-30s cold start on every session. I'd rather keep the container running and release VRAM when idle.

Proposal: POST /dev/unload

An endpoint that releases the model from GPU memory without stopping the container. The HTTP server stays alive (no cold start on the next request), and the model reloads lazily when the next inference request arrives.

Implementation sketch (based on reading model_manager.py):

@router.post("/dev/unload")
async def unload_model(
    tts_service: TTSService = Depends(get_tts_service),
):
    """Release the model from GPU VRAM. Reloads automatically on next request."""
    manager = tts_service._model_manager
    async with manager._lock:
        if manager._backend is not None:
            await manager._backend.unload()
            manager._backend = None
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    logger.info("Model unloaded from GPU memory")
    return {"status": "unloaded"}

The lazy-reload path would trigger in generate() when _backend is None — currently it raises; that guard would become an await self.initialize() call instead.

Questions:

• Is _backend.unload() + setting to None the right cleanup, or is there additional teardown?
• Any concern about thread safety beyond the existing _lock?
• Is /dev/unload the right path, or should it live elsewhere?

I have a fork at sageframe-no-kaji/Kokoro-FastAPI with a feature/model-unload branch ready to work on once the design is confirmed.