fix: auto-unload model after idle timeout to reduce memory by VirenMohindra · Pull Request #1051 · cjpais/Handy

VirenMohindra · 2026-03-16T00:34:18Z

summary

handy holds the transcription model in memory indefinitely (default: Never unload). on a 16 GB machine, a parakeet model uses ~1 GB, even when the user hasn't transcribed in hours

this PR adds automatic model unloading after a configurable timeout, reducing idle memory from ~1.1 GB to ~80 MB

before	after

changes

default timeout: Never -> Min5: model auto-unloads after 5 minutes of inactivity
fix: watcher thread killed by Drop on clones: initiate_model_load() clones TranscriptionManager, and when the clone drops, Drop::drop sets shutdown_signal = true, killing the watcher. fixed with an Arc::strong_count guard so only the last clone shuts down the watcher
fix: reset last_activity on model load: without this, switching models after timeout elapsed would immediately unload the just-loaded model
unload logging upgraded to info!() level: was debug!(), invisible at default log level
removed duplicate "unloaded" event: unload_model() already emits it
error handling on unload: match instead of if let Ok so failures are logged

test results (macOS, parakeet v2, physical footprint = activity monitor)

state	memory
baseline (no model)	46 MB
model loaded	1.1 GB
after auto-unload (5s timeout)	80-88 MB
unload time	33-44ms

actual time to reach 88mb is ~30seconds

full lifecycle:

46 MB -> 1.1 GB -> 660 MB -> 330 MB -> 88 MB
(idle)  (loaded)  (unloading... OS reclaiming pages)

test plan

start app, timeout=sec5, transcribe -> model loads, unloads after ~12s
transcribe again -> model reloads, unloads after ~7s
watcher survives both initiate_model_load clone/drop cycles
activity monitor confirms 80-88 MB after unload (not 700+ MB)
unload duration logged at info level

cjpais · 2026-03-16T00:39:00Z

I was noticing on the latest build the timeout was actually not working at all. Have you experienced this?

At least in the case of changing models from the menubar.

VirenMohindra · 2026-03-16T01:14:39Z

I was noticing on the latest build the timeout was actually not working at all. Have you experienced this?
At least in the case of changing models from the menubar.

think i found the bug. load_model() wasn't updating last_activity, so when you switch models from the menubar, here's what happens~

you transcribe at T=0 -> last_activity = T=0
some time passes (longer than the timeout)
you switch models from the tray and load_model() loads the new model
but last_activity is still T=0
idle watcher checks on its next 10s tick: now - last_activity > timeout -> true -> immediately unloads the model you just loaded

so the model loads, then gets unloaded on the very next watcher cycle. it looks like the timeout "doesn't work" but really the timer was just starting from the wrong point

pushed a fix, load_model() now resets last_activity after successfully loading, so the idle timer starts fresh from the moment the model is ready. this applies to both tray model switches and the background initiate_model_load() path

cjpais · 2026-03-17T01:08:04Z

the changes as they are don't work for me

I dont see prints in the console that model unloading is happening. I am also not seeing memory being freed, nor is the UI updating to show the model is unloaded.

I think the most critical paths to verify are:

start the app
set to never unload
trigger transcription start/stop
model doesnt unload
set to 5 sec

after 5-10 sec should see print in console, and indicated in the UI that the model is unloaded (including the status bar)

set to never (model does not load and loads next time)
set to 2 minutes
trigger transcription start/stop
set to 5 seconds
model should unload

I think realistically we might need a small state machine or something... Or at least very clear logic what happens when transitioning between states. It may be worth making a diagram at least... Posting that here for verification and then having an LLM go implement and verify it ideally. would be great to have it in a testable unit.

VirenMohindra · 2026-03-17T03:37:09Z

tested all 10 steps - everything works now. the main issue was a pre-existing bug in Drop that was killing the idle watcher thread before it could ever fire.

script here
test-model-unload.sh

root cause: initiate_model_load() clones TranscriptionManager and moves it into a spawned thread. when that thread finishes loading, the clone is dropped -> Drop::drop fires -> sets shutdown_signal = true -> watcher dies. so the watcher was alive for ~60s after startup and then permanently dead. no auto-unload ever happened

fix: added an Arc::strong_count guard in Drop - clones skip shutdown since other owners (tauri state, watcher thread) still exist. only the very last clone shuts down the watcher

test results (physical footprint, matches activity monitor)~

state	memory
baseline (no model)	46 MB
model loaded	1.1 GB
after auto-unload (sec5)	80-88 MB
unload time	33-44ms

full lifecycle:

46 MB -> 1.1 GB -> 660 MB -> 330 MB -> 88 MB
(idle)  (loaded)  (unloading... OS reclaiming pages)

two back-to-back transcriptions, both auto-unloaded, watcher survived the entire session~

04:17:23  Transcription #1
04:17:35  Model idle 12s > 5s -> unloaded (44ms)
04:18:58  Transcription #2
04:19:05  Model idle 7s > 5s -> unloaded (33ms)

also upgraded unload logging from debug!() to info!() so it's visible at default log level, and replaced if let Ok with match to log unload errors

- Default model_unload_timeout from Never to Min5 - Fix Drop impl: use take() on watcher handle so clones from initiate_model_load() don't kill the watcher thread - Reset last_activity on model load to prevent immediate unload - Upgrade watcher logging from debug to info level - Remove duplicate "unloaded" event (unload_model already emits it)

VirenMohindra · 2026-03-17T04:33:49Z

src-tauri/src/managers/transcription.rs

+        if Arc::strong_count(&self.engine) > 1 {
+            return;
+        }


Arc::strong_count docs warn it shouldn't be used for synchronization and that the count can change between the check and the shutdown_signal.store. in practice this only runs during app teardown so it's fine today, but adding this comment so future clone paths don't break this silently. alternatively, an explicit AtomicUsize refcount we control would be more robust, but YAGNI for now since i doubt we will have more clone paths

VirenMohindra · 2026-03-17T04:34:15Z

src-tauri/src/settings.rs

 impl Default for ModelUnloadTimeout {
    fn default() -> Self {
-        ModelUnloadTimeout::Never
+        ModelUnloadTimeout::Min5


this only affects fresh installs right? prob worth confirming existing users who never touched this setting won't suddenly get auto-unload behavior after upgrading

- Default model_unload_timeout from Never to Min5 - Fix Drop impl: use take() on watcher handle so clones from initiate_model_load() don't kill the watcher thread - Reset last_activity on model load to prevent immediate unload - Upgrade watcher logging from debug to info level - Remove duplicate "unloaded" event (unload_model already emits it) Co-authored-by: CJ Pais <[email protected]>

VirenMohindra force-pushed the vm/memory-optimization branch from 9ff999c to 417f576 Compare March 16, 2026 01:14

VirenMohindra mentioned this pull request Mar 16, 2026

[BUG] Partial downloads prevent the model from ever being downloaded again #858

Closed

VirenMohindra force-pushed the vm/memory-optimization branch from 0c14750 to 417f576 Compare March 16, 2026 03:04

ruszabarov mentioned this pull request Mar 16, 2026

chore: improve type safety of unload timeout options #1071

Closed

3 tasks

VirenMohindra force-pushed the vm/memory-optimization branch 2 times, most recently from 3a2639e to 7006902 Compare March 17, 2026 03:30

VirenMohindra changed the title ~~fix: reduce idle memory by auto-unloading model after timeout~~ fix: auto-unload model after idle timeout to reduce memory Mar 17, 2026

VirenMohindra force-pushed the vm/memory-optimization branch from 7006902 to 838857a Compare March 17, 2026 03:34

VirenMohindra force-pushed the vm/memory-optimization branch 7 times, most recently from 8e845e2 to 62a3153 Compare March 17, 2026 04:13

VirenMohindra force-pushed the vm/memory-optimization branch from 62a3153 to 6523980 Compare March 17, 2026 04:14

Merge branch 'main' into vm/memory-optimization

aaaaae4

VirenMohindra commented Mar 17, 2026

View reviewed changes

cjpais merged commit d1da935 into cjpais:main Mar 17, 2026
5 checks passed

VirenMohindra mentioned this pull request Mar 18, 2026

fix: prevent idle watcher from unloading model during recording #1085

Merged

7 tasks

VirenMohindra mentioned this pull request Mar 19, 2026

[BUG] Pastes clipboard instead of spoken text #502

Open

VirenMohindra deleted the vm/memory-optimization branch March 19, 2026 05:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: auto-unload model after idle timeout to reduce memory#1051

fix: auto-unload model after idle timeout to reduce memory#1051
cjpais merged 2 commits intocjpais:mainfrom
VirenMohindra:vm/memory-optimization

VirenMohindra commented Mar 16, 2026 •

edited

Loading

Uh oh!

cjpais commented Mar 16, 2026

Uh oh!

VirenMohindra commented Mar 16, 2026 •

edited

Loading

Uh oh!

cjpais commented Mar 17, 2026

Uh oh!

VirenMohindra commented Mar 17, 2026 •

edited

Loading

Uh oh!

VirenMohindra Mar 17, 2026

Uh oh!

VirenMohindra Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

VirenMohindra commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

summary

changes

test results (macOS, parakeet v2, physical footprint = activity monitor)

test plan

Uh oh!

cjpais commented Mar 16, 2026

Uh oh!

VirenMohindra commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjpais commented Mar 17, 2026

Uh oh!

VirenMohindra commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VirenMohindra Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

VirenMohindra Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

VirenMohindra commented Mar 16, 2026 •

edited

Loading

VirenMohindra commented Mar 16, 2026 •

edited

Loading

VirenMohindra commented Mar 17, 2026 •

edited

Loading