Fix engine keep-warm, model-switch UX, and local mmap defaults in the TUI#600
Merged
Conversation
tobocop2
changed the base branch from
feat/ingest-symlink-parallel-discovery
to
main
July 23, 2026 04:10
… TUI Hold the chat engine resident while an interactive session owns the process, and honor keep_engine_warm as weight-residency rather than just process lifetime. The intent is recorded on the services state before the container builds, threaded through create_provider -> RoutingProvider -> FleetProvider, and ends up as provider instance state; _warm_ttl_seconds stays a pure function taking that flag, so it returns 0 for an interactive session, keep_engine_warm, or an active ingest. A TUI session keeps the engine warm for its lifetime; closing lilbee releases it. Model switching now keeps the chat input disabled until the new model actually serves, with a placeholder naming the target and saying why the box is locked, and eagerly warms the new model instead of deferring its cold load to the user's next prompt. Previously the input came back before the model was ready, so typing silently did nothing. The warm indicator shows a spinner, the model name, and the phase (starting engine / reading weights NN% / loading into VRAM). Spinner frames come from Rich's built-in "dots" spinner instead of a hand-rolled tuple, and the catalog screen's duplicate braille frames are consolidated onto the shared helper. Local-disk chat weights now use mmap; --no-mmap is kept only for a network filesystem, where it avoids wedging the loader in uninterruptible I/O. mmap reaches a first token faster on a cold cache, which is the common desktop first launch. The onefile bootstrap hides the parked cursor at handoff so it no longer sits blinking on the wordmark before the Python splash takes over.
tobocop2
force-pushed
the
fix/tui-warm-switch-load-ux
branch
from
July 23, 2026 04:20
54dc1e3 to
05555b9
Compare
The swap now holds the chat input disabled with a placeholder naming the target model, and warms the new model before unblocking, so pin both: the input reason text, and the reload -> warm -> wait ordering that keeps the input from re-enabling in front of an unloaded model.
SPINNERS entry values are typed as object, so tuple() on the frames string failed the type check.
test_cli_model patched create_provider by string name, so the earlier sweep for call sites missed it and CI caught the TypeError.
…weep bar Two defects found by driving a real switch on the HP with the worker instrumented. The chat warm line was computed only when the task bar was idle, so a background document sync (which starts on launch) suppressed it entirely. A swap holds the chat input disabled for as long as the model takes to load -- measured at 37s on a 4B model -- so the user sat in front of a dead input with no explanation while the bar showed an unrelated sync summary. The warm line is now computed regardless of task activity and outranks the task summary, which stays available in the Task Center. The indeterminate phases go back to the sweeping bar rather than a spinner glyph: in use, the moving bar reads as progress at a glance where the spinner did not. The determinate byte bar and the model name and phase wording are unchanged.
warm_chat_after_swap was added while chasing the swap-feedback bug and left unused once the fix landed in the task bar instead; it is dead code and it is what dropped coverage below the 100% gate. Also covers the branches the new code added but no test reached: the placeholder shown during a placement reload, a warm that fails mid-swap, a reload that raises, the warm line naming the model, and a finished warm reporting no active progress.
chat.py:2050 was the last uncovered statement: a swap superseded by a newer one returns without toasting a completion or releasing the input gate, which the newer swap now owns.
tobocop2
marked this pull request as ready for review
July 23, 2026 07:11
The existing test sends a real SIGTERM and skips on Windows, where the signal is never delivered, so the handler body went unexercised there and Windows CI failed the 100% gate on setup.py:276. Drive the registered handler directly so the conversion that makes the self-check teardown run is covered on every platform.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a real box driving the TUI over ssh, four things read as regressions:
engine_idle_ttl_minutes(default 5) regardless of the setting, and nothing kept the engine resident while the TUI was open.--no-mmapon local disk, forcing a full upfront read of the weights instead of mmap's lazy paging.Solution
_warm_ttl_secondsreturns 0 (weights resident) whenever an interactive session owns the process,keep_engine_warmis set, or a bulk ingest is holding the fleet. A TUI session keeps the engine warm for its lifetime; closing lilbee releases it. The intent is recorded on the services state before the container builds and threaded throughcreate_provider->RoutingProvider->FleetProvider, so it ends up as provider instance state rather than a process-wide flag.--no-mmapis kept only for a network filesystem, where it avoids wedging the loader in uninterruptible I/O.Verified on hardware
Measured on a GTX 1070 Ti box running Jan v3.5 4B and Qwen3 0.6B, from source:
keep_engine_warm = false, the live llama-swap configs wrotettl: 0and both servers stayed resident continuously past the 5-minute idle window (GPU held at 4917 MiB). Under the previousttl: 300they were evicted at 5 minutes.wait_chat_readyreports the model can serve — so it never re-enables in front of an unloaded model.llama-serverargv no longer carries--no-mmapon this local-disk host.