Skip to content

Fix engine keep-warm, model-switch UX, and local mmap defaults in the TUI#600

Merged
tobocop2 merged 8 commits into
mainfrom
fix/tui-warm-switch-load-ux
Jul 23, 2026
Merged

Fix engine keep-warm, model-switch UX, and local mmap defaults in the TUI#600
tobocop2 merged 8 commits into
mainfrom
fix/tui-warm-switch-load-ux

Conversation

@tobocop2

@tobocop2 tobocop2 commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Problem

On a real box driving the TUI over ssh, four things read as regressions:

  • "Keep engine warm" re-warmed on every return. The idle TTL unloaded the model after engine_idle_ttl_minutes (default 5) regardless of the setting, and nothing kept the engine resident while the TUI was open.
  • Switching chat models locked the input with no indication of why, and the new model only loaded on the next prompt, so the input came back before the model was ready and typing silently did nothing.
  • Cold model load felt slower than it should. The chat server defaulted to --no-mmap on local disk, forcing a full upfront read of the weights instead of mmap's lazy paging.
  • The first-start splash left a cursor blinking on the wordmark.

Solution

  • _warm_ttl_seconds returns 0 (weights resident) whenever an interactive session owns the process, keep_engine_warm is set, or a bulk ingest is holding the fleet. A TUI session keeps the engine warm for its lifetime; closing lilbee releases it. The intent is recorded on the services state before the container builds and threaded through create_provider -> RoutingProvider -> FleetProvider, so it ends up as provider instance state rather than a process-wide flag.
  • The model swap keeps the chat input disabled until the new model actually serves, with a placeholder naming the target and saying why the box is locked, and eagerly warms the new model instead of deferring its cold load to the next prompt.
  • The warm line is computed regardless of task activity and outranks the task summary. It was previously computed only when the task bar was idle, so the document sync that starts on launch suppressed it entirely — leaving a locked input with no explanation for the whole load.
  • The indeterminate phases use the sweeping bar (a spinner glyph read worse in use); the line now also carries the model name and phase.
  • Local-disk chat weights use mmap; --no-mmap is kept only for a network filesystem, where it avoids wedging the loader in uninterruptible I/O.
  • The onefile bootstrap hides the parked cursor at handoff so it stays hidden until Textual takes over.

Verified on hardware

Measured on a GTX 1070 Ti box running Jan v3.5 4B and Qwen3 0.6B, from source:

  • Keep-warm: with keep_engine_warm = false, the live llama-swap configs wrote ttl: 0 and both servers stayed resident continuously past the 5-minute idle window (GPU held at 4917 MiB). Under the previous ttl: 300 they were evicted at 5 minutes.
  • Model switch: instrumented end to end. The input is disabled at +0.00s and released at +37.1s, exactly when wait_chat_ready reports the model can serve — so it never re-enables in front of an unloaded model.
  • mmap: the chat llama-server argv no longer carries --no-mmap on this local-disk host.
  • Splash cursor: not verifiable from source, since it lives in the C onefile bootstrap that only exists in a built executable. The patch hunk arithmetic was corrected and validated; the visual check needs a build.

@tobocop2
tobocop2 changed the base branch from feat/ingest-symlink-parallel-discovery to main July 23, 2026 04:10
… TUI

Hold the chat engine resident while an interactive session owns the process, and honor keep_engine_warm as weight-residency rather than just process lifetime. The intent is recorded on the services state before the container builds, threaded through create_provider -> RoutingProvider -> FleetProvider, and ends up as provider instance state; _warm_ttl_seconds stays a pure function taking that flag, so it returns 0 for an interactive session, keep_engine_warm, or an active ingest. A TUI session keeps the engine warm for its lifetime; closing lilbee releases it.

Model switching now keeps the chat input disabled until the new model actually serves, with a placeholder naming the target and saying why the box is locked, and eagerly warms the new model instead of deferring its cold load to the user's next prompt. Previously the input came back before the model was ready, so typing silently did nothing.

The warm indicator shows a spinner, the model name, and the phase (starting engine / reading weights NN% / loading into VRAM). Spinner frames come from Rich's built-in "dots" spinner instead of a hand-rolled tuple, and the catalog screen's duplicate braille frames are consolidated onto the shared helper.

Local-disk chat weights now use mmap; --no-mmap is kept only for a network filesystem, where it avoids wedging the loader in uninterruptible I/O. mmap reaches a first token faster on a cold cache, which is the common desktop first launch.

The onefile bootstrap hides the parked cursor at handoff so it no longer sits blinking on the wordmark before the Python splash takes over.
@tobocop2
tobocop2 force-pushed the fix/tui-warm-switch-load-ux branch from 54dc1e3 to 05555b9 Compare July 23, 2026 04:20
tobocop2 added 5 commits July 23, 2026 00:35
The swap now holds the chat input disabled with a placeholder naming the target model, and warms the new model before unblocking, so pin both: the input reason text, and the reload -> warm -> wait ordering that keeps the input from re-enabling in front of an unloaded model.
SPINNERS entry values are typed as object, so tuple() on the frames string failed the type check.
test_cli_model patched create_provider by string name, so the earlier sweep for call sites missed it and CI caught the TypeError.
…weep bar

Two defects found by driving a real switch on the HP with the worker instrumented.

The chat warm line was computed only when the task bar was idle, so a background document sync (which starts on launch) suppressed it entirely. A swap holds the chat input disabled for as long as the model takes to load -- measured at 37s on a 4B model -- so the user sat in front of a dead input with no explanation while the bar showed an unrelated sync summary. The warm line is now computed regardless of task activity and outranks the task summary, which stays available in the Task Center.

The indeterminate phases go back to the sweeping bar rather than a spinner glyph: in use, the moving bar reads as progress at a glance where the spinner did not. The determinate byte bar and the model name and phase wording are unchanged.
warm_chat_after_swap was added while chasing the swap-feedback bug and left
unused once the fix landed in the task bar instead; it is dead code and it is
what dropped coverage below the 100% gate.

Also covers the branches the new code added but no test reached: the placeholder
shown during a placement reload, a warm that fails mid-swap, a reload that
raises, the warm line naming the model, and a finished warm reporting no active
progress.
chat.py:2050 was the last uncovered statement: a swap superseded by a newer one
returns without toasting a completion or releasing the input gate, which the
newer swap now owns.
@tobocop2
tobocop2 marked this pull request as ready for review July 23, 2026 07:11
The existing test sends a real SIGTERM and skips on Windows, where the signal is
never delivered, so the handler body went unexercised there and Windows CI failed
the 100% gate on setup.py:276. Drive the registered handler directly so the
conversion that makes the self-check teardown run is covered on every platform.
@tobocop2
tobocop2 merged commit 91797fa into main Jul 23, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant