Inference broker step 1: leases + idle unload#17
Merged
Conversation
Pure types for broker step 1: priorities, the three shared llama-server roles, the lease value, and clock-free bookkeeping. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Pinned residency ids are never eviction victims but still count toward the RAM budget; the ledger publishes pin state and lease descriptions for admission and the resource monitor. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Leases pin models for each request; the last release starts a per-role linger, and optional TTLs release abandoned leases. Injectable runtime hooks keep tests independent of llama-server. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Wrap every generate, schema, completion, and streaming call in a broker lease while preserving the base engine's own protocol overrides. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
makeTextEngine returns a LeasedTextEngine for the .builtIn backend: the server boots on first request under a lease instead of at engine creation, and chat/cotyping/model-test calls carry interactive priority. External backends (Apple Intelligence, Ollama, custom OpenAI-compatible) are untouched — they are not our servers. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Each batch pins the embedding server while it runs and releases it after, so indexing can no longer be evicted mid-batch and the embedder unloads after ten idle minutes instead of staying resident forever. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
The controller leases the Main LLM from resolveEndpoint until shutdown or failure, so a pi conversation can't have its model evicted mid-session by an unrelated load. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Each external wake acquires a fresh 600 s lease on the Main LLM and releases the previous one, so a crashed server is revived per wake, questions in a burst share one warm model, and the RAM returns ten minutes after the last question instead of never. Disabling the Privacy toggle releases the lease immediately. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Resident model rows read "in use — chat (interactive)" while the broker holds leases on them, and idle rows disappear on their own when the linger unload fires. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Keep the implementation's source specification in-tree so the broker PR can link its scope, decisions, and follow-up boundaries directly. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
Agent Mode and external wake handling now release a lease that finishes acquiring after shutdown or Privacy disable already ran. Focused race tests hold broker ensure in flight to cover both late-assignment paths. Claude-Session: https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
stevyhacker
marked this pull request as ready for review
July 12, 2026 10:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ask_librarywakes through broker leasesDesign and scope:
Docs/superpowers/specs/2026-07-11-inference-broker-design.mdVerification
LeaseBook,ModelResidency,InferenceBroker,LeasedTextEngine, Agent Access, Agent Mode, and resource-monitor presentationChatStoreTests, two real-GGUF in-process cotyping classes, and the real Bun/pi integration classLokalBot Devbuild succeeded with code signing disabled; the signed build compiled and then waited on the local development-certificate Keychain duringcodesignask_libraryskipped because no long-lived app watcher was serving port 17872https://claude.ai/code/session_011BB4m5LTgHFUBH7odNkfRW