Repository navigation
server : host-memory prompt caching - #16391
Merged
Merged
Conversation
ggerganov
force-pushed
the
gg/prompt-cache-ext
branch
2 times, most recently
from
October 3, 2025 18:49
0787f03 to
5c0cec4
Compare
This comment was marked as spam.
This comment was marked as spam.
ggerganov
force-pushed
the
gg/prompt-cache-ext
branch
from
October 7, 2025 07:40
5c0cec4 to
1440ec5
Compare
ggerganov
changed the base branch from
master
to
gg/server-checkpoints-improve
October 7, 2025 07:41
3 tasks done
ggerganov
force-pushed
the
gg/prompt-cache-ext
branch
from
October 7, 2025 15:09
9de8392 to
cf7dd4b
Compare
Member
Author
ggerganov
force-pushed
the
gg/prompt-cache-ext
branch
from
October 8, 2025 08:24
65e8991 to
264d2c3
Compare
ggerganov
marked this pull request as ready for review
October 8, 2025 12:53
1 task
4 tasks
trestoncuzzort
added a commit
to trestoncuzzort/dawnr
that referenced
this pull request
Oct 2, 2026
…, github.com/ggml-org/llama.cpp/pull/16391); the student's server reached 10.8 GB with the default
frostyautumnleaf
pushed a commit
to frostyautumnleaf/llama.cpp
that referenced
this pull request
Oct 5, 2026
* minor : code style * server : fix prompt similarity calculation * server : initial host-memory prompt caching * cont * server : refactor * cont * cont : make the server task of the slot const * cont : minor [no ci] * server : cache prompts and checkpoints only for completion tasks * server : improve prompt caching logic * cont : fix check for number of cached prompts [no ci] * server : improve caching logic, add -cram CLI arg * server : print prompt mismatch info * cont : better naming [no ci] * server : improve prompt cache loading logic * server : add option to debug the slot contents (ggml-org#16482) * server : add option to debug the slot contents * Update tools/server/server.cpp --------- Co-authored-by: Xuan-Son Nguyen <[email protected]> * server : add option to disable prompt cache --------- Co-authored-by: Xuan-Son Nguyen <[email protected]>
trestoncuzzort
added a commit
to trestoncuzzort/dawnr
that referenced
this pull request
Oct 6, 2026
…, github.com/ggml-org/llama.cpp/pull/16391); the student's server reached 10.8 GB with the default
trestoncuzzort
added a commit
to trestoncuzzort/dawnr
that referenced
this pull request
Oct 18, 2026
…, github.com/ggml-org/llama.cpp/pull/16391); the student's server reached 10.8 GB with the default
Vect0rM
added a commit
to AtomicBot-ai/atomic-chat-core
that referenced
this pull request
Nov 3, 2026
…#68) * fix(hermes): patch only the top-level model block of config.yaml The writer rewrote the first `default:`, `provider:` and `base_url:` lines anywhere in ~/.hermes/config.yaml. Hermes repeats those keys under auxiliary, delegation, cron, moa, tts, memory and others, so on any config where one of them comes before `model:` (or where `model:` is a scalar or missing) the patch landed in another block, sometimes breaking the YAML, and left the chat route untouched. Hermes then ran on its built-in defaults (empty model, provider auto), fell through to OpenRouter and got "HTTP 400: No models provided" before any request reached Atomic Chat. Edit only direct children of the top-level `model:` key: replace `default`, `provider` and `base_url` in place (dropping a nested value a scalar replaces), add the missing ones at the block's indent, turn a scalar `model: <id>` into a block, and append a model section when there is none. Also never add a second top-level `providers:` or `custom_providers:` key (Hermes' ruamel reader rejects duplicate keys and then runs on defaults), and recognise those keys with a trailing comment. Golden fixture outputs are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(server): turn llama.cpp thinking off for Anthropic requests that ask for none The Anthropic API thinks only when a request sets `thinking` to enabled or adaptive. llama.cpp's /v1/messages maps only `enabled` and otherwise leaves the chat template at the server default, which --jinja turns on for every template that can think. Claude Code's session-title request (no `thinking`, max_tokens 32000) made Qwen3.5 4B think for 20k tokens, and on a one-slot server the user's own turn waited behind it (QA B08). For a /messages request to a llama.cpp engine (llamacpp, llamacpp-upstream, atomic-prism) that has no `thinking` or `thinking: {type: "disabled"}` and does not set the switch itself, the core now adds `chat_template_kwargs: {enable_thinking: false}`, which llama.cpp's Anthropic conversion passes to the template. The chat-completions fallback gets the same kwargs. Requests that ask for thinking pass byte for byte; other engines are untouched. The proxy-http contract records the divergence. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(server): pin the byte-exact relay of llama.cpp streams to usage-asking clients B11 (Kilo Code showed no reply while the inspector recorded a completed stream): Kilo (AI SDK openai-compatible with includeUsage) and OpenClaude both send stream_options.include_usage themselves, so the proxy neither rewrites their request nor strips anything from the reply. Pin that with a llama.cpp b12019-shaped thinking-model reply written in small pieces cut mid-line and between an event's two newlines: the client gets the upstream bytes unchanged, watched or not, and a client that did not ask loses only the usage trailer the proxy asked for, with the finish chunk and [DONE] intact across split reads. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(server): keep max_tokens on a llama.cpp /messages fallback A /messages request that llama.cpp's template rejects (Claude Code sends mid-conversation system messages, which Qwen3.5's template refuses) is retried as chat completions. The translation dropped max_tokens, as the Rust proxy did, so on a server started with n_predict -1 the reply ran until the context was full. llama.cpp's own /v1/messages honours the cap; its fallback now keeps it too. Other engines keep the translation as it was. The proxy-http contract records the divergence. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(server): merge mid-conversation system messages before llama.cpp sees them Claude Code sends its reminders as role "system" messages between turns to a model it does not know. llama.cpp's /v1/messages passes them to the chat template where they stand, and a strict template (Qwen3.5: "System message must be at the beginning.") answers 500, so every later turn went through the chat-completions fallback. That fallback merges the system messages into the leading one but loses the reply's token usage (Claude Code reported input_tokens 0 and never learned how full its context was) and the turn's thinking blocks. For a /messages request to a llama.cpp engine, the core now merges those messages into the top-level system the same way before forwarding, so the engine answers natively. Other engines and conversations without system messages are sent as they came. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(integrations): hand the served context window to the agent writers Agents told nothing about the window assume 128K-272K and never compact before a 16K-64K local server refuses. ConfigureInput gains an optional contextLength, and `atomic-chat launch` reads the window the session was loaded with from its /props (falling back to the asked size without fit). The writers use it in follow-up commits. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): Zed's real config dir and the served window for Zed and ZCode - zedConfigDir(): %APPDATA%\Zed on Windows, $XDG_CONFIG_HOME/zed on Linux, ~/.config/zed on macOS, as Zed resolves it (parity with the app's zed_config_dir). - Zed: max_tokens = served window, max_output_tokens = min(8192, ctx/4), images: false (Zed's own default); 32768 when the window is unknown, since Zed requires a value. - ZCode: contextWindow = served window, omitted when unknown so ZCode's default stands; output cap min(8192, ctx/4). - Both keep the user's default model they replace in a <config>.atomic-chat-backup.json sidecar for the app's Disconnect. - Golden zed fixtures: images false, and the sidecar where the seed had another provider as the default model. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): find Zed and ZCode Preview off PATH in `launch` Mirrors the app's zed_cli_candidates / zed_app_candidates and the ZCode Preview build of zcode_app_candidates_for. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): tell Kilo, OpenCode, MiMo and pi the served window and keep what Connect replaces The OpenCode-family writers now describe the model the way these agents need for a local server: - `limit: { context, output: min(8192, ctx/4) }` when the window is known (omitted when not), so OpenCode and Kilo compact instead of running with `limit.context = 0`, and MiMo stops assuming a 1M window; - `reasoning: true` and `interleaved: { field: "reasoning_content" }`, so a thinking model's answer is shown as thoughts plus text and its reasoning is sent back within a tool loop (B11); - provider options `timeout: false, headerTimeout: false`, verified in the Kilo, OpenCode 1.15 and MiMo schemas: a long prompt on a slow machine no longer hits the 5-minute first-byte timeout. pi gets `name`, `contextWindow`, `maxTokens` and a per-model `compaction.modelOverrides["atomic/<id>"]` (reserve and recent history a quarter of the window), the schema documented in pi's settings reference. OpenCode and MiMo are read as JSON5 like Kilo (comments and trailing commas are dropped on write); an existing `opencode.jsonc` / `mimocode.jsonc` is written instead of the `.json` it overrides, and our leftovers in that `.json` are removed. KILO_CONFIG_DIR, OPENCODE_CONFIG, OPENCODE_CONFIG_DIR, MIMOCODE_HOME, PI_CODING_AGENT_DIR and XDG_CONFIG_HOME are honoured. MiMo's `$schema` is the one MiMo serves. Before `model`, `provider.atomic`, pi's default provider/model or its `providers.atomic` are overwritten, a value that is not ours is kept in `<file>.atomic-chat-backup.json` for the app's Disconnect. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): run Goose with its own environment, start Kilo on our model Goose no longer gets OPENAI_API_KEY, OPENAI_HOST and GOOSE_* written user-wide (shell rc block, `setx`), which pointed every OpenAI SDK tool on the machine at the local server. `launch goose` sets them on the child only: GOOSE_PROVIDER, GOOSE_MODEL, OPENAI_BASE_URL with the server's API prefix (instead of the bare host plus a hard-coded OPENAI_BASE_PATH), OPENAI_API_KEY, and GOOSE_CONTEXT_LIMIT / GOOSE_MAX_TOKENS when the window is known (names checked against goose 1.57). An ambient OPENAI_HOST / OPENAI_BASE_PATH is cleared, since Goose prefers it to OPENAI_BASE_URL. Configuring Goose now only removes what earlier releases wrote: our marked block in .zshenv/.bashrc/.bash_profile (our export lines only) and, on Windows, the user variables recognisably set by `setx`. Kilo's saved per-agent pick beats `model` in its config, so `launch kilo` passes `--model atomic/<id>` (the TUI's own flag). Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(contract): agent-config fixtures for the new Kilo, OpenCode, MiMo, pi and Goose output Regenerated by the app's Rust writers (`opencode_family.rs` `dump_family_fixtures`) and replayed byte for byte here: the model entry with thinking, no timeouts and the window, pi's window and compaction override, the backup files Connect leaves for Disconnect, the `.jsonc` target with its `.json` cleaned, MiMo's corrected `$schema`, and Goose writing nothing but removing what earlier releases wrote. Cases may carry `context_length` (index note). CHECKSUM recomputed over this tree. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(models): count sliding-window KV layer by layer, with a window per slot The KV estimate averaged a sliding-window model over every layer, (full + window) / 2. llama.cpp b12019 keeps the whole window only in the global layers and pad256(n_swa x sequences + n_ubatch) cells in each sliding one (src/llama-kv-cache-iswa.cpp:73), so Gemma 3 was counted at two and a half times its real cache at 32K and small Macs were narrowed far more than needed. The estimate now reads the per-layer sliding_window_pattern (inlined by the GGUF reader at any length, like head_count_kv), a period from the GGUF, or the architecture's built-in pattern (gemma2/3/3n, gpt-oss, cohere2, exaone4, afmoe, smallthinker); Gemma 4's own SWA head sizes and shared KV layers count too. An architecture with a window but no known pattern is counted in full, the safe side. A new layout argument gives the sequences sharing a unified cache and the micro-batch size. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(llamacpp): agent-friendly defaults: shared KV slots, answer and thinking caps Integrations audit 2026-11-03 (S3, S4, S5, X6, P3), llama.cpp b12019: - Slots: -kvu is always sent, so --parallel N shares one KV pool of the whole window (src/llama-context.cpp:389-390) and costs no window memory beyond each sliding layer's per-slot window. parallel defaults to 2, so a title or classifier request no longer waits behind an agent turn. The fit margin and the small-Mac ceiling no longer multiply the KV cache by the slots (load-plan.ts); the ceiling is the pool itself. - Continuous batching is sent both ways (--cont-batching / --no-cont-batching): llama.cpp has it on unless told otherwise, so the setting could never turn it off. Its default is now on, as llama.cpp's. - n_predict 0 is automatic: 16384 tokens, never above the window core knows. Clients without max_tokens (Codex via Responses, Kilo, OpenCode) no longer generate until the window is full. -1 stays "no limit". - New reasoning_budget setting (default 8192) sent as --reasoning-budget with --reasoning-budget-message from b8287, the first build that takes a token count (ggml-org/llama.cpp#20297); requests' own budgets win. - --cache-ram 1024/2048 MiB on machines with at most 8/16 GiB of RAM (from b6723, ggml-org/llama.cpp#16391) instead of llama.cpp's 8 GiB. - DEFAULT_CTX_LEN is 32768: a fit-off load without a window gets 32K (or the trained context) instead of the whole trained context, and MLX's default window is 32K; on a Mac below 16 GiB both are narrowed toward the previous 16K by the KV estimate, never below it. - isContextOverflowFinishLength takes the engine's own answer cap into account: a length stop at --n-predict is no longer a context overflow. The args fixtures from the Rust builder are compared without -kvu and --no-cont-batching, recorded as deliberate divergences; the engine contract gains the new flags and build gates. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(managed): automatic context length and request count for vLLM and SGLang Integrations audit 2026-11-03, S2: vLLM and SGLang defaulted to 8192 tokens and one request, so every coding agent's 13-27K-token prompt was refused with context_length_exceeded and side requests queued behind the agent's turn. context_length and max_num_seqs / max_running_requests now default to 0, automatic: the load picks the widest pair the engine's own memory rule passes on the card it re-probes right before the container — 64K with two requests, 32K with two, 32K alone, 16K, then the old 8K alone, at which the memory check refuses exactly as before. The window never exceeds the checkpoint's own (max_position_embeddings and the other keys vLLM derives it from), leaves room for the output cap, and with a KV size fixed by hand holds one full sequence of it. Both engines keep one paged pool sized by their memory share, so the window only bounds a request. The chosen pair reaches the container through the launch plan and the session limits (/v1/models context) through planContextLength. Values set by hand stay as they are. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(settings): move stored values off the defaults core no longer ships settings.json stores every provider value, defaults included, so a new schema default reached fresh installs only. On open, the store now moves a stored value that still equals an old default to the new one, once, and records the migration under state.default_migrations; a value the user sets afterwards, the old default included, stays. A moved value bumps the revision so the app mirrors it. Migrations (integrations audit 2026-11-03): llama.cpp parallel 1 -> 2, cont_batching false -> true (llama.cpp always batched continuously), n_predict -1 -> 0 (automatic cap); vLLM/SGLang context_length 8192 -> 0 and request count 1 -> 0 (automatic). Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): keep Claude Code and OpenClaude writers in step with the app The `launch` writers for Claude Code and OpenClaude now match the app's claude_config.rs: each keeps a `<file>.atomic-chat-backup.json` sidecar recording what every key held before, so the app's Disconnect can undo a connection the CLI made; a reconnect undoes the previous one first, and values an earlier release wrote are taken out rather than recorded. Claude Code: empty CLAUDE_CODE_USE_BEDROCK/VERTEX/FOUNDRY, the terminal title off, and with a known window CLAUDE_CODE_MAX_CONTEXT_TOKENS plus CLAUDE_CODE_MAX_OUTPUT_TOKENS (min(8192, ctx/4)); $CLAUDE_CONFIG_DIR is honoured. OpenClaude: the profile carries the server's key again (OC-2: the keyless profile got a 401 from a keyed server), maxContextLength and a per-model output cap with a known window, empty values for the shell variables that make OpenClaude skip its active profile, and the startup profile its OPENAI_API_KEY. Our profile is matched by its id only, and $OPENCLAUDE_CONFIG_DIR and the legacy .config.json are honoured. Golden fixtures for both agents regenerated; the in-place replacement case became openclaude__users_own_atomic_chat_profile_kept. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(server): keep mid-conversation system messages in place for llama.cpp 6f87e30 moved every system message inside an Anthropic conversation into the top-level system. That rewrites the start of the prompt whenever Claude Code sends a reminder, and llama.cpp then processes the whole conversation again (13-27k tokens on hybrid and sliding-window models, whose prompt cache cannot reuse anything past the change). For a llama.cpp engine, only system messages before the first turn now join `system`. A later one stays where it is, as user text wrapped in <system-reminder>, merged into a neighbouring user turn so the roles still alternate: before the text of the next user message, or at the end of its last tool_result when it answers tool calls (llama.cpp puts a user message's text before its tool results, so a text block there would land between the call and its result). With no user message after it, it goes to the end of the one before, or into a user message of its own. A reminder sent with clear_at "next_user_message" is dropped once a user message follows it. The chat-completions fallback gets the same messages. The Responses shim (Codex to llama.cpp) placed developer and system items the same hoisting way; it now keeps later ones in place too, never between a tool call and its output. The shim contract case that recorded the Rust hoisting is corrected as a known divergence. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(server): advertise the served window and answer overflows as prompt too long Agents did not know the window the local engine really serves, so none of them compacted in time (audit X1). GET /v1/models now gives each local model whose window is known `context_length` and `meta.n_ctx` (the field llama-server's own list uses), and Muse Code's catalogue advertises the real window instead of its 32k guess. The window comes from the session: a declared session (vLLM, SGLang, TensorRT-LLM) states it; a llama.cpp session is asked once per loaded generation through /props (default_generation_settings.n_ctx, answered from metadata without a slot). The core wires SessionWindows into the server as `contextWindow`; a session target now carries its generation. Other engines advertise none rather than a guess. On /v1/messages, a llama.cpp context overflow (exceed_context_size_error with n_prompt_tokens/n_ctx, or its message) on the native call or on the chat-completions fallback is answered as the Anthropic API's 400 invalid_request_error "prompt is too long: N tokens > M maximum", the wording Claude Code compacts on; the fallback is no longer tried for it. OpenAI routes keep llama.cpp's own error. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(server): keep each request's context window and cached prompt tokens The Activity inspector showed the window of whatever session served the model when the row was opened, not the one the request ran with (QA B04). The trace now records the window of the session a request goes to when it is routed (the session's declared window or the one SessionWindows read) and sends it as `ctx_len` in the finish fields, so the log keeps it after a reload or an unload. The finish fields also carry `cached_tokens`, the part of the prompt the engine reused, from llama.cpp's timings.cache_n, OpenAI's prompt_tokens_details.cached_tokens and Anthropic's cache_read_input_tokens. Prompt tokens now count the whole prompt: Anthropic's input_tokens leaves out cache reads and writes, and llama.cpp's timings.prompt_n only what it processed. A whole Anthropic reply is read for usage and stop reason too. The Responses shim carries the cached part into input_tokens_details instead of a hard-coded 0. The inspector-telemetry contract records the new field as a known divergence (null in every recorded case). Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(server): give Codex's MCP namespace tools to llama.cpp Codex sends the tools of each MCP server as a Responses namespace ({type: "namespace", name, tools: [...]}) and runs a call to one only with that namespace. The Responses shim kept only top-level `function` tools, so a local model never saw a Codex MCP tool (audit S7). The shim now offers every function in a namespace as a chat function named <ns>__<name> (the plain name in Codex's default `functions` namespace; no second delimiter after a namespace that already ends in one), described by its own description or else the namespace's. A function_call item replayed from history and a namespaced tool_choice use the same name. In the reply, whole or streamed, a call to such a name is given back its `name` and `namespace`, so Codex routes it to the MCP server. Freeform tools inside a namespace are dropped like top-level ones. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(server): cover the system-reminder wrapper Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): tell Codex the served window in its profile With the session's window known, the Atomic Chat profile carries model_context_window and model_auto_compact_token_limit (three quarters of it), so Codex compacts before a 16K-64K local server refuses instead of assuming 272K. Unknown, neither line is written. Byte-identical to the desktop writer. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): Cline, Droid, Copilot and dsh get the window and a backup Ports the desktop writers (app wip/fam-codex), byte for byte: - A known window is written with a reply cap of a quarter of it (8192 at most), and nothing is invented when it is unknown: Cline's contextWindow/maxTokens on the entry `cline auth` saved, Droid's maxContextLimit/maxOutputTokens (instead of a fixed 16384), Copilot's maxPromptTokens/maxOutputTokens/maxContextWindowTokens, dsh's route. - Each writer records what it overwrote in a <config>.atomic-chat-backup.json sidecar, which the desktop's Disconnect restores from. - Droid names its entry (id, index) and selects it through sessionDefaultSettings.model; the legacy top-level selector is removed. - Copilot writes an atomic-chat provider and model into .copilot/providers.json instead of persisting COPILOT_* in the shell rc, removes exactly the rc block an earlier release wrote, and `launch` selects the model with COPILOT_MODEL=atomic-chat/<model>. - dsh 0.2 imports settings.yaml once into the profile, so the route and the default model go into profiles/web/cordis.patch.yml, re-serializing only our rows; a settings.yaml still present is kept in step, never created, and settings.yaml.imported is never touched. The cline/copilot/droid/dsh agent-config fixtures change accordingly (index note); the desktop writers produce the same bytes for every copilot, droid and dsh case. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(cli): start npm-shim agents through cmd.exe on Windows `atomic-chat launch` spawned the detected launcher without a shell. On Windows an npm-installed agent is a `.cmd` shim, which Node refuses to spawn directly (EINVAL) and does not find by its bare name (ENOENT), so every npm agent failed to start. Shims and bare names now run through `cmd.exe /d /s /c "<line>"` with each token quoted for cmd; executables and other platforms spawn as before. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): Hermes, OpenClaw and Atomic Agent writers record what they replace Hermes: one route for the app and the CLI, bare `custom` with model.base_url, model.api_key and model.context_length, which needs no custom_providers entry (an inline list cannot take one) and carries the server key. The served window is written as served, 65536 only when unknown. HERMES_HOME is honoured on every platform (~, $NAME, ${NAME}, %NAME% expanded) and the sticky profile from `hermes profile use` is the one written. No skeleton is seeded. OpenClaw: the model entry carries contextWindow and maxTokens when the window is known; the request budget moves to the provider (timeoutSeconds 600) and the agents.defaults.timeoutSeconds: 240 older releases seeded is dropped; gateway.auth.mode "none" is seeded only when no token or password is set. Atomic Agent: llm.fallback.appendLocal false when there is no fallback block, so a failure of ours is not retried on local-llama :8080; userModels carries the served window. Each writer records the keys it overwrote in <config>.atomic-chat-backup.json (json-key-backup.ts), the same sidecar the app writes, so the app's Disconnect restores a connection made by `atomic-chat launch`. The golden fixtures of the three agents are regenerated and replayed byte for byte against the app's Rust writers. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(integrations): OpenHands, Poolside and Muse get their environment at launch only Their writers persist nothing any more: the variables go on the child `launch` starts (and inline in the app's terminal). Connect removes the marked shell-rc block an older release wrote, from any rc file it may have used, and nothing else; the old prefix sweep that deleted the user's own `export META_*` / `POOLSIDE_*` / `LLM_*` lines is gone with the writer. Poolside keeps `/v1` in POOLSIDE_STANDALONE_BASE_URL as its docs give it and is told the served window (POOLSIDE_STANDALONE_CONTEXT_LENGTH) through childEnv, which `launch` now passes the session's window to. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(contract): recompute the app fixture checksum Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(integrations): cover the per-key backup and the legacy shell block Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs: record the agent-serving defaults and the window the server advertises Co-Authored-By: Claude Opus 5.5 <[email protected]> * test: keep context-growth tests independent of host RAM, /props and path separators - sessions: mock node:os totalmem so the 32K default is not narrowed on small CI runners (7 GiB macOS, Linux arm64). - public-server: load the overflowing model at an explicit 4K window instead of relying on the old 8K default. - api-events: the external engine stub answers the /props window read and counts only chat requests. - launch: ZCode's window is the one /v1/models advertises. - json-backup: `~/` joins with the platform separator on Windows. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test: bring the new agent writers and overflow mapping up to their coverage floors The macOS unit job runs test:coverage, which failed its per-file floors on the code this branch added. Tests now cover: - goose: every legacy local-host range, the Windows reg/PowerShell path and its IO_ERROR, the placeholder vs server key. - hermes: CRLF input, %YAML directives, wrapped scalars, the providers block, the Disconnect backup reader, ~ and %NAME% homes, .env proxy bypass. - opencode-style, openclaude, pi: stale $schema, unreadable backups, the sibling .json beside a .jsonc, earlier startup profiles. - errors: anthropicPromptTooLong from fields and both message wordings, MLX KV-cache wordings, clientMaxTokens. - mlx model-files, fit-margin, catalog: memory facts, shard renames, the 256-token grid, Zed under LOCALAPPDATA. goose.ts drops a `?? ''` fallback on a regex group that always matches. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(e2e): give a cold Windows daemon 45 s to print its ready line On windows-11-arm the first daemon of each e2e file spends 10-15 s in the PowerShell hardware probe before it is ready (main included); on a slow runner that crossed the 20 s wait and failed the start-up tests. Other platforms keep 20 s. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

target #16440
rel #16117
Initial version of automatic memory offloading to host memory using an extended logic for minimizing the prompt reprocessing. The host-memory prompt cache acts as "extra slots" with which we can calculate prefix similarity and decide to hot-swap them into the
llama_contextif it would reduce the processing. The cache is stored in regular RAM.The RAM size that is used for caching prompts has 2 limits:
--cache-ram, -cramCLI arg)--context-size)The server logs provide detailed prompt cache information each time the cache is updated:
A small QoL improvement is that
update_slots()now also logs the old and new prompt for each task aroundn_past(up to 10 tokens) so we can have a better understanding what caused the particular choice of then_pastvalue for the new task.Setting
LLAMA_SERVER_SLOTS_DEBUG=1env will make the/slotsendpoint output a more detailed output containing the prompt and the generated text of the current or last task. This is useful for debugging purposes.Note: mtmd workarounds are starting to cause some headaches. For example
server_tokensis not copyable which complicates the cache logic and makes the prompt caching feature incompatible with mtmd.Usage
Server refactor
server_slotmembers with a singleserver_taskserver_slot.n_predictslot.taskis nowconst ptrto reflect that the task parameters should not change when it is passed to the slotTODOs