Skip to content

server : host-memory prompt caching - #16391

Merged
ggerganov merged 17 commits into
masterfrom
gg/prompt-cache-ext
Oct 9, 2025
Merged

ggerganov merged 17 commits into
masterfrom
gg/prompt-cache-ext

Conversation

@ggerganov

@ggerganov ggerganov commented Oct 2, 2025 •

Copy link
Copy Markdown
Member

target #16440
rel #16117

Initial version of automatic memory offloading to host memory using an extended logic for minimizing the prompt reprocessing. The host-memory prompt cache acts as "extra slots" with which we can calculate prefix similarity and decide to hot-swap them into the llama_context if it would reduce the processing. The cache is stored in regular RAM.

The RAM size that is used for caching prompts has 2 limits:

  • Max size in bytes (controlled with new --cache-ram, -cram CLI arg)
  • Max number of cached tokens (by default, equal to --context-size)

The server logs provide detailed prompt cache information each time the cache is updated:

image
  • A small QoL improvement is that update_slots() now also logs the old and new prompt for each task around n_past (up to 10 tokens) so we can have a better understanding what caused the particular choice of the n_past value for the new task.

  • Setting LLAMA_SERVER_SLOTS_DEBUG=1 env will make the /slots endpoint output a more detailed output containing the prompt and the generated text of the current or last task. This is useful for debugging purposes.

Note: mtmd workarounds are starting to cause some headaches. For example server_tokens is not copyable which complicates the cache logic and makes the prompt caching feature incompatible with mtmd.

Usage

# use 8192 MiB of host RAM for caching prompts
llama-server ... --cache-ram 8192

# use as much host RAM is available (i.e. no limit)
llama-server ... -cram -1

# disable prompt caching in RAM
llama-server ... -cram 0

Server refactor

  • Replace server_slot members with a single server_task
  • Remove server_slot.n_predict
  • Remove prompt truncation logic (obsolete and not useful anymore)
  • slot.task is now const ptr to reflect that the task parameters should not change when it is passed to the slot
  • Bump default context checkpoints from 3 to 8

TODOs

  • Set memory limit for the host-memory cache from CLI
  • Clean-up implementation
  • Test with agentic workflows
  • Multi-slot tests
  • Fix progress report

@ggerganov
ggerganov force-pushed the gg/prompt-cache-ext branch 2 times, most recently from 0787f03 to 5c0cec4 Compare October 3, 2025 18:49
@tommarques56

This comment was marked as spam.

@ggerganov
ggerganov force-pushed the gg/prompt-cache-ext branch from 5c0cec4 to 1440ec5 Compare October 7, 2025 07:40
@ggerganov
ggerganov changed the base branch from master to gg/server-checkpoints-improve October 7, 2025 07:41
@github-actions github-actions Bot added the python python script changes label Oct 7, 2025
@ggerganov ggerganov mentioned this pull request Oct 7, 2025
3 tasks done
@ggerganov
ggerganov force-pushed the gg/prompt-cache-ext branch from 9de8392 to cf7dd4b Compare October 7, 2025 15:09
@ggerganov

Copy link
Copy Markdown
Member Author

Looking for some feedback of how this new logic performs in different use cases. I've been testing it with the llama.vscode agent and it significantly improves the experience since we can now use a single server slot without trashing the prompt cache.

The current implementation should work with any model (dense, MoE, SWA, SSM, etc.). I think the default settings should be good for most use cases, though we'll probably add some options to adjust cache limits if needed.

Pay attention to these new messages in the logs:

image

Interested in testing agentic use cases, such as Claude Code and similar, where we have a single large context with various auxilary calls (keyword extraction, summarization, etc.) interleaved. The expectation is that prompt reprocessing should be significantly reduces in such cases.

Base automatically changed from gg/server-checkpoints-improve to master October 8, 2025 07:57
@ggerganov
ggerganov force-pushed the gg/prompt-cache-ext branch from 65e8991 to 264d2c3 Compare October 8, 2025 08:24
@ggerganov
ggerganov marked this pull request as ready for review October 8, 2025 12:53
@ggerganov
ggerganov requested a review from ngxson as a code owner October 8, 2025 12:53
trestoncuzzort added a commit to trestoncuzzort/dawnr that referenced this pull request Oct 2, 2026
…, github.com/ggml-org/llama.cpp/pull/16391); the student's server reached 10.8 GB with the default
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
* minor : code style

* server : fix prompt similarity calculation

* server : initial host-memory prompt caching

* cont

* server : refactor

* cont

* cont : make the server task of the slot const

* cont : minor [no ci]

* server : cache prompts and checkpoints only for completion tasks

* server : improve prompt caching logic

* cont : fix check for number of cached prompts [no ci]

* server : improve caching logic, add -cram CLI arg

* server : print prompt mismatch info

* cont : better naming [no ci]

* server : improve prompt cache loading logic

* server : add option to debug the slot contents (ggml-org#16482)

* server : add option to debug the slot contents

* Update tools/server/server.cpp

---------

Co-authored-by: Xuan-Son Nguyen <[email protected]>

* server : add option to disable prompt cache

---------

Co-authored-by: Xuan-Son Nguyen <[email protected]>
trestoncuzzort added a commit to trestoncuzzort/dawnr that referenced this pull request Oct 6, 2026
…, github.com/ggml-org/llama.cpp/pull/16391); the student's server reached 10.8 GB with the default
trestoncuzzort added a commit to trestoncuzzort/dawnr that referenced this pull request Oct 18, 2026
…, github.com/ggml-org/llama.cpp/pull/16391); the student's server reached 10.8 GB with the default
Vect0rM added a commit to AtomicBot-ai/atomic-chat-core that referenced this pull request Nov 3, 2026
…#68)

* fix(hermes): patch only the top-level model block of config.yaml

The writer rewrote the first `default:`, `provider:` and `base_url:` lines
anywhere in ~/.hermes/config.yaml. Hermes repeats those keys under
auxiliary, delegation, cron, moa, tts, memory and others, so on any config
where one of them comes before `model:` (or where `model:` is a scalar or
missing) the patch landed in another block, sometimes breaking the YAML,
and left the chat route untouched. Hermes then ran on its built-in
defaults (empty model, provider auto), fell through to OpenRouter and got
"HTTP 400: No models provided" before any request reached Atomic Chat.

Edit only direct children of the top-level `model:` key: replace
`default`, `provider` and `base_url` in place (dropping a nested value a
scalar replaces), add the missing ones at the block's indent, turn a
scalar `model: <id>` into a block, and append a model section when there
is none. Also never add a second top-level `providers:` or
`custom_providers:` key (Hermes' ruamel reader rejects duplicate keys and
then runs on defaults), and recognise those keys with a trailing comment.

Golden fixture outputs are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(server): turn llama.cpp thinking off for Anthropic requests that ask for none

The Anthropic API thinks only when a request sets `thinking` to enabled or
adaptive. llama.cpp's /v1/messages maps only `enabled` and otherwise leaves
the chat template at the server default, which --jinja turns on for every
template that can think. Claude Code's session-title request (no `thinking`,
max_tokens 32000) made Qwen3.5 4B think for 20k tokens, and on a one-slot
server the user's own turn waited behind it (QA B08).

For a /messages request to a llama.cpp engine (llamacpp, llamacpp-upstream,
atomic-prism) that has no `thinking` or `thinking: {type: "disabled"}` and
does not set the switch itself, the core now adds
`chat_template_kwargs: {enable_thinking: false}`, which llama.cpp's Anthropic
conversion passes to the template. The chat-completions fallback gets the
same kwargs. Requests that ask for thinking pass byte for byte; other
engines are untouched. The proxy-http contract records the divergence.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(server): pin the byte-exact relay of llama.cpp streams to usage-asking clients

B11 (Kilo Code showed no reply while the inspector recorded a completed
stream): Kilo (AI SDK openai-compatible with includeUsage) and OpenClaude
both send stream_options.include_usage themselves, so the proxy neither
rewrites their request nor strips anything from the reply. Pin that with a
llama.cpp b12019-shaped thinking-model reply written in small pieces cut
mid-line and between an event's two newlines: the client gets the upstream
bytes unchanged, watched or not, and a client that did not ask loses only
the usage trailer the proxy asked for, with the finish chunk and [DONE]
intact across split reads.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(server): keep max_tokens on a llama.cpp /messages fallback

A /messages request that llama.cpp's template rejects (Claude Code sends
mid-conversation system messages, which Qwen3.5's template refuses) is
retried as chat completions. The translation dropped max_tokens, as the
Rust proxy did, so on a server started with n_predict -1 the reply ran
until the context was full. llama.cpp's own /v1/messages honours the cap;
its fallback now keeps it too. Other engines keep the translation as it
was. The proxy-http contract records the divergence.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(server): merge mid-conversation system messages before llama.cpp sees them

Claude Code sends its reminders as role "system" messages between turns
to a model it does not know. llama.cpp's /v1/messages passes them to the
chat template where they stand, and a strict template (Qwen3.5: "System
message must be at the beginning.") answers 500, so every later turn went
through the chat-completions fallback. That fallback merges the system
messages into the leading one but loses the reply's token usage (Claude
Code reported input_tokens 0 and never learned how full its context was)
and the turn's thinking blocks.

For a /messages request to a llama.cpp engine, the core now merges those
messages into the top-level system the same way before forwarding, so the
engine answers natively. Other engines and conversations without system
messages are sent as they came.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(integrations): hand the served context window to the agent writers

Agents told nothing about the window assume 128K-272K and never compact
before a 16K-64K local server refuses. ConfigureInput gains an optional
contextLength, and `atomic-chat launch` reads the window the session was
loaded with from its /props (falling back to the asked size without fit).
The writers use it in follow-up commits.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): Zed's real config dir and the served window for Zed and ZCode

- zedConfigDir(): %APPDATA%\Zed on Windows, $XDG_CONFIG_HOME/zed on
  Linux, ~/.config/zed on macOS, as Zed resolves it (parity with the
  app's zed_config_dir).
- Zed: max_tokens = served window, max_output_tokens = min(8192, ctx/4),
  images: false (Zed's own default); 32768 when the window is unknown,
  since Zed requires a value.
- ZCode: contextWindow = served window, omitted when unknown so ZCode's
  default stands; output cap min(8192, ctx/4).
- Both keep the user's default model they replace in a
  <config>.atomic-chat-backup.json sidecar for the app's Disconnect.
- Golden zed fixtures: images false, and the sidecar where the seed had
  another provider as the default model.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): find Zed and ZCode Preview off PATH in `launch`

Mirrors the app's zed_cli_candidates / zed_app_candidates and the ZCode
Preview build of zcode_app_candidates_for.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): tell Kilo, OpenCode, MiMo and pi the served window and keep what Connect replaces

The OpenCode-family writers now describe the model the way these agents
need for a local server:

- `limit: { context, output: min(8192, ctx/4) }` when the window is known
  (omitted when not), so OpenCode and Kilo compact instead of running with
  `limit.context = 0`, and MiMo stops assuming a 1M window;
- `reasoning: true` and `interleaved: { field: "reasoning_content" }`, so
  a thinking model's answer is shown as thoughts plus text and its
  reasoning is sent back within a tool loop (B11);
- provider options `timeout: false, headerTimeout: false`, verified in
  the Kilo, OpenCode 1.15 and MiMo schemas: a long prompt on a slow
  machine no longer hits the 5-minute first-byte timeout.

pi gets `name`, `contextWindow`, `maxTokens` and a per-model
`compaction.modelOverrides["atomic/<id>"]` (reserve and recent history a
quarter of the window), the schema documented in pi's settings reference.

OpenCode and MiMo are read as JSON5 like Kilo (comments and trailing
commas are dropped on write); an existing `opencode.jsonc` /
`mimocode.jsonc` is written instead of the `.json` it overrides, and our
leftovers in that `.json` are removed. KILO_CONFIG_DIR, OPENCODE_CONFIG,
OPENCODE_CONFIG_DIR, MIMOCODE_HOME, PI_CODING_AGENT_DIR and
XDG_CONFIG_HOME are honoured. MiMo's `$schema` is the one MiMo serves.

Before `model`, `provider.atomic`, pi's default provider/model or its
`providers.atomic` are overwritten, a value that is not ours is kept in
`<file>.atomic-chat-backup.json` for the app's Disconnect.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): run Goose with its own environment, start Kilo on our model

Goose no longer gets OPENAI_API_KEY, OPENAI_HOST and GOOSE_* written
user-wide (shell rc block, `setx`), which pointed every OpenAI SDK tool on
the machine at the local server. `launch goose` sets them on the child
only: GOOSE_PROVIDER, GOOSE_MODEL, OPENAI_BASE_URL with the server's API
prefix (instead of the bare host plus a hard-coded OPENAI_BASE_PATH),
OPENAI_API_KEY, and GOOSE_CONTEXT_LIMIT / GOOSE_MAX_TOKENS when the
window is known (names checked against goose 1.57). An ambient
OPENAI_HOST / OPENAI_BASE_PATH is cleared, since Goose prefers it to
OPENAI_BASE_URL. Configuring Goose now only removes what earlier
releases wrote: our marked block in .zshenv/.bashrc/.bash_profile (our
export lines only) and, on Windows, the user variables recognisably set
by `setx`.

Kilo's saved per-agent pick beats `model` in its config, so `launch kilo`
passes `--model atomic/<id>` (the TUI's own flag).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(contract): agent-config fixtures for the new Kilo, OpenCode, MiMo, pi and Goose output

Regenerated by the app's Rust writers (`opencode_family.rs`
`dump_family_fixtures`) and replayed byte for byte here: the model entry
with thinking, no timeouts and the window, pi's window and compaction
override, the backup files Connect leaves for Disconnect, the `.jsonc`
target with its `.json` cleaned, MiMo's corrected `$schema`, and Goose
writing nothing but removing what earlier releases wrote. Cases may carry
`context_length` (index note). CHECKSUM recomputed over this tree.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(models): count sliding-window KV layer by layer, with a window per slot

The KV estimate averaged a sliding-window model over every layer,
(full + window) / 2. llama.cpp b12019 keeps the whole window only in the
global layers and pad256(n_swa x sequences + n_ubatch) cells in each
sliding one (src/llama-kv-cache-iswa.cpp:73), so Gemma 3 was counted at
two and a half times its real cache at 32K and small Macs were narrowed
far more than needed.

The estimate now reads the per-layer sliding_window_pattern (inlined by
the GGUF reader at any length, like head_count_kv), a period from the
GGUF, or the architecture's built-in pattern (gemma2/3/3n, gpt-oss,
cohere2, exaone4, afmoe, smallthinker); Gemma 4's own SWA head sizes and
shared KV layers count too. An architecture with a window but no known
pattern is counted in full, the safe side. A new layout argument gives
the sequences sharing a unified cache and the micro-batch size.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(llamacpp): agent-friendly defaults: shared KV slots, answer and thinking caps

Integrations audit 2026-11-03 (S3, S4, S5, X6, P3), llama.cpp b12019:

- Slots: -kvu is always sent, so --parallel N shares one KV pool of the
  whole window (src/llama-context.cpp:389-390) and costs no window memory
  beyond each sliding layer's per-slot window. parallel defaults to 2, so
  a title or classifier request no longer waits behind an agent turn.
  The fit margin and the small-Mac ceiling no longer multiply the KV
  cache by the slots (load-plan.ts); the ceiling is the pool itself.
- Continuous batching is sent both ways (--cont-batching /
  --no-cont-batching): llama.cpp has it on unless told otherwise, so the
  setting could never turn it off. Its default is now on, as llama.cpp's.
- n_predict 0 is automatic: 16384 tokens, never above the window core
  knows. Clients without max_tokens (Codex via Responses, Kilo, OpenCode)
  no longer generate until the window is full. -1 stays "no limit".
- New reasoning_budget setting (default 8192) sent as --reasoning-budget
  with --reasoning-budget-message from b8287, the first build that takes
  a token count (ggml-org/llama.cpp#20297); requests' own budgets win.
- --cache-ram 1024/2048 MiB on machines with at most 8/16 GiB of RAM
  (from b6723, ggml-org/llama.cpp#16391) instead of llama.cpp's 8 GiB.
- DEFAULT_CTX_LEN is 32768: a fit-off load without a window gets 32K (or
  the trained context) instead of the whole trained context, and MLX's
  default window is 32K; on a Mac below 16 GiB both are narrowed toward
  the previous 16K by the KV estimate, never below it.
- isContextOverflowFinishLength takes the engine's own answer cap into
  account: a length stop at --n-predict is no longer a context overflow.

The args fixtures from the Rust builder are compared without -kvu and
--no-cont-batching, recorded as deliberate divergences; the engine
contract gains the new flags and build gates.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(managed): automatic context length and request count for vLLM and SGLang

Integrations audit 2026-11-03, S2: vLLM and SGLang defaulted to 8192
tokens and one request, so every coding agent's 13-27K-token prompt was
refused with context_length_exceeded and side requests queued behind the
agent's turn.

context_length and max_num_seqs / max_running_requests now default to 0,
automatic: the load picks the widest pair the engine's own memory rule
passes on the card it re-probes right before the container — 64K with
two requests, 32K with two, 32K alone, 16K, then the old 8K alone, at
which the memory check refuses exactly as before. The window never
exceeds the checkpoint's own (max_position_embeddings and the other keys
vLLM derives it from), leaves room for the output cap, and with a KV
size fixed by hand holds one full sequence of it. Both engines keep one
paged pool sized by their memory share, so the window only bounds a
request. The chosen pair reaches the container through the launch plan
and the session limits (/v1/models context) through planContextLength.
Values set by hand stay as they are.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(settings): move stored values off the defaults core no longer ships

settings.json stores every provider value, defaults included, so a new
schema default reached fresh installs only. On open, the store now moves
a stored value that still equals an old default to the new one, once,
and records the migration under state.default_migrations; a value the
user sets afterwards, the old default included, stays. A moved value
bumps the revision so the app mirrors it.

Migrations (integrations audit 2026-11-03): llama.cpp parallel 1 -> 2,
cont_batching false -> true (llama.cpp always batched continuously),
n_predict -1 -> 0 (automatic cap); vLLM/SGLang context_length 8192 -> 0
and request count 1 -> 0 (automatic).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): keep Claude Code and OpenClaude writers in step with the app

The `launch` writers for Claude Code and OpenClaude now match the app's
claude_config.rs: each keeps a `<file>.atomic-chat-backup.json` sidecar
recording what every key held before, so the app's Disconnect can undo a
connection the CLI made; a reconnect undoes the previous one first, and
values an earlier release wrote are taken out rather than recorded.

Claude Code: empty CLAUDE_CODE_USE_BEDROCK/VERTEX/FOUNDRY, the terminal
title off, and with a known window CLAUDE_CODE_MAX_CONTEXT_TOKENS plus
CLAUDE_CODE_MAX_OUTPUT_TOKENS (min(8192, ctx/4)); $CLAUDE_CONFIG_DIR is
honoured.

OpenClaude: the profile carries the server's key again (OC-2: the
keyless profile got a 401 from a keyed server), maxContextLength and a
per-model output cap with a known window, empty values for the shell
variables that make OpenClaude skip its active profile, and the startup
profile its OPENAI_API_KEY. Our profile is matched by its id only, and
$OPENCLAUDE_CONFIG_DIR and the legacy .config.json are honoured.

Golden fixtures for both agents regenerated; the in-place replacement
case became openclaude__users_own_atomic_chat_profile_kept.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(server): keep mid-conversation system messages in place for llama.cpp

6f87e30 moved every system message inside an Anthropic conversation into
the top-level system. That rewrites the start of the prompt whenever Claude
Code sends a reminder, and llama.cpp then processes the whole conversation
again (13-27k tokens on hybrid and sliding-window models, whose prompt
cache cannot reuse anything past the change).

For a llama.cpp engine, only system messages before the first turn now
join `system`. A later one stays where it is, as user text wrapped in
<system-reminder>, merged into a neighbouring user turn so the roles still
alternate: before the text of the next user message, or at the end of its
last tool_result when it answers tool calls (llama.cpp puts a user
message's text before its tool results, so a text block there would land
between the call and its result). With no user message after it, it goes
to the end of the one before, or into a user message of its own. A
reminder sent with clear_at "next_user_message" is dropped once a user
message follows it. The chat-completions fallback gets the same messages.

The Responses shim (Codex to llama.cpp) placed developer and system items
the same hoisting way; it now keeps later ones in place too, never between
a tool call and its output. The shim contract case that recorded the Rust
hoisting is corrected as a known divergence.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(server): advertise the served window and answer overflows as prompt too long

Agents did not know the window the local engine really serves, so none
of them compacted in time (audit X1). GET /v1/models now gives each local
model whose window is known `context_length` and `meta.n_ctx` (the field
llama-server's own list uses), and Muse Code's catalogue advertises the
real window instead of its 32k guess.

The window comes from the session: a declared session (vLLM, SGLang,
TensorRT-LLM) states it; a llama.cpp session is asked once per loaded
generation through /props (default_generation_settings.n_ctx, answered
from metadata without a slot). The core wires SessionWindows into the
server as `contextWindow`; a session target now carries its generation.
Other engines advertise none rather than a guess.

On /v1/messages, a llama.cpp context overflow (exceed_context_size_error
with n_prompt_tokens/n_ctx, or its message) on the native call or on the
chat-completions fallback is answered as the Anthropic API's 400
invalid_request_error "prompt is too long: N tokens > M maximum", the
wording Claude Code compacts on; the fallback is no longer tried for it.
OpenAI routes keep llama.cpp's own error.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(server): keep each request's context window and cached prompt tokens

The Activity inspector showed the window of whatever session served the
model when the row was opened, not the one the request ran with (QA B04).
The trace now records the window of the session a request goes to when it
is routed (the session's declared window or the one SessionWindows read)
and sends it as `ctx_len` in the finish fields, so the log keeps it after
a reload or an unload.

The finish fields also carry `cached_tokens`, the part of the prompt the
engine reused, from llama.cpp's timings.cache_n, OpenAI's
prompt_tokens_details.cached_tokens and Anthropic's
cache_read_input_tokens. Prompt tokens now count the whole prompt:
Anthropic's input_tokens leaves out cache reads and writes, and llama.cpp's
timings.prompt_n only what it processed. A whole Anthropic reply is read
for usage and stop reason too. The Responses shim carries the cached part
into input_tokens_details instead of a hard-coded 0.

The inspector-telemetry contract records the new field as a known
divergence (null in every recorded case).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(server): give Codex's MCP namespace tools to llama.cpp

Codex sends the tools of each MCP server as a Responses namespace
({type: "namespace", name, tools: [...]}) and runs a call to one only
with that namespace. The Responses shim kept only top-level `function`
tools, so a local model never saw a Codex MCP tool (audit S7).

The shim now offers every function in a namespace as a chat function
named <ns>__<name> (the plain name in Codex's default `functions`
namespace; no second delimiter after a namespace that already ends in
one), described by its own description or else the namespace's. A
function_call item replayed from history and a namespaced tool_choice
use the same name. In the reply, whole or streamed, a call to such a name
is given back its `name` and `namespace`, so Codex routes it to the MCP
server. Freeform tools inside a namespace are dropped like top-level ones.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(server): cover the system-reminder wrapper

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): tell Codex the served window in its profile

With the session's window known, the Atomic Chat profile carries
model_context_window and model_auto_compact_token_limit (three quarters
of it), so Codex compacts before a 16K-64K local server refuses instead
of assuming 272K. Unknown, neither line is written. Byte-identical to
the desktop writer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): Cline, Droid, Copilot and dsh get the window and a backup

Ports the desktop writers (app wip/fam-codex), byte for byte:

- A known window is written with a reply cap of a quarter of it (8192
  at most), and nothing is invented when it is unknown: Cline's
  contextWindow/maxTokens on the entry `cline auth` saved, Droid's
  maxContextLimit/maxOutputTokens (instead of a fixed 16384), Copilot's
  maxPromptTokens/maxOutputTokens/maxContextWindowTokens, dsh's route.
- Each writer records what it overwrote in a
  <config>.atomic-chat-backup.json sidecar, which the desktop's
  Disconnect restores from.
- Droid names its entry (id, index) and selects it through
  sessionDefaultSettings.model; the legacy top-level selector is removed.
- Copilot writes an atomic-chat provider and model into
  .copilot/providers.json instead of persisting COPILOT_* in the shell
  rc, removes exactly the rc block an earlier release wrote, and
  `launch` selects the model with COPILOT_MODEL=atomic-chat/<model>.
- dsh 0.2 imports settings.yaml once into the profile, so the route and
  the default model go into profiles/web/cordis.patch.yml, re-serializing
  only our rows; a settings.yaml still present is kept in step, never
  created, and settings.yaml.imported is never touched.

The cline/copilot/droid/dsh agent-config fixtures change accordingly
(index note); the desktop writers produce the same bytes for every
copilot, droid and dsh case.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(cli): start npm-shim agents through cmd.exe on Windows

`atomic-chat launch` spawned the detected launcher without a shell. On
Windows an npm-installed agent is a `.cmd` shim, which Node refuses to
spawn directly (EINVAL) and does not find by its bare name (ENOENT), so
every npm agent failed to start. Shims and bare names now run through
`cmd.exe /d /s /c "<line>"` with each token quoted for cmd; executables
and other platforms spawn as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): Hermes, OpenClaw and Atomic Agent writers record what they replace

Hermes: one route for the app and the CLI, bare `custom` with model.base_url,
model.api_key and model.context_length, which needs no custom_providers entry
(an inline list cannot take one) and carries the server key. The served
window is written as served, 65536 only when unknown. HERMES_HOME is honoured
on every platform (~, $NAME, ${NAME}, %NAME% expanded) and the sticky profile
from `hermes profile use` is the one written. No skeleton is seeded.

OpenClaw: the model entry carries contextWindow and maxTokens when the window
is known; the request budget moves to the provider (timeoutSeconds 600) and
the agents.defaults.timeoutSeconds: 240 older releases seeded is dropped;
gateway.auth.mode "none" is seeded only when no token or password is set.

Atomic Agent: llm.fallback.appendLocal false when there is no fallback block,
so a failure of ours is not retried on local-llama :8080; userModels carries
the served window.

Each writer records the keys it overwrote in <config>.atomic-chat-backup.json
(json-key-backup.ts), the same sidecar the app writes, so the app's Disconnect
restores a connection made by `atomic-chat launch`. The golden fixtures of the
three agents are regenerated and replayed byte for byte against the app's
Rust writers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(integrations): OpenHands, Poolside and Muse get their environment at launch only

Their writers persist nothing any more: the variables go on the child
`launch` starts (and inline in the app's terminal). Connect removes the
marked shell-rc block an older release wrote, from any rc file it may have
used, and nothing else; the old prefix sweep that deleted the user's own
`export META_*` / `POOLSIDE_*` / `LLM_*` lines is gone with the writer.

Poolside keeps `/v1` in POOLSIDE_STANDALONE_BASE_URL as its docs give it and
is told the served window (POOLSIDE_STANDALONE_CONTEXT_LENGTH) through
childEnv, which `launch` now passes the session's window to.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(contract): recompute the app fixture checksum

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(integrations): cover the per-key backup and the legacy shell block

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs: record the agent-serving defaults and the window the server advertises

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test: keep context-growth tests independent of host RAM, /props and path separators

- sessions: mock node:os totalmem so the 32K default is not narrowed on
  small CI runners (7 GiB macOS, Linux arm64).
- public-server: load the overflowing model at an explicit 4K window
  instead of relying on the old 8K default.
- api-events: the external engine stub answers the /props window read and
  counts only chat requests.
- launch: ZCode's window is the one /v1/models advertises.
- json-backup: `~/` joins with the platform separator on Windows.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test: bring the new agent writers and overflow mapping up to their coverage floors

The macOS unit job runs test:coverage, which failed its per-file floors
on the code this branch added. Tests now cover:

- goose: every legacy local-host range, the Windows reg/PowerShell path
  and its IO_ERROR, the placeholder vs server key.
- hermes: CRLF input, %YAML directives, wrapped scalars, the providers
  block, the Disconnect backup reader, ~ and %NAME% homes, .env proxy
  bypass.
- opencode-style, openclaude, pi: stale $schema, unreadable backups, the
  sibling .json beside a .jsonc, earlier startup profiles.
- errors: anthropicPromptTooLong from fields and both message wordings,
  MLX KV-cache wordings, clientMaxTokens.
- mlx model-files, fit-margin, catalog: memory facts, shard renames, the
  256-token grid, Zed under LOCALAPPDATA.

goose.ts drops a `?? ''` fallback on a regex group that always matches.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(e2e): give a cold Windows daemon 45 s to print its ready line

On windows-11-arm the first daemon of each e2e file spends 10-15 s in
the PowerShell hardware probe before it is ready (main included); on a
slow runner that crossed the 20 s wait and failed the start-up tests.
Other platforms keep 20 s.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python python script changes server

Projects

None yet

Development

Successfully merging this pull request may close these issues.