Session warming misses the cache it is meant to keep, and warms providers that don't need it #61775

Activity

1. opencodejr commented on Nov 4, 2026

   

   opencodejrBot

   on Nov 4, 2026

   Contributor

   More actions

   Thanks for the detailed write-up. I reproduced the timing problem (point 2) on 2.2.5 and on the latest dev (bd420e0).

   What I did, using the simulated model:

   • Set warming: { interval: "10 seconds", duration: "1 minute" }, standing in for the default 4 minutes.
   • Sent one prompt whose response streams for about 15s.
   • Logged when each request was sent.

   The step request went out at 1.7s and the response finished at 16.7s. The warm went out at 26.7s, which is 25s after the step's request. So the interval runs from the end of the response, not from when the prompt was read. With the default 4 minutes and a 5-minute TTL, any step streaming longer than about a minute makes the first warm arrive late. With "interval": "auto", the warm is late whenever the step takes more than 10% of the TTL. The script below reproduces it.

   On point 1:

   • In a plain session the warm request had the same system prompt, tools and history prefix as the step's request. Only the output limit differed, and the keep-alive prompt was added at the end.
   • The goal case I confirmed in the code but didn't drive end to end. The goal reminder is added only in the context hook, while the warm goes through generate. So whenever the step carries the reminder (for example after compaction), the warm's prefix differs.

   I couldn't measure real cache hit rates without a provider account. Stopping after a miss, a shorter idle window and skipping implicit-cache providers would change how warming is designed, so maintainers will need to weigh those separately.

   Related: #43628 (Gemini/Vertex misses with the 4-minute interval), #60969 ("auto" interval) and #61724 / #61737 (one warming loop per session).

   Reproduction script (drive.ts)

   // Shows when session warming fires relative to the step's model request, and diffs the two requests.
   // Config: warming interval 10s (stands in for the default 4 minutes). The step's response streams for ~15s.
   // Run: OPENCODE_DRIVE_MEDIA_DIR=/workspace/media opencode-drive run drive.ts   (OC_DEV=<checkout> to run from source)
   import { Effect, Stream } from "effect"
   import { OpenCodeDriver, Llm } from "opencode-drive"
   import { writeFileSync } from "node:fs"
   
   const log: Array<{ at: number; event: string; body?: any }> = []
   const start = Date.now()
   const mark = (event: string, body?: unknown) => log.push({ at: (Date.now() - start) / 1000, event, body })
   
   export default OpenCodeDriver.use(
     {
       ...(process.env.OC_DEV ? { opencode: { dev: process.env.OC_DEV } } : {}),
       config: { autoupdate: false, warming: { interval: "10 seconds", duration: "1 minute" } },
       tui: { recording: true },
     },
     ({ ui, llm }) =>
       Effect.gen(function* () {
         let step = false
         yield* llm.serve((request) => {
           const body = request.body as any
           if (!body?.tools) return Stream.make(Llm.text("Title"))
           if (!step) {
             step = true
             mark("step request sent", body)
             // A long-streaming step response: ~15 seconds.
             return Stream.make(Llm.text("Step answer part 1. "), Llm.pause(15_000), Llm.text("Step answer done.")).pipe(
               Stream.concat(Stream.fromEffect(Effect.sync(() => mark("step response finished"))).pipe(Stream.drain)),
             )
           }
           mark("warm request sent", body)
           return Stream.make(Llm.text("OK"))
         })
         yield* ui.submit("Say hello")
         yield* Effect.sleep(32_000)
         const out = process.env.LOG ?? "/workspace/repro/log.json"
         writeFileSync(out, JSON.stringify(log, null, 2))
         for (const e of log) console.log(`${e.at.toFixed(1)}s  ${e.event}`)
         const [s, w] = log.filter((e) => e.body).map((e) => e.body)
         if (s && w) {
           console.log("warm - step request gap:", (log.find((e) => e.event === "warm request sent")!.at - log[0]!.at).toFixed(1), "s")
           console.log("system/tools identical:", JSON.stringify(s.messages[0]) === JSON.stringify(w.messages[0]), JSON.stringify(s.tools) === JSON.stringify(w.tools))
           console.log("max tokens: step", s.max_completion_tokens, "warm", w.max_completion_tokens)
           console.log("warm tail:", JSON.stringify(w.messages.at(-1)).slice(0, 160))
         }
       }),
   )

   Run it with opencode-drive: opencode-drive run ./drive.ts. It drives opencode from your PATH; set OC_DEV to the path of an OpenCode checkout to run that from source instead.
2. opencodejr

   added

   reproduced

   on Nov 4, 2026
3. houleiyx commented on Nov 5, 2026

   

   houleiyx

   on Nov 5, 2026

   More actions

   The timing repro is useful. For the proposed fix, I’d also check one real Anthropic session with a goal set: capture the normal request, its warm request, and
   the next normal request, then compare their system prompt, tools, cache breakpoints, and cache read/write usage. That should show whether the warm actually
   preserves the prefix the next step uses.

   I work on ccglass. If the provider’s base URL can be pointed through its local proxy, its request diff and usage view
   can help with this check. A redacted comparison of those three requests would be more useful here than sharing the full prompt.
4. cris005 commented on Nov 6, 2026

   

   cris005

   on Nov 6, 2026

   More actions

   I maintain a plugin that does its own keepalive (opencode-cache-manager), and it already handles the points you listed, so it may be a useful reference for the fix:

   • The ping is aligned in the http.request hook. Its message history is replaced with the exact messages of the session's last request, cache_control included, so anything a context hook added to the history is in the ping too.
   • The timer runs from when the last request was sent, not from the end of the response.
   • A ping counts as a hit if it reads at least 90% of the previous context from cache. If one misses, it stops pinging that session until the next real request, so a miss costs one ping.
   • It stops after a set number of pings per idle stretch (5 by default), and it skips sessions under a minimum context size (50k tokens by default). Both are configurable per session.
   • It only pings models it sets breakpoints on itself: Claude, and GPT-5.6+ with explicit breakpoints. It doesn't ping other models at all.

   It also keeps a per-session savings ledger in the sidebar. It weighs what the pings cost, wasted ones included, against the rewrites they avoided when you came back, so you can see whether keepalive is paying off on that session and turn it down or off if it isn't.