Repository navigation
Conversation
Anthropic looks back at most 20 block positions from each breakpoint for an earlier cache entry. When one new message holds 20 or more blocks, such as a prompt with about ten attached images or twenty text files, the tail breakpoint no longer reaches the previous request's entry and the whole conversation is written to the cache again. On Anthropic's own route, auto now also marks the second newest message, which is where the previous request's tail breakpoint sat. The slot comes from the first system part, which only adds the base prompt to the tools prefix; it keeps its breakpoint whenever one is left over, as on a Session's first request, so cross-session reuse is unchanged. Breakpoints are now filled in the order tools, last system part, newest messages, first system part, and a tail longer than the budget keeps its newest messages. Previously the oldest message in the tail took the last slot, leaving the newest content uncached. The long tool turn cassette pins the single-tail placement it was recorded with until it is re-recorded. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01X7arppEkkX1wg1Yy1nq7tw
![opencodejr[bot]](https://avatars.githubusercontent.com/in/5497873?s=60&v=4){kind=small}
There was a problem hiding this comment.
The change looks sound. I copied the new cache-policy.test.ts into the base checkout and ran it: 4 tests fail there, including the 25-block case where only the newest message is marked and the previous entry is out of reach. All of them pass on this branch. The packages/ai tests closest to the change and typecheck:tsc pass too. I could not check it against the live Anthropic API, so the ~56k → ~14k write saving isn't confirmed.
Public API: no schema, config or SDK changes. The only behaviour change is where the default auto policy places breakpoints on the anthropic-messages route, plus the newest-first fill order when an explicit tail runs out of breakpoints. Existing configs keep working.
Nothing blocks a merge. One optional point, inline: Vertex and Bedrock Mantle use Anthropic's caching too, so they probably miss the previous entry in the same way. Also, the recorded test is now pinned to the old one-message setting, so the new Anthropic default isn't covered by a recorded test until it is re-recorded.
Related: #61282
The second newest message is usually the reply the model just generated, not the previous request's tail. In a tool loop that reply often holds only thinking and tool_use blocks, which never carry cache_control, so its hint spent a slot, was dropped when lowering, and pushed out the first system part's breakpoint. With parallel tool calls, the second newest message was another result from the same step. Auto on Anthropic's route now marks the message before the newest assistant reply instead. The previous request put its tail breakpoint there, so the new request reads that entry directly, however many blocks the reply and the new message add. Message hints now go only on blocks a protocol can send a marker on: non-blank text, media and client tool results. A message without one keeps its slot. Anthropic rejects a ttl="1h" marker after a 5m one. An auto hint now takes the longest lifetime of any manual hint after it, so a manual 1h hint on the newest message no longer turns the request into a 400. The prefix up to that hint is written at 1h anyway, so this costs nothing. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01X7arpnFR8AZSgGeYEyZcfh
There was a problem hiding this comment.
Re-reviewed at 1f45968a3. I ran the #61689 reproduction (a mock Anthropic endpoint, two plain prompts, then one prompt with 10 attached images) on both checkouts. On the base, the only conversation breakpoint lands 21 blocks after the previous tail. With this PR, the previous tail (block 2) is marked again, so the entry is read directly. cache-policy, effort-updates and the provider tests pass (1506), and typecheck:tsc passes. I didn't measure real cache usage, since there are no credentials.
The public API is unchanged. Only where the default auto policy places breakpoints changes, plus the internal lifetime upgrade. No schema, config or SDK changes.
Earlier points: leaving out Vertex, Bedrock Mantle and Converse for now is fine given your reasons, so that's resolved. Keeping the cassette pinned to the single-tail setting is fine too. In the new commit, marking the message before the newest reply is correct for tool loops and parallel tool results. Skipping tool-call, reasoning and blank text matches what the Anthropic lowering can actually mark. The lifetime upgrade fixes auto markers placed before a manual 1h one. Nothing blocking.
Related: #61282
The previous commit raised an auto hint to the longest manual hint after it in
request order, but requests could still send a 1h marker after a 5m one, which
Anthropic rejects with a 400:
- A shorter manual hint before an auto hint: `cache: { ttl: "1h" }` with a
manual 5m hint on an earlier message sent the tail and the previous tail at
1h after it.
- Manual tool hints, which only counted against the budget.
- System updates on models that take native system messages. Anthropic
Messages sends an update after the user turn that follows it, so a 1h hint on
the update went out after the 5m marker the cache policy had placed on that
turn.
Request order can't settle this, so the cache policy gives every auto hint the
requested lifetime again, and Anthropic Messages, Bedrock Converse and OpenAI
Chat raise each marker to the longest lifetime after it once the body is
built. Raising never moves the last 1h marker, so it costs nothing, and it also
repairs manual hints the caller placed out of order.
A wrapped system update kept only its last part's hint, so the tail breakpoint
the cache policy puts on its last non-blank part was dropped when a blank part
followed. The wrapped block now takes the last hint on any of its parts.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01X7arpmRb395Ye7ukWMUdSp
There was a problem hiding this comment.
Re-reviewed the new commit 98a6ec9b8. The #61689 reproduction (mock Anthropic endpoint, two prompts, then 10 attached images) still marks the previous tail at block 2 next to the new tail at block 23. The full packages/ai suite passes (2059), and so does typecheck:tsc. Nothing was measured against the live API.
The public API is unchanged: the only changes are where auto places breakpoints and how cache lifetimes are ordered in the request body. No schema, config or SDK changes.
Ordering lifetimes once the body is built looks right. Every marker it changes in place is a fresh object on each route (Anthropic, the shared cacheControl() used by OpenRouter, Venice, Alibaba and DigitalOcean, and Bedrock cache points), so nothing shared is changed. Bedrock models without 1h support never emit a 1h marker for it to raise. Wrapped system updates taking the last hint on any part is also correct.
One note for maintainers rather than a blocker: the lifetime ordering now touches three protocols and only affects callers that set manual hints with mixed lifetimes. That goes beyond #61689 and could ship separately if preferred. Nothing blocking.
Resolves the conflict with anomalyco#60971 in the long-tool-turn cassette test: it keeps the pinned single-tail cache policy and takes the flattened maxTokens and temperature. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01X7arpmRb395Ye7ukWMUdSp
There was a problem hiding this comment.
The new commit only merges dev. The PR's own changes are the same as at 98a6ec9b8, and the recorded Anthropic cache test now sits alongside the top-level maxTokens and temperature fields that came from dev. On the merged head, the full packages/ai suite passes (2060), typecheck:tsc passes, and the #61689 reproduction still marks the previous tail (block 2) next to the new tail (block 23). The public API is unchanged. No earlier threads are open, and there is nothing new to raise.
|
Checked the current head, Could a maintainer approve these runs so CI can validate the current head? I don't have Actions write access on this repository. |
Issue for this PR
Closes #61689
Type of change
What does this PR do?
On Anthropic's own route, auto now also marks the message before the newest assistant reply. The previous request ended there and wrote its tail entry there, so the new request reads that entry directly, however many blocks the reply and the new message add. Before, a prompt with 20+ blocks (10 attached images, say) left the only conversation breakpoint outside the 20-block lookback, and the whole conversation was written again.
The slot comes from the first system part, which only adds the base prompt to the tools prefix. It still gets a breakpoint whenever one is left over, as on a session's first request, so cross-session reuse doesn't change. Anthropic-compatible routes keep the current placement, since they may bill breakpoints differently.
Two smaller fixes in the same code, for every route:
tool_useblocks no longer takes a slot whose marker is dropped when lowering. A wrapped system update takes the last hint on any of its parts, so a trailing blank part doesn't drop the tail breakpoint either.ttl: "1h"marker after a 5m one with a 400. Anthropic Messages, Bedrock Converse and OpenAI Chat now raise each marker to the longest lifetime after it once the body is built, in the order the provider reads them. Only the finished body has that order: on models that take native system messages, Anthropic Messages sends a system update after the user turn that follows it. Raising never moves the last 1h marker, and the prefix up to it is written at 1h anyway, so this costs nothing. It also repairs manual hints placed out of order.Breakpoints are filled in the order tools, last system part, conversation tail, previous tail, first system part. A tail longer than the budget keeps its newest messages; before, the oldest message in the tail took the last slot and the newest content stayed uncached.
The first commit marked the second newest message instead. That is usually the reply the model just generated: in a tool loop its hint landed on a
tool_useblock and was never sent, and with parallel tool calls it marked another result of the same step. The second commit fixes that and raised auto lifetimes before a manual hint in request order. The third commit moves lifetime ordering to the finished body. Request order had missed a shorter manual hint before an auto marker, manual tool hints, and a system update moved after the next user turn, and each of these still sent a 1h marker after a 5m one.Mixed lifetimes need manual hints. OpenCode itself doesn't set any (Core passes one request-level TTL), but plugins and direct
@opencode/aicallers can.How did you verify your code works?
bun test(2059 pass),bun typecheckandbun run typecheck:tscinpackages/ai; oxlint and prettier on the changed files.Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_01X7arpnFR8AZSgGeYEyZcfh
https://claude.ai/code/session_01X7arpmRb395Ye7ukWMUdSp
Added by the opencode bot (AI-generated).
Related: #61282