Skip to content

Commit 7cc1126

Browse files
atishpatelclaude
andauthored
feat(agent): fix Anthropic prompt caching with Databricks (+ MCP proxy/TLS passthrough) (#3463)
> On 8 tasks matched by name across the two runs, cost fell $8.36 → $1.77 (4.71×) and wall-clock 12,423 s → 1,085 s (11.45×). ## Summary Two independent, self-contained fixes to `buzz-agent`/`buzz-acp`, split out of the benchmark branch so they can land while the harness work continues: 1. **Request and surface Anthropic prompt caching.** buzz never sent a `cache_control` breakpoint, so on the Databricks Anthropic route `cache_read_input_tokens` was **structurally always 0** and the ~10× cache-read discount was never claimed. This teaches `anthropic_body()` to mark the cacheable prefix, and plumbs the cache split end-to-end so accounting can price it. 2. **Pass proxy + TLS-trust env into MCP tool subprocesses**, so agent tools on a proxy-only host stop reporting a live network as offline. ## Why the caching gap matters The Anthropic Messages API does **not** cache unless the request carries a `cache_control` breakpoint, and the Databricks AI Gateway — a third-party proxy in front of the model, in the same category as Bedrock/Vertex — does **not** auto-cache (only the first-party Anthropic API and Claude-on-AWS do zero-config caching). So every request was billed cold. Measured live against the Databricks gateway (`databricks-claude-opus-5`, 2026-07-28), the same call with and without a single `cache_control` marker: | Run | `input_tokens` | `cache_creation` | `cache_read` | latency | |---|---|---|---|---| | No `cache_control`, two byte-identical calls | 121,625 | 0 | **0** | ~9.3 s | | With one marker — cold (write) | 4 | 121,625 | 0 | 9.3 s | | With one marker — warm (read) | 4 | 0 | **121,625** | **4.5 s** | One marker moved 121,625 tokens from full-price input to a 0.1× cache read and roughly halved latency (a clean, isolated ~2.07× prefill speedup on this single-threaded microbenchmark). The gateway honours `cache_control`; buzz simply never sent it. At fleet scale this was a real budget item. Across matched Terminal-Bench solo sweeps (89 tasks, `-n 20`, before the fix), the two OpenAI-route models independently landed at ~86–87% cache reads — the expected shape for an agentic loop, where system + tools + append-only history repeat every turn — while the Anthropic route returned a hard 0% on every receipt: | Condition | Route | Input tokens | Cache reads | Cost | Cost if uncached | Discount | |---|---|---|---|---|---|---| | luna (`gpt-5-6`) | OpenAI | 20,320,818 | **17.7M (87.0%)** | $6.96 | $22.87 | **3.28×** | | sol (`gpt-5-6`) | OpenAI | 22,312,290 | **19.2M (85.9%)** | $37.07 | $123.35 | **3.33×** | | opus (`claude-opus-5`) | Anthropic | 12,459,822 | **0 (0.0%)** | $81.31 | $81.31 | **1.00×** | Applying luna's measured 87% read rate to the opus token counts at list prices (`input $5/M`, `cached_input $0.5/M`, `output $25/M`) puts the opus run at **~$32.53 vs the $81.31 actually paid — a ~60% overspend on those 49 trials (~$89 on a full sweep)**. That is an upper bound (it prices every cached token at the 0.1× read rate and ignores the 1.25× write premium), and the opus discount is structurally smaller than luna/sol's because opus emits ~3.5× more uncacheable output per trial, which sets a floor on what caching can recover. There is also a plausible **second-order effect**: Databricks appears to meter its per-minute rate limit on *uncached* input tokens, so the missing cache also cost rate-limit headroom — the opus endpoint lost 63% of its trials to fatal 429s while running alone at one-third of a GPT endpoint's raw throughput. This is a hypothesis, not a proven mechanism (the only zero-cache condition is also the only Anthropic endpoint), but it is the reading that explains the throttling with one rule instead of two. ## Post-fix results (provisional — first trials of an in-flight re-run) On 8 tasks matched by name across the two runs, cost fell **$8.36 → $1.77 (4.71×)** and wall-clock **12,423 s → 1,085 s (11.45×)**. | Metric | before (`4a955a858`) | after (`3bef1f6a`) | |---|---|---| | Cache reads as % of input | **0.0%** | **78.7%** (still climbing toward the ~86% steady state) | | `cost_usd_no_cache_discount / cost_usd` | **1.00×** | **2.18×** (tracking the projected ~2.5×) | | Trials with a fatal 429 (same `-n 20`) | **63%** | **15–19%** | To be clear about attribution: **~2× of that is the clean prefill saving from caching itself**; the rest is second-order — cached requests burn far less rate-limit budget, so they stall less and redo less destroyed work. The 11.45× is a system-level result specific to this throttled workspace, not a caching benchmark. Quality held (7/8 solved in each run). A controlled low-`-n` A/B (neither arm hitting a 429), which the `BUZZ_AGENT_PROMPT_CACHING` opt-out exists to enable, is still owed before this becomes a published claim. ## What changed ### 1. Request caching (`llm.rs`, `config.rs`) `anthropic_body()` emits ephemeral `cache_control` breakpoints, gated by `BUZZ_AGENT_PROMPT_CACHING` (**default on**, `=0` to opt out): - **Static prefix** — marker on the `system` block. Prefix order is `tools → system → messages`, so this single marker caches **tools + system** together. Byte-identical on every turn of a run, and survives a context handoff (system/tools come from cfg/mcp, not `self.history`). - **Rolling tail + leapfrog** — marker on the last block of the last **two** messages. The append-only history re-reads the prior turn's prefix from cache; marking two messages (not one) keeps consecutive breakpoints inside Anthropic's **20-block lookback window** even as tool parallelism rises, avoiding a silent full-price miss. An empty system prompt stays a bare string (Anthropic rejects empty text blocks), and below-threshold prefixes are silently not cached, so the flag is safe on by default. ### 2. Surface the cache split end-to-end — the plumbing (`types.rs`, `llm.rs`, `agent.rs`, `lib.rs`, `usage.rs`, `acp.rs`) This is the part that makes gaps like the one above **visible** instead of silent. A consumer that prices all of `input_tokens` at the full rate can't tell a route that's caching from one that isn't — the total looks right either way. So: - `LlmResponse` gains `cached_input_tokens` (a **subset** of `input_tokens`, never an addition); `parse_anthropic` / `parse_openai` / `parse_responses` each populate it. - A `usage_first()` helper reads the cache count wherever a provider hides it — flat `cache_read_input_tokens` (Anthropic), `prompt_tokens_details.cached_tokens` (OpenAI chat), `input_tokens_details.cached_tokens` (Responses) — taking the **first present value, never a sum**. Reading only flat keys is exactly why the OpenAI route's nested `cached_tokens` had *also* been going unclaimed: `prompt_tokens` is already inclusive, so the total looked correct while the discount silently went unreported. - The per-turn/per-session accumulators and the goose `usage_update` payload now carry `accumulatedCachedInputTokens`; `buzz-acp` deserializes it (`serde` default `0` for goose, which doesn't send it) and logs `cached=<n>`. ### 3. Fix a Databricks MLflow-route double-count (`llm.rs`) The Databricks MLflow route reports the flat Anthropic-spelled `cache_read_input_tokens` *alongside* an already-inclusive `prompt_tokens`, so the old code summed them and nearly doubled the count — inflating both the context-budget gate and cost. `openai_chat_input_tokens()` now reads `prompt_tokens` alone. Verified on a live `databricks-glm-5-2` response where `prompt_tokens + completion == total` proves inclusivity. (Anthropic's native route genuinely *excludes* the cache fields and is still summed — the two never collide, because `claude*` models route to the Anthropic path.) ### 4. Proxy + TLS-trust passthrough into MCP tools (`mcp.rs`) — independent fix `buzz-agent` `env_clear()`s each MCP child, and the allowlist carried no proxy/TLS vars. On a proxy-only host that doesn't degrade the tools, it **blinds** them: apt, curl, pip, git connect directly, the egress firewall resets the socket, and the agent reports "Connection reset by peer" — indistinguishable from a genuinely offline task. Adds both spellings of `HTTP(S)_PROXY`/`NO_PROXY`/`ALL_PROXY` (curl/git read lowercase; Go/Python read uppercase; libcurl ignores uppercase `HTTP_PROXY`) plus `SSL_CERT_FILE`/`SSL_CERT_DIR` for TLS-terminating proxies that present their own CA. ## Testing - `cargo fmt --all -- --check`, `cargo clippy -p buzz-agent -p buzz-acp --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent -p buzz-acp` — **all green** (632 + 299 lib tests plus integration suites, 0 failures). New tests cover: the three breakpoints and the disabled/empty-system/single-message edge cases; nested-vs-flat cache parsing for all three routes; the Databricks inclusive-`prompt_tokens` fix; wire deserialization of `accumulatedCachedInputTokens`; and the proxy/TLS passthrough allowlist. - Pre-push lefthook suite green (branch-skew, rust-tests, test, desktop-check/test/tauri). ## Relationship to the benchmark branch These are the non-`benchmarks/` changes from `benchmark/harness-accounting-and-solo`, lifted onto a clean base off `main` so they can merge independently. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Atish Patel <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent aa1c316 commit 7cc1126

8 files changed

Lines changed: 519 additions & 23 deletions

File tree

‎crates/buzz-acp/src/acp.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1848,6 +1848,11 @@ impl AcpClient {
18481848
session_id = %notif.session_id,
18491849
input = payload.accumulated_input_tokens,
18501850
output = payload.accumulated_output_tokens,
1851+
// A subset of `input`, logged so downstream accounting can
1852+
// price it at the provider's cached rate. Always emitted,
1853+
// including as 0, so a parser can tell "no cache hits"
1854+
// apart from "this build predates the field".
1855+
cached = payload.accumulated_cached_input_tokens,
18511856
"goose usage update"
18521857
);
18531858
self.goose_usage.record(&notif.session_id, payload);

‎crates/buzz-acp/src/usage.rs‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,12 @@ pub(crate) struct UsageUpdatePayload {
8585
pub context_limit: u64,
8686
pub accumulated_input_tokens: u64,
8787
pub accumulated_output_tokens: u64,
88+
/// The cache-served subset of `accumulated_input_tokens`. Optional — goose
89+
/// does not send it, and buzz-agent only reports a non-zero value when the
90+
/// provider returned a cache split, so `0` legitimately means either "no
91+
/// cache hits" or "provider reported none".
92+
#[serde(default)]
93+
pub accumulated_cached_input_tokens: u64,
8894
pub accumulated_cost: Option<f64>,
8995
/// Effective model id for this turn. Optional — goose payloads that
9096
/// predate this field deserialize cleanly as `None`.
@@ -323,12 +329,44 @@ impl UsageTracker {
323329
mod tests {
324330
use super::*;
325331

332+
/// The camelCase key buzz-agent actually puts on the wire must land on the
333+
/// field. A rename mismatch here would deserialize to the serde default of
334+
/// 0, and every trial would price as if nothing had ever been cached — the
335+
/// exact silent failure this field was added to remove.
336+
#[test]
337+
fn cached_input_tokens_deserialize_from_the_wire_key() {
338+
let p: UsageUpdatePayload = serde_json::from_value(serde_json::json!({
339+
"used": 15_247,
340+
"contextLimit": 0,
341+
"accumulatedInputTokens": 15_091,
342+
"accumulatedOutputTokens": 156,
343+
"accumulatedCachedInputTokens": 5_033,
344+
}))
345+
.expect("payload must deserialize");
346+
assert_eq!(p.accumulated_cached_input_tokens, 5_033);
347+
assert!(p.accumulated_cached_input_tokens <= p.accumulated_input_tokens);
348+
}
349+
350+
/// goose does not send the field; its payloads must still deserialize.
351+
#[test]
352+
fn a_payload_without_the_cache_field_defaults_to_zero() {
353+
let p: UsageUpdatePayload = serde_json::from_value(serde_json::json!({
354+
"used": 500,
355+
"contextLimit": 200_000,
356+
"accumulatedInputTokens": 400,
357+
"accumulatedOutputTokens": 100,
358+
}))
359+
.expect("payload must deserialize without the cache field");
360+
assert_eq!(p.accumulated_cached_input_tokens, 0);
361+
}
362+
326363
fn payload(input: u64, output: u64, cost: Option<f64>) -> UsageUpdatePayload {
327364
UsageUpdatePayload {
328365
used: input + output,
329366
context_limit: 200_000,
330367
accumulated_input_tokens: input,
331368
accumulated_output_tokens: output,
369+
accumulated_cached_input_tokens: 0,
332370
accumulated_cost: cost,
333371
model: None,
334372
}
@@ -340,6 +378,7 @@ mod tests {
340378
context_limit: 0,
341379
accumulated_input_tokens: input,
342380
accumulated_output_tokens: output,
381+
accumulated_cached_input_tokens: 0,
343382
accumulated_cost: cost,
344383
model: None,
345384
}
@@ -836,6 +875,7 @@ mod tests {
836875
context_limit: 200_000,
837876
accumulated_input_tokens: input,
838877
accumulated_output_tokens: output,
878+
accumulated_cached_input_tokens: 0,
839879
accumulated_cost: cost,
840880
model: model.map(str::to_string),
841881
}

‎crates/buzz-agent/src/agent.rs‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,11 @@ pub struct RunCtx<'a> {
6060
/// Accumulated output tokens across all LLM rounds in this turn, for
6161
/// NIP-AM metric publishing. Reset to `None` at turn start in `run()`.
6262
pub turn_output_tokens: &'a mut Option<u64>,
63+
/// The cache-served subset of `turn_input_tokens`, accumulated across all
64+
/// LLM rounds in this turn. Reset to `None` at turn start in `run()`.
65+
/// Consumers price this slice at the provider's cached rate; without it
66+
/// every round of a growing conversation is billed at full price.
67+
pub turn_cached_input_tokens: &'a mut Option<u64>,
6368
}
6469

6570
impl RunCtx<'_> {
@@ -78,6 +83,7 @@ impl RunCtx<'_> {
7883
// Reset per-turn token accumulators for this prompt.
7984
*self.turn_input_tokens = None;
8085
*self.turn_output_tokens = None;
86+
*self.turn_cached_input_tokens = None;
8187

8288
let mut round = 0u32;
8389
// Per-prompt `_Stop` objection count. Bounded per prompt (not per
@@ -175,6 +181,17 @@ impl RunCtx<'_> {
175181
*self.turn_output_tokens =
176182
Some(self.turn_output_tokens.unwrap_or(0).saturating_add(out));
177183
}
184+
// Accumulate the cache-served subset of this turn's input. Tracked
185+
// separately from `turn_input_tokens` rather than subtracted from
186+
// it: the input total must stay inclusive for the handoff gate,
187+
// which cares how much context was sent, not what it cost.
188+
if let Some(cached) = response.cached_input_tokens {
189+
*self.turn_cached_input_tokens = Some(
190+
self.turn_cached_input_tokens
191+
.unwrap_or(0)
192+
.saturating_add(cached),
193+
);
194+
}
178195

179196
if !response.reasoning.is_empty() {
180197
wire::send(

‎crates/buzz-agent/src/config.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -738,6 +738,13 @@ pub struct Config {
738738
/// Thinking/reasoning effort level. `None` = use provider default (no
739739
/// thinking config sent). Set via `BUZZ_AGENT_THINKING_EFFORT`.
740740
pub thinking_effort: Option<ThinkingEffort>,
741+
/// Emit Anthropic `cache_control` breakpoints on the stable prefix
742+
/// (tools + system prompt) and the rolling conversation tail. Default on;
743+
/// disable with `BUZZ_AGENT_PROMPT_CACHING=0`. Only consulted on Anthropic
744+
/// Messages routes (first-party Anthropic and the DatabricksV2 Claude
745+
/// route) — the Databricks gateway does not auto-cache, so without this the
746+
/// surfaced `cache_read_input_tokens` is structurally always 0.
747+
pub prompt_caching: bool,
741748
}
742749

743750
impl Config {
@@ -833,6 +840,7 @@ impl Config {
833840
hook_servers: parse_hook_servers_env("MCP_HOOK_SERVERS"),
834841
hints_enabled: parse_env("BUZZ_AGENT_NO_HINTS", 0u8)? == 0,
835842
thinking_effort: parse_thinking_effort(env("BUZZ_AGENT_THINKING_EFFORT").as_deref())?,
843+
prompt_caching: parse_env("BUZZ_AGENT_PROMPT_CACHING", 1u8)? != 0,
836844
};
837845
cfg.validate()?;
838846
Ok(cfg)
@@ -874,6 +882,7 @@ impl Config {
874882
hook_servers: HookServers::None,
875883
hints_enabled: false,
876884
thinking_effort: None,
885+
prompt_caching: false,
877886
}
878887
}
879888

‎crates/buzz-agent/src/lib.rs‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,11 @@ struct Session {
100100
accumulated_input_tokens: u64,
101101
/// Session-cumulative output tokens across all turns.
102102
accumulated_output_tokens: u64,
103+
/// Session-cumulative cache-served input tokens across all turns — a subset
104+
/// of `accumulated_input_tokens`, not an addition to it. Emitted alongside
105+
/// it so a consumer can price the cached slice at the provider's discounted
106+
/// rate instead of assuming every input token cost full price.
107+
accumulated_cached_input_tokens: u64,
103108
}
104109

105110
fn die(msg: String) -> ! {
@@ -426,6 +431,7 @@ async fn session_new(app: &Arc<App>, id: Value, params: Value, wire_tx: &WireSen
426431
effective_model: None,
427432
accumulated_input_tokens: 0,
428433
accumulated_output_tokens: 0,
434+
accumulated_cached_input_tokens: 0,
429435
},
430436
);
431437
drop(sessions);
@@ -672,6 +678,7 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
672678
.unwrap_or(&app.cfg.model);
673679
let mut turn_input_tokens: Option<u64> = None;
674680
let mut turn_output_tokens: Option<u64> = None;
681+
let mut turn_cached_input_tokens: Option<u64> = None;
675682
let mut ctx = RunCtx {
676683
cfg: &app.cfg,
677684
effective_model: effective_model_str,
@@ -690,6 +697,7 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
690697
last_request_history_bytes: &mut last_request_history_bytes,
691698
turn_input_tokens: &mut turn_input_tokens,
692699
turn_output_tokens: &mut turn_output_tokens,
700+
turn_cached_input_tokens: &mut turn_cached_input_tokens,
693701
};
694702
let result = ctx.run(p.prompt).await;
695703
if let Some(s) = app.sessions.lock().await.get_mut(&sid) {
@@ -722,14 +730,21 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
722730
s.accumulated_output_tokens = s
723731
.accumulated_output_tokens
724732
.saturating_add(turn_output_tokens.unwrap_or(0));
725-
Some((s.accumulated_input_tokens, s.accumulated_output_tokens))
733+
s.accumulated_cached_input_tokens = s
734+
.accumulated_cached_input_tokens
735+
.saturating_add(turn_cached_input_tokens.unwrap_or(0));
736+
Some((
737+
s.accumulated_input_tokens,
738+
s.accumulated_output_tokens,
739+
s.accumulated_cached_input_tokens,
740+
))
726741
} else {
727742
// Session is gone — the accumulated baseline no longer exists, so
728743
// there is nothing correct to emit. Skip the usage notification.
729744
None
730745
}
731746
};
732-
if let Some((accumulated_in, accumulated_out)) = accumulated {
747+
if let Some((accumulated_in, accumulated_out, accumulated_cached)) = accumulated {
733748
wire::send(
734749
&wire_tx,
735750
goose_session_update(
@@ -742,6 +757,11 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
742757
"contextLimit": 0u64,
743758
"accumulatedInputTokens": accumulated_in,
744759
"accumulatedOutputTokens": accumulated_out,
760+
// A subset of accumulatedInputTokens, not an addition to
761+
// it. Extends goose's usage_update shape; a consumer that
762+
// does not know the field ignores it and prices exactly as
763+
// it did before.
764+
"accumulatedCachedInputTokens": accumulated_cached,
745765
"model": effective_model_str,
746766
}),
747767
),

0 commit comments

Comments
 (0)