feat(llm-drivers): system_and_3 prompt cache stamping for Anthropic (M1)#3126
Conversation
Borrow hermes-agent's rolling-window cache_control strategy
(`apply_anthropic_cache_control` in agent/prompt_caching.py:41-72)
to fill all 4 of Anthropic's per-request cache breakpoint slots
instead of the previous single-point stamping that left 2 unused.
Before: system + tools-last + last-message → 3 breakpoints used.
After: system + tools-last + trailing 2-3 messages → 4 breakpoints
(rolls forward each turn so mid-conversation tool_use /
tool_result rounds enter the cached prefix).
Expected savings: 30-50% additional input-token discount on
multi-turn sessions with tool_use loops, on top of what PR baseline
caching already delivered.
Changes:
- librefang-llm-driver: CompletionRequest grows
`cache_ttl: Option<&'static str>`. None (default) ≡ 5m. Some("1h")
→ 1h cache. Other values collapse to 5m. Doc updated.
- librefang-llm-drivers/anthropic.rs:
* new `CacheTtl` enum (Short/Long) and helpers
`apply_cache_markers_system_and_3`, `stamp_block_with_marker`,
`request_uses_1h_cache`.
* build_anthropic_request now stamps system + tools-last + last
2-3 messages (one message slot reserved for tools when present).
* 1h TTL auto-injects the
`anthropic-beta: extended-cache-ttl-2025-04-11` header on both
/v1/messages call sites (complete + stream).
- All other LLM drivers (openai, gemini, bedrock, copilot, groq,
qwen-code, claude-code, chatgpt, fallback, fallback-chain,
token-rotation) are unaffected — `cache_ttl` is a dead field
there, matching the existing `prompt_caching: false` pattern.
- 44 CompletionRequest construction sites across kernel, runtime,
testing, and driver tests get `cache_ttl: None` added.
Tests: 6 new anthropic unit tests cover rolling-window math
(stamps last 3 with no tools / last 2 with tools), ToolResult
block stamping, system-always-stamped, 1h ttl propagation, and
default-5m omits the ttl key. Existing tools-last and caching-off
assertions still pass.
Verification:
- cargo check --workspace --lib: clean
- cargo test -p librefang-llm-drivers --lib: 374 ok
- cargo test -p librefang-runtime --lib: 1248 ok (1 preexisting
flaky tts test passes when run alone)
- cargo clippy -p librefang-llm-driver -p librefang-llm-drivers
--all-targets -- -D warnings: clean
M2 (cache_hit_ratio metric + trajectory field) is a follow-up PR.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Review fix (high severity) — follow-up commit
|
…s (review fix) (#3161) High-severity review fix on top of #3126. The `apply_cache_markers_system_and_3` algorithm reserved a fixed `take = remaining.min(n)` budget and then blindly called `stamp_block_with_marker` for each tail message. When a message's content was an empty `ApiContent::Blocks(vec![])` — produced by `convert_message` for assistant turns whose only content was a Thinking block — the stamp silently no-op'd while the budget slot was already consumed. The rolling window therefore shrunk from the promised 2-3 message stamps to 1-2 in real conversations, and the 30-50% input-token savings claimed by the PR were not realised. Algorithm change: - Walk `api_messages` from tail → head and only count messages where a marker actually landed. - Replace `stamp_block_with_marker` with `try_stamp_block_with_marker` returning `bool`; skipped (empty) messages do NOT decrement the budget, so the window keeps walking back until 2-3 markers are successfully placed. - All-empty pathological case still terminates cleanly with zero stamps — no panic, no out-of-bounds. Tests added in `anthropic.rs::tests`: - `empty_blocks_message_does_not_consume_breakpoint` - `total_cache_control_breakpoints_at_most_4_invariant` - `rolling_window_when_all_messages_have_thinking_only_falls_back_gracefully` All 27 anthropic driver tests pass; clippy clean; `cargo check --workspace --lib` clean.
* feat(runtime): cache_hit_ratio metric + trajectory field (M2/2) Builds on PR-1 (#3126, the system_and_3 stamping). Adds the single-value observability metric the dashboard surfaces so users can see at a glance whether prompt caching is paying off for their workload. Pure observability — no behaviour change. `cache_hit_ratio = cache_read / (cache_read + cache_creation)`, in `[0.0, 1.0]`. Returns `0.0` when both numerators are zero (caching not active for this turn / model doesn't support it). Wire-up: - `crates/librefang-runtime/src/agent_loop.rs`: new `cache_hit_ratio` helper alongside `accumulate_token_usage`. Per-turn `tracing::info!` on `librefang::cache` target with agent / hit_ratio / creation / read so existing log pipelines (Tempo, Grafana) pick it up without dashboard changes. - `crates/librefang-kernel/src/trajectory/mod.rs`: `TrajectoryMetadata` grows `cache_hit_ratio: Option<f32>` (`#[serde(default, skip_serializing_if = "Option::is_none")]`) so legacy trajectories deserialise to None and replays don't need to know about caching to round-trip. Tests (6 new): - cache_hit_ratio_zero_when_no_caching (both 0) - cache_hit_ratio_full_hit (read=100, creation=0 → 1.0) - cache_hit_ratio_partial (read=70, creation=30 → 0.7) - cache_hit_ratio_cold_start (read=0, creation=100 → 0.0) - trajectory_metadata_cache_hit_ratio_serde_round_trip - trajectory_metadata_cache_hit_ratio_legacy_compat (legacy JSON without the field deserialises to None, no panic) Companion fix: `scheduler.rs:555` baseline `clippy::manual_pattern_char_comparison` (`s.find(|c: char| c == '/' || c == '?' || c == '#')` → `s.find(['/', '?', '#'])`). Same one-line fix is also queued in PR #3144 / #3147; whichever lands first the others rebase cleanly (identical content). Verification: - cargo test -p librefang-runtime --lib agent_loop: 167 ok - cargo test -p librefang-kernel --lib trajectory: 14 ok - cargo clippy -p librefang-runtime -p librefang-kernel --all-targets -- -D warnings: clean Stack: independent, base main. Closes the prompt-caching-system-and-3 plan — system_and_3 stamping (PR-1) plus this PR cover the full "borrow hermes-agent's cache strategy" scope. * refactor(types): consolidate cache_hit_ratio onto TokenUsage Two parallel implementations of the same cache-ratio math (one in agent_loop with f32/0.0-on-empty, one in trajectory with Option<f32>) were drifting apart in semantics. Move the authoritative implementation to TokenUsage::cache_hit_ratio() in librefang-types and have both callers delegate: - runtime: tracing log uses .unwrap_or(0.0) for the f32 field, with a comment explaining how to disambiguate "no caching" from "0% hit" via the creation/read totals. - kernel: trajectory::compute_cache_hit_ratio() becomes a thin re-export; with_cache_hit_ratio builder doc clarified that the exporter never sees TokenUsage and call-site wiring is a follow-up. Test coverage for the math is now centralized in librefang-types (4 tests), runtime drops its 4 redundant tests, kernel keeps a smoke test for the re-export plus the builder/serde/legacy-compat tests.
Summary
Borrow hermes-agent's
system_and_3rolling-window strategy (agent/prompt_caching.py:41-72) so the Anthropic driver fills all 4 of Anthropic's per-requestcache_controlbreakpoints instead of the previous single-point stamping that left 2 slots unused.Before:
system+tools.last()+messages.last()→ 3 breakpoints used.After:
system+tools.last()+ trailing 2-3 messages → 4 breakpoints (rolls forward each turn so mid-conversationtool_use/tool_resultrounds enter the cached prefix).Expected savings: 30-50% additional input-token discount on multi-turn sessions with tool loops, on top of what the existing single-point caching already delivered.
Changes
librefang-llm-driverCompletionRequestgrowscache_ttl: Option<&'static str>.None(default) ≡ 5m;Some("1h")→ 1-hour cache. Other values collapse to 5m. Doc onprompt_cachingupdated to describe the new stamping shape.librefang-llm-drivers/drivers/anthropic.rsCacheTtlenum (Short/Long) + helpersapply_cache_markers_system_and_3,stamp_block_with_marker,request_uses_1h_cache.build_anthropic_requeststampssystem+tools.last()+ last 2-3 messages (one message slot is reserved for tools when present, so the 4-breakpoint cap is never exceeded).anthropic-beta: extended-cache-ttl-2025-04-11header on both/v1/messagescall sites (completeandstream).All other drivers (openai, gemini, bedrock, copilot, groq, qwen-code, claude-code, chatgpt, fallback, fallback-chain, token-rotation) are unaffected —
cache_ttlis a dead field there, matching the existingprompt_caching: falsepattern.44
CompletionRequestconstruction sites across kernel, runtime, testing, and driver tests getcache_ttl: Noneadded.Test plan
cargo check --workspace --lib— cleancargo test -p librefang-llm-drivers --lib— 374 ok (6 new tests + all existing)cargo test -p librefang-runtime --lib— 1248 ok (1 preexisting flakytts::test_synthesize_no_provider, passes when run alone)cargo clippy -p librefang-llm-driver -p librefang-llm-drivers --all-targets -- -D warnings— clean/v1/messages2-turn cache hit > 50%) — localcargo buildis sandbox-blocked; defer to CI withANTHROPIC_API_KEYNew unit tests
multi_turn_rolling_window_stamps_last_threerolling_window_reserves_slot_for_toolstool_result_block_in_window_is_stampedContentBlock::ToolResultat the tail receivescache_control(not justTextarm)system_block_always_stamped_when_caching_onttl_1h_propagates_into_all_markerscache_ttl: Some("1h")→ every marker carries"ttl":"1h"andrequest_uses_1h_cachereturns truettl_default_omits_ttl_field_and_skips_beta_headercache_ttl: None→ markers omit thettlkey; 1h beta gate stays closedRisks (from the design plan)
prompt_cachingmanifest switch (one-shot agents should keep it off).const ANTHROPIC_1H_CACHE_BETAso a future GA rename is one-line.message_count < 3—take = remaining.min(n)degrades naturally (no underflow).convert_messagealready filters them at the boundary;stamp_block_with_marker's last-block scan can only land on a serializable variant.Stack
Independent — base
main. Companion follow-up PR (M2) will addcache_hit_ratioruntime metric + trajectory field per the design plan; this PR ships only the wire-level stamping change.See
.plans/prompt-caching-system-and-3.mdfor full design.