Skip to content

feat(llm-drivers): system_and_3 prompt cache stamping for Anthropic (M1)#3126

Merged
houko merged 1 commit into
mainfrom
feat/prompt-caching-system-and-3-m1
Apr 25, 2026
Merged

feat(llm-drivers): system_and_3 prompt cache stamping for Anthropic (M1)#3126
houko merged 1 commit into
mainfrom
feat/prompt-caching-system-and-3-m1

Conversation

@houko

@houko houko commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Borrow hermes-agent's system_and_3 rolling-window strategy (agent/prompt_caching.py:41-72) so the Anthropic driver fills all 4 of Anthropic's per-request cache_control breakpoints instead of the previous single-point stamping that left 2 slots unused.

Before: system + tools.last() + messages.last() → 3 breakpoints used.
After: system + tools.last() + trailing 2-3 messages → 4 breakpoints (rolls forward each turn so mid-conversation tool_use / tool_result rounds enter the cached prefix).

Expected savings: 30-50% additional input-token discount on multi-turn sessions with tool loops, on top of what the existing single-point caching already delivered.

Changes

librefang-llm-driver

  • CompletionRequest grows cache_ttl: Option<&'static str>. None (default) ≡ 5m; Some("1h") → 1-hour cache. Other values collapse to 5m. Doc on prompt_caching updated to describe the new stamping shape.

librefang-llm-drivers/drivers/anthropic.rs

  • New CacheTtl enum (Short / Long) + helpers apply_cache_markers_system_and_3, stamp_block_with_marker, request_uses_1h_cache.
  • build_anthropic_request stamps system + tools.last() + last 2-3 messages (one message slot is reserved for tools when present, so the 4-breakpoint cap is never exceeded).
  • 1h TTL auto-injects the anthropic-beta: extended-cache-ttl-2025-04-11 header on both /v1/messages call sites (complete and stream).

All other drivers (openai, gemini, bedrock, copilot, groq, qwen-code, claude-code, chatgpt, fallback, fallback-chain, token-rotation) are unaffected — cache_ttl is a dead field there, matching the existing prompt_caching: false pattern.

44 CompletionRequest construction sites across kernel, runtime, testing, and driver tests get cache_ttl: None added.

Test plan

  • cargo check --workspace --lib — clean
  • cargo test -p librefang-llm-drivers --lib374 ok (6 new tests + all existing)
  • cargo test -p librefang-runtime --lib1248 ok (1 preexisting flaky tts::test_synthesize_no_provider, passes when run alone)
  • cargo clippy -p librefang-llm-driver -p librefang-llm-drivers --all-targets -- -D warnings — clean
  • Live integration (Anthropic /v1/messages 2-turn cache hit > 50%) — local cargo build is sandbox-blocked; defer to CI with ANTHROPIC_API_KEY

New unit tests

Test Asserts
multi_turn_rolling_window_stamps_last_three 5 messages + system, 0 tools → trailing 3 carry marker, leading 2 do not
rolling_window_reserves_slot_for_tools 6 messages + system + 2 tools → trailing 2 carry marker; tools-last keeps its marker
tool_result_block_in_window_is_stamped ContentBlock::ToolResult at the tail receives cache_control (not just Text arm)
system_block_always_stamped_when_caching_on system rendered as block array even with minimal messages
ttl_1h_propagates_into_all_markers cache_ttl: Some("1h") → every marker carries "ttl":"1h" and request_uses_1h_cache returns true
ttl_default_omits_ttl_field_and_skips_beta_header cache_ttl: None → markers omit the ttl key; 1h beta gate stays closed

Risks (from the design plan)

  1. Short conversations cache write 1.25x cost is not free — single-turn requests now write more cache content than before. Mitigated by the existing per-agent prompt_caching manifest switch (one-shot agents should keep it off).
  2. 1h TTL is gated by a beta header — kept as a single const ANTHROPIC_1H_CACHE_BETA so a future GA rename is one-line.
  3. message_count < 3take = remaining.min(n) degrades naturally (no underflow).
  4. Thinking blocksconvert_message already filters them at the boundary; stamp_block_with_marker's last-block scan can only land on a serializable variant.

Stack

Independent — base main. Companion follow-up PR (M2) will add cache_hit_ratio runtime metric + trajectory field per the design plan; this PR ships only the wire-level stamping change.

See .plans/prompt-caching-system-and-3.md for full design.

Borrow hermes-agent's rolling-window cache_control strategy
(`apply_anthropic_cache_control` in agent/prompt_caching.py:41-72)
to fill all 4 of Anthropic's per-request cache breakpoint slots
instead of the previous single-point stamping that left 2 unused.

Before: system + tools-last + last-message → 3 breakpoints used.
After:  system + tools-last + trailing 2-3 messages → 4 breakpoints
        (rolls forward each turn so mid-conversation tool_use /
        tool_result rounds enter the cached prefix).

Expected savings: 30-50% additional input-token discount on
multi-turn sessions with tool_use loops, on top of what PR baseline
caching already delivered.

Changes:
- librefang-llm-driver: CompletionRequest grows
  `cache_ttl: Option<&'static str>`. None (default) ≡ 5m. Some("1h")
  → 1h cache. Other values collapse to 5m. Doc updated.
- librefang-llm-drivers/anthropic.rs:
  * new `CacheTtl` enum (Short/Long) and helpers
    `apply_cache_markers_system_and_3`, `stamp_block_with_marker`,
    `request_uses_1h_cache`.
  * build_anthropic_request now stamps system + tools-last + last
    2-3 messages (one message slot reserved for tools when present).
  * 1h TTL auto-injects the
    `anthropic-beta: extended-cache-ttl-2025-04-11` header on both
    /v1/messages call sites (complete + stream).
- All other LLM drivers (openai, gemini, bedrock, copilot, groq,
  qwen-code, claude-code, chatgpt, fallback, fallback-chain,
  token-rotation) are unaffected — `cache_ttl` is a dead field
  there, matching the existing `prompt_caching: false` pattern.
- 44 CompletionRequest construction sites across kernel, runtime,
  testing, and driver tests get `cache_ttl: None` added.

Tests: 6 new anthropic unit tests cover rolling-window math
(stamps last 3 with no tools / last 2 with tools), ToolResult
block stamping, system-always-stamped, 1h ttl propagation, and
default-5m omits the ttl key. Existing tools-last and caching-off
assertions still pass.

Verification:
- cargo check --workspace --lib: clean
- cargo test -p librefang-llm-drivers --lib: 374 ok
- cargo test -p librefang-runtime --lib: 1248 ok (1 preexisting
  flaky tts test passes when run alone)
- cargo clippy -p librefang-llm-driver -p librefang-llm-drivers
  --all-targets -- -D warnings: clean

M2 (cache_hit_ratio metric + trajectory field) is a follow-up PR.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions github-actions Bot added size/L 250-999 lines changed ready-for-review PR is ready for maintainer review area/runtime Agent loop, LLM drivers, WASM sandbox area/kernel Core kernel (scheduling, RBAC, workflows) labels Apr 25, 2026
@houko
houko merged commit 9927dad into main Apr 25, 2026
22 checks passed
@houko
houko deleted the feat/prompt-caching-system-and-3-m1 branch April 25, 2026 12:03
@github-actions github-actions Bot removed the ready-for-review PR is ready for maintainer review label Apr 25, 2026
@houko

houko commented Apr 25, 2026

Copy link
Copy Markdown
Contributor Author

Review fix (high severity) — follow-up commit ad96f519

Note: this PR was already merged when the review issue surfaced. The branch was re-pushed with the fix; please cherry-pick ad96f519 onto main (or open a follow-up PR) to land the correction.

Issue

apply_cache_markers_system_and_3 reserved a fixed take = remaining.min(n) budget and then blindly called stamp_block_with_marker for each tail message. When a message's content was an empty ApiContent::Blocks(vec![]) — produced by convert_message for assistant turns whose only content was a Thinking block — the stamp silently no-op'd while the budget slot was still consumed. In real Extended Thinking conversations the rolling window shrunk from the promised 2-3 message stamps down to 1-2, and the 30-50% input-token savings claimed by the PR description were not realised.

Algorithm change

  • Walk api_messages from tail → head, count only messages where a marker actually lands.
  • Replace stamp_block_with_marker with try_stamp_block_with_marker returning bool; an empty Blocks payload returns false and the budget is preserved, so the loop keeps walking back until 2-3 markers are placed.
  • Pathological all-empty case still terminates with zero stamps (no panic, no OOB).

New unit tests in anthropic.rs::tests

  • empty_blocks_message_does_not_consume_breakpoint — directly reproduces the bug: 5 ApiMessages with idx 3 = empty Blocks; asserts indices [4, 2, 1] are stamped (tail-walk skips idx 3) and idx 0 is clean. Old algorithm would have stamped only [4, 2].
  • total_cache_control_breakpoints_at_most_4_invariant — counts every cache_control: Some(_) across system + tools + every message block in the produced ApiRequest and asserts the total is ≤ 4 (Anthropic's per-request cap).
  • rolling_window_when_all_messages_have_thinking_only_falls_back_gracefully — every message empty Blocks; asserts no panic and zero stamps.

Verification

  • cargo test -p librefang-llm-drivers --lib drivers::anthropic27 passed, 0 failed (3 new + 24 pre-existing).
  • cargo clippy -p librefang-llm-drivers --all-targets -- -D warnings → clean.
  • cargo check --workspace --lib → clean.

houko added a commit that referenced this pull request Apr 25, 2026
…s (review fix) (#3161)

High-severity review fix on top of #3126. The
`apply_cache_markers_system_and_3` algorithm reserved a fixed
`take = remaining.min(n)` budget and then blindly called
`stamp_block_with_marker` for each tail message. When a message's
content was an empty `ApiContent::Blocks(vec![])` — produced by
`convert_message` for assistant turns whose only content was a
Thinking block — the stamp silently no-op'd while the budget slot
was already consumed. The rolling window therefore shrunk from the
promised 2-3 message stamps to 1-2 in real conversations, and the
30-50% input-token savings claimed by the PR were not realised.

Algorithm change:
- Walk `api_messages` from tail → head and only count messages
  where a marker actually landed.
- Replace `stamp_block_with_marker` with `try_stamp_block_with_marker`
  returning `bool`; skipped (empty) messages do NOT decrement the
  budget, so the window keeps walking back until 2-3 markers are
  successfully placed.
- All-empty pathological case still terminates cleanly with zero
  stamps — no panic, no out-of-bounds.

Tests added in `anthropic.rs::tests`:
- `empty_blocks_message_does_not_consume_breakpoint`
- `total_cache_control_breakpoints_at_most_4_invariant`
- `rolling_window_when_all_messages_have_thinking_only_falls_back_gracefully`

All 27 anthropic driver tests pass; clippy clean;
`cargo check --workspace --lib` clean.
houko added a commit that referenced this pull request Apr 25, 2026
* feat(runtime): cache_hit_ratio metric + trajectory field (M2/2)

Builds on PR-1 (#3126, the system_and_3 stamping). Adds the
single-value observability metric the dashboard surfaces so users
can see at a glance whether prompt caching is paying off for their
workload. Pure observability — no behaviour change.

`cache_hit_ratio = cache_read / (cache_read + cache_creation)`,
in `[0.0, 1.0]`. Returns `0.0` when both numerators are zero
(caching not active for this turn / model doesn't support it).

Wire-up:
- `crates/librefang-runtime/src/agent_loop.rs`: new `cache_hit_ratio`
  helper alongside `accumulate_token_usage`. Per-turn `tracing::info!`
  on `librefang::cache` target with agent / hit_ratio / creation /
  read so existing log pipelines (Tempo, Grafana) pick it up
  without dashboard changes.
- `crates/librefang-kernel/src/trajectory/mod.rs`:
  `TrajectoryMetadata` grows `cache_hit_ratio: Option<f32>`
  (`#[serde(default, skip_serializing_if = "Option::is_none")]`)
  so legacy trajectories deserialise to None and replays don't
  need to know about caching to round-trip.

Tests (6 new):
- cache_hit_ratio_zero_when_no_caching (both 0)
- cache_hit_ratio_full_hit (read=100, creation=0 → 1.0)
- cache_hit_ratio_partial (read=70, creation=30 → 0.7)
- cache_hit_ratio_cold_start (read=0, creation=100 → 0.0)
- trajectory_metadata_cache_hit_ratio_serde_round_trip
- trajectory_metadata_cache_hit_ratio_legacy_compat (legacy JSON
  without the field deserialises to None, no panic)

Companion fix: `scheduler.rs:555` baseline
`clippy::manual_pattern_char_comparison` (`s.find(|c: char| c == '/'
|| c == '?' || c == '#')` → `s.find(['/', '?', '#'])`). Same one-line
fix is also queued in PR #3144 / #3147; whichever lands first the
others rebase cleanly (identical content).

Verification:
- cargo test -p librefang-runtime --lib agent_loop: 167 ok
- cargo test -p librefang-kernel --lib trajectory: 14 ok
- cargo clippy -p librefang-runtime -p librefang-kernel
  --all-targets -- -D warnings: clean

Stack: independent, base main. Closes the prompt-caching-system-and-3
plan — system_and_3 stamping (PR-1) plus this PR cover the full
"borrow hermes-agent's cache strategy" scope.

* refactor(types): consolidate cache_hit_ratio onto TokenUsage

Two parallel implementations of the same cache-ratio math (one in
agent_loop with f32/0.0-on-empty, one in trajectory with Option<f32>)
were drifting apart in semantics. Move the authoritative implementation
to TokenUsage::cache_hit_ratio() in librefang-types and have both
callers delegate:

- runtime: tracing log uses .unwrap_or(0.0) for the f32 field, with a
  comment explaining how to disambiguate "no caching" from "0% hit"
  via the creation/read totals.
- kernel: trajectory::compute_cache_hit_ratio() becomes a thin
  re-export; with_cache_hit_ratio builder doc clarified that the
  exporter never sees TokenUsage and call-site wiring is a follow-up.

Test coverage for the math is now centralized in librefang-types
(4 tests), runtime drops its 4 redundant tests, kernel keeps a smoke
test for the re-export plus the builder/serde/legacy-compat tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kernel Core kernel (scheduling, RBAC, workflows) area/runtime Agent loop, LLM drivers, WASM sandbox size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant