Skip to content

feat(openai): support explicit prompt cache breakpoints#3719

Merged
dgageot merged 2 commits into
mainfrom
feat/openai-explicit-prompt-caching
Jul 17, 2026
Merged

feat(openai): support explicit prompt cache breakpoints#3719
dgageot merged 2 commits into
mainfrom
feat/openai-explicit-prompt-caching

Conversation

@Sayt-0

@Sayt-0 Sayt-0 commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

  • translate existing stable CacheControl message boundaries into OpenAI explicit prompt-cache breakpoints on GPT-5.6+
  • support both Chat Completions and Responses content shapes, including multipart content and function-call outputs
  • retain OpenAI's default implicit cache policy while preventing OpenAI-only fields from reaching older, third-party, or self-hosted endpoints
  • report cache-write tokens without double-counting prompt usage

Issue expectations

Expectation Implementation
Determine whether explicit caching can improve caching Existing stable session boundaries now become provider-visible cache breakpoints on supported OpenAI models
Support recent OpenAI models Gated to genuine OpenAI GPT-5.6+ models, including Azure, provider-qualified gateway models, and trusted custom OpenAI providers
Preserve compatibility Older models, unrelated OpenAI-compatible providers, and openai configurations with a custom base_url receive no explicit-cache fields
Cover both OpenAI APIs Implemented for Chat Completions and Responses
Measure cache behavior Cache reads and writes are partitioned into mutually exclusive usage buckets for accurate costs and context accounting

Design

  • prompt_cache_options remains omitted, preserving the default implicit breakpoint while adding explicit stable boundaries.
  • String content is converted to typed content blocks only when a marked message needs to carry a breakpoint.
  • Breakpoints remain scoped to the source message so filtered or expanded messages cannot mark neighboring content.
  • Eligible breakpoint conversion takes precedence over generic consecutive-message merging because merging would destroy stable cache boundaries.

Validation

Check Result
go build ./... pass
go test -count=1 ./... pass
Focused prompt-cache tests with -race pass
task --force lint pass, 0 issues
Adversarial review two cycles completed; blockers resolved

Closes #3705

@Sayt-0
Sayt-0 requested a review from a team as a code owner July 17, 2026 11:22
@Sayt-0
Merge remote-tracking branch 'origin/main' into feat/openai-explicit-…
82f5b3a 
…prompt-caching

# Conflicts:
#	pkg/chat/chat.go
@rumpl

rumpl commented Jul 17, 2026

Copy link
Copy Markdown
Member

I would love to see some before/after numbers

@aheritier aheritier added area/models LLM model integrations and model providers area/providers/openai For features/issues/fixes related to the usage of OpenAI models kind/feat PR adds a new feature (maps to feat:). Use on PRs only. labels Jul 17, 2026
@Sayt-0

Sayt-0 commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

Measured against the Docker staging models gateway with openai/gpt-5.6-luna.

The benchmark used 10 interleaved requests per variant: 1 cold request and 9 warm requests, a stable ~4.7k-token system prefix, isolated per-variant cache namespaces, a changing user nonce, thinking_budget: none, and a 32-token output cap. main was patched locally only to expose cache_write_tokens; its request construction was unchanged.

metric main, implicit cache PR, implicit + explicit breakpoints change
cold cache-write tokens 4,676 4,678 equivalent
warm cached-input tokens 0 4,657 +4,657
warm cache-write tokens 4,676 21 -99.55%
warm fresh-input tokens 3 3 unchanged
warm cache-read ratio 0% 99.49% +99.49 pp
normalized warm input cost/request $0.005848 $0.00049495 -91.54%, 11.8x lower
normalized input cost over 10 requests $0.05848 $0.010305 -82.38%, 5.67x lower
median warm time to first chunk 1.535s 1.494s -2.68%
median warm stream duration 1.814s 1.794s -1.09%

Input cost uses $1/M fresh tokens, $0.10/M cache reads, and $1.25/M cache writes; output cost was excluded because output length was fixed. Latency was effectively flat, but the cache behavior and input cost improved substantially: main rewrote the full stable prompt on every changing suffix, while the PR reused the stable prefix and wrote only the dynamic tail.

@dgageot
dgageot merged commit 5a244ac into main Jul 17, 2026
12 checks passed
@dgageot
dgageot deleted the feat/openai-explicit-prompt-caching branch July 17, 2026 14:53
@BrewTestBot BrewTestBot mentioned this pull request Jul 20, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dgageot dgageot dgageot approved these changes

Assignees

@dgageot dgageot

Labels

area/models LLM model integrations and model providers area/providers/openai For features/issues/fixes related to the usage of OpenAI models kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Should we support OpenAI explicit caching

4 participants

@Sayt-0 @rumpl @dgageot @aheritier