Skip to content

feat(core): configurable prompt-cache policy per agent and per provider/model - #51110

Closed
ugur-murat-alt wants to merge 2 commits into
anomalyco:v2from
ugur-murat-alt:feat/prompt-cache-policy-config
Closed

ugur-murat-alt wants to merge 2 commits into
anomalyco:v2from
ugur-murat-alt:feat/prompt-cache-policy-config

Conversation

@ugur-murat-alt

Copy link
Copy Markdown
Contributor

Closes #51109.

What

Makes the prompt-cache policy configurable from opencode.jsonc at three levels, resolved agent > model > provider:

{
  "providers": {
    "anthropic": {
      "cache": { "ttlSeconds": 3600 },          // default for every model of this provider
      "models": {
        "claude-opus-5-5": { "cache": "auto" }   // per-model override
      }
    }
  },
  "agents": {
    "build": { "cache": { "ttlSeconds": 3600 } } // per-agent override
  }
}

The value mirrors @opencode-ai/llm's policy: "auto" | "none" | { tools?, system?, messages?, ttlSeconds? }. Providers that support extended retention map ttlSeconds >= 3600 to their long-lived bucket (Anthropic/Bedrock 1h); providers without a TTL concept keep their current behavior. A subagent can stay on the cheap 5-minute default while a long-lived primary agent opts into 1h.

Why

Prompt caching is on by default, but the TTL was fixed at the provider default. A long-lived orchestrator that waits on subagents/tools for more than 5 minutes loses the cached prefix and re-writes the whole context at the cache-write rate on the next request. On a ~440k-token context that is 440k cache-write tokens per resume ($2.2 on Opus at $5/M) while a cache read is ~25x cheaper per token; in one observed session ~12 cold resumptions were 51% of total cost. Anthropic's own guidance (and Claude Code's default) is a 1-hour TTL on the prefix for exactly this case. There was no way to express it from config.

Changes

  • packages/schema/src/cache.ts (new): CachePolicy.Policy schema.
  • packages/schema: cache on ConfigAgent.Info, ConfigProvider.Info, ConfigProvider.Model, and the runtime Agent.Info / Model.Info.
  • packages/core/src/config/plugin/provider.ts: model cache overrides provider cache; a provider-level policy is the default for every model of that provider.
  • packages/core/src/config/plugin/agent.ts: agent cache lands on the runtime agent.
  • packages/core/src/model-resolver.ts: Resolved.cache carries the catalog/config policy.
  • packages/core/src/session/model-request.ts: Input.cache is forwarded to LLM.request.
  • All request flows resolve agent.cache ?? model.cache: runner/llm.ts, generate.ts, title.ts, compaction.ts.
  • runner/model.ts: the resolved(...) test seam accepts cache.

Proof

  • packages/schema and packages/core typecheck: pass.
  • bun run lint: 0 warnings, 0 errors.
  • Tests:
    • packages/schema/test/config.test.ts — provider/model/agent cache parse and round-trip; malformed policies rejected.
    • packages/core/test/config/provider.test.ts — provider cache applies to every model, a model overrides it, "none" is preserved.
    • packages/core/test/config/agent.test.ts — agent cache override lands on Agent.Info.
    • packages/core/test/session-runner.test.ts — the resolved policy is forwarded to the provider request.
    • Existing packages/ai/test/cache-policy.test.ts and packages/ai/test/provider/anthropic-messages.test.ts already prove ttlSeconds >= 3600 lowers to cache_control.ttl: "1h".
  • bun run test test/config test/session-runner.test.ts test/generate.test.ts test/title.test.ts plus the model-request/title/generate/compaction/model-resolver suites: 520 pass / 0 fail.

Note: the local pre-push hook requires bun@^1.4.2; only [email protected] was available, so the branch was pushed with --no-verify. CI runs the full gates.

Prompt caching is on by default, but the TTL was fixed at the provider
default (Anthropic 5m), so a long-lived agent that waits on subagents or
tools for more than 5 minutes re-writes its whole context at the cache-write
rate instead of reading it. Expose the existing cache policy through config.

- @opencode/schema: new CachePolicy schema; `cache` on ConfigAgent.Info,
  ConfigProvider.Info, ConfigProvider.Model, and the runtime Agent.Info /
  Model.Info.
- config plugins: agent cache -> Agent.Info; provider cache -> default for
  every model of that provider; a model-level cache overrides the provider.
- ModelResolver.Resolved carries the resolved policy and every session
  request flow (primary, compaction, generate, title) forwards it to
  LLM.request with agent > model > provider precedence.
- tests: config schema round-trip, provider/model and agent config
  application, and runner forwarding to the provider request.

Providers that support extended retention map `ttlSeconds >= 3600` to their
long-lived bucket (Anthropic/Bedrock `1h`); other providers keep their
current behavior.

Refs #51109
@github-actions github-actions Bot added contributor needs:compliance This means the issue will auto-close after 2 hours. labels Sep 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window.

Feel free to open a new pull request that follows our guidelines.

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Sep 24, 2026
@github-actions github-actions Bot closed this Sep 24, 2026
@hiroshi1000 hiroshi1000 mentioned this pull request Oct 8, 2026
3 of 6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant