Skip to content

fix(agents): add per-install prompt cache partition key to prevent cross-install cache-slot contention#25977

Closed
Water-Enjoyer wants to merge 3 commits into
openclaw:mainfrom
Water-Enjoyer:prompt-cache-partition
Closed

fix(agents): add per-install prompt cache partition key to prevent cross-install cache-slot contention#25977
Water-Enjoyer wants to merge 3 commits into
openclaw:mainfrom
Water-Enjoyer:prompt-cache-partition

Conversation

@Water-Enjoyer

@Water-Enjoyer Water-Enjoyer commented Feb 25, 2026

Copy link
Copy Markdown

Summary

Code changes in this PR were made with the help of AI.

Describe the problem and fix in 2–5 bullets:

  • Problem: prompt-cache locality was weak because OpenClaw did not enforce a stable per-install cache partition at the very start of the system prompt.
  • Why it matters: shared early prompt prefixes can cause cache slot contention across installs, reducing cache-read tokens and increasing cost/latency.
  • What changed: added an auto-generated, persisted agents.defaults.promptCachePartition, injected as the first system-prompt line, and threaded into OpenRouter prompt_cache_key as an additive routing hint.
  • What did NOT change (scope boundary): Anthropic-specific caching behavior and existing cache-control handling were not modified.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

User-visible / Behavior Changes

  • New config field: agents.defaults.promptCachePartition.
  • If missing, it is auto-generated and persisted on config load.
  • System prompt now starts with <!-- openclaw-cache-partition:<key> --> (including promptMode: "none").
  • For OpenRouter only, outbound payload now includes prompt_cache_key from this partition unless already set or overridden.

Security Impact (required)

  • New permissions/capabilities? (No)
  • Secrets/tokens handling changed? (No)
  • New/changed network calls? (No)
  • Command/tool execution surface changed? (No)
  • Data access scope changed? (No)
  • If any Yes, explain risk + mitigation:

Repro + Verification

Environment

  • OS: Linux
  • Runtime/container: local dev runtime
  • Model/provider: OpenRouter (openai-completions), Anthropic path unchanged
  • Integration/channel (if any): N/A
  • Relevant config (redacted):
    • agents.defaults.promptCachePartition = <auto-generated 32-hex>

Steps

  1. Start with config missing agents.defaults.promptCachePartition.
  2. Load config once.
  3. Build system prompt and inspect first line; run OpenRouter payload path.

Expected

  • Config gets persisted with stable partition key.
  • System prompt first line contains partition marker.
  • OpenRouter payload gets prompt_cache_key (unless already present/overridden).

Actual

  • Matches expected in tests.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios:
    • Personally verified cache tokens working via manual testing on OpenRouter.
  • What you did not verify:
    • Providers other than OpenRouter

Compatibility / Migration

  • Backward compatible? (Yes)
  • Config/env changes? (Yes)
  • Migration needed? (No)

Failure Recovery (if this breaks)

  • How to disable/revert this change quickly:
    • Remove/ignore agents.defaults.promptCachePartition and revert this PR.
  • Files/config to restore:
    • src/agents/prompt-cache-partition.ts
    • src/config/io.ts
    • src/agents/system-prompt.ts
    • src/agents/pi-embedded-runner/extra-params.ts
  • Known bad symptoms reviewers should watch for:
    • System prompt missing the partition marker on line 1.
    • OpenRouter payload missing prompt_cache_key despite a configured partition.

Risks and Mitigations

List only real risks for this PR. Add/remove entries as needed. If none, write None.

  • Risk: HTML comment visible to LLM adds ~10 tokens per request
    • Mitigation: negligible vs. the thousands of tokens saved per turn by proper caching

Greptile Summary

This PR adds per-install prompt cache partitioning to improve cache-read token efficiency and reduce cross-install cache slot contention. The implementation introduces an auto-generated, persisted agents.defaults.promptCachePartition config field that is injected as the first line of system prompts and threaded into OpenRouter's prompt_cache_key parameter.

Key changes:

  • New prompt-cache-partition.ts module with ensurePromptCachePartition generates a 32-character hex key if missing
  • Config I/O automatically generates and persists the partition key on first load (similar to ownerDisplaySecret pattern)
  • System prompt prepends <!-- openclaw-cache-partition:<key> --> as line 1 for all prompt modes including "none"
  • OpenRouter wrapper injects prompt_cache_key from partition unless already set or overridden via model params
  • Anthropic-specific caching behavior unchanged (scope boundary respected)

The implementation follows existing patterns for auto-generated config fields, includes comprehensive test coverage, and maintains backward compatibility.

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • The implementation follows established patterns in the codebase (similar to ownerDisplaySecret auto-generation), includes comprehensive unit tests for all new functionality, maintains backward compatibility, and has a clear scope boundary that explicitly excludes Anthropic-specific behavior. The changes are additive and non-breaking, with proper test coverage for edge cases like missing configs, existing values, and provider-specific integration.
  • No files require special attention

Last reviewed commit: 52b6e53

(2/5) Greptile learns from your feedback when you react with thumbs up/down!

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M labels Feb 25, 2026
@svenssonaxel

Copy link
Copy Markdown
Contributor

The scope is good and I expect the solution to partly solve the problem. However, it's not obvious that the prompt cache partition key should be unique per install. I can see a few scenarios where more granular partitions would be more suitable:

  • (A) A user runs several agents on the same installation. They are similar enough to receive the same partition key. As mentioned in Feature Request: Support prompt_cache_key metadata for provider-side cache routing #16357, the request rate can then overwhelm a single GPU and break prompt-cache locality. In short, a per-install partition key does nothing to solve Feature Request: Support prompt_cache_key metadata for provider-side cache routing #16357. In this scenario, a per-agent partition key would be more suitable.
  • (B) A user runs several sessions for the same agent. Analogously with (A), the request rate could then break prompt-cache locality. A per-session partition key would be more suitable.
  • (C) A provider could conceivably evict all previous cache items for the same cache key, at every request. (I say "conceivably" because I don't know how common this is.) On an install using such a provider, it'd be enough to use two sessions to break the cache. Imagine sending a message to session A, then a message to session B. The message to session B would then experience a cache miss for all content that differs from session A, meaning the entire conversation history. A per-session partition key would be more suitable.

It is tempting to conclude that a per-session partition key makes the most sense, but there will be scenarios where it causes a cache miss for the system prompt when a per-install key would not. In order to decide on the best way forward, I believe we first need to know how common scenario (C) is:

  • If (C) is very common, then I believe a per-session partition key is the most suitable.
  • If (C) is very uncommon, then the suitable cache granularity will depend on the usage pattern. It would make sense to let the user configure the cache key granularity by one of three options: install, agent or session, defaulting to install. Users with usage patterns matching (A) or (B) could then configure it to agent or session, respectively.

@svenssonaxel

Copy link
Copy Markdown
Contributor

Discussion continues here: #31708 (comment)

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added stale Marked as stale due to inactivity and removed stale Marked as stale due to inactivity labels Mar 24, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Mar 30, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #pr-thunderdome-dangerzone on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

@openclaw-barnacle openclaw-barnacle Bot closed this Apr 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: M stale Marked as stale due to inactivity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants