perf(system-prompt): move dynamic sections after static content for prefix cache stability#189
Open
BingqingLyu wants to merge 2 commits into
Open
Conversation
…refix cache stability Dynamic per-channel/session sections (Messaging, Group Chat Context, Reactions) were placed before the large static Project Context block, causing a full cache miss on every request where channel context differed. Reorder so all static content comes first, dynamic content comes last: Before: [static] → [dynamic: Messaging/Reactions] → [Project Context] → [Runtime] After: [static] → [Project Context] → [Runtime] → [dynamic: Messaging/Reactions] Also moves the "Workspace Files (injected)" header to sit immediately before Project Context, where it belongs semantically. Fixes openclaw#40256 Measured improvement on local models (Qwen3-Coder-30B via MLX): - Cache hit rate: ~10-16% → ~95%+ - Follow-up turn latency: 10-16s → 1-2s - Prompt tokens reprocessed per session: ~800K → ~200K (75% reduction) Adds a test asserting Project Context always precedes dynamic sections so this ordering cannot silently regress.
Fixes robustness issue flagged in code review: indexOf could match inside injected context-file content if a file path or body contained a string like '## Messaging'. Using lastIndexOf for the dynamic sections (which are always appended last) makes the ordering assertion reliable regardless of what context files are injected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Dynamic per-channel/session sections (
## Messaging,## Group Chat Context,## Reactions) were placed before the large static# Project Contextblock inbuildAgentSystemPrompt(). This caused a full prefix cache miss on every request where channel context differed — the LLM had to reprocess the entire Project Context block from scratch.Measured on a local Qwen3-Coder-30B agent (MLX, Apple M3 Ultra), as reported in openclaw#40256:
Fix
Reorder sections so all static content comes first, dynamic content comes last:
The LLM reads the full system prompt regardless of order, so behaviour is identical — only caching changes.
Also moves the
## Workspace Files (injected)header to sit immediately before# Project Context, where it belongs semantically (previously it appeared before the dynamic sections, not the content it was announcing).Changes
src/agents/system-prompt.ts— reorder section assembly inbuildAgentSystemPrompt()src/agents/system-prompt.e2e.test.ts— add ordering test so this cannot silently regressTest
Added a test that asserts
# Project Contextalways appears before## Messaging,## Group Chat Context, and## Reactions:All existing tests pass unchanged — none checked section ordering, only presence.
Closes openclaw#40256