Skip to content

perf(system-prompt): move dynamic sections after static content for prefix cache stability#189

Open
BingqingLyu wants to merge 2 commits into
mainfrom
fork-pr-40296-fix-system-prompt-prefix-cache-ordering
Open

perf(system-prompt): move dynamic sections after static content for prefix cache stability#189
BingqingLyu wants to merge 2 commits into
mainfrom
fork-pr-40296-fix-system-prompt-prefix-cache-ordering

Conversation

@BingqingLyu

@BingqingLyu BingqingLyu commented Apr 27, 2026

Copy link
Copy Markdown
Owner

Problem

Dynamic per-channel/session sections (## Messaging, ## Group Chat Context, ## Reactions) were placed before the large static # Project Context block in buildAgentSystemPrompt(). This caused a full prefix cache miss on every request where channel context differed — the LLM had to reprocess the entire Project Context block from scratch.

Measured on a local Qwen3-Coder-30B agent (MLX, Apple M3 Ultra), as reported in openclaw#40256:

Scenario Cache rate Latency
Before (cache miss) 10–16% 10–16s
After (cache hit) ~95%+ 1–2s

Fix

Reorder sections so all static content comes first, dynamic content comes last:

Before: [static] → [dynamic: Messaging/Reactions] → [Project Context] → [Runtime]
After:  [static] → [Project Context] → [Runtime]   → [dynamic: Messaging/Reactions]

The LLM reads the full system prompt regardless of order, so behaviour is identical — only caching changes.

Also moves the ## Workspace Files (injected) header to sit immediately before # Project Context, where it belongs semantically (previously it appeared before the dynamic sections, not the content it was announcing).

Changes

  • src/agents/system-prompt.ts — reorder section assembly in buildAgentSystemPrompt()
  • src/agents/system-prompt.e2e.test.ts — add ordering test so this cannot silently regress

Test

Added a test that asserts # Project Context always appears before ## Messaging, ## Group Chat Context, and ## Reactions:

it("places static Project Context before dynamic Messaging section for prefix cache stability", () => {
  // ...
  expect(messagingPos).toBeGreaterThan(projectContextPos);
  expect(groupChatPos).toBeGreaterThan(projectContextPos);
  expect(reactionsPos).toBeGreaterThan(projectContextPos);
});

All existing tests pass unchanged — none checked section ordering, only presence.

Closes openclaw#40256


  • AI-assisted (Claude Sonnet 4.6)
  • Fully tested — all unit tests pass, ordering test added
  • No behaviour change — only section ordering, LLM sees identical content

…refix cache stability

Dynamic per-channel/session sections (Messaging, Group Chat Context,
Reactions) were placed before the large static Project Context block,
causing a full cache miss on every request where channel context differed.

Reorder so all static content comes first, dynamic content comes last:

  Before: [static] → [dynamic: Messaging/Reactions] → [Project Context] → [Runtime]
  After:  [static] → [Project Context] → [Runtime] → [dynamic: Messaging/Reactions]

Also moves the "Workspace Files (injected)" header to sit immediately
before Project Context, where it belongs semantically.

Fixes openclaw#40256

Measured improvement on local models (Qwen3-Coder-30B via MLX):
- Cache hit rate: ~10-16% → ~95%+
- Follow-up turn latency: 10-16s → 1-2s
- Prompt tokens reprocessed per session: ~800K → ~200K (75% reduction)

Adds a test asserting Project Context always precedes dynamic sections
so this ordering cannot silently regress.
Fixes robustness issue flagged in code review: indexOf could match inside
injected context-file content if a file path or body contained a string
like '## Messaging'. Using lastIndexOf for the dynamic sections (which are
always appended last) makes the ordering assertion reliable regardless of
what context files are injected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

System prompt section ordering breaks LLM prefix caching for local models

2 participants