Skip to content

fix(agents): close remaining prompt cache boundary gaps#60691

Merged
vincentkoc merged 6 commits into
mainfrom
vk/pr-53225-cache-split
Apr 4, 2026
Merged

fix(agents): close remaining prompt cache boundary gaps#60691
vincentkoc merged 6 commits into
mainfrom
vk/pr-53225-cache-split

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

  • Problem: #59054 fixed the cache boundary on several transport builders, but three request paths still bypassed that shaping: compaction, OpenAI WS HTTP fallback, and later turns reusing a previously wrapped session stream.
  • Why it matters: those paths could still leak <!-- OPENCLAW_CACHE_BOUNDARY --> into provider-visible prompt text or skip the intended Anthropic/OpenAI cache shaping, which risks cache misses and divergent prompt bytes across turns.
  • What changed: moved embedded stream resolution into a shared helper, rebuilt each turn from the session's original base stream, routed compaction through the same stream/extra-param/transport stack as normal turns, and made OpenAI WS HTTP fallback use the boundary-aware HTTP builder instead of raw streamSimple.
  • What did NOT change (scope boundary): no new provider-specific cache policy was added for custom plugin streams beyond boundary stripping; no live-cache assertions or pricing logic changed.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause (if applicable)

  • Root cause: cache-boundary logic was added at transport-builder seams, but some agent request paths still bypassed those seams and talked to provider streams directly.
  • Missing detection / guardrail: there was no regression coverage for compaction transport shaping, WebSocket HTTP fallback shaping, or stale wrapped session streams after model/provider changes.
  • Contributing context (if known): #59054 merged before these follow-up path divergences were fully enumerated.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Seam / integration test
    • Unit test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file:
    • src/agents/pi-embedded-runner/run/attempt.test.ts
    • src/agents/pi-embedded-runner/compact.hooks.test.ts
    • src/agents/openai-ws-stream.test.ts
  • Scenario the test should lock in: default turns rebuild from the session's original base stream, compaction goes through the shared transport/extra-param stack, and WS fallback uses the boundary-aware HTTP builder instead of raw streamSimple.
  • Why this is the smallest reliable guardrail: the bug is in orchestration seams between session setup and provider request shaping, not in the individual transport builders themselves.
  • Existing test that already covers this (if any): transport-specific stripping tests already existed for OpenAI/Google/Anthropic builders.
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

  • Provider-visible system prompts no longer diverge across compaction, WS fallback, and later-turn stream reuse paths.
  • Anthropic/OpenAI cache shaping is now consistent across those paths.

Diagram (if applicable)

Before:
[embedded turn/compaction/ws fallback] -> [path-specific stream fn] -> [raw provider request]

After:
[embedded turn/compaction/ws fallback] -> [shared stream resolution] -> [boundary-aware transport shaping] -> [provider request]

Security Impact (required)

  • New permissions/capabilities? (No)
  • Secrets/tokens handling changed? (No)
  • New/changed network calls? (No)
  • Command/tool execution surface changed? (No)
  • Data access scope changed? (No)
  • If any Yes, explain risk + mitigation:

Repro + Verification

Environment

  • OS: macOS
  • Runtime/container: Node 22 / pnpm worktree
  • Model/provider: OpenAI Responses, OpenAI WS fallback, Anthropic-family, provider-owned streams
  • Integration/channel (if any): N/A
  • Relevant config (redacted): default embedded-agent transport paths

Steps

  1. Build an embedded system prompt containing the internal cache boundary.
  2. Exercise compaction, WebSocket fallback, or later-turn session stream reuse.
  3. Inspect the provider-facing stream/payload context.

Expected

  • Provider-visible prompt text does not contain the internal boundary marker.
  • Shared transport shaping is reused across all affected paths.

Actual

  • Some paths bypassed the shared shaping layer and could emit raw boundary text or stale transport behavior.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

  • Verified scenarios: manual code-path audit against Claude Code / Codex cache patterns; diff review of all request-entry seams; regression tests added for stale stream reuse, compaction shaping, and WS fallback builder selection.
  • Edge cases checked: provider-owned createStreamFn, default streamSimple fallback, WS fallback, compaction, later-turn stream wrapper reuse.
  • What you did not verify: live provider cache hit-rate runs; full pnpm build and targeted Vitest were non-reporting in this environment during the final pass.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? (Yes)
  • Config/env changes? (No)
  • Migration needed? (No)
  • If yes, exact upgrade steps:

Risks and Mitigations

  • Risk: session stream resolution now caches the original base stream per session object.
    • Mitigation: cache is a WeakMap, only used to rebuild from the original base stream, and is resettable in tests.
  • Risk: compaction now uses more of the normal embedded turn setup path.
    • Mitigation: scope stays limited to stream resolution, extra params, and transport override; compaction-specific behavior remains unchanged otherwise.

@vincentkoc vincentkoc self-assigned this Apr 4, 2026
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M maintainer Maintainer-authored PR labels Apr 4, 2026
@vincentkoc
vincentkoc marked this pull request as ready for review April 4, 2026 05:24
@greptile-apps

greptile-apps Bot commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR closes three prompt-cache boundary gaps left open after #59054: compaction now flows through the shared resolveEmbeddedAgentStreamFn / applyExtraParamsToAgent / resolveAgentTransportOverride stack, the OpenAI WS HTTP fallback now calls createBoundaryAwareStreamFnForModel instead of raw streamSimple, and a per-session WeakMap cache ensures each normal turn is rebuilt from the original base stream rather than accumulating prior-turn wrappers. The one minor callout is the as never cast in the WS fallback deps initialiser (see inline), which is a style concern and not blocking.

Confidence Score: 5/5

Safe to merge — all three cache-boundary gaps are correctly closed, tests validate the key seams, and no P0/P1 issues were found.

The fix is well-scoped: each affected path (compaction, WS HTTP fallback, later-turn stream reuse) is routed through the boundary-aware transport stack with matching test coverage. The only finding is a P2 style concern (as never cast) that doesn't affect correctness at runtime.

No files require special attention beyond the minor as never cast in src/agents/openai-ws-stream.ts.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/agents/openai-ws-stream.ts
Line: 90-91

Comment:
**`as never` type cast hiding a model type mismatch**

`createBoundaryAwareStreamFnForModel` likely expects a narrower model type than `Parameters<StreamFn>[0]`, so the cast silences a real type incompatibility rather than a false positive. If the shapes are genuinely compatible, a shared intermediate type (e.g. `ProviderRuntimeModel`) would be safer than `as never` here.

```suggestion
  createHttpFallbackStreamFn: (model) =>
    createBoundaryAwareStreamFnForModel(model as ProviderRuntimeModel),
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "Merge branch 'main' into vk/pr-53225-cac..." | Re-trigger Greptile

Comment thread src/agents/openai-ws-stream.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4189946a8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/pi-embedded-runner/compact.ts Outdated
@vincentkoc
vincentkoc merged commit fd01561 into main Apr 4, 2026
18 of 31 checks passed
@vincentkoc
vincentkoc deleted the vk/pr-53225-cache-split branch April 4, 2026 05:41
KimGLee pushed a commit to KimGLee/openclaw that referenced this pull request Apr 4, 2026
* fix(agents): route default stream fallbacks through boundary shapers

* fix(agents): close remaining cache boundary gaps

* chore(changelog): note cache prefix follow-up rollout

* fix(agents): preserve cache-safe fallback stream bases
lovewanwan pushed a commit to lovewanwan/openclaw that referenced this pull request Apr 28, 2026
* fix(agents): route default stream fallbacks through boundary shapers

* fix(agents): close remaining cache boundary gaps

* chore(changelog): note cache prefix follow-up rollout

* fix(agents): preserve cache-safe fallback stream bases
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
* fix(agents): route default stream fallbacks through boundary shapers

* fix(agents): close remaining cache boundary gaps

* chore(changelog): note cache prefix follow-up rollout

* fix(agents): preserve cache-safe fallback stream bases
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
* fix(agents): route default stream fallbacks through boundary shapers

* fix(agents): close remaining cache boundary gaps

* chore(changelog): note cache prefix follow-up rollout

* fix(agents): preserve cache-safe fallback stream bases
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
* fix(agents): route default stream fallbacks through boundary shapers

* fix(agents): close remaining cache boundary gaps

* chore(changelog): note cache prefix follow-up rollout

* fix(agents): preserve cache-safe fallback stream bases
Nachx639 pushed a commit to Nachx639/clawdbot that referenced this pull request Jun 17, 2026
* fix(agents): route default stream fallbacks through boundary shapers

* fix(agents): close remaining cache boundary gaps

* chore(changelog): note cache prefix follow-up rollout

* fix(agents): preserve cache-safe fallback stream bases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant