Skip to content

fix(agents): strip system-event prefix from modelPrompt when before_prompt_build hooks add context#95349

Merged
steipete merged 11 commits into
openclaw:mainfrom
openperf:fix/95323-system-event-double-render
Jul 5, 2026
Merged

fix(agents): strip system-event prefix from modelPrompt when before_prompt_build hooks add context#95349
steipete merged 11 commits into
openclaw:mainfrom
openperf:fix/95323-system-event-double-render

Conversation

@openperf

@openperf openperf commented Jun 20, 2026

Copy link
Copy Markdown
Member

Summary

  • Problem: Issue Inbound system event rendered twice per turn (runtime-context message + reply-body prepend) on Slack #95323 reports that a Slack DM (or any channel message that drains a system event) delivers the inbound label — e.g. System: [2026-06-20 13:59:51] Slack DM from Alice — to the model twice in the same turn: once inside the hidden runtime-context custom message (Message A) and again inline in the model-facing user turn (Message B). Both occurrences carry the same timestamp, confirming a single event rendered at two sites.
  • Root Cause: When before_prompt_build hooks (or queued next-turn injections) return prependContext or appendContext — as active-memory, memory-lancedb, and agent_turn_prepare hooks do — attempt.ts sets hasPromptBuildContext = true and passes promptForModelBeforeRuntimeContextSplit (the full effective prompt after hooks, which includes the system-event prefix already prepended to queuedBody) as the modelPrompt argument to resolveRuntimeContextPromptParts. The function correctly extracts the system-event prefix into hiddenRuntimeContext / runtimeContext (→ Message A via the hidden custom transcript message). However, modelPromptText is taken from modelPrompt.text, which still contains the prefix verbatim. Because modelPromptText !== transcriptPrompt, the function returns it as promptSubmission.modelPrompt, which buildCurrentInboundPrompt wraps with currentInboundContext to form Message B — event prefix intact.
  • Fix: After computing hiddenRuntimeContext, strip it from modelPromptText using removeLastPromptOccurrence before returning. lastIndexOf targets the last occurrence so hook content that coincidentally contains the same literal is not mis-targeted; the ?? modelPromptText fallback makes the branch a safe no-op when the substring is absent. The guard hiddenRuntimeContext && modelPrompt is falsy in every pre-existing code path (either hiddenRuntimeContext resolves to "" when modelPrompt.text === extracted.text, or modelPrompt is undefined), so the new branch is unreachable for all existing callers.
  • What changed:
    • src/agents/embedded-agent-runner/run/runtime-context-prompt.ts — compute returnModelPromptText by stripping hiddenRuntimeContext from modelPromptText when both are non-empty; use it in the final return.
    • src/agents/embedded-agent-runner/run/runtime-context-prompt.test.ts — two regression tests: prepend-context hook case and append-context hook case; both fail on main and pass after the fix.
    • src/agents/embedded-agent-runner/run/attempt.llm-boundary.test.ts — model-boundary regression: asserts that after the fix, when insertRuntimeContextMessageForPrompt + normalizeMessagesForLlmBoundary assemble the LLM-bound message array, the system-event label appears only in the runtime-context custom message (Message A) and not in the model-facing user turn (Message B).
  • What did NOT change: Config surface unchanged. Plugin surface unchanged. Gateway protocol unchanged. The !prompt.trim() (runtime-event) and emptyTranscriptMode === "model-prompt" early-return branches are untouched.

Reproduction

  1. Configure a Slack session with the active-memory or memory-lancedb plugin so before_prompt_build returns prependContext on a memory hit (or any plugin that queues next-turn context injections).
  2. Send a Slack DM to the agent.
  3. Inspect the model API request (provider payload or transcript).
  4. Before this PR: the model-facing user turn contains System: [timestamp] Slack DM from <user> inline, before the actual message text; the same label also appears in the preceding hidden runtime-context custom message — two occurrences, same timestamp.
  5. After this PR: the system-event label appears only in the hidden runtime-context message (Message A); the model-facing user turn (Message B) contains only the hook context and the user text.

Real behavior proof

Behavior addressed (#95323, Slack inbound system event double-render): when a before_prompt_build hook (active-memory, memory-lancedb, or any next-turn injection) adds prepend or append context, resolveRuntimeContextPromptParts now returns modelPrompt without the system-event prefix and places it exclusively in runtimeContext; the model receives the event exactly once per turn.

Real environment tested (Linux, Node 22 — Vitest against the production resolveRuntimeContextPromptParts and insertRuntimeContextMessageForPrompt + normalizeMessagesForLlmBoundary implementations with inputs matching the attempt.ts call site for the hook+system-event scenario): promptForRuntimeContextSplit as the pre-hook queuedBody (system-event prefix + user text), transcriptPrompt as transcriptCommandBody (user text only), modelPrompt as promptForModelBeforeRuntimeContextSplit (hook context + queuedBody); model-boundary assembly uses insertRuntimeContextMessageForPrompt with the extracted runtime-context message and the stripped model-facing user message.

Exact steps or command run after this patch: node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/runtime-context-prompt.test.ts src/agents/embedded-agent-runner/run/attempt.llm-boundary.test.ts extensions/slack/src/monitor/message-handler/prepare.test.ts src/auto-reply/reply/get-reply-run.media-only.test.ts.

Evidence after fix (Vitest output):

runtime-context-prompt.test.ts     Tests  23 passed (23)   (includes 2 new)
attempt.llm-boundary.test.ts       Tests  24 passed (24)   (includes 1 new)
prepare.test.ts                    Tests  85 passed (85)
get-reply-run.media-only.test.ts   Tests  82 passed (82)

Observed result after fix: in the new model-boundary test, contextMsg.content contains the system-event label (Message A); String(userTurn.content) does not (Message B). In the two new helper tests, resolveRuntimeContextPromptParts returns modelPrompt: "prependContext\n\nuser text" and runtimeContext: "System: [ts] Slack DM from Alice" — event absent from the model prompt, present only in runtime context.

What was not tested: a live Slack session with a real active-memory hit end-to-end. The production flow from drainFormattedSystemEventsbuildReplyPromptBodiesqueuedBodyattempt.ts call site → resolveRuntimeContextPromptPartsinsertRuntimeContextMessageForPromptnormalizeMessagesForLlmBoundary is covered at each layer by the focused unit tests; a full gateway round-trip was not driven.

Repro confirmation: the two new helper tests exercise the exact production input shapes (promptForRuntimeContextSplit, transcriptPromptForRuntimeSplit, promptForModelBeforeRuntimeContextSplit) from attempt.ts:4193-4210; both fail on main without the stripping step and pass after it. The model-boundary test confirms the resulting message structure has the event label only once.

Risk / Mitigation

  • Risk: removeLastPromptOccurrence could incorrectly target hook-injected content that happens to be identical to the system-event text. Mitigation: lastIndexOf removes the last occurrence; since the system event is prepended to queuedBody before hooks wrap it, the system-event text appears after (in string position) any hook prependContext — so the last occurrence is always in the queuedBody portion, not the hook portion.
  • Risk: stripping could fire on a modelPromptText that does not actually contain the system-event label (e.g. a path where hooks rewrote the body). Mitigation: removeLastPromptOccurrence returns null when the substring is absent; the ?? modelPromptText fallback leaves returnModelPromptText unchanged, and the final return behaves identically to the pre-fix code.

Change Type (select all)

  • feat
  • fix
  • improve
  • refactor
  • docs
  • chore

Scope (select all touched areas)

  • Agents / runner
  • Tests

Linked Issue/PR

Fixes #95323

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 20, 2026
@clawsweeper

clawsweeper Bot commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 11:28 AM ET / 15:28 UTC.

Summary
The PR changes runtime-context prompt splitting to strip hidden system-event context from hook-mutated model prompts and adds focused helper and LLM-boundary regression tests.

PR surface: Source +56, Tests +175. Total +231 across 3 files.

Reproducibility: yes. for the reported behavior via the linked live 2026.6.10 Slack DM repro. In this read-only PR pass I did not rerun Slack, but current main source still matches the duplicate path.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: vector/embedding metadata: src/agents/embedded-agent-runner/run/attempt.llm-boundary.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #95323
Summary: This PR is the candidate fix for the canonical Slack duplicate system-event model prompt issue; related PRs overlap prompt/context symptoms but target different layers or already-fixed preview behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] No live Slack end-to-end provider-payload capture from this PR branch was reviewed; confidence rests on source trace, the linked live 2026.6.10 repro, and the PR-body focused test output.

Maintainer options:

  1. Decide the mitigation before merge
    Land the narrow runtime-context splitter fix after maintainer review, letting the linked issue close when this PR merges.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P2] This member-authored implementation PR is already the concrete candidate fix; the remaining action is maintainer review and normal merge decision, not a new repair job.

Security
Cleared: Cleared: the diff only changes internal prompt string splitting and colocated tests; it does not touch secrets, dependencies, workflows, package metadata, or code-execution surfaces.

Review details

Best possible solution:

Land the narrow runtime-context splitter fix after maintainer review, letting the linked issue close when this PR merges.

Do we have a high-confidence way to reproduce the issue?

Yes for the reported behavior via the linked live 2026.6.10 Slack DM repro. In this read-only PR pass I did not rerun Slack, but current main source still matches the duplicate path.

Is this the best way to solve the issue?

Yes. The splitter sees both hidden runtime context and the hook-mutated modelPrompt, so removing only the extracted hidden prompt context there is narrower than changing Slack enqueue or drained-event formatting.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 4010b81a77f4.

Label changes

Label justifications:

  • P2: The PR addresses a normal-priority agents prompt correctness bug with limited Slack/channel blast radius and no data-loss or security impact.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Not applicable to this member-authored PR gate; the body includes focused post-fix Vitest output but no live Slack branch capture.
Evidence reviewed

PR surface:

Source +56, Tests +175. Total +231 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 1 71 15 +56
Tests 2 175 0 +175
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 246 15 +231

What I checked:

  • Repository policy applied: Root and scoped agents/embedded-runner AGENTS.md files were read fully; their guidance requires deep prompt/runtime review and favors focused helper tests for prompt assembly behavior. (AGENTS.md:10, 4010b81a77f4)
  • Current main still returns hook-mutated modelPrompt unchanged: On current main, resolveRuntimeContextPromptParts extracts runtimeContext but returns modelPromptText whenever it differs from prompt, so an extracted System line can remain in the model-facing prompt. (src/agents/embedded-agent-runner/run/runtime-context-prompt.ts:137, 4010b81a77f4)
  • Caller passes hook-mutated prompt to the splitter: The embedded runner sets hasPromptBuildContext from before_prompt_build prepend/append context and passes promptForModelBeforeRuntimeContextSplit as modelPrompt while using the pre-hook prompt for runtime-context splitting. (src/agents/embedded-agent-runner/run/attempt.ts:4207, 4010b81a77f4)
  • Slack/system-event path matches the reported duplicate label: Slack enqueues a label-only inbound system event, drainFormattedSystemEvents formats it as a System line, and buildReplyPromptBodies prepends drained events to queued/model bodies while keeping transcript text separate. (extensions/slack/src/monitor/message-handler/prepare.ts:1139, 4010b81a77f4)
  • PR-head implementation strips only hidden prompt context around the active prompt: The PR-head helper computes hiddenPromptParts, removes matching hidden before/after context around the active prompt in modelPromptText, and falls back to the original modelPromptText if no anchored match exists. (src/agents/embedded-agent-runner/run/runtime-context-prompt.ts:183, c560f9f82b9f)
  • PR-head tests cover the core edge cases: The PR adds helper tests for prepend context, append context, repeated hook text, repeated prompt text, and a boundary test asserting the system event appears in runtime context but not the model-facing user turn. (src/agents/embedded-agent-runner/run/runtime-context-prompt.test.ts:102, c560f9f82b9f)

Likely related people:

  • vincentkoc: Recent file history and the current PR assignment point to repeated work on the runtime-context splitter and embedded attempt caller, including current-main commits immediately around this surface. (role: recent runtime-context contributor; confidence: high; commits: 7b9ddbda997a, 2a93d7b9c53f, b0ecf6e1e745; files: src/agents/embedded-agent-runner/run/runtime-context-prompt.ts, src/agents/embedded-agent-runner/run/attempt.ts)
  • Alix-007: Commit 8b7a482 added the before_prompt_build model/transcript split behavior that this PR repairs for system-event prefixes. (role: introduced hook prompt-local behavior; confidence: high; commits: 8b7a4826a1ff; files: src/agents/embedded-agent-runner/run/attempt.ts, src/agents/embedded-agent-runner/run/runtime-context-prompt.ts)
  • steipete: Recent file history shows Slack inbound context and auto-reply prompt-prelude work adjacent to the affected system-event pipeline, and steipete merged the related preview-removal PR. (role: adjacent prompt and Slack context contributor; confidence: medium; commits: 82e5dd4da74f, 1507a9701b83, 47545e04c4bd; files: src/auto-reply/reply/prompt-prelude.ts, extensions/slack/src/monitor/message-handler/prepare.ts)
  • hugenshen: Authored the merged Slack/Teams preview-removal work that the linked issue explicitly distinguishes from this remaining label-only double-render bug. (role: adjacent prior fix author; confidence: medium; commits: 47545e04c4bd; files: extensions/slack/src/monitor/message-handler/prepare.ts, extensions/slack/src/monitor/message-handler/prepare.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jun 20, 2026
@openperf
openperf force-pushed the fix/95323-system-event-double-render branch from 54c6c84 to 6eda65e Compare June 21, 2026 05:36
@vincentkoc
vincentkoc force-pushed the fix/95323-system-event-double-render branch 2 times, most recently from 9a659df to ce36a1e Compare June 22, 2026 05:51
@vincentkoc vincentkoc self-assigned this Jun 23, 2026
@steipete
steipete force-pushed the fix/95323-system-event-double-render branch from c560f9f to 8fb67c7 Compare July 5, 2026 08:45
@steipete

steipete commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Maintainer fixups are pushed on the PR branch.

Implementation review:

  • Replaced post-hoc prompt occurrence inference with explicit pre-hook, pre-transform, and pre-annotation boundaries owned by the embedded runner.
  • The model sanitizer now replaces only the active prompt occurrence between the actual prepend/append hook additions, preserving quoted hook text and user whitespace.
  • Queued/orphan transforms, steering-style prefixes, internal-context normalization, and inter-session provenance retain their intended visible/hidden ordering.
  • Removed the synthetic boundary test that did not exercise the production splitter; added focused regressions for the concrete ambiguity classes.

Verification:

  • node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/runtime-context-prompt.test.ts src/agents/embedded-agent-runner/run/attempt-system-prompt.test.ts src/agents/embedded-agent-runner/run/attempt.prompt-helpers.test.ts — 3 files, 44 tests passed.
  • Blacksmith Testbox tbx_01kwrpwnpxs4kbbtefdma7m9yq: pnpm check:changed passed; Actions run https://github.com/openclaw/openclaw/actions/runs/28735045489.
  • Fresh structured autoreview of the fixup: clean (0.93 confidence).
  • Fresh structured autoreview of the full branch: clean (0.88 confidence).
  • Review artifacts validated with zero findings and READY FOR /prepare-pr.

Known proof gap: I did not repeat the live Slack model turn because the defect is in generic prompt construction; issue #95323 already contains a live reproduction, and the exact prompt algebra is covered by focused tests.

Fresh hosted CI is now running on exact head 8fb67c7d866d70f239c327d18f87da596093041d.

@steipete
steipete merged commit e1eac7a into openclaw:main Jul 5, 2026
96 checks passed
@steipete

steipete commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 6, 2026
…rompt_build hooks add context (openclaw#95349)

* fix(agents): strip system-event prefix from modelPrompt when before_prompt_build hooks add context

* fix(agents): preserve prompt boundaries when stripping runtime context

* test(agents): assert nested prompt text in boundary regression

* fix(agents): anchor runtime context stripping to prompt boundaries

* fix(agents): prefer the active prompt boundary

* fix(agents): avoid rewriting prompts without hidden context

* fix(agents): make prompt stripping helper total

* test(agents): drop synthetic prompt boundary proof

* fix(agents): anchor prompt stripping to hook boundaries

* fix(agents): preserve normalized prompt boundaries

* fix(agents): carry prompt build boundaries

---------

Co-authored-by: Vincent Koc <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inbound system event rendered twice per turn (runtime-context message + reply-body prepend) on Slack

3 participants