fix(agents): strip system-event prefix from modelPrompt when before_prompt_build hooks add context#95349
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 11:28 AM ET / 15:28 UTC. Summary PR surface: Source +56, Tests +175. Total +231 across 3 files. Reproducibility: yes. for the reported behavior via the linked live 2026.6.10 Slack DM repro. In this read-only PR pass I did not rerun Slack, but current main source still matches the duplicate path. Review metrics: none identified. Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the narrow runtime-context splitter fix after maintainer review, letting the linked issue close when this PR merges. Do we have a high-confidence way to reproduce the issue? Yes for the reported behavior via the linked live 2026.6.10 Slack DM repro. In this read-only PR pass I did not rerun Slack, but current main source still matches the duplicate path. Is this the best way to solve the issue? Yes. The splitter sees both hidden runtime context and the hook-mutated modelPrompt, so removing only the extracted hidden prompt context there is narrower than changing Slack enqueue or drained-event formatting. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 4010b81a77f4. Label changesLabel justifications:
Evidence reviewedPR surface: Source +56, Tests +175. Total +231 across 3 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
54c6c84 to
6eda65e
Compare
9a659df to
ce36a1e
Compare
…rompt_build hooks add context
c560f9f to
8fb67c7
Compare
|
Maintainer fixups are pushed on the PR branch. Implementation review:
Verification:
Known proof gap: I did not repeat the live Slack model turn because the defect is in generic prompt construction; issue #95323 already contains a live reproduction, and the exact prompt algebra is covered by focused tests. Fresh hosted CI is now running on exact head |
|
Merged via squash.
|
…rompt_build hooks add context (openclaw#95349) * fix(agents): strip system-event prefix from modelPrompt when before_prompt_build hooks add context * fix(agents): preserve prompt boundaries when stripping runtime context * test(agents): assert nested prompt text in boundary regression * fix(agents): anchor runtime context stripping to prompt boundaries * fix(agents): prefer the active prompt boundary * fix(agents): avoid rewriting prompts without hidden context * fix(agents): make prompt stripping helper total * test(agents): drop synthetic prompt boundary proof * fix(agents): anchor prompt stripping to hook boundaries * fix(agents): preserve normalized prompt boundaries * fix(agents): carry prompt build boundaries --------- Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>
Summary
System: [2026-06-20 13:59:51] Slack DM from Alice— to the model twice in the same turn: once inside the hidden runtime-context custom message (Message A) and again inline in the model-facing user turn (Message B). Both occurrences carry the same timestamp, confirming a single event rendered at two sites.before_prompt_buildhooks (or queued next-turn injections) returnprependContextorappendContext— asactive-memory,memory-lancedb, andagent_turn_preparehooks do —attempt.tssetshasPromptBuildContext = trueand passespromptForModelBeforeRuntimeContextSplit(the full effective prompt after hooks, which includes the system-event prefix already prepended toqueuedBody) as themodelPromptargument toresolveRuntimeContextPromptParts. The function correctly extracts the system-event prefix intohiddenRuntimeContext/runtimeContext(→ Message A via the hidden custom transcript message). However,modelPromptTextis taken frommodelPrompt.text, which still contains the prefix verbatim. BecausemodelPromptText !== transcriptPrompt, the function returns it aspromptSubmission.modelPrompt, whichbuildCurrentInboundPromptwraps withcurrentInboundContextto form Message B — event prefix intact.hiddenRuntimeContext, strip it frommodelPromptTextusingremoveLastPromptOccurrencebefore returning.lastIndexOftargets the last occurrence so hook content that coincidentally contains the same literal is not mis-targeted; the?? modelPromptTextfallback makes the branch a safe no-op when the substring is absent. The guardhiddenRuntimeContext && modelPromptis falsy in every pre-existing code path (eitherhiddenRuntimeContextresolves to""whenmodelPrompt.text === extracted.text, ormodelPromptis undefined), so the new branch is unreachable for all existing callers.src/agents/embedded-agent-runner/run/runtime-context-prompt.ts— computereturnModelPromptTextby strippinghiddenRuntimeContextfrommodelPromptTextwhen both are non-empty; use it in the final return.src/agents/embedded-agent-runner/run/runtime-context-prompt.test.ts— two regression tests: prepend-context hook case and append-context hook case; both fail onmainand pass after the fix.src/agents/embedded-agent-runner/run/attempt.llm-boundary.test.ts— model-boundary regression: asserts that after the fix, wheninsertRuntimeContextMessageForPrompt+normalizeMessagesForLlmBoundaryassemble the LLM-bound message array, the system-event label appears only in the runtime-context custom message (Message A) and not in the model-facing user turn (Message B).!prompt.trim()(runtime-event) andemptyTranscriptMode === "model-prompt"early-return branches are untouched.Reproduction
before_prompt_buildreturnsprependContexton a memory hit (or any plugin that queues next-turn context injections).System: [timestamp] Slack DM from <user>inline, before the actual message text; the same label also appears in the preceding hidden runtime-context custom message — two occurrences, same timestamp.Real behavior proof
Behavior addressed (#95323, Slack inbound system event double-render): when a
before_prompt_buildhook (active-memory, memory-lancedb, or any next-turn injection) adds prepend or append context,resolveRuntimeContextPromptPartsnow returnsmodelPromptwithout the system-event prefix and places it exclusively inruntimeContext; the model receives the event exactly once per turn.Real environment tested (Linux, Node 22 — Vitest against the production
resolveRuntimeContextPromptPartsandinsertRuntimeContextMessageForPrompt+normalizeMessagesForLlmBoundaryimplementations with inputs matching theattempt.tscall site for the hook+system-event scenario):promptForRuntimeContextSplitas the pre-hook queuedBody (system-event prefix + user text),transcriptPromptas transcriptCommandBody (user text only),modelPromptaspromptForModelBeforeRuntimeContextSplit(hook context + queuedBody); model-boundary assembly usesinsertRuntimeContextMessageForPromptwith the extracted runtime-context message and the stripped model-facing user message.Exact steps or command run after this patch:
node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/runtime-context-prompt.test.ts src/agents/embedded-agent-runner/run/attempt.llm-boundary.test.ts extensions/slack/src/monitor/message-handler/prepare.test.ts src/auto-reply/reply/get-reply-run.media-only.test.ts.Evidence after fix (Vitest output):
Observed result after fix: in the new model-boundary test,
contextMsg.contentcontains the system-event label (Message A);String(userTurn.content)does not (Message B). In the two new helper tests,resolveRuntimeContextPromptPartsreturnsmodelPrompt: "prependContext\n\nuser text"andruntimeContext: "System: [ts] Slack DM from Alice"— event absent from the model prompt, present only in runtime context.What was not tested: a live Slack session with a real active-memory hit end-to-end. The production flow from
drainFormattedSystemEvents→buildReplyPromptBodies→queuedBody→attempt.tscall site →resolveRuntimeContextPromptParts→insertRuntimeContextMessageForPrompt→normalizeMessagesForLlmBoundaryis covered at each layer by the focused unit tests; a full gateway round-trip was not driven.Repro confirmation: the two new helper tests exercise the exact production input shapes (
promptForRuntimeContextSplit,transcriptPromptForRuntimeSplit,promptForModelBeforeRuntimeContextSplit) fromattempt.ts:4193-4210; both fail onmainwithout the stripping step and pass after it. The model-boundary test confirms the resulting message structure has the event label only once.Risk / Mitigation
removeLastPromptOccurrencecould incorrectly target hook-injected content that happens to be identical to the system-event text. Mitigation:lastIndexOfremoves the last occurrence; since the system event is prepended toqueuedBodybefore hooks wrap it, the system-event text appears after (in string position) any hook prependContext — so the last occurrence is always in the queuedBody portion, not the hook portion.modelPromptTextthat does not actually contain the system-event label (e.g. a path where hooks rewrote the body). Mitigation:removeLastPromptOccurrencereturnsnullwhen the substring is absent; the?? modelPromptTextfallback leavesreturnModelPromptTextunchanged, and the final return behaves identically to the pre-fix code.Change Type (select all)
Scope (select all touched areas)
Linked Issue/PR
Fixes #95323