Skip to content

Commit 793b65b

Browse files
DaubnerFedelauna
andauthored
fix: validate tools against the request's policy snapshot instead of live settings (#1749)
* fix: validate tools against the request's policy snapshot instead of live settings The system prompt for a request is built once, from the settings at that moment, but tool-call validation re-read live settings for every tool call during the streaming response. A disabledTools or custom-mode change landing mid-request therefore made the validator contradict the prompt: it rejected a tool the prompt advertised, or accepted a tool the prompt never advertised (#1642). Capture a RequestPolicySnapshot (disabledTools, experiments, customModes, modelInfo) once per API attempt when the request is built, and thread it as a required parameter through the presentation chain into validation, failing fast when it is missing instead of falling back to a live read. Each API attempt, retries included, captures its own snapshot, so a settings change can affect a later attempt but never the one in flight. Add three src-layer tests covering both drift directions under a mid-stream settings mutation and the missing-snapshot fail-fast case. Closes #1642 * test: cover presenter recursion paths introduced by the policy-snapshot change presentAssistantMessage threads the request policy snapshot through its two self-recursions: draining a completed non-final block into the next block, and re-presenting the current block when an update arrives mid-presentation. Those two call sites were the only changed lines in the file that no test executed, which held the patch coverage report at 77.77% against the 80% codecov target. Add two focused tests to presentAssistantMessage-unknown-tool.spec.ts. The first drains a two-block message through the block-drain recursion and asserts both blocks present in order; the second races a stream update against an in-flight presentation and asserts the pending-updates recursion re-presents the block. Each test executes exactly one of the two recursion lines. * test: add focused Task-level coverage for policy-snapshot wiring attemptApiRequest captures the request policy snapshot and hands it to the presenter, but no Task-level test asserted what the presenter actually receives: the existing tests call the presenter directly with hand-built snapshots, so a wrong or missing Task-side snapshot could still pass the suite. Add a request policy snapshot publication describe block to Task.spec.ts with three tests. The first drives attemptApiRequest with request-time provider state and asserts the presenter receives exactly the snapshot read when the request was built (disabledTools, experiments, customModes, modelInfo) despite later setting changes, and that validation rejects a tool the snapshot disables while live settings no longer do. The second makes the first attempt fail with a context-window error and asserts the retry captures the settings that changed in between while the first attempt's snapshot stays frozen. The third calls the presenter wrapper with no snapshot published and asserts the missing-snapshot throw is logged as a real failure instead of being treated as an abort. Model resolution runs once per request by design: every attempt publishes a snapshot carrying that same resolution, so the prompt and the validator always agree on the model. * test: tighten snapshot-vs-live discrimination in policy-snapshot specs Three test-only changes to the specs covering the request policy snapshot, so each test fails when the code reads live provider state instead of the snapshot published for the request: - The pending-updates re-presentation test (unknown-tool spec) now mutates the streamed block while the first presentation awaits say(), and asserts the content of both say() calls: a presenter that loops with a stale copy of the block fails instead of passing. - The retry test (Task.spec.ts) now asserts experiments and customModes on the re-captured snapshot and on the frozen first-attempt snapshot, not only disabledTools. - A new custom-tool test poisons the live model metadata with an inclusion list that omits a tool the snapshot advertises, and asserts validation receives the snapshot's alias-resolved includedTools: a live read would reject the tool instead of running it. Model resolution stays resolved once per request: every attempt publishes a snapshot carrying that same resolution. * fix(core): validate native mcp_tool_use calls against the request policy snapshot The native mcp_tool_use validation path added in #1724 re-read live provider state (disabled tools, custom modes, experiments) and the live model info on every dispatch. Every other validation path in the streaming presentation chain consumes the policy snapshot published once per API request, so this one could contradict the prompt its own request was generated from: a settings change landing mid-stream would silently change which MCP calls are accepted for the request already in flight. It now validates against the same frozen snapshot, honoring the snapshot-consistency requirement. Task-local mode resolution and the MCP hub server-name lookup stay live reads by design - neither is a policy input, matching the converted tool_use arm. Tests: the native-mcp attribution tests now carry their policy inputs in the snapshot handed to the presenter instead of the provider-state double, and three new tests pin the behavior directly - the snapshot wins in both directions (a snapshot that disables a tool live state allows, and a snapshot that allows a tool live state disables), and a spy proves the native validation path performs no provider-state read at all. The truncated-tool-call test (issue #1221) now has its mocked attemptApiRequest publish the snapshot the way the real generator does, so the presenter's entry guard no longer fires before the guard under test is reached. * test: cover legacy read_file telemetry event's live model id The presenter emits READ_FILE_LEGACY_FORMAT_USED when a completed read_file block used the legacy format, sourcing the event's `model` field from the live api.getModel() read; no test asserted the event or its payload. Add a test that processes a usedLegacyFormat read_file block and pins the captureEvent call to the task id plus the model id returned by the live getModel() double. The double's id is distinct from every other model id the fixture carries, so the assertion fails if the payload ever switches to a different source. --------- Co-authored-by: edelauna <[email protected]>
1 parent adee4d2 commit 793b65b

10 files changed

Lines changed: 1093 additions & 164 deletions

‎src/core/assistant-message/__tests__/presentAssistantMessage-auto-deny.spec.ts‎

Lines changed: 23 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,19 @@ import type { Anthropic } from "@anthropic-ai/sdk"
44
import { describe, it, expect, beforeEach, vi } from "vitest"
55
import { presentAssistantMessage } from "../presentAssistantMessage"
66
import { validateToolUse } from "../../tools/validateToolUse"
7+
import type { RequestPolicySnapshot } from "../../prompts/tools/effective-tool-policy"
78
import type { Task } from "../../task/Task"
89
import type { AskApproval } from "../../../shared/tools"
910

11+
// The presenter validates against this frozen request snapshot; the mocked
12+
// validateToolUse never reads policy fields, and the provider-state double in
13+
// the task mock stays inert.
14+
const baseSnapshot: RequestPolicySnapshot = {
15+
disabledTools: [],
16+
customModes: [],
17+
experiments: {},
18+
}
19+
1020
vi.mock("../../task/Task")
1121
vi.mock("../../tools/validateToolUse", async (importOriginal) => {
1222
const actual = await importOriginal<typeof import("../../tools/validateToolUse")>()
@@ -256,7 +266,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
256266
.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: NOT_ALLOWLISTED_DETAIL })
257267
.mockResolvedValueOnce({ response: "yesButtonClicked" })
258268

259-
await presentAssistantMessage(asTask(mockTask))
269+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
260270

261271
expect(mockTask.userMessageContent).toHaveLength(2)
262272

@@ -302,7 +312,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
302312
.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: GUARD_UNAVAILABLE_DETAIL })
303313
.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: NOT_ALLOWLISTED_DETAIL })
304314

305-
await presentAssistantMessage(asTask(mockTask))
315+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
306316

307317
expect(mockTask.userMessageContent).toHaveLength(2)
308318

@@ -380,7 +390,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
380390
autoDenyDetail: { kind: "denylist", command: "rm x", pattern: "rm" },
381391
})
382392

383-
await presentAssistantMessage(asTask(mockTask))
393+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
384394

385395
expect(mockTask.userMessageContent).toHaveLength(2)
386396

@@ -437,7 +447,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
437447
autoDenyDetail: { kind: "denylist", command: "rm x", pattern: "rm" },
438448
})
439449

440-
await presentAssistantMessage(asTask(mockTask))
450+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
441451

442452
expect(mockTask.userMessageContent).toHaveLength(1)
443453
const payload = JSON.parse(mockTask.userMessageContent[0].content as string)
@@ -457,7 +467,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
457467
// Real user click: noButtonClicked with no structured detail.
458468
mockTask.ask.mockResolvedValueOnce({ response: "noButtonClicked" })
459469

460-
await presentAssistantMessage(asTask(mockTask))
470+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
461471

462472
expect(mockTask.userMessageContent).toHaveLength(2)
463473

@@ -486,7 +496,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
486496

487497
mockTask.ask.mockResolvedValueOnce({ response: "noButtonClicked", text: "do not run that" })
488498

489-
await presentAssistantMessage(asTask(mockTask))
499+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
490500

491501
expect(mockTask.say).toHaveBeenCalledWith("user_feedback", "do not run that", undefined)
492502
expect(mockTask.didRejectTool).toBe(true)
@@ -508,7 +518,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
508518
},
509519
)
510520

511-
await presentAssistantMessage(asTask(mockTask))
521+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
512522

513523
expect(mockTask.ask).toHaveBeenCalledWith("command", "rm x", false, undefined, false, AUTO_APPROVAL_CONTEXT)
514524
})
@@ -541,7 +551,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
541551
},
542552
)
543553

544-
await presentAssistantMessage(asTask(mockTask))
554+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
545555

546556
expect(mockTask.ask).toHaveBeenCalledWith(
547557
"use_mcp_server",
@@ -583,7 +593,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
583593
// wording, and the rest of the turn aborts.
584594
mockTask.ask.mockResolvedValueOnce({ response: "noButtonClicked" })
585595

586-
await presentAssistantMessage(asTask(mockTask))
596+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
587597

588598
expect(mockTask.userMessageContent).toHaveLength(1)
589599
const payload = JSON.parse(mockTask.userMessageContent[0].content as string)
@@ -598,7 +608,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
598608

599609
mockTask.ask.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: DCG_DENY_DETAIL })
600610

601-
await presentAssistantMessage(asTask(mockTask))
611+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
602612

603613
expect(mockTask.userMessageContent).toHaveLength(1)
604614
const payload = JSON.parse(mockTask.userMessageContent[0].content as string)
@@ -646,7 +656,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
646656

647657
mockTask.ask.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: DCG_DENY_DETAIL })
648658

649-
await presentAssistantMessage(asTask(mockTask))
659+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
650660

651661
expect(mockTask.userMessageContent).toHaveLength(1)
652662
const payload = JSON.parse(mockTask.userMessageContent[0].content as string)
@@ -665,7 +675,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
665675
.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: DANGEROUS_SUBSTITUTION_DETAIL })
666676
.mockResolvedValueOnce({ response: "yesButtonClicked" })
667677

668-
await presentAssistantMessage(asTask(mockTask))
678+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
669679

670680
expect(mockTask.userMessageContent).toHaveLength(2)
671681

@@ -698,7 +708,7 @@ describe("presentAssistantMessage - automatic (policy) denials", () => {
698708

699709
mockTask.ask.mockResolvedValueOnce({ response: "noButtonClicked", autoDenyDetail: MALFORMED_COMMAND_DETAIL })
700710

701-
await presentAssistantMessage(asTask(mockTask))
711+
await presentAssistantMessage(asTask(mockTask), baseSnapshot)
702712

703713
expect(mockTask.userMessageContent).toHaveLength(1)
704714
const denialPayload = JSON.parse(mockTask.userMessageContent[0].content as string)

0 commit comments

Comments
 (0)