Repository navigation
Commit 793b65b
fix: validate tools against the request's policy snapshot instead of live settings (#1749)
* fix: validate tools against the request's policy snapshot instead of live settings
The system prompt for a request is built once, from the settings at that
moment, but tool-call validation re-read live settings for every tool call
during the streaming response. A disabledTools or custom-mode change
landing mid-request therefore made the validator contradict the prompt: it
rejected a tool the prompt advertised, or accepted a tool the prompt never
advertised (#1642).
Capture a RequestPolicySnapshot (disabledTools, experiments, customModes,
modelInfo) once per API attempt when the request is built, and thread it
as a required parameter through the presentation chain into validation,
failing fast when it is missing instead of falling back to a live read.
Each API attempt, retries included, captures its own snapshot, so a
settings change can affect a later attempt but never the one in flight.
Add three src-layer tests covering both drift directions under a
mid-stream settings mutation and the missing-snapshot fail-fast case.
Closes #1642
* test: cover presenter recursion paths introduced by the policy-snapshot change
presentAssistantMessage threads the request policy snapshot through its two
self-recursions: draining a completed non-final block into the next block,
and re-presenting the current block when an update arrives
mid-presentation. Those two call sites were the only changed lines in the
file that no test executed, which held the patch coverage report at 77.77%
against the 80% codecov target.
Add two focused tests to
presentAssistantMessage-unknown-tool.spec.ts. The first drains a two-block
message through the block-drain recursion and asserts both blocks present
in order; the second races a stream update against an in-flight
presentation and asserts the pending-updates recursion re-presents the
block. Each test executes exactly one of the two recursion lines.
* test: add focused Task-level coverage for policy-snapshot wiring
attemptApiRequest captures the request policy snapshot and hands it to the
presenter, but no Task-level test asserted what the presenter actually
receives: the existing tests call the presenter directly with hand-built
snapshots, so a wrong or missing Task-side snapshot could still pass the
suite.
Add a request policy snapshot publication describe block to Task.spec.ts
with three tests. The first drives attemptApiRequest with request-time
provider state and asserts the presenter receives exactly the snapshot
read when the request was built (disabledTools, experiments, customModes,
modelInfo) despite later setting changes, and that validation rejects a
tool the snapshot disables while live settings no longer do. The second
makes the first attempt fail with a context-window error and asserts the
retry captures the settings that changed in between while the first
attempt's snapshot stays frozen. The third calls the presenter wrapper
with no snapshot published and asserts the missing-snapshot throw is
logged as a real failure instead of being treated as an abort.
Model resolution runs once per request by design: every attempt publishes
a snapshot carrying that same resolution, so the prompt and the validator
always agree on the model.
* test: tighten snapshot-vs-live discrimination in policy-snapshot specs
Three test-only changes to the specs covering the request policy
snapshot, so each test fails when the code reads live provider state
instead of the snapshot published for the request:
- The pending-updates re-presentation test (unknown-tool spec) now
mutates the streamed block while the first presentation awaits
say(), and asserts the content of both say() calls: a presenter
that loops with a stale copy of the block fails instead of passing.
- The retry test (Task.spec.ts) now asserts experiments and
customModes on the re-captured snapshot and on the frozen
first-attempt snapshot, not only disabledTools.
- A new custom-tool test poisons the live model metadata with an
inclusion list that omits a tool the snapshot advertises, and
asserts validation receives the snapshot's alias-resolved
includedTools: a live read would reject the tool instead of
running it.
Model resolution stays resolved once per request: every attempt
publishes a snapshot carrying that same resolution.
* fix(core): validate native mcp_tool_use calls against the request policy snapshot
The native mcp_tool_use validation path added in #1724 re-read live provider
state (disabled tools, custom modes, experiments) and the live model info on
every dispatch. Every other validation path in the streaming presentation
chain consumes the policy snapshot published once per API request, so this
one could contradict the prompt its own request was generated from: a
settings change landing mid-stream would silently change which MCP calls are
accepted for the request already in flight. It now validates against the
same frozen snapshot, honoring the snapshot-consistency requirement.
Task-local mode resolution and the MCP hub server-name lookup stay live
reads by design - neither is a policy input, matching the converted
tool_use arm.
Tests: the native-mcp attribution tests now carry their policy inputs in the
snapshot handed to the presenter instead of the provider-state double, and
three new tests pin the behavior directly - the snapshot wins in both
directions (a snapshot that disables a tool live state allows, and a
snapshot that allows a tool live state disables), and a spy proves the
native validation path performs no provider-state read at all. The
truncated-tool-call test (issue #1221) now has its mocked attemptApiRequest
publish the snapshot the way the real generator does, so the presenter's
entry guard no longer fires before the guard under test is reached.
* test: cover legacy read_file telemetry event's live model id
The presenter emits READ_FILE_LEGACY_FORMAT_USED when a completed
read_file block used the legacy format, sourcing the event's `model`
field from the live api.getModel() read; no test asserted the event or
its payload.
Add a test that processes a usedLegacyFormat read_file block and pins
the captureEvent call to the task id plus the model id returned by the
live getModel() double. The double's id is distinct from every other
model id the fixture carries, so the assertion fails if the payload
ever switches to a different source.
---------
Co-authored-by: edelauna <[email protected]>1 parent adee4d2 commit 793b65b
10 files changed
Lines changed: 1093 additions & 164 deletions
File tree
- src/core
- assistant-message
- __tests__
- prompts/tools
- task
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
10 | 20 | | |
11 | 21 | | |
12 | 22 | | |
| |||
256 | 266 | | |
257 | 267 | | |
258 | 268 | | |
259 | | - | |
| 269 | + | |
260 | 270 | | |
261 | 271 | | |
262 | 272 | | |
| |||
302 | 312 | | |
303 | 313 | | |
304 | 314 | | |
305 | | - | |
| 315 | + | |
306 | 316 | | |
307 | 317 | | |
308 | 318 | | |
| |||
380 | 390 | | |
381 | 391 | | |
382 | 392 | | |
383 | | - | |
| 393 | + | |
384 | 394 | | |
385 | 395 | | |
386 | 396 | | |
| |||
437 | 447 | | |
438 | 448 | | |
439 | 449 | | |
440 | | - | |
| 450 | + | |
441 | 451 | | |
442 | 452 | | |
443 | 453 | | |
| |||
457 | 467 | | |
458 | 468 | | |
459 | 469 | | |
460 | | - | |
| 470 | + | |
461 | 471 | | |
462 | 472 | | |
463 | 473 | | |
| |||
486 | 496 | | |
487 | 497 | | |
488 | 498 | | |
489 | | - | |
| 499 | + | |
490 | 500 | | |
491 | 501 | | |
492 | 502 | | |
| |||
508 | 518 | | |
509 | 519 | | |
510 | 520 | | |
511 | | - | |
| 521 | + | |
512 | 522 | | |
513 | 523 | | |
514 | 524 | | |
| |||
541 | 551 | | |
542 | 552 | | |
543 | 553 | | |
544 | | - | |
| 554 | + | |
545 | 555 | | |
546 | 556 | | |
547 | 557 | | |
| |||
583 | 593 | | |
584 | 594 | | |
585 | 595 | | |
586 | | - | |
| 596 | + | |
587 | 597 | | |
588 | 598 | | |
589 | 599 | | |
| |||
598 | 608 | | |
599 | 609 | | |
600 | 610 | | |
601 | | - | |
| 611 | + | |
602 | 612 | | |
603 | 613 | | |
604 | 614 | | |
| |||
646 | 656 | | |
647 | 657 | | |
648 | 658 | | |
649 | | - | |
| 659 | + | |
650 | 660 | | |
651 | 661 | | |
652 | 662 | | |
| |||
665 | 675 | | |
666 | 676 | | |
667 | 677 | | |
668 | | - | |
| 678 | + | |
669 | 679 | | |
670 | 680 | | |
671 | 681 | | |
| |||
698 | 708 | | |
699 | 709 | | |
700 | 710 | | |
701 | | - | |
| 711 | + | |
702 | 712 | | |
703 | 713 | | |
704 | 714 | | |
| |||
0 commit comments