Skip to content

Commit 3700cf9

Browse files
committed
test(task): model and verify native tool-call stream isolation
1 parent 72d0ab5 commit 3700cf9

10 files changed

Lines changed: 756 additions & 17 deletions

File tree

‎.github/workflows/code-qa.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ jobs:
9090
run: pnpm lint
9191
- name: Check types
9292
run: pnpm check-types
93-
- name: Model-check concurrent task lifecycle
93+
- name: Model-check task lifecycle protocols
9494
run: pnpm lifecycle:model-check
9595

9696
build-vsix:
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Native tool-call parser request-scope model check
2+
3+
Zoo Code checks native tool-call parser request isolation with a bounded, exhaustive replay model. It is a child submodel in the umbrella task lifecycle verification suite, which runs in CI and locally with:
4+
5+
```sh
6+
pnpm lifecycle:model-check
7+
```
8+
9+
For focused debugging, run this submodel directly with:
10+
11+
```sh
12+
pnpm parser-scope:model-check
13+
```
14+
15+
The command is composed into the same verification suite, but this remains a separate protocol and state space from the persisted task lifecycle model and shared-store concurrency model. It owns its parser-scoping invariants and adds no parser state to `HistoryItem` or `taskLifecycle.ts`; instead, it replays the public production `NativeToolCallParser` APIs using two independent scope objects.
16+
17+
## Bounds and replay
18+
19+
The source of truth is `scripts/check-native-tool-call-parser-scoping.ts`. The model has two request scopes, A and B. Both receive provider raw tool index zero, but each has a distinct tool-call ID and two distinct JSON argument fragments. Each scope follows this local order:
20+
21+
1. open the request scope;
22+
2. start raw call index zero and its streaming accumulator;
23+
3. add two distinct argument fragments through both production accumulation APIs;
24+
4. finalize the raw call and reject duplicate raw finalization;
25+
5. finalize the streaming call, reject duplicate streaming finalization, and clear both kinds of state; and
26+
6. deliver late raw and streaming fragments.
27+
28+
The checker exhausts all 924 order-preserving interleavings of those two six-action sequences. Opening, raw start, fragment delivery, raw finalization, streaming finalization/cleanup, and late fragment delivery are independently schedulable protocol phases. Fragment delivery remains one bounded action per scope and replays both argument fragments through both production accumulation APIs; streaming cleanup remains attached to streaming finalization because late delivery is the only valid following local phase. This preserves each request's local order while keeping CI runtime bounded. The expected schedule count, maximum schedule budget, scope count, raw index, and actions per scope are explicit. It fails if schedule enumeration differs from the binomial bound or exceeds the budget, so truncated exploration cannot pass.
29+
30+
Each schedule uses fresh production scope objects and calls `processRawChunk`, `startStreamingToolCall`, `processStreamingChunk`, `finalizeRawChunks`, `finalizeStreamingToolCall`, `clearRawChunkState`, `clearAllStreamingToolCalls`, and `hasActiveStreamingToolCalls`. It neither inspects private parser maps nor duplicates their transition logic.
31+
32+
## Invariants and landmarks
33+
34+
Every replay checks:
35+
36+
1. emitted start, delta, and end events retain the owning scope's call ID;
37+
2. finalized arguments contain only the owning scope's fragments;
38+
3. cleanup in one scope cannot change the other scope's active streaming state;
39+
4. each scope emits exactly one raw end and one streaming final result;
40+
5. repeated finalization is empty/null rather than duplicate;
41+
6. late raw and streaming fragments are ignored after cleanup;
42+
7. every modeled action is reachable.
43+
44+
Named landmarks require simultaneous active scopes, B opening while A has received its fragments, either scope raw-finalizing while the other remains active, either scope streaming-finalizing and cleaning up while the other remains active, and symmetric late-fragment schedules in which the other scope is still active.
45+
46+
These are finite safety claims only. The model does not claim provider transport ordering, retry liveness, fairness, persistence, or arbitrary call counts. Provider suites separately test their public stream contracts with two overlapping streams, while focused parser and Task tests cover production integration.

‎docs/architecture/task-lifecycle-model.md‎

Lines changed: 25 additions & 13 deletions
Large diffs are not rendered by default.

‎package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@
1212
"lint": "turbo lint --log-order grouped --output-logs new-only",
1313
"check-types": "turbo check-types --log-order grouped --output-logs new-only",
1414
"test": "turbo test --log-order grouped --output-logs new-only",
15-
"lifecycle:model-check": "tsx scripts/check-task-lifecycle.ts && tsx scripts/check-task-store-concurrency.ts",
15+
"lifecycle:model-check": "tsx scripts/check-task-lifecycle.ts && tsx scripts/check-task-store-concurrency.ts && pnpm parser-scope:model-check",
16+
"parser-scope:model-check": "rm -f .parser-scope-model-check.cjs; esbuild scripts/check-native-tool-call-parser-scoping.ts --bundle --platform=node --format=cjs --external:vscode --outfile=.parser-scope-model-check.cjs && node .parser-scope-model-check.cjs; status=$?; rm -f .parser-scope-model-check.cjs; exit $status",
1617
"test:coverage": "turbo test:coverage --log-order grouped --output-logs new-only",
1718
"format": "turbo format --log-order grouped --output-logs new-only",
1819
"build": "turbo build --log-order grouped --output-logs new-only",
Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
import assert from "node:assert/strict"
2+
3+
import { NativeToolCallParser, type ToolCallStreamEvent } from "../src/core/assistant-message/NativeToolCallParser"
4+
5+
const scopeIds = ["A", "B"] as const
6+
type ScopeId = (typeof scopeIds)[number]
7+
8+
const localActions = [
9+
"open",
10+
"start-raw-call",
11+
"add-fragments",
12+
"finalize-raw-call",
13+
"finalize-streaming-call-and-cleanup",
14+
"late-fragments",
15+
] as const
16+
type LocalAction = (typeof localActions)[number]
17+
18+
interface ScheduledAction {
19+
scopeId: ScopeId
20+
action: LocalAction
21+
}
22+
23+
interface ScopeReplayState {
24+
scope?: object
25+
rawEndCount: number
26+
streamFinalizationCount: number
27+
lateFragmentsIgnored: boolean
28+
}
29+
30+
interface ReplayState {
31+
scopes: Record<ScopeId, ScopeReplayState>
32+
events: Array<{ owner: ScopeId; event: ToolCallStreamEvent }>
33+
}
34+
35+
const RAW_TOOL_INDEX = 0
36+
const MAX_ACTIONS_PER_SCOPE = localActions.length
37+
const MAX_TOTAL_ACTIONS = MAX_ACTIONS_PER_SCOPE * scopeIds.length
38+
const EXPECTED_SCHEDULES = binomial(MAX_TOTAL_ACTIONS, MAX_ACTIONS_PER_SCOPE)
39+
const MAX_SCHEDULES = EXPECTED_SCHEDULES
40+
41+
const callIds = { A: "call_scope_a", B: "call_scope_b" } satisfies Record<ScopeId, string>
42+
const paths = { A: "scope-a.ts", B: "scope-b.ts" } satisfies Record<ScopeId, string>
43+
const fragments = {
44+
A: ['{"path":"scope-', 'a.ts"}'],
45+
B: ['{"path":"scope-', 'b.ts"}'],
46+
} satisfies Record<ScopeId, readonly [string, string]>
47+
48+
const expectedActions = new Set<LocalAction>(localActions)
49+
const reachedActions = new Set<LocalAction>()
50+
const reachedLandmarks = new Set<string>()
51+
52+
const landmarkNames = [
53+
"simultaneous-active-scopes",
54+
"B-opens-while-A-is-partial",
55+
"A-raw-finalizes-while-B-is-active",
56+
"B-raw-finalizes-while-A-is-active",
57+
"A-stream-finalizes-while-B-is-active",
58+
"B-stream-finalizes-while-A-is-active",
59+
"A-late-fragment-while-B-is-active",
60+
"B-late-fragment-while-A-is-active",
61+
] as const
62+
63+
function binomial(n: number, k: number): number {
64+
let result = 1
65+
for (let index = 1; index <= k; index++) {
66+
result = (result * (n - k + index)) / index
67+
}
68+
return result
69+
}
70+
71+
function initialReplayState(): ReplayState {
72+
return {
73+
scopes: {
74+
A: { rawEndCount: 0, streamFinalizationCount: 0, lateFragmentsIgnored: false },
75+
B: { rawEndCount: 0, streamFinalizationCount: 0, lateFragmentsIgnored: false },
76+
},
77+
events: [],
78+
}
79+
}
80+
81+
function activeAtProgress(progress: number): boolean {
82+
return (
83+
progress >= localActions.indexOf("start-raw-call") + 1 &&
84+
progress < localActions.indexOf("finalize-streaming-call-and-cleanup") + 1
85+
)
86+
}
87+
88+
function appendOwnedEvents(state: ReplayState, owner: ScopeId, events: ToolCallStreamEvent[]): void {
89+
for (const event of events) {
90+
state.events.push({ owner, event })
91+
assert.equal(event.id, callIds[owner], `${owner} emitted an event owned by the other request scope`)
92+
}
93+
}
94+
95+
function requireScope(state: ReplayState, scopeId: ScopeId): object {
96+
const scope = state.scopes[scopeId].scope
97+
assert.ok(scope, `${scopeId} must be opened before ${scopeId}'s parser APIs are replayed`)
98+
return scope
99+
}
100+
101+
function replayAction(state: ReplayState, scheduled: ScheduledAction): void {
102+
const { scopeId, action } = scheduled
103+
const scopeState = state.scopes[scopeId]
104+
reachedActions.add(action)
105+
106+
switch (action) {
107+
case "open":
108+
scopeState.scope = NativeToolCallParser.createScope()
109+
break
110+
case "start-raw-call": {
111+
const scope = requireScope(state, scopeId)
112+
const events = NativeToolCallParser.processRawChunk(
113+
{ index: RAW_TOOL_INDEX, id: callIds[scopeId], name: "read_file" },
114+
scope,
115+
)
116+
assert.deepEqual(events, [{ type: "tool_call_start", id: callIds[scopeId], name: "read_file" }])
117+
appendOwnedEvents(state, scopeId, events)
118+
NativeToolCallParser.startStreamingToolCall(callIds[scopeId], "read_file", scope)
119+
break
120+
}
121+
case "add-fragments": {
122+
const scope = requireScope(state, scopeId)
123+
for (const fragment of fragments[scopeId]) {
124+
const events = NativeToolCallParser.processRawChunk(
125+
{ index: RAW_TOOL_INDEX, arguments: fragment },
126+
scope,
127+
)
128+
assert.deepEqual(events, [{ type: "tool_call_delta", id: callIds[scopeId], delta: fragment }])
129+
appendOwnedEvents(state, scopeId, events)
130+
assert.notEqual(
131+
NativeToolCallParser.processStreamingChunk(callIds[scopeId], fragment, scope),
132+
null,
133+
`${scopeId}'s fragment was not accepted by its streaming accumulator`,
134+
)
135+
}
136+
break
137+
}
138+
case "finalize-raw-call": {
139+
const scope = requireScope(state, scopeId)
140+
const events = NativeToolCallParser.finalizeRawChunks(scope)
141+
assert.deepEqual(events, [{ type: "tool_call_end", id: callIds[scopeId] }])
142+
appendOwnedEvents(state, scopeId, events)
143+
scopeState.rawEndCount += events.length
144+
assert.deepEqual(
145+
NativeToolCallParser.finalizeRawChunks(scope),
146+
[],
147+
`${scopeId} emitted a duplicate raw end`,
148+
)
149+
break
150+
}
151+
case "finalize-streaming-call-and-cleanup": {
152+
const scope = requireScope(state, scopeId)
153+
const result = NativeToolCallParser.finalizeStreamingToolCall(callIds[scopeId], scope)
154+
assert.equal(result?.type, "tool_use")
155+
if (result?.type !== "tool_use" || result.name !== "read_file") {
156+
throw new Error(`${scopeId}'s streaming result was not a read_file tool use`)
157+
}
158+
if (!result.nativeArgs || !("path" in result.nativeArgs)) {
159+
throw new Error(`${scopeId}'s streaming result did not use current read_file arguments`)
160+
}
161+
assert.equal(result.nativeArgs?.path, paths[scopeId], `${scopeId}'s arguments crossed request scopes`)
162+
scopeState.streamFinalizationCount += 1
163+
assert.equal(
164+
NativeToolCallParser.finalizeStreamingToolCall(callIds[scopeId], scope),
165+
null,
166+
`${scopeId} finalized its streaming call twice`,
167+
)
168+
NativeToolCallParser.clearRawChunkState(scope)
169+
NativeToolCallParser.clearAllStreamingToolCalls(scope)
170+
break
171+
}
172+
case "late-fragments": {
173+
const scope = requireScope(state, scopeId)
174+
const rawEvents = NativeToolCallParser.processRawChunk(
175+
{ index: RAW_TOOL_INDEX, arguments: `late-${scopeId}` },
176+
scope,
177+
)
178+
const streamingResult = NativeToolCallParser.processStreamingChunk(
179+
callIds[scopeId],
180+
`late-${scopeId}`,
181+
scope,
182+
)
183+
assert.deepEqual(rawEvents, [], `${scopeId} accepted a late raw fragment`)
184+
assert.equal(streamingResult, null, `${scopeId} accepted a late streaming fragment`)
185+
scopeState.lateFragmentsIgnored = true
186+
break
187+
}
188+
}
189+
}
190+
191+
function checkInvariants(state: ReplayState, progress: Record<ScopeId, number>, trace: ScheduledAction[]): void {
192+
for (const scopeId of scopeIds) {
193+
const scopeState = state.scopes[scopeId]
194+
const scope = scopeState.scope
195+
const expectedActive = activeAtProgress(progress[scopeId])
196+
assert.equal(
197+
scope ? NativeToolCallParser.hasActiveStreamingToolCalls(scope) : false,
198+
expectedActive,
199+
`${scopeId}'s active streaming state was changed by the other request scope`,
200+
)
201+
assert.ok(scopeState.rawEndCount <= 1, `${scopeId} emitted duplicate raw finalization events`)
202+
assert.ok(scopeState.streamFinalizationCount <= 1, `${scopeId} finalized its streaming call more than once`)
203+
}
204+
205+
for (const { owner, event } of state.events) {
206+
assert.equal(event.id, callIds[owner], `${owner}'s event log contains another scope's call ID`)
207+
}
208+
209+
const last = trace.at(-1)
210+
if (!last) return
211+
if (activeAtProgress(progress.A) && activeAtProgress(progress.B)) reachedLandmarks.add("simultaneous-active-scopes")
212+
if (last.scopeId === "B" && last.action === "open" && progress.A === 3) {
213+
reachedLandmarks.add("B-opens-while-A-is-partial")
214+
}
215+
if (last.action === "finalize-raw-call" && activeAtProgress(progress[last.scopeId === "A" ? "B" : "A"])) {
216+
reachedLandmarks.add(`${last.scopeId}-raw-finalizes-while-${last.scopeId === "A" ? "B" : "A"}-is-active`)
217+
}
218+
if (
219+
last.action === "finalize-streaming-call-and-cleanup" &&
220+
activeAtProgress(progress[last.scopeId === "A" ? "B" : "A"])
221+
) {
222+
reachedLandmarks.add(`${last.scopeId}-stream-finalizes-while-${last.scopeId === "A" ? "B" : "A"}-is-active`)
223+
}
224+
if (last.action === "late-fragments" && activeAtProgress(progress[last.scopeId === "A" ? "B" : "A"])) {
225+
reachedLandmarks.add(`${last.scopeId}-late-fragment-while-${last.scopeId === "A" ? "B" : "A"}-is-active`)
226+
}
227+
}
228+
229+
function cleanupReplay(state: ReplayState): void {
230+
for (const scopeId of scopeIds) {
231+
const scope = state.scopes[scopeId].scope
232+
if (!scope) continue
233+
NativeToolCallParser.clearRawChunkState(scope)
234+
NativeToolCallParser.clearAllStreamingToolCalls(scope)
235+
}
236+
}
237+
238+
function replaySchedule(trace: ScheduledAction[]): void {
239+
const state = initialReplayState()
240+
const progress: Record<ScopeId, number> = { A: 0, B: 0 }
241+
try {
242+
for (const scheduled of trace) {
243+
replayAction(state, scheduled)
244+
progress[scheduled.scopeId] += 1
245+
checkInvariants(state, progress, trace.slice(0, progress.A + progress.B))
246+
}
247+
for (const scopeId of scopeIds) {
248+
assert.equal(state.scopes[scopeId].rawEndCount, 1, `${scopeId} did not emit exactly one raw end`)
249+
assert.equal(state.scopes[scopeId].streamFinalizationCount, 1, `${scopeId} did not finalize exactly once`)
250+
assert.equal(
251+
state.scopes[scopeId].lateFragmentsIgnored,
252+
true,
253+
`${scopeId}'s late fragments were not checked`,
254+
)
255+
}
256+
} catch (error) {
257+
const formattedTrace = trace
258+
.map(({ scopeId, action }, index) => `${index + 1}. ${scopeId}.${action}`)
259+
.join("\n")
260+
throw new Error(
261+
`Native tool-call parser scope invariant failed within bounds scopes=${scopeIds.length}, actions-per-scope=${MAX_ACTIONS_PER_SCOPE}, schedules=${MAX_SCHEDULES}\n${formattedTrace}`,
262+
{ cause: error },
263+
)
264+
} finally {
265+
cleanupReplay(state)
266+
}
267+
}
268+
269+
function enumerateSchedules(): number {
270+
const trace: ScheduledAction[] = []
271+
const progress: Record<ScopeId, number> = { A: 0, B: 0 }
272+
let exploredSchedules = 0
273+
274+
function visit(): void {
275+
if (trace.length === MAX_TOTAL_ACTIONS) {
276+
exploredSchedules += 1
277+
if (exploredSchedules > MAX_SCHEDULES) {
278+
throw new Error(`Parser-scope exploration exceeded its ${MAX_SCHEDULES}-schedule budget`)
279+
}
280+
replaySchedule(trace)
281+
return
282+
}
283+
284+
for (const scopeId of scopeIds) {
285+
const localProgress = progress[scopeId]
286+
if (localProgress === MAX_ACTIONS_PER_SCOPE) continue
287+
const action = localActions[localProgress]
288+
if (!action) throw new Error(`${scopeId} has no modeled action at local progress ${localProgress}`)
289+
trace.push({ scopeId, action })
290+
progress[scopeId] += 1
291+
visit()
292+
progress[scopeId] -= 1
293+
trace.pop()
294+
}
295+
}
296+
297+
visit()
298+
return exploredSchedules
299+
}
300+
301+
const exploredSchedules = enumerateSchedules()
302+
assert.equal(
303+
exploredSchedules,
304+
EXPECTED_SCHEDULES,
305+
`Parser-scope exploration truncated: expected ${EXPECTED_SCHEDULES} schedules, explored ${exploredSchedules}`,
306+
)
307+
308+
const unreachableActions = [...expectedActions].filter((action) => !reachedActions.has(action))
309+
assert.deepEqual(unreachableActions, [], `Parser-scope model has unreachable actions: ${unreachableActions.join(", ")}`)
310+
const missingLandmarks = landmarkNames.filter((name) => !reachedLandmarks.has(name))
311+
assert.deepEqual(missingLandmarks, [], `Parser-scope model has unreachable landmarks: ${missingLandmarks.join(", ")}`)
312+
313+
console.log(
314+
`Native tool-call parser scope model check passed: ${exploredSchedules}/${EXPECTED_SCHEDULES} valid local-order interleavings, ${localActions.length}/${localActions.length} actions reachable, ${landmarkNames.length}/${landmarkNames.length} landmarks reached, scopes=${scopeIds.length}, raw-index=${RAW_TOOL_INDEX}, actions-per-scope=${MAX_ACTIONS_PER_SCOPE}`,
315+
)

0 commit comments

Comments
 (0)