Skip to content

Commit 220ee89

Browse files
author
Bertan Ari
committed
fix(vscode-lm): address review feedback on leaked tool-call recovery
- dispose the probe CancellationTokenSource in a finally block
1 parent cbac74d commit 220ee89

4 files changed

Lines changed: 78 additions & 20 deletions

File tree

‎.roo/skills/probe-vscode-lm-api/SKILL.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,17 @@ description: How to empirically probe the VS Code Language Model API (`vscode.lm
88
## When to Use This Skill
99

1010
- A claim is being made about what a Copilot-backed model _actually_ emits over `vscode.lm` (tool-call parts vs. text), and it needs evidence rather than inference.
11-
- Changing [`extractLeakedToolCalls()`](src/api/providers/vscode-lm.ts) or its guards, and you need a false-positive corpus.
11+
- Changing [`extractLeakedToolCalls()`](../../../src/api/providers/vscode-lm.ts) or its guards, and you need a false-positive corpus.
1212
- Any question that can only be answered by real `model.sendRequest()` traffic — the mocked unit tests cannot answer it.
1313

1414
## When NOT to Use This Skill
1515

16-
- Ordinary provider work covered by [`src/api/providers/__tests__/vscode-lm.spec.ts`](src/api/providers/__tests__/vscode-lm.spec.ts). Live probing is slow and burns Copilot quota.
16+
- Ordinary provider work covered by [`src/api/providers/__tests__/vscode-lm.spec.ts`](../../../src/api/providers/__tests__/vscode-lm.spec.ts). Live probing is slow and burns Copilot quota.
1717
- Anything about non-`vscode-lm` providers. Anthropic-API behavior does not transfer.
1818

1919
## Running the Probe
2020

21-
Scripts live in [`scripts/`](.roo/skills/probe-vscode-lm-api/scripts/) next to this file.
21+
Scripts live in [`scripts/`](scripts/) next to this file.
2222

2323
1. Copy `scripts/package.json` and `scripts/extension.js` into a scratch directory, e.g. `<repo>\.tmp\lmprobe\`. No build, no `npm install` — it is plain CommonJS against the `vscode` module.
2424
2. Adjust `OUT_DIR` at the top of `extension.js` to the transcript output directory.
@@ -65,7 +65,7 @@ Never `npx vitest` — it resolves a wrong hoisted 3.2.4 instead of the pinned 4
6565

6666
## Measured Findings
6767

68-
Sample: 210 live requests, 7 Copilot Claude models x 6 scenarios x 5 repeats, 0 errors. Raw evidence in [`transcripts/`](.roo/skills/probe-vscode-lm-api/transcripts/).
68+
Sample: 210 live requests, 7 Copilot Claude models x 6 scenarios x 5 repeats, 0 errors. Raw evidence in [`transcripts/`](transcripts/).
6969

7070
| Scenario | Setup | Runs | `<invoke` in text |
7171
| -------- | ----------------------------------------- | ---- | ----------------- |
@@ -90,7 +90,7 @@ Observations:
9090

9191
## Reproducing the False-Positive Replay
9292

93-
`scripts/probe-false-positives.spec.ts` replays [`extractLeakedToolCalls()`](src/api/providers/vscode-lm.ts) over a transcript directory and writes a `RECOVERED`/`passthrough` report. Drop it into `src/api/providers/__tests__/`, point `TRANSCRIPTS` at the transcript directory, then:
93+
[`scripts/probe-false-positives.spec.ts`](scripts/probe-false-positives.spec.ts) replays [`extractLeakedToolCalls()`](../../../src/api/providers/vscode-lm.ts) over a transcript directory and writes a `RECOVERED`/`passthrough` report. Drop it into `src/api/providers/__tests__/`, point `TRANSCRIPTS` at the transcript directory, then:
9494

9595
```
9696
pnpm --dir src exec vitest run api/providers/__tests__/probe-false-positives.spec.ts

‎.roo/skills/probe-vscode-lm-api/scripts/extension.js‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,8 @@ async function runOnce(model, scenario) {
5151
if (scenario.tools) {
5252
options.tools = [READ_TOOL]
5353
}
54+
const source = new vscode.CancellationTokenSource()
5455
try {
55-
const source = new vscode.CancellationTokenSource()
5656
const response = await model.sendRequest(messages, options, source.token)
5757
for await (const chunk of response.stream) {
5858
const typeName = chunk && chunk.constructor ? chunk.constructor.name : typeof chunk
@@ -69,6 +69,8 @@ async function runOnce(model, scenario) {
6969
}
7070
} catch (error) {
7171
record.error = { name: error && error.name, message: error && error.message, stack: error && error.stack }
72+
} finally {
73+
source.dispose()
7274
}
7375

7476
const text = record.concatenatedText

‎src/api/providers/__tests__/vscode-lm.spec.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1331,6 +1331,10 @@ describe("leaked tool-call recovery", () => {
13311331
it("does not hold back an over-long trailing fragment", () => {
13321332
expect(trailingPartialToolMarkerLength("<invoke " + "x".repeat(200))).toBe(0)
13331333
})
1334+
1335+
it("does not hold back an over-long generic tag fragment", () => {
1336+
expect(trailingPartialToolMarkerLength("text <" + "a".repeat(200))).toBe(0)
1337+
})
13341338
})
13351339

13361340
describe("quoted markup", () => {
@@ -1350,6 +1354,24 @@ describe("leaked tool-call recovery", () => {
13501354

13511355
expect(calls).toHaveLength(0)
13521356
})
1357+
1358+
it("does not recover an invoke block quoted in unfenced, backtick-free prose", () => {
1359+
const text = "You must never emit " + invoke("update_todo_list", param("todos", "x")) + " directly."
1360+
1361+
const { calls, leftoverText } = extractLeakedToolCalls(text, new Set(["update_todo_list"]))
1362+
1363+
expect(calls).toHaveLength(0)
1364+
expect(leftoverText).toBe(text)
1365+
})
1366+
1367+
it("keeps wrapper tags around a block that was not recovered", () => {
1368+
const text = `<function_calls>${invoke("some_other_tool", param("x", "1"))}</function_calls>`
1369+
1370+
const { calls, leftoverText } = extractLeakedToolCalls(text, new Set(["update_todo_list"]))
1371+
1372+
expect(calls).toHaveLength(0)
1373+
expect(leftoverText).toBe(text)
1374+
})
13531375
})
13541376
})
13551377

‎src/api/providers/vscode-lm.ts‎

Lines changed: 48 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ const MAX_PARTIAL_INVOKE_CARRY = 64
9393
export function trailingPartialToolMarkerLength(text: string): number {
9494
const partialTag = text.match(/<(?:antml:)?[a-zA-Z_]*$/)
9595
if (partialTag) {
96-
return partialTag[0].length
96+
return partialTag[0].length <= MAX_PARTIAL_INVOKE_CARRY ? partialTag[0].length : 0
9797
}
9898
// An `<invoke` whose `name="` attribute hasn't arrived yet: hold it back so the marker can
9999
// latch on the next chunk, bounded so ordinary prose is never swallowed.
@@ -102,17 +102,25 @@ export function trailingPartialToolMarkerLength(text: string): number {
102102
}
103103

104104
/**
105-
* True when `index` falls inside a fenced code block or an inline code span, where `<invoke>`
106-
* markup is being quoted (e.g. a file excerpt or a "do NOT emit this" example) rather than
107-
* actually invoked.
105+
* True when the block spanning `[index, endIndex)` is being quoted — inside a fenced code block,
106+
* inside an inline code span, or embedded mid-sentence in plain prose — rather than invoked.
108107
*/
109-
function isQuotedAsCode(text: string, index: number): boolean {
108+
function isQuotedAsCode(text: string, index: number, endIndex: number): boolean {
110109
const before = text.slice(0, index)
111110
if ((before.match(/```/g)?.length ?? 0) % 2 === 1) {
112111
return true
113112
}
114113
const lineStart = before.lastIndexOf("\n") + 1
115-
return (before.slice(lineStart).match(/`/g)?.length ?? 0) % 2 === 1
114+
if ((before.slice(lineStart).match(/`/g)?.length ?? 0) % 2 === 1) {
115+
return true
116+
}
117+
// A genuine leak ends the turn's line — nothing follows it but the optional closing wrapper.
118+
// Narrative words after the block on the same line mean the markup is being talked about
119+
// (e.g. "never emit <invoke ...> directly"), which must not be replayed as a live call.
120+
const after = text.slice(endIndex)
121+
const lineEnd = after.indexOf("\n")
122+
const restOfLine = lineEnd === -1 ? after : after.slice(0, lineEnd)
123+
return restOfLine.replace(/<[^<>]*>/g, "").trim().length > 0
116124
}
117125

118126
function parseLeakedInvokeParams(body: string): Record<string, string> {
@@ -136,28 +144,54 @@ export function extractLeakedToolCalls(
136144
precedingText = "",
137145
): { calls: Array<{ name: string; input: Record<string, string> }>; leftoverText: string } {
138146
const calls: Array<{ name: string; input: Record<string, string> }> = []
139-
let leftover = ""
147+
// Text between recovered/passed-through blocks, kept as segments so the wrapper cleanup below
148+
// only touches segments adjacent to a block that was actually recovered.
149+
const segments: Array<{ text: string; nearRecovery: boolean }> = []
150+
let pending = ""
140151
let lastIndex = 0
141152

142153
LEAKED_INVOKE_BLOCK.lastIndex = 0
143154
let match: RegExpExecArray | null
144155
while ((match = LEAKED_INVOKE_BLOCK.exec(text)) !== null) {
145-
leftover += text.slice(lastIndex, match.index)
156+
pending += text.slice(lastIndex, match.index)
146157
const name = match[1]
147158
// Quote detection needs the text streamed before the buffer, since a fence may have opened there.
148-
if (validToolNames.has(name) && !isQuotedAsCode(precedingText + text, precedingText.length + match.index)) {
159+
if (
160+
validToolNames.has(name) &&
161+
!isQuotedAsCode(
162+
precedingText + text,
163+
precedingText.length + match.index,
164+
precedingText.length + match.index + match[0].length,
165+
)
166+
) {
149167
calls.push({ name, input: parseLeakedInvokeParams(match[2]) })
168+
segments.push({ text: pending, nearRecovery: true })
169+
pending = ""
170+
// The segment that follows a recovery also holds that call's closing wrapper.
171+
segments.push({ text: "", nearRecovery: true })
150172
} else {
151173
// Not one of our tools, or quoted as code — keep the block as literal text.
152-
leftover += match[0]
174+
pending += match[0]
153175
}
154176
lastIndex = match.index + match[0].length
155177
}
156-
leftover += text.slice(lastIndex)
178+
pending += text.slice(lastIndex)
179+
const trailing =
180+
segments.length > 0 && segments[segments.length - 1].nearRecovery && segments[segments.length - 1].text === ""
181+
if (trailing) {
182+
segments[segments.length - 1].text = pending
183+
} else {
184+
segments.push({ text: pending, nearRecovery: false })
185+
}
157186

158-
// Remove bare function-call wrapper tags left behind (cosmetic; also avoids re-teaching
159-
// the model this format when the turn is later sent back as history).
160-
leftover = leftover.replace(/<\/?(?:antml:)?function_calls\s*>/gi, "")
187+
// Remove the wrapper tags belonging to a recovered call (cosmetic; also avoids re-teaching the
188+
// model this format when the turn is sent back as history). Wrappers around blocks that were
189+
// NOT recovered are user-visible text and must survive verbatim.
190+
const leftover = segments
191+
.map((segment) =>
192+
segment.nearRecovery ? segment.text.replace(/<\/?(?:antml:)?function_calls\s*>/gi, "") : segment.text,
193+
)
194+
.join("")
161195

162196
return { calls, leftoverText: leftover }
163197
}

0 commit comments

Comments
 (0)