Skip to content

Commit baf65ee

Browse files
test(evals): busy preview tab recovery case and an ordinary-timeout control
The busy control reason added for #32849 is new agent-facing tool-result text. preview-busy-tab-opens-new-tab resumes a thread right after preview_act returned the busy error, and passes when the first tab-touching preview call is preview_open with reuseExistingTab=false or t3_preview_close on the stuck tab. preview-wait-timeout-keeps-tab is its control: after a plain wait_for timeout the agent keeps working in the same tab. The runner now replays a prior result that decodes as a contracts PreviewAutomationError as T3 sends it, an error result carrying the class's own message, so the case never copies the text. A new firstCall check grades the first call among a set of tools against alternatives. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent b088b88 commit baf65ee

4 files changed

Lines changed: 56 additions & 3 deletions

File tree

‎evals/mcp-tool-use/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,9 @@ Cases are JSON lines in `cases/`, one file per area. The case schema in `run.ts`
2727
- `kind`: `bad` reproduces a mistake, `guard` protects correct behavior, `control` is ordinary work.
2828
- `experience`: the real mistake, in a sentence. `source`: where it happened.
2929
- `setup`: the fake thread, its `world` (repo, branch, files, PRs, pages), `priorTurns`, and `mocks` for tool results.
30+
- `setup.priorTurns[].toolCalls[].result`: a T3 preview error (`{"_tag": "PreviewAutomationControlInterruptedError", ...}`, every field the contracts class needs) replays as T3 sends it: an error result with the class's own message.
3031
- `setup.priorTurns[].toolCalls[].resultFile`: load a repository-relative file as the tool result instead of copying its contents into the case. An empty `prompt` resumes a top-level thread after its last prior tool result without adding a user message. Side-chat and subagent setups require a nonempty prompt.
31-
- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).
32+
- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `firstCall` (the first call of any of its `tools` matches one of its `calls`), `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).
3233

3334
Check that the case builds with `--dry-run --case <id>`. Then run it on the text before your fix, where it must fail, and on the text after, where it must pass.
3435

0 commit comments

Comments
 (0)