Skip to content

Commit 6557665

Browse files
fix(provider): agents wait on a deploy instead of ending the turn (#26681)
Co-authored-by: maria-rcks <[email protected]>
1 parent ea3767b commit 6557665

11 files changed

Lines changed: 36 additions & 32 deletions

File tree

‎apps/server/src/orchestration-v2/testkit/fixtures/multi_turn/muse_transcript.ndjson‎

Lines changed: 6 additions & 6 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/muse_permission/muse_transcript.ndjson‎

Lines changed: 6 additions & 6 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/muse_subagent_approval/muse_transcript.ndjson‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/muse_subagent_interrupt/muse_transcript.ndjson‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/muse_workflow/muse_transcript.ndjson‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/simple/muse_transcript.ndjson‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/subagent_v2/muse_transcript.ndjson‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎apps/server/src/orchestration-v2/testkit/fixtures/turn_interrupt/muse_transcript.ndjson‎

Lines changed: 3 additions & 3 deletions
Large diffs are not rendered by default.

‎evals/mcp-tool-use/cases/pull-requests.ndjson‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,4 @@
2626
{"id": "control-link-means-url", "area": "pull-requests", "kind": "control", "pattern": "'link the PR' means show the URL", "experience": "'Link the PR that did this' about a merged culprit PR means give the URL in the reply, not link_pull_request.", "rule": "Do not link unrelated PRs mentioned only as background.", "privacy": "commit-safe", "setup": {"role": "top", "priorTurns": [{"role": "user", "text": "Why isn't the new thread selector grouping projects anymore?"}, {"role": "assistant", "text": "#120, merged yesterday, removed the grouping step when it moved the selector to the new list component.", "toolCalls": [{"tool": "Bash", "input": {"command": "git log --oneline -S groupByProject -- apps/web/src/ThreadSelector.tsx"}, "result": "8e1f2a3 refactor(web): move thread selector to the shared list (#120)"}, {"tool": "Bash", "input": {"command": "git show 8e1f2a3 --stat"}, "result": "commit 8e1f2a3\n\n refactor(web): move thread selector to the shared list (#120)\n\n apps/web/src/ThreadSelector.tsx | 14 ++------------"}]}], "world": {"prs": [{"number": 120, "title": "refactor(web): move thread selector to the shared list", "state": "MERGED", "headRefName": "refactor/selector"}]}}, "prompt": "Link the pr that did this", "expect": {"notCalls": ["link_pull_request", "watch_pull_request"], "textMust": ["github\\.com/acme/app/pull/120"]}, "notes": "Least certain control: 'link' is ambiguous. The user's next message was 'File revert pr', which shows he wanted to see it, not track it."}
2727
{"id": "control-nightly-time-question", "area": "pull-requests", "kind": "control", "pattern": "question about a CI schedule is not a schedule ask", "experience": "'When does the next nightly go out?' is answered from the workflow file, with no scheduler tools.", "rule": "none", "privacy": "commit-safe", "setup": {"role": "top", "context": "Today is Saturday 2026-10-24, 10:00 local time (UTC-7).", "shellMocks": [{"match": "^date\\b", "output": "Sat Oct 24 10:00:00 PDT 2026"}, {"match": "gh run list", "output": "completed\tsuccess\tnightly\tmain\tschedule\t991\t21m\t2026-10-24T09:00:00Z"}], "world": {"files": {".github/workflows/nightly.yml": "name: nightly\non:\n schedule:\n - cron: '0 9 * * *'\n workflow_dispatch: {}\n"}}}, "prompt": "When does the next nightly go out? Want to start testing this asap", "expect": {"notCalls": ["schedule_task", "list_scheduled_tasks", "run_scheduled_task_now"]}, "notes": ""}
2828
{"id": "control-fix-with-background-pr", "area": "pull-requests", "kind": "control", "pattern": "ordinary code fix that names a PR", "experience": "A plain code fix that mentions the PR which added the bug needs no PR tools.", "rule": "Do not link unrelated PRs mentioned only as background.", "privacy": "commit-safe", "setup": {"role": "top", "world": {"files": {"src/session.ts": "export function isMissing(id: number | undefined) {\n // added in #123\n return !id;\n}\n"}}}, "prompt": "The isMissing guard that #123 added in src/session.ts treats an id of 0 as missing. Fix it. Don't open a PR, I'll review the diff here.", "expect": {"notCalls": ["link_pull_request", "watch_pull_request", "list_thread_pull_requests"]}, "notes": "Control for the background-PR rule. Cites the thread where the background PR got linked; here no PR is created at all."}
29+
{"id": "deploy-wait-ends-turn-unwatched", "area": "pull-requests", "kind": "bad", "pattern": "says it is waiting on a deploy, then ends the turn with nothing to wake it", "experience": "On 'ship it', the agent merged and said 'Waiting on the Vercel production deploy to finish building.', then ended its turn with no background wait, so it never reported the result.", "rule": "Never wait with sleep, loops, gh --watch, Monitor, or ScheduleWakeup.", "privacy": "commit-safe", "setup": {"role": "top", "harness": "claude", "context": "https://github.com/acme/app/pull/123 is linked to this thread and is not watched.", "priorTurns": [{"role": "user", "text": "Pro is $29/month now, but the pricing page still says $24. Fix it."}, {"role": "assistant", "text": "Fixed in [#123](https://github.com/acme/app/pull/123): the Pro plan now costs 2900 cents ($29/month). All checks pass, including the Vercel preview build. Ship it?", "toolCalls": [{"tool": "Edit", "input": {"file_path": "/home/user/.t3/worktrees/t3code/t3-f8cdd940/apps/web/src/pricing/plans.ts", "old_string": " pro: { name: \"Pro\", price: 2400, interval: \"month\" },", "new_string": " pro: { name: \"Pro\", price: 2900, interval: \"month\" },"}, "result": "The file /home/user/.t3/worktrees/t3code/t3-f8cdd940/apps/web/src/pricing/plans.ts has been updated successfully."}, {"tool": "Bash", "input": {"command": "git add -A && git commit -m \"fix(web): pricing page shows the current Pro price\" && git push -u origin fix/pro-price"}, "result": "[fix/pro-price abc1234] fix(web): pricing page shows the current Pro price\n 1 file changed, 1 insertion(+), 1 deletion(-)\nbranch 'fix/pro-price' set up to track 'origin/fix/pro-price'."}, {"tool": "Bash", "input": {"command": "gh pr create --fill"}, "result": "https://github.com/acme/app/pull/123"}, {"tool": "link_pull_request", "input": {"url": "https://github.com/acme/app/pull/123"}, "result": {"host": "github.com", "repository": "acme/app", "number": 123, "url": "https://github.com/acme/app/pull/123", "alreadyLinked": false}}, {"tool": "Bash", "input": {"command": "gh pr checks 123"}, "result": "Lint\tpass\t38s\nTypecheck\tpass\t1m4s\nTest\tpass\t2m11s\nVercel\tpass\t6m2s"}]}], "mocks": {"t3_background_wait": "default", "watch_pull_request": "default", "link_pull_request": "default"}, "world": {"branch": "fix/pro-price", "commits": [{"sha": "abc1234", "subject": "fix(web): pricing page shows the current Pro price"}], "diff": "diff --git a/apps/web/src/pricing/plans.ts b/apps/web/src/pricing/plans.ts\n--- a/apps/web/src/pricing/plans.ts\n+++ b/apps/web/src/pricing/plans.ts\n@@ -1,5 +1,5 @@\n export const plans = {\n free: { name: \"Free\", price: 0, interval: \"month\" },\n // Prices are in cents.\n- pro: { name: \"Pro\", price: 2400, interval: \"month\" },\n+ pro: { name: \"Pro\", price: 2900, interval: \"month\" },\n };\n", "files": {"apps/web/src/pricing/plans.ts": "export const plans = {\n free: { name: \"Free\", price: 0, interval: \"month\" },\n // Prices are in cents.\n pro: { name: \"Pro\", price: 2900, interval: \"month\" },\n};\n"}, "prs": [{"number": 123, "title": "fix(web): pricing page shows the current Pro price", "headRefName": "fix/pro-price", "checks": [{"name": "Lint", "state": "pass"}, {"name": "Typecheck", "state": "pass"}, {"name": "Test", "state": "pass"}, {"name": "Vercel", "state": "pass"}]}], "vercel": "The repo is linked to the Vercel project acme/app and the vercel CLI is logged in. Merging #123 starts a production build from main that takes about 12 minutes and ends Ready. Until then `vercel ls` and `vercel inspect` show it as Building. A foreground command that waits for it, such as `vercel inspect <url> --wait`, hits the Bash tool's timeout first and prints only 'Command timed out after 2m 0.0s' (10m 0.0s with the longest timeout). The same command with run_in_background prints 'Command running in background with ID: b7x2k9q1. Output is being written to: /tmp/claude/tasks/b7x2k9q1.output'."}, "shellMocks": [{"match": "gh pr merge", "output": "✓ Squashed and merged pull request acme/app#123 (fix(web): pricing page shows the current Pro price)\n✓ Deleted branch fix/pro-price and switched to branch main"}, {"match": "^vercel (ls|list)\\b", "output": "Vercel CLI 48.2.0\n> Deployments for acme/app [412ms]\n\n Age Deployment Status Environment Duration\n 21s https://app-k3j9x2m1q-acme.vercel.app ● Building Production --\n 2h https://app-p8d7f6e5w-acme.vercel.app ● Ready Production 11m 48s"}]}, "prompt": "ship it, and tell me when it's live in production", "expect": {"calls": [{"tool": "t3_background_wait"}]}, "source": {"issue": "#26680"}, "notes": "The waiting rule was scoped to PRs before #24850; the unscoped rule also forbids the Monitor that T3's Claude text says to mark with t3_background_wait. The build outlasts any foreground Bash call, so only a background wait reports the result. Claude only: t3_background_wait is a Claude tool.", "maxSteps": 16}

‎packages/provider-core/src/server/runtimeInstructions.test.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,10 @@ describe("buildRuntimeInstructions", () => {
3535
instructions,
3636
"After PR work, a child's last line is: PR <full URL>, head <SHA>, checks and reviews <state>.",
3737
);
38-
assert.include(instructions, "Never wait with sleep");
38+
assert.include(
39+
instructions,
40+
"For PR waits, no sleep, loops, gh --watch, Monitor or ScheduleWakeup.",
41+
);
3942
assert.include(instructions, "call unwatch_pull_request");
4043
});
4144
});

0 commit comments

Comments
 (0)