fix(server): the auto-settle sweep no longer rereads every idle branch every few minutes - #36371

The real-app finding is right and is addressed. Why it still happened: the recheck deadline (max(30 minutes, idle time / 12), capped at a day) left every idle branch due in the same minute, so with 1,030 threads the settlement sweep, and the discovery sweep while a client is in the foreground, reread all of them in one burst at the 30-minute mark, and would again after a restart. Change in this commit (apps/server/src/orchestration-v2/noPullRequestRechecks.ts, used by ThreadSettlementService.ts and ThreadPullRequestService.ts): a scheduled sweep selects at most four due no-PR branch groups, longest unread first and most recently active first among never-read ones, and leaves the rest to later sweeps, so a worker's scheduled lookups are bounded at four a minute regardless of thread count. A selected group is stamped before its lookup so a group that keeps failing cannot hold a place in every sweep. Not capped: groups whose last lookup found a pull request (looked up every sweep as on main), a merge event, a close command in the thread's shell, per-thread sweeps from a thread's own events, the discovery backfill's settled threads, and discovery groups with a saved branch pull request. I did not rerun the 1,030-thread, 37-minute scenario (it does not fit this step's budget). What I ran: an isolated real server (T3CODE_DEV_SEED=0, auto-settle 7 days, local bare origin) with 24 devReplay threads in separate worktrees, restarted so no lookups were remembered; each of the first two settlement sweeps made 4 branchPullRequest lookups and 60 git commands, where every due group was looked up in one sweep before. A new unit test covers the cap with 10 branches (4 on the first sweep, all 10 once by the sixth). Trade-off: with many idle branches a full pass is slow, about 4.3 hours per worker for 1,030 groups, so a pull request opened, merged or closed outside T3 on an idle branch with no known pull request can go unnoticed by the scheduled sweeps for that long, or up to a day for long-idle threads. Macroscope's approvability verdict on 53b36c8 is Needs Human Approval for exactly that trade-off; it raised no code finding. CodeRabbit raised one new finding rated Minor, left because it is below the high-or-above bar, and it is accurate: a failed lookup keeps its pre-lookup stamp, so a transient failure delays that branch's next scheduled lookup by the full deadline instead of a shorter retry. Its earlier Minor finding (the discovery recheck key omits the project's workspace root) is also still open. All 35 checks passed on 53b36c8 and the pull request is no longer watched.

Fixes #35319

Reproduction on main

Commit 9efd960 (origin/main). Started an isolated server: node apps/server/src/bin.ts serve --base-dir <fresh home> --dev-url <dummy> with T3CODE_DEV_SEED=0 and settings.json {sidebarAutoSettleAfterDays: 7, default model devReplay, real providers disabled}. Created a git repo with a local bare origin, t3 project add it, then ran t3 thread start <repo> 'hello N' --provider devReplay --model multi-turn --worktree main --branch wt-N --wait 30 times (plus one thread that failed to provision) and pushed wt-1..wt-30 to origin. Left the server idle about 9 minutes and counted runGitCommand spans per minute in userdata/logs/server.trace.ndjson, attributing them by traceId to ThreadSettlementServiceV2.sweep. Result: one full sweep every minute; minutes 2-4 ran 0 git commands (cache hits); at minute 5-6 the sweep reran 186 git commands for 31 (cwd, branch) groups (138 + 48), then 0 again until the next 5-minute window. Per group: GitManager.branchPullRequest.remotes, branchPullRequest.branchRef, resolveDefaultBranchName, 3 readConfigValue calls and probeWorkTree/detectRepository. So the cost is about 6 git commands per idle unsettled branch thread every 5 minutes, linear in thread count; at the reporter's scale (thousands of threads) that is hundreds a minute. I did not reproduce the queue starvation itself (that needs thousands of threads), and my remote was a local path, not GitHub.

Evidence

Create 120 completed replay threads, observe 31.50 real minutes, then add 1,880 idle MCP-launched branch threads per side. Keep foreground activity active so scheduled PR discovery runs; issue command and VCS probes throughout. Use real authenticated GitHub reads and local-only branch pushes. Compare the final 8.50-minute window after event discovery drained.

Before	After
c83e78fe073d	53b36c8d8a01
Idle recording, 60x	Idle recording, 60x
30,557 scheduled git commands; 453 GitHub queries in 8.50 minutes	960 scheduled git commands; 16 GitHub queries in the same window
Full scheduled sweeps reached 2,000 branch lookups	Both scheduled services stayed at four branch lookups per sweep

Native traces show a completed base settlement sweep with 28,349 git commands over 557 seconds and a completed base PR discovery sweep with 30,000 git commands over 457 seconds. Head also performed 32 scheduled PR discovery rechecks after actual gaps of 30.44–44.57 minutes. Final-window command dispatch p95 was 358.6 ms on base versus 44.6 ms on head; local VCS status p95 was 1,113.9 ms versus 305.4 ms. All probes succeeded, so indefinite starvation was not demonstrated.

Driver and seed scripts · Analyzed native span data · Redacted base traces · Redacted head traces · Full-speed recordings.

The scale fixture contains 120 completed replay threads and 1,880 empty idle launches. Live dashboard totals include foreground VCS probes; the comparison totals above include only scheduled descendants. Shutdown interrupted one subsequent base sweep in each service; their completed child commands remain included in the observation window.

Not shown:

• Indefinite command-queue starvation was not reproduced: every foreground command and VCS probe succeeded, although base latency was higher.
• The 2,000-thread case contains 120 completed replay turns plus 1,880 idle MCP launches without prompts; 2,000 completed agent turns were not run.
• Same-branch scheduled settlement rechecks after 30 minutes were not directly observed: provider idle-shutdown events refreshed initial threads, and newly created never-checked branches took priority. Scheduled PR discovery did show 32 actual rechecks after 30.44–44.57-minute gaps.
• Exhaustive revisiting of all 2,000 branches, longer adaptive recheck delays, and the 24-hour ceiling were outside the observation budget.

Tests

vp test run src/orchestration-v2/ThreadPullRequestService.test.ts src/orchestration-v2/ThreadSettlementService.test.ts: 62 passed (one new test for the per-sweep cap). vp run check:changed: pass (format, lint, knip, graphql, typecheck of 3 packages, tests in 4 files). Real server, 24 idle replay threads, restarted: 4 branch lookups and 60 git commands per settlement sweep in each of the first two minutes. CI on 53b36c8: all 35 checks passed, including the 8 required ones. Not run: the 1,030-thread, 37-minute scenario with foreground clients and a real GitHub origin.

Not in this change

• Threads whose branch has an open pull request still get a lookup on every minute sweep; GitManager caches an open answer for 60 seconds, so those keep costing a host query a minute each. The issue is about threads without a pull request, so this was left alone.
• Groups whose lookup fails (git error, ambiguous remotes) are retried on every sweep; only GitManager's host-failure backoff limits them.
• GitManager's PR lookup cache TTLs and 2,048-entry capacities are unchanged.
• The sweep still reads all settlement candidates and checks each worktree's .git on disk every minute.
• Threads whose branch has an open pull request are still looked up on every minute sweep by both workers; GitManager caches an open answer for 60 seconds.
• Discovery groups with a saved merged or closed branchPullRequest are still looked up on every sweep.
• Groups whose lookup fails are retried on every sweep; only GitManager's host-failure backoff limits them.
• The two sweeps keep separate recheck state instead of one shared service, so each reads a no-PR branch once per deadline.
• Both sweeps still read all candidate threads and check each worktree's .git on disk every minute.
• CodeRabbit's Minor finding: the discovery sweep's recheck key omits the project's workspace root, so a changed project root can inherit the old root's no-PR deadline for threads without a usable worktree.
• A single no-PR branch lookup cost 15 git commands in my run, against the 6 the triage counted; GitManager's per-lookup cost, cache TTLs and capacities are unchanged.
• CodeRabbit's Minor finding: a failed lookup is delayed like a no-PR answer (up to a day) instead of getting a shorter retry window.
• CodeRabbit's Minor finding: the discovery sweep's recheck key omits the project's workspace root.
• Threads whose branch lookup last found a pull request are still looked up on every minute sweep by both workers, uncapped; GitManager caches an open answer for 60 seconds.
• The two sweeps keep separate recheck state instead of one shared service, so each has its own cap of four and reads a no-PR branch once per deadline.
• A single no-PR branch lookup costs about 15 git commands in the real server; GitManager's per-lookup cost, cache TTLs and capacities are unchanged.

Who did the work

• Written by Claude Fable 5.2, in Claude Code
• Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
• Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5

Run by the t3code factory in T3 Code.