Skip to content

fix(server): turns recreate a worktree folder that lost its checkout - #34894

Merged
juliusmarminge merged 4 commits into
mainfrom
factory/issue-24390
Nov 13, 2026
Merged

juliusmarminge merged 4 commits into
mainfrom
factory/issue-24390

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

The failing Agent context budget check was not caused by this change. The PR does not change any agent context text. After the previous merge, main lowered apps/server/src/provider/agentContextBudget.json by 3 bytes in 0568a63 (#30371), so the branch's older copy looked like growth. Merging origin/main again (merge commit, no conflicts) brings the budget file to main's values. The PR diff is unchanged: four server files covering worktree recovery in the shared restoreMissingWorktree helper (clears a folder only when it has no .git and holds only node_modules/.next directories), storage cleanup keeping such folders, and their tests.

Fixes #24390

Reproduction on main

In the app, create a worktree thread in a pnpm project and let setup install dependencies. Stop the turn, record its branch, then delete everything in that worktree except node_modules, including .git; optionally run git worktree prune in the project repository. Send another message to the same thread and ask the agent to report its current directory, branch, and tracked files. On main, recovery is skipped and the folder remains without a checkout; with this change, the leftover directory is removed and the thread's checkout is recreated before provider startup. Separately, trigger storage cleanup for an eligible archived thread in the same broken-folder state: main reports a missing-.git failure, whereas this change reports the folder as kept. These steps were read from the issue and PR, not executed.

Evidence

Evidence captured in Chromium at 1280×800, light theme, against fresh isolated servers with synthetic data and Replay. Create a worktree, install pnpm dependencies, leave only node_modules, remove .git, prune, and resume the thread. Separately archive another broken-folder thread and run storage cleanup. Filesystem observations are measured by the driver and displayed in chat, not generated by Replay. Absolute paths are redacted.

Before b5c0ee56fa After 6facbfe5e1
Resumed checkout Before: missing checkout After: original branch and tracked files restored
Archived-folder cleanup Before: missing-.git stat failure, 1 kept and 1 failed After: 2 kept, no failures
Recording Before After

Scenario source · Capture redaction script

scenarios:prove exited 0: failed on base and passed on head.

Not shown:

  • Automatic project setup and interrupting an active turn were not exercised: pnpm was invoked directly in the app-created checkout, and the initial Replay turn finished before folder corruption. A live provider's directory report was not exercised; the driver measured the actual filesystem and git state.

Tests

Compared agentContextBudget.json on the branch and origin/main (5597/7327 vs 5594/7324) and found the main commit that lowered it. After the merge, vp run check:changed passed: format, lint, knip, typecheck (3 packages), tests (3 files). The budget check itself was not rerun locally; CI runs it on the new head.

Not in this change

  • Already-broken folders that hold user files besides node_modules/.next are still not recovered; turns keep running there. Deciding what to do with them needs a maintainer decision.

Who did the work

  • Written by Claude Opus 5.5, in Claude Code
  • Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
  • Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5

Run by the t3code factory in T3 Code.

…d kept

Repro for #24390: turn start skips recreation when the managed worktree
folder exists without a checkout, and the storage cleanup sweep fails on
the missing .git instead of keeping the folder.

AI-assisted; verified with the tests and tools listed above.
A failed removal can leave a thread's managed worktree folder behind with no
.git link. Turn start only checked that the folder existed, so the agent ran
outside any checkout and git commands reached the repository around it.
Turn start now checks for the folder's .git, clears the leftover folder, and
recreates the checkout. Storage cleanup keeps such a folder as "not a linked
worktree" instead of failing the sweep on the missing .git.

Fixes #24390

AI-assisted; verified with the tests and tools listed above.
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Nov 12, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 72db875

Macroscope's review found this PR approvable — Restores managed worktrees whose checkout disappeared but left dependency or build directories behind. Recovery preserves existing checkouts and folders containing other entries, while cleanup safely skips folders missing .git.

No code changes detected at 6facbfe. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: b5c0ee5 · PR result: 6facbfe · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Managed-worktree recovery identifies existing directories without .git that contain only node_modules or .next directories, removes eligible managed leftovers, and recreates the worktree. Storage cleanup treats a missing .git entry as a non-linked worktree and propagates other stat errors.

Changes: Managed worktree recovery and cleanup

Layer / File(s) Summary
Detect and recreate incomplete worktrees
apps/server/src/workspace/restoreMissingWorktree.ts, apps/server/src/orchestration-v2/ProviderTurnStartService.test.ts
Recovery recognizes eligible managed directories without .git, removes them before recreation, and leaves paths with other contents untouched. Tests cover leftover recovery and preservation of a user file.
Handle missing worktree links during cleanup
apps/server/src/storageCleanup.ts, apps/server/src/storageCleanup.test.ts
A missing .git entry is treated as a non-linked worktree. Other stat errors still propagate. A test verifies that cleanup keeps the directory and its node_modules contents.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: t3dotgg, stienswout, tonybynmp4

Merge Risk: Minimal · up to 72db8

This change lets turn start recover a managed worktree folder that lost its checkout. It also stops storage cleanup from failing when .git is missing. No actionable merge-blocking risk remains.

Pre-merge checks: Passed 3 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Linked Issues check Inconclusive [#24390] Requirement A, turn-start recovery: the whole-PR diff (merge base f313d0a to head 72db875) changes apps/server/src/workspace/restoreMissingWorktree.ts, not ProviderTurnStartService.ts. The … 1. Confirm that the turn-start path reaches makeRestoreMissingWorktree for an existing managed folder without .git. Name the call site in the explanation or the PR text. 2. Update the PR description to name restoreMissingWorktree.ts and rem…
Passed Passed checks: 3 passed

Check name Status Explanation
Title check Passed The title names the main change: turn starts recreate a managed worktree folder that lost its checkout. It is specific and conventional, and matches the code, tests, and objectives. It does not descri…
Out of Scope Changes check Passed Every changed file maps to #24390. restoreMissingWorktree.ts adds leftover detection, a managed-path guard, and recursive removal before recreation. storageCleanup.ts handles NotFound for the .git sta…
Description check Passed The description covers the required sections: Problem, Change, Scope, and Verification. It explains that the failing Agent context budget check comes from an older copy of agentContextBudget.json, not…
Full details: Title check

Explanation

The title names the main change: turn starts recreate a managed worktree folder that lost its checkout. It is specific and conventional, and matches the code, tests, and objectives. It does not describe the storage cleanup change, but that is a secondary aspect.

Full details: Out of Scope Changes check

Explanation

Every changed file maps to #24390. restoreMissingWorktree.ts adds leftover detection, a managed-path guard, and recursive removal before recreation. storageCleanup.ts handles NotFound for the .git stat. The tests cover both behaviors. The change to restoreMissingWorktree.ts also includes a comment and a path import, which support the same behavior. No unrelated file changes appear in the whole-PR diff for these paths. The PR description is stale about ProviderTurnStartService.ts, but this is a description mismatch, not an out-of-scope code change.

Full details: Description check

Explanation

The description covers the required sections: Problem, Change, Scope, and Verification. It explains that the failing Agent context budget check comes from an older copy of agentContextBudget.json, not from this change, and that a merge of origin/main resolves it. It states the problem and reproduction steps, links the issue #24390, and describes the change to restoreMissingWorktree and storage cleanup. Verification lists focused commands, the observed results, and the items not exercised. Evidence includes before and after captures. The title is a conventional commit form in the linked PR objectives. The description has minor gaps: it does not state a conventional commit title in the description itself, and the Verification section is detailed but the wording is long. The author also states that the reproduction steps were not executed. The Who did the work section names models and harnesses, as the template asks.

Full details: Linked Issues check

Explanation

[#24390] Requirement A, turn-start recovery: the whole-PR diff (merge base f313d0a to head 72db875) changes apps/server/src/workspace/restoreMissingWorktree.ts, not ProviderTurnStartService.ts. The head adds isLeftoverFolder. It treats a folder as a leftover only when it has no .git entry and every top-level entry is a directory named node_modules or .next. Such folders are removed with fileSystem.remove(recursive) before recreateMissingWorktree runs. This matches the issue's node_modules-only reproduction. The PR description and the previous assessment cite ProviderTurnStartService.ts changes (path.join(worktreePath, ".git") check, forced remove inside recreateMissingWorktree). Those changes are absent from the reviewed diff, so the description is stale. I did not confirm that the turn-start path calls makeRestoreMissingWorktree. Requirement B, storage cleanup: storageCleanup.ts now uses Effect.catchReason("PlatformError", "NotFound") on stat(<worktree>/.git) and returns 'not a linked worktree'. This meets the issue's requirement. Open question: the issue's expected behavior covers any managed path that is not a valid checkout. A managed folder without .git that holds any other entry, such as a user file, is still not recovered. The test 'keeps user file' confirms this is intended. The issue text does not settle whether that is in scope. Tests: the test changes are reported as passing. I did not run them.

Resolution

  1. Confirm that the turn-start path reaches makeRestoreMissingWorktree for an existing managed folder without .git. Name the call site in the explanation or the PR text. 2. Update the PR description to name restoreMissingWorktree.ts and remove the ProviderTurnStartService.ts claims, or move the change to the file the description names. 3. State explicitly whether a managed folder without .git that holds non-leftover content must stay unrecovered. If the issue requires recovery for it, extend the rule and add a test.
Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2ae1676 and 72db875.

Files selected for processing: 4

  • apps/server/src/orchestration-v2/ProviderTurnStartService.test.ts
  • apps/server/src/storageCleanup.test.ts
  • apps/server/src/storageCleanup.ts
  • apps/server/src/workspace/restoreMissingWorktree.ts

Limit details: You’ve used all 10 included reviews currently available.

Main moved worktree recovery into workspace/restoreMissingWorktree.ts, so
the turn-start change now lives in that shared helper and also covers
terminal launches. The helper only clears a folder without .git when it
holds nothing but node_modules or .next directories; any other content is
left alone and recovery is skipped.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions github-actions Bot added size:M 30-99 changed lines (additions + deletions). and removed size:S 10-29 changed lines (additions + deletions). labels Nov 13, 2026
@juliusmarminge juliusmarminge added takeover:ready Takeover done; auto-lands when gates pass and removed takeover:in-progress A takeover lead is working on this PR labels Nov 13, 2026
Picks up main's lowered agent context budget.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@juliusmarminge
juliusmarminge merged commit 5aea99a into main Nov 13, 2026
35 checks passed
@juliusmarminge
juliusmarminge deleted the factory/issue-24390 branch November 13, 2026 17:31
@juliusmarminge juliusmarminge removed the takeover:ready Takeover done; auto-lands when gates pass label Nov 13, 2026
Pipyakas pushed a commit to Pipyakas/t3code that referenced this pull request Nov 13, 2026
kylebegeman pushed a commit to kylebegeman/loom that referenced this pull request Nov 14, 2026
…6 (Running the fork's tests in apps/server)

Failing check: Running the fork's tests in apps/server

The apps/server fork tests failed in three places. Each was a fork seam that no longer fit upstream's new code, and all three now pass.

**Switchboard auth showed the native endpoint** (`fork/switchboard/claudeDriverRouting.test.ts`). Upstream now adds an `endpoint` field to Claude's custom-endpoint auth info. The fork's Switchboard seam overrode the auth type and label but still let the native auth info through, so a routed probe reported `endpoint: http://native.example`. In `ClaudeProvider.ts` I now drop the native auth info when routed, the same way the fork already drops email and workspace. That line is marked `fork: switchboard`.

**A worktree folder left with only `node_modules` was not rebuilt** (`orchestration-v2/ProviderTurnStartService.test.ts`). Upstream pingdotgg#34894 made `restoreMissingWorktree` clear a managed worktree folder that holds only `node_modules`/`.next` and recreate the checkout. The fork's project-lifecycle seam calls its own `restoreWorktree` instead, and that returned early whenever the folder existed, so upstream's new `leftover` case failed. I carried upstream's leftover check into `workspace/restoreWorktree.ts`, which is fork-only. It runs only when the caller passes `clearLeftovers`, and the turn-start seam sets that only for server-managed paths, matching upstream's guard.

**Stop on a page's first load hung for 120s** (`preview/ServerBrowser.test.ts`, 4 cases). Upstream pingdotgg#29475 lets a viewer press Stop while a tab's first page load is still pending. The fork's `browser-timeout-recovery` seam in `findTab` made every tab wait for that first load before a viewer could attach, so the viewer never connected. The wait is only needed when a tab is being rebuilt after a stall, so `findTab` now checks for that first and waits only then; new tabs behave as upstream. The marker is kept and no `seams.tsv` row changed.

Rerun results:
- The two failing test files plus `restoreWorktree.test.ts`: 93/93 pass.
- `ServerBrowser.test.ts`: 106/106 pass, including the fork's recovery tests.
- Claude provider, Switchboard and project-lifecycle tests: 191/191 pass.
- `check_seams`, the apps/server typecheck, lint and format checks on the four changed files: all clean.

I did not rerun the whole apps/server suite.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Thread resumes in a worktree folder without .git instead of recreating the checkout

1 participant