[Bug]: Checkpoint capture hits the 30 s git timeout in large workspaces and never converges; every killed attempt leaves a tmp_objdir-bulk-fsync-* directory behind #26227

Activity

1. markshan00 commented on Oct 28, 2026

   

   markshan00

   on Oct 28, 2026

   More actions

   Another macOS data point with the same failure mode, on the 0.1.2 stable line. This one is a media folder, so each leaked directory is much larger.

   Environment

   • T3 Code 0.1.2 desktop. It's a rebuild that only changes UI strings; the server/VCS code is identical to upstream. macOS 27.0.1, Apple Silicon, git 2.54.0 (Apple Git-157)
   • Workspace: a non-code project folder of about 60 GiB and 40,000 files, mostly .mp4, .mov and .wav (82 files over 100 MiB), with no .gitignore. It had become a git repository with an unborn HEAD (no commits, no index, no refs) shortly before the problem started. None of the agent session logs contain a git init, so the in-app "Initialize Git" action is the most likely cause, but I can't confirm it because the server logs from that time had already rotated.

   What happened

   • Each turn spawned git -C <workspace> -c core.fsmonitor=false … -c core.fsync=objects,reference -c core.fsyncMethod=fsync -c core.fsyncMethod=batch add -A -- . against a .git/t3-checkpoint-index-* and was killed at 30 s (VcsProcessTimeoutError … GitVcsDriver.checkpoints.captureCheckpoint … after 30000ms). The trace had rotated except for the last few minutes; all 4 captures in that window timed out at 30.0 s. No checkpoint ever succeeded: there were no refs/t3 refs, and the real object store held only 73 loose objects (2.3 MiB) and no packs.
   • 70 tmp_objdir-bulk-fsync-* directories accumulated over about 16 hours, most of them during roughly 6 hours of active use. With several threads active, two sometimes appeared in the same minute. Each was about 1.5 GiB (a sampled one held 585 objects), about 94 GiB in total. Free disk space fell to about 8 GB before I noticed.
   • Each attempt gets about 1.5 GiB into the roughly 60 GiB tree before it is killed, and the next attempt starts over, so it can never converge.
   • An ordinary code repository on the same machine also had one 94 MiB tmp_objdir-bulk-fsync-* left from a single interrupted capture. This leak isn't limited to unusual workspaces; it is just much faster in them.

   Recovery
   I stopped the running capture and deleted the whole .git. With no commits there was nothing to lose, and the workspace files were untouched. That brought free space back to about 98 GiB.

   Notes on fixes

   • Per @rileyjshaw's test on fix(server): failed checkpoints no longer leave temporary packs in the repository #15400, the per-capture quarantine also covers this tmp_objdir path. That would stop the leak, but on a workspace like this the capture would still burn 30 s of I/O on every turn.
   • A back-off after N consecutive capture timeouts, or a cap on untracked size (fix(server): a large untracked file no longer fills the disk with checkpoint packs #15297) for unborn-HEAD repos, would have avoided this entirely. If "Initialize Git" was the trigger, a warning when the folder is very large or mostly media would also help.
2. dpreil-ydlp commented on Oct 29, 2026

   

   dpreil-ydlp

   on Oct 29, 2026

   Contributor

   More actions

   Another macOS data point. The issue is still present on 0.1.4 stable.

   Environment

   • Incident build: 0.1.3-nightly.20261026.3710. I also checked the installed 0.1.4 bundle (/Applications/T3 Code (Beta).app): it has the same captureCheckpoint / stageFiles / cleanupTempIndex code.
   • macOS, APFS, core.fsyncMethod=batch on the capture command.
   • Workspace: an ordinary code repo with a normal HEAD, not unborn. Agents had written large untracked build and data output into it.

   What happened (2026-10-26)

   • Free disk fell to about 0.39 GiB. The machine crashed three times in 24 hours. I did not inspect the panic logs, so I can't say that disk pressure caused every crash.
   • lsof showed one git process writing both <repo>/.git/t3-checkpoint-index-<id>.lock and <repo>/.git/objects/tmp_objdir-bulk-fsync-<x>/….
   • The repo's .git was 46.1 GB. After I removed the leaked tmp_objdir-bulk-fsync-* dirs it was 1.4 GB, and free space came back to about 43 GiB. Other large directories (Codex worktrees) did not change in size over the same samples, so the checkpoint writer was the active drain.

   Code notes (bundled names, 0.1.3 nightly; same code in 0.1.4)

   • stageFiles runs git add -A -- . against the temporary index. Nothing checks the size of the untracked input first.
   • cleanupTempIndex removes only the temp index and its .lock. It never removes a tmp_objdir-bulk-fsync-* created by the attempt that timed out, so every killed attempt leaks one.
   • The darwin writeout-only re-run in VcsProcess.runUnbounded (gated by canRetryFullFsync) is a second full add -A, but only for the two fsync/lock stderr patterns. The isTransientGitExit path in VcsProcess.run retries up to 2 more times. Neither path runs on a timeout, so the leak comes from capture being attempted again turn after turn, not from these retries.

   Suggested fix, in addition to the quarantine cleanup in #15400

   1. Before git add, measure the selected untracked input (respecting ignore rules) plus tracked changes. If it is over a budget, skip the capture or mark it incomplete. Don't edit user ignore files, and don't claim recovery coverage for content that was skipped.
   2. After a timeout or kill, once the child process has exited, remove only the tmp_objdir-* that this attempt created.
   3. Don't retry a full stage after a size or timeout failure. Back off instead (this matches the back-off suggested above).

   Test idea: use a small synthetic repo with a small byte limit and force repeated stage timeouts. Assert that no new tmp_objdir-* remain, that a concurrent writer's files and the live index survive, and that a normal checkpoint still restores the source.
3. klinerj commented on Oct 30, 2026

   

   klinerj

   on Oct 30, 2026

   More actions

   Another macOS data point, on 0.1.5 stable, in an ordinary code repository (normal HEAD, actively developed), not an unborn or media folder. Here the tracked files alone exceed the timeout.

   Environment

   • T3 Code 0.1.5 server (background service), macOS 27.0.1, Apple M1 Max, git 2.54.0 (Apple Git-157)
   • Workspace: 32,252 tracked files, 21.3 GB (mostly committed screenshots), plus 1 to 2 GB of untracked output that agents write. Threads run in the project's main checkout.

   What happened

   • In about 18 hours (Oct 29 15:49 to Oct 30 10:00), 44 tmp_objdir-bulk-fsync-* directories built up in .git/objects, about 41 GB, roughly 1 GB per turn.
   • The trace shows the same failure: VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint: git (<workspace>) after 30000ms. While it ran, git add -A -- . (with core.fsyncMethod=batch) showed up as a child of t3 serve.
   • Hashing the tracked files from an index freshly read from HEAD (git status, git update-index --refresh) takes about 100 s here. If capture starts from a fresh index each time, as the report above reads the source, it can't finish in 30 s in this repo even with no untracked files at all.

   Removing the untracked output made each leftover smaller but didn't stop them. Reusing the previous checkpoint index, or cleaning up tmp_objdir-* when the timeout fires, would fix it for us.
4. mateodelcon1 commented on Oct 30, 2026

   

   mateodelcon1

   on Oct 30, 2026

   More actions

   Windows data point for the same failure mode, on 0.1.5 stable (desktop). It's the first one here that isn't macOS: core.fsyncMethod=batch is also applied on win32, so NTFS leaks the same tmp_objdir-bulk-fsync-* directories. It adds two details: the trigger was T3's own scratch folder, and the temp index .lock files leak too.

   What happened

   • Threads with no project run in ~/.t3/scratch. On 2026-10-20 something ran git init in ~ (unborn HEAD, no commits, no remote). From then on, git rev-parse --show-toplevel in ~/.t3/scratch resolved to ~, so every capture ran git add -A -- . for the scratch folder against ~/.git.
   • The scratch folder holds about 200k files and 1.5 GB of agent output: downloads, an 888 MB partial file and a 289 MB executable. There's no .gitignore, because it isn't meant to be a repo. Hashing it doesn't fit in 30 s.
   • The last checkpoint refs (7 under refs/t3/orchestration-v2/checkpoints/…) are from 2026-10-22 19:06 local. No capture succeeded after that, so revert didn't work for any scratch thread for 8 days.

   Diagnosis (source at v0.1.5, 6d450a4284)

   Same as the original report:

   • stageFiles in apps/server/src/vcs/GitVcsDriver.ts (~L1040) runs add -A with durableWrite (core.fsyncMethod=batch on darwin and win32, ~L814) and no timeoutMs. So VcsProcess kills it at DEFAULT_TIMEOUT_MS = 30_000 (VcsProcess.ts:64).
   • Nothing in apps/server/src cleans up objects/tmp_objdir-* after a timeout or interruption.

   Windows-specific:

   • cleanupTempIndex runs in Effect.ensuring, but its remove(…).pipe(Effect.ignore) left 6 stale 0-byte t3-checkpoint-index-<uuid>.lock files in ~/.git, aged 31–199 h.
   • My guess, not confirmed: the timed-out git.exe still holds the lock when the cleanup runs, Windows refuses the delete, and the error is swallowed.
   • Measured durations go up to 45.8 s for a 30 s timeout, so the kill isn't immediate.

   main at 2e860972ae (2026-10-30) still has no tmp_objdir cleanup, and the capture still uses the default timeout. #21315 and #21329 reduce git process count and scan cost, but don't change this.

   Steps to reproduce (Windows)

   1. git init in your home folder (or have any repo at ~).
   2. Use threads with no project, so the workspace is ~/.t3/scratch, and let agents write a few GB of files there.
   3. Each turn: VcsProcessTimeoutError … captureCheckpoint: git (~\.t3\scratch) after 30000ms, plus one new ~/.git/objects/tmp_objdir-bulk-fsync-* directory, and sometimes a leftover t3-checkpoint-index-*.lock.

   Version

   T3 Code (Beta) 0.1.5 desktop (server 0.1.5, per the triage context). Triage CLI 0.1.5.

   Environment

   Windows 11 Pro 26H2 (10.0.26300), NTFS on NVMe, Git for Windows 2.55.0.windows.5, Ryzen 9 7950X3D, 63 GB RAM. Desktop app with a local server; several agent threads running at once.

   Evidence

   • Leaked directories: 948 tmp_objdir-bulk-fsync-* in ~/.git/objects, the first dated 2026-10-22 19:12. Per day: 80 / 68 / 110 / 183 / 161 / 200 / 130. The count rose from 932 to 948 within about 40 minutes today.
   • Total size: ~/.git was 48 GB in about 9.7 million files. A sample of 7 directories held 5.6k–13.6k files and 31–92 MB each. git count-objects saw only 1,096 loose objects; everything else sat in the leaked directories.
   • Trace: 119 VcsProcessTimeoutError spans for GitVcsDriver.checkpoints.captureCheckpoint in the retained trace files, all between 20:23 and 21:02 UTC on 2026-10-30, about 3 per minute with 6–7 threads active. Durations ran from 30,136 to 45,773 ms.
   • Windows side effects:
     • Defender (MsMpEng) averaged about one full core over 2 days, with samples at 171–218%.
     • The NTFS MFT on C: grew to 13.9 GB.
     • A git add was running at about 98% of a core most of the time.

   Related issues

   #26227 (this issue), #3646 and #15400 (tmp_pack litter from failed checkpoints), #1228 (checkpoint timeout when running from the home directory), #24083.

   Fix applied or workaround

   • Moved the accidental ~/.git out of the profile. ~/.t3/scratch is no longer inside a repo, so captures there now fail fast as "not a git repository" instead of timing out. After later turns, T3 had not re-created a .git in ~ or in the scratch folder.
   • One leaked tmp_objdir stayed locked by another process and needs deleting after a reboot.
   • Suggestions beyond the original report:
     • Retry the .lock cleanup after the child has fully exited, at least on Windows.
     • Consider not checkpointing T3's own scratch folder into an ancestor repo it doesn't own, such as ~. Anyone with a repo at their home folder would otherwise hit this on every scratch thread.

   Filed by

   Claude Opus 5.5 (claude-opus-5-5) in Claude Code, following the t3 triage 0.1.5 playbook, on the affected machine.
5. Yclpaiva commented on Oct 31, 2026

   

   Yclpaiva

   on Oct 31, 2026

   More actions

   Another macOS case on October 31: my disk fell to about 0.5 GiB free. I found 273 abandoned .git/objects/tmp_objdir-bulk-fsync-* directories holding 82.5 GiB and 161,853 files.

   Environment: T3 Code Nightly 0.1.4-nightly.20261027.3770, macOS 26.4, APFS, Apple Silicon arm64, Git 2.55.0. The workspace has an unborn HEAD, contains generated media, and its 494 refs are all T3 checkpoints.

   The retained server traces contain six capture timeouts at approximately 30 seconds, reported as VcsProcessTimeoutError in GitVcsDriver.checkpoints.captureCheckpoint. Each interval matches the modification time of one abandoned directory. A Git object hash also confirmed that the same original video had been copied into multiple abandoned directories.

   I tested the interruption mechanism separately in disposable repositories on this machine:

   • git add with core.fsync=objects,reference and core.fsyncMethod=batch left a tmp_objdir-bulk-fsync-* directory after SIGTERM. SIGKILL did the same.
   • With core.fsyncMethod=fsync, SIGTERM left no bulk-fsync directory.
   • With batch and no interruption, git add completed without leaving a temporary directory.

   After checking that the old directories were not in use, I removed only those 273 directories. All 494 checkpoint refs remained identical, the normal object count stayed at 11,674, and git fsck --connectivity-only --no-dangling passed before and after. Original workspace files were preserved.

   Linking #23715 as the relevant proposed fix. I can help validate its batch-fsync timeout and cancellation cleanup on native macOS arm64 with Git 2.55.0, along with successful capture and restore. I have not tested that PR yet. The affected T3 build above is the one investigated; I have not reproduced this incident on a newer release.
6. archuity commented on Nov 3, 2026

   

   archuity

   on Nov 3, 2026

   More actions

   Another Windows data point, from a normal checkout with HEAD (not an unborn repo), on desktop 0.1.4-nightly.20261028.3841.

   What we saw. The repo has about 300k tracked paths. At the worst point it also had about 9,000 nonignored untracked files (0.9 GB, none over 50 MB). .git/objects had collected 349 tmp_objdir-bulk-fsync-* folders, 39.7 GB in total. On a busy day, about 82 were created in one hour. We assume most came from checkpoint git add -A runs killed at the 30 s limit. We didn't match each folder to a trace.

   Why it seems to happen. As far as we can read the installed build, each capture makes a fresh UUID temp index, seeds it from the workspace index, and removes it afterwards. The workspace index doesn't hold the untracked files, so a fresh capture index doesn't keep their cached metadata from the last capture. That fits the cold and warm difference we measured. (#23624 now covers skip-worktree and assume-unchanged entries; ours is the untracked files that aren't in the workspace index.) We replayed the add on a copy of the index, with a temp object dir:

   • First trial: 15.5 s cold, then 1.1 s reusing the same temp index.
   • Second trial, about 15 minutes later under heavier load: 29.6 s cold, right at the 30 s limit.

   These are git add timings, not whole captures. Refreshing the real index (update-index --refresh) didn't improve this replay. Reusing one temp index is evidence of the direction, not a safe implementation.

   What helped. We hid untracked files over 1.5 MB with .git/info/exclude. In T3's own trace (GitVcsDriver.checkpoints.captureCheckpoint), captures went from a 32.5 s median with 23 of 29 failed (19 minutes before) to an 11.0 s median with 0 of 9 failed (5 minutes after). New leftover folders dropped from about 82 an hour to about one. Later we cut the untracked files from about 9,000 to under 200. A sparse worktree replayed in 2.9 to 5.7 s, against 17.6 to 22.1 s on the full checkout.

   What would help us. The proposals already in this thread match what we need. Our addition is the measured cold and warm comparison above.

   1. Keep the capture index between captures, or seed it from the previous one, so untracked files already hashed aren't hashed again every turn. This matches the index-reuse idea earlier in this thread.
   2. After a timeout or kill, remove that attempt's own object storage. We'd prefer a per-capture quarantine, as proposed in the still-open fix(server): interrupted checkpoints no longer leak temporary packs #23715 (the earlier fix(server): failed checkpoints no longer leave temporary packs in the repository #15400 was closed unmerged), to a broad sweep. One older t3-checkpoint-index-*.lock was also left from 10-28, though the existing finalizer should remove it. We can't say why it stayed.
   3. When a capture can't finish in time, skip it with a visible notice instead of retrying every turn.

   Environment: Windows 11 Pro, Git 2.54.0.windows.1, T3 Code desktop 0.1.4-nightly.20261028.3841 (server bundle SHA-256 0a3cfede...7464). We haven't tied the running backend version to logs yet, and we haven't tried 0.1.7. Reading the latest nightly's source, it still uses a fresh capture index and the 30 s staging default, but we haven't reproduced the incident on it. Happy to run a check on a newer nightly.
7. romanolivera commented on Nov 4, 2026

   

   romanolivera

   on Nov 4, 2026

   More actions

   Additional incident on macOS: about 1.28 TB of abandoned checkpoint objects nearly filled a 2 TB system drive. Adding the measurements here because they match this issue.

   Environment

   • T3 Code desktop: 0.1.7 installed at diagnosis.
   • macOS 27.0.1 (26A434), Apple Silicon, APFS.
   • Git 2.55.0.
   • Investigation date: 4 November 2026. The accumulation began on 30 October; I have not established which T3 version created every older directory.

   Measured disk usage

   • 1,025 directories matching .git/objects/tmp_objdir-bulk-fsync-*.
   • Approximately 1,279,754,428,416 allocated bytes (1.28 TB) attributable to those directories. This was calculated from the measured object-directory total minus all non-temporary object entries, rather than extrapolated from a sample.
   • The other Git objects occupied approximately 9.04 GB.
   • 552 of the temporary directories were created on 4 November.
   • Free space fell to approximately 160 MB, and other local services reported No space left on device.

   Three sampled temporary directories contained 5,210 common Git object names. Matching examples had the same sizes but different inodes and a link count of one, consistent with repeated separate copies. One sample alone occupied approximately 1.61 GB.

   Correlation with T3 checkpoint failures

   The retained T3 server traces contained 82 checkpoint attempts, all failed:

   • 70 VcsProcessTimeoutError failures at the 30,000 ms limit.
   • 12 VcsProcessExitError failures with exit code 128.
   • 74 of the temporary-directory creation timestamps fell inside these failed GitVcsDriver.checkpoints.captureCheckpoint intervals.

   Representative error, with the private path removed:

   VcsProcessTimeoutError: VCS process timed out in
   GitVcsDriver.checkpoints.captureCheckpoint:
   git (<workspace>) after 30000ms
   

   The timestamp comparison supports the same checkpoint leak described above. Retained logs do not cover the full accumulation period, so I am not claiming an individual trace match for all 1,025 directories.

   Impact and requested fix

   This exhausted disk capacity and disrupted work outside T3. T3 has been stopped. A completed cleanup and a fixed release have not yet been verified.

   Please prioritise preventing each failed or interrupted capture from leaving its temporary object storage behind. Cleanup should remove only storage owned by that capture, after its Git child has exited, and preserve concurrent writers. A regression test should force repeated capture timeouts/cancellation and confirm that temporary storage does not accumulate. Backoff or a visible suspension after repeated failures would also limit the continuing I/O load.

   No private workspace names, file names, file contents, absolute paths, thread identifiers, or full logs are included in this report.
8. lyndon050516 commented on Nov 4, 2026

   

   lyndon050516

   on Nov 4, 2026

   More actions

   Another macOS data point. The leak is still present on the 2026-11-03 nightly, and it used about 449 GB in 25 hours.

   Environment

   • T3 Code (Nightly) 0.1.7-nightly.20261103.4119, macOS 26.6.2, git 2.55.0
   • Workspace: a video production project with a normal HEAD (one commit, 2 tracked files). Agents write renders into it, and several threads run at once.
   • Files that git add -A must process: 18,008 untracked, non-ignored files, 31.5 GB, of which 25.9 GB is rendered video output.

   What happened

   • .git/objects held 457 tmp_objdir-bulk-fsync-* directories, 448.9 GB in total. The real object store held 632 MiB.
   • The first directory appeared on Nov 3 at 06:36 and the newest on Nov 4 at 07:55. The directories had a median size of 1.05 GB and a maximum of 1.68 GB. The peak was 52 new directories in one hour.
   • Free disk space fell to 12 GB on a 1 TB disk. The growth over the last 2 hours measured was 36.7 GB.
   • The last checkpoint ref is from Nov 2 at 22:17, so no capture has succeeded in this project since then.

   Link between timeouts and directories
   The retained trace covers 06:59 to 07:56 on Nov 4. It has 53 VcsProcessTimeoutError failures from GitVcsDriver.checkpoints.captureCheckpoint in this workspace. In the same window, 53 directories appeared, and each one appeared within 35 seconds before a failure.

   Note on #29259
   As I read that PR, it stops the leak but does not change the timeout. This workspace would still have no checkpoints and would still spend 30 seconds on each failed capture. The back-off or size budget suggested above would still be needed for this type of project.
9. dl-pt commented on Nov 5, 2026

   

   dl-pt

   on Nov 5, 2026

   Last edited by dl-pt

   More actions

   I observe the same issue on macOS Apple Silicon with T3 0.1.7 using Codex. Repeated checkpoint timeouts leave eight tmp_objdir-bulk-fsync-* directories behind, and block each prompt for 30 seconds before it reaches Codex. #28996 is the same issue.
10. Chiptonk commented on Nov 6, 2026

    

    Chiptonk

    on Nov 6, 2026

    More actions

    Another Windows data point, with a workaround that has held.

    Environment: T3 Code Nightly (currently 0.1.8.0; the build at the time of the incidents wasn't recorded), Windows 11, NTFS, git 2.55.0.windows.5. An ordinary repo with commits, where other agents commit to main.

    What happened: about 5.6 GB of untracked, non-ignored files made captureCheckpoint hit the 30-second timeout. Each killed git add left a tmp_objdir-bulk-fsync-* folder of about 1 GB and about 3,400 files. It happened three times: 2026-10-29 (size not recorded), 2026-11-01 (376.7 GB) and 2026-11-04 (352.6 GB in 358 folders, up to about 37 GB an hour).

    Workaround: adding the idle bulky folders to .git/info/exclude brought the files to save under 1 GB. After that, 377 checkpoints were recorded with no leftover folders, saves took about 3 seconds, and the trace logs showed no failures. Those logs only reach back about 4 hours here. It only works for folders that don't need saving, and it doesn't fix the timeout.

    For #29259: on Windows, each killed attempt here cost about 1 GB, so a fix that removes the batch folder would have prevented all of this.
11. janoschbl commented on Nov 6, 2026

    

    janoschbl

    on Nov 6, 2026

    More actions

    Linux data point, on 0.1.5 stable. On Linux the leak takes a different form: there are no tmp_objdir-bulk-fsync-* directories, and the killed captures leave .git/objects/pack/tmp_pack_* files instead. Here they filled the disk that also holds the T3 data directory, and T3 Connect stopped answering.

    What happened

    Diagnosed and written by Claude Opus 5.5, based on a t3 triage session.

    The mobile app showed "T3 Code on omarchy sent an unexpected answer". The desktop app had hung for about 11 minutes before that.

    Diagnosis

    • The workspace is an ordinary repo with commits. It holds about 54 GB of untracked, non-ignored training checkpoints. Each optimizer.pt is about 839 MB.
    • captureCheckpoint runs git add -A with the default DEFAULT_TIMEOUT_MS = 30_000 (apps/server/src/vcs/VcsProcess.ts:64). The trace shows VcsProcessTimeoutError ... captureCheckpoint ... after 30000ms.
    • GitVcsDriver.ts adds core.fsyncMethod=batch only on darwin and win32 (apps/server/src/vcs/GitVcsDriver.ts:819, v0.1.5). On Linux, Git does not use a tmp_objdir quarantine. Files above core.bigFileThreshold (512 MiB by default) go through bulk-checkin, which streams them into .git/objects/pack/tmp_pack_*. A killed git add leaves that file behind, and the next capture writes it again.
    • From 2026-11-05 11:47 to 2026-11-06 15:36, 171 tmp_pack_* files built up, 107 GB in total. They arrived in pairs every 15 minutes, about 700 MB each.
    • The disk reached 0 bytes free. The T3 data directory is on the same disk, so the server could not write. The last capture failed with exit 128 instead of a timeout, and the server trace stopped there.
    • With the disk full, cloudflared stayed connected, but the origin answered 77 relay requests with HTTP 500. The mobile app reported invalid_response and endpoint_request_failed with the tunnel marked "ok". The desktop hang probably has the same cause. I did not confirm that.

    Steps to reproduce

    1. On Linux, open a project whose repo has a few untracked, non-ignored files larger than 512 MiB, enough that git add -A takes longer than 30 seconds.
    2. Run turns in a thread in that project.
    3. Watch .git/objects/pack/. Each timed-out capture leaves a tmp_pack_* file that is never cleaned up.

    Version

    T3 Code 0.1.5 desktop (AUR t3code-bin 0.1.5-1), host server 0.1.5. Mobile app 2.2.0.

    Environment

    Arch Linux (Omarchy), kernel 7.2.8, x64, git 2.56.0, ext4.

    Evidence

    VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint: git (<repo>) after 30000ms   # 2026-11-06 02:12
    VcsProcessExitError: VCS process failed in GitVcsDriver.checkpoints.captureCheckpoint: git (<repo>) exited with 128       # 2026-11-06 15:36, disk full
    $ ls <repo>/.git/objects/pack/tmp_pack_* | wc -l
    171      # 107 GB, mtimes 11-05 11:47 to 11-06 15:36
    $ df -h <data disk>
    917G  870G  0  100%
    cloudflared_tunnel_response_by_code{status_code="500"} 77
    

    Fix applied

    • Deleted the tmp_pack_* files while no Git process was running. That freed 107 GB.
    • Added *.pt, *.safetensors, *.onnx, *.bin, checkpoint-*/ and similar patterns to the repo's .git/info/exclude. git status now finishes in milliseconds and captures stop timing out.

    Cleaning up on timeout should cover objects/pack/tmp_pack_* on Linux as well as tmp_objdir-*.

    Sent from Claude Opus 5.5 (Claude Code, via t3 triage) for janosch.
12. davidalexander commented on Nov 6, 2026

    

    davidalexander

    on Nov 6, 2026

    More actions

    Confirmed on macOS/Apple Silicon with 0.1.8-nightly.20261104.4165: 790 leftover tmp_objdir-bulk-fsync-* directories occupied 27.4 GB.

    Excluding specific generated QA/source copies through .git/info/exclude restored successful checkpointing. This workaround reduces checkpoint coverage.
13. EvilHumphrey commented on Nov 7, 2026

    

    EvilHumphrey

    on Nov 7, 2026

    Last edited by EvilHumphrey

    More actions

    Windows checkpoint-coverage trap worth knowing when validating a fix here: a capture can succeed quickly and still omit all non-ignored untracked files.

    Why: before staging, captureCheckpoint asks filterIgnoredPaths(input.cwd, ["."]) whether the workspace root itself is ignored. If it is, it stages with git add -u instead of -A (v0.1.7 L1105 and L1177; same on main, L1111/L1183). Git answers "ignored" for . when a root-level ignore source has a blank line ending in CRLF. Git's parser skips an empty LF line, but for an otherwise empty CRLF line it strips the \r and adds an empty pattern, and that pattern matches ..

    Repro (Git 2.56.0.windows.4, Git Bash, in a disposable directory):

    git init repro && cd repro
    printf 'sentinel\n' > new.txt
    printf 'a.txt\n\nb.txt\n' > .gitignore
    git check-ignore -v --no-index .                  # exit 1: not ignored
    printf 'a.txt\r\n\r\nb.txt\r\n' > .gitignore
    git check-ignore -v --no-index .                  # .gitignore:2:<TAB>.   exit 0: ignored
    git ls-files --others --exclude-standard          # still lists new.txt

    Ordinary untracked listing is unaffected; only asking git about . itself trips. T3's NUL-delimited form of the probe gives the same answer, and replaying the selected staging command omits new.txt with -u.

    Scope, as far as we tested:

    • .git/info/exclude (the file this thread suggests for the workaround) and a user-wide core.excludesFile trigger it the same way, so if an exclude workaround made your captures fast, git check-ignore -v --no-index . will tell you whether it did so by switching to -u.
    • A line of only spaces triggers it even with LF endings. A blank line in a subdirectory's .gitignore doesn't.
    • core.autocrlf=true (Git for Windows' installer default) checks a LF .gitignore out with CRLF endings unless .gitattributes pins it (for example .gitignore text eol=lf). We haven't tested other Git builds or operating systems.

    Why it matters for this issue: changing nothing but line endings can switch which staging branch runs, so fast, successful capture spans don't show that untracked capture works. Check that a non-ignored sentinel file actually appears in the checkpoint. In our own repo every checkpoint tree we sampled held only the tracked files. We haven't established what caused the capture timeouts we hit earlier.

    A worse consequence on restore: restoreCheckpoint runs git restore --source <checkpoint> --worktree --staged -- . and then git clean -fd -- . (v0.1.7 L1317, L1346). When the capture took the -u branch, the clean can delete non-ignored untracked files, including ones that existed before the turn and that the agent never touched. We reproduced this in a scratch repo with T3's command sequence, not by driving T3. That data-loss path is now tracked separately in #32360.

    Sent via Claude Opus 5.5 in Claude Code. Other AI models reviewed it before posting.
14. ridhitb commented on Nov 7, 2026

    

    ridhitb

    on Nov 7, 2026

    More actions

    Another macOS case, on 0.1.7. It adds a trigger and a second kill path not covered above, and a test of #29259 on 0.1.7.

    The project is a plain folder holding 14 repositories and 28 worktrees, plus about 7.8 GB of notes and outputs outside them. Between 2 and 5 November, Initialize Git in the thread details card made it a repository with no commits five times, because the card shows a thread's linked PRs only in git projects (#27626). Each time, every turn left a tmp_objdir-bulk-fsync-* of about 800 MB, up to 177 GB in total.

    Checkpoints can't do anything useful in such a folder. git add -A stores each nested repository as a gitlink, so the agent's work inside them never reaches the checkpoint. The capture only hashes everything around them, and here that never fits in 30 s.

    There is also a second kill path. When a nested repository has no commits, the first add -A fails, and the retry that excludes it runs under CHECKPOINT_RECOVERY_TIMEOUT (5 s, still on main). On a large tree that retry is killed every time and leaks as well. In a synthetic copy (770 MB in 31,360 files, one nested repository without commits), every turn took this path: the first add failed after about 21 s, the retry was killed at 5 s, and each turn left 147 MB.

    I ported #29259 to v0.1.7, reading the environment from process.env because 0.1.7 has no HostProcess.Environment. Its tests pass there, including the ones for a killed add and for large-file packs. In a commit-less repository of 30,000 files, captures killed at 4 s left 118 MB each before the port and nothing after it. As noted above, turns still spend the timeout.

    For the folder-of-repositories case, capture could skip a repository that has no commits when one of its untracked folders holds a .git, and record a missing checkpoint as it does for a folder that isn't a repository. The check is one git ls-files --others --exclude-standard --directory, which took 0.04 s on the real folder.

    If you want the git UI on such a folder before a fix ships, a repository-local excludes file that ignores everything works: printf '/*\n' > .git/t3-ignore-all && git config core.excludesFile "$PWD/.git/t3-ignore-all". Checkpoints then commit an empty tree in about 50 ms. Don't put /* in .git/info/exclude instead: ripgrep reads that file too, so agents' search tools would see no files at the root. With this setup git check-ignore --no-index . exits 1, so the -u branch described above doesn't apply. To undo it, delete the whole .git. Removing only the setting turns it back into the repository that leaks.

    Written by Claude Opus 5.5 in Claude Code.
15. edemaine commented on Nov 9, 2026

    

    edemaine

    on Nov 9, 2026

    Contributor

    More actions

    FWIW, I just ran into this on my Windows machine, where one (large) repo with a lot of turns had accumulated 52 GB of these temporary directories over just a week:

    
16. kaijiabo commented on Nov 12, 2026

    

    kaijiabo

    on Nov 12, 2026

    More actions

    Another macOS data point, on 0.1.7 stable (desktop), with two triggers not covered above. In both, the workspace is small enough for the fast capture path, but capture silently falls back to the slow full-rehash path.

    Environment: T3 Code 0.1.7, macOS, Apple Silicon, git 2.54.0 (Apple Git-157). An ordinary repo with a normal HEAD and about 200k tracked files, where several agents commit to main in the same checkout.

    What we saw: 66 of 73 captures in the retained traces timed out at 30 s, and ProviderTurnStartService.start took 92–124 s. The chat showed "streaming" with no output for that whole time.

    Trigger 1: more than 1 MB of skip-worktree entries disables the fast path

    captureCheckpoint copies the user's index and collects flagged entries from ls-files -v (skip-worktree S and lowercase assume-unchanged). Once they pass FLAGGED_INDEX_ENTRIES_MAX_BYTES (1 MiB), it sets fallback = true and rebuilds the temp index with a plain read-tree HEAD. That index has no cached stat data, so git add -A re-reads and re-hashes every tracked file, LFS clean filters included.

    Our repo had 15,815 skip-worktree entries (1.7 MB). They were archived files that had already been deleted on disk and were hidden from git status with update-index --skip-worktree. Measured on the same repo:

    Path	Time
    Fallback (read-tree HEAD + add -A)	81.4 s
    Fast path (copied index + read-tree --reset HEAD + add -A)	8.8 s
    Fast path, tracked files only (add -u)	0.5 s

    Fix on our side: we committed the deletions, which left 0 flagged entries. Captures now succeed in 1–10 s.

    Trigger 2: a plain git read-tree <commit> in user tooling makes the next capture slow

    We had a sync script that moved HEAD and then ran git read-tree <new-commit>. A plain read-tree drops cached stat data from the user's index, and capture copies that index, so the next capture re-hashed everything. With git read-tree --reset <new-commit>, the same step afterwards took 0.8 s instead of 81 s. Tools that sync the checkout in the background can cause this without anyone seeing it.

    Suggestions

    • When the flagged-entry cap is hit, log it at warn level, or keep the copied index's stat data for the unflagged entries instead of a full read-tree HEAD. Today the fallback is silent and costs about 10× the fast path.
    • Show capture as its own state in the thread (for example "Taking snapshot…"), so a slow capture doesn't look like a stuck stream. fix(server): a slow baseline checkpoint no longer holds every prompt for 30 seconds #34849 removes the wait from turn start, which fixes most of the impact.
    • A per-project setting for the capture timeout, or a way to turn capture off, would still help for very large workspaces.
17. WhiteBalance2800K commented on Nov 13, 2026

    

    WhiteBalance2800K

    on Nov 13, 2026

    More actions

    Windows data point on 0.1.9-nightly.20261110.4465: a 4 GB workspace filled a 118 GB system drive, and a second, tighter timeout is involved

    Adding this because it differs from the reports above in three ways: the workspace is small (4 GB), the system drive is small (118 GB), and the kill appears to come from a 5-second budget rather than the 30 s VcsProcess default.

    Environment

    • T3 Code (Nightly) 0.1.9-nightly.20261110.4465, desktop, Windows 11, Git for Windows 2.53.0, NTFS on NVMe
    • Workspace: a plain folder (C:\Users\<user>\Downloads) containing a .git with an unborn HEAD — no commits, no remote, no .gitignore
    • 483 files / 3.98 GB, 415 untracked, of which 119 binaries account for 3.77 GB (896 MB zip, 453 MB mp4, 423 MB exe, 362 MB mp4, …)
    • System drive: 118 GB total; free space fell to 40 MB

    What happened

    • git rev-list --all --count = 0, so headExists is false and every capture takes the full-add path.
    • 33 tmp_objdir-bulk-fsync-* directories, 19.43 GB, created between 2026-11-11 09:46 and 2026-11-12 20:51 (~35 h).
    • Each held 128–215 files, 93–672 MB; average 0.59 GB. That is ~15% of the workspace per killed attempt, and 4.9× the workspace size leaked in total.
    • git count-objects -vH reported only 34 objects / 13.65 MiB, so the leak is invisible to normal git tooling — that is why it went unnoticed until the drive was full.
    • Two stale t3-checkpoint-index-<uuid>.lock files were present, matching the Windows report above.

    A second, tighter timeout

    The reports above attribute the kill to DEFAULT_TIMEOUT_MS = 30_000 in VcsProcess. The installed 0.1.9-nightly bundle has an additional budget in CheckpointService.ts:

    const BASELINE_CAPTURE_BUDGET = "5 seconds";
    ...
    yield* checkpointStore.captureCheckpoint({
        cwd: input.scope.cwd,
        checkpointRef
    }).pipe(timeout(BASELINE_CAPTURE_BUDGET), catchTags({ VcsUnsupportedOperationError: () => void_ }));

    So baseline capture is interrupted after 5 s, not 30 s. A 4 GB tree cannot be hashed and written in 5 s, so this path fails even in a workspace that would comfortably finish inside the 30 s git timeout. If this budget is the timer that fires, raising or scaling the 30 s default would not help this case.

    I could not confirm from the outside which of the two timers killed each individual attempt (no retained traces on this install), so I am flagging it rather than asserting it.

    Mechanism confirmed locally (matches archuity)

    In a disposable repo with 30 × 30 MB files:

    • git -c core.fsync=objects,reference -c core.fsyncMethod=batch add -A -- . created tmp_objdir-bulk-fsync-4xsnF0 at 0.3 s and removed it on successful completion.
    • Killing the same process with TerminateProcess right after the directory appeared left tmp_objdir-bulk-fsync-TShaEt behind permanently.

    Self-reinforcing loop

    Because the leak is what fills the disk, the failure gets worse over time: less free space → slower writes → less progress per 5 s (or 30 s) → killed again → another directory. At the point I found it, the leak was still adding ~0.6 GB per attempt while the drive had 40 MB left. Removing the directories restored 19.5 GB.

    Workaround that worked here

    Removing .git/objects/tmp_objdir-* was safe: 0 commits, no refs, git count-objects unaffected. Beyond that, the same remedies as above apply — a .gitignore for the 3.77 GB of binaries (so git add only needs ~215 MB) and an initial commit so headExists is true and the incremental path is used.

    Suggestion

    If the 5 s baseline budget is intentional, it needs the same size-awareness as the 30 s default, or baseline capture should be skippable/deferred above some workspace size. A folder whose .git has an unborn HEAD is arguably the worst case for this bug: there is no incremental path at all, so it can never converge.
18. ethernetz commented on Nov 13, 2026

    

    ethernetz

    on Nov 13, 2026

    More actions

    Linux data point on 0.1.9-nightly.20261110.4478: about 670 GB of tmp_pack_* / tmp_obj_* overnight, from a few large untracked videos

    Environment: Ubuntu 24.04, t3 serve, git 2.43, ext4. Ordinary repo with a normal HEAD, about 7,100 tracked files (0.5 GB), and several agent threads in the same checkout.

    What we saw

    • Untracked, non-ignored content had grown to about 67k files and 13.4 GB, mostly video frames (.jpg) plus a few 0.5–2.4 GB .mp4 renders.
    • The server trace shows 729 checkpoint capture failed errors with VcsProcessTimeoutError ... captureCheckpoint: git (<repo>) after 30000ms, and 661 checkpoint baseline capture failed; starting provider without a baseline.
    • On Linux the leftovers differ from the macOS bulk-fsync case. Each killed git add -A left a .git/objects/pack/tmp_pack_* (files over core.bigFileThreshold are streamed straight into a pack) and/or loose .git/objects/??/tmp_obj_*. The same ~564 MB file was re-streamed about every 30 s. In about 22 hours that came to 941 tmp_pack_* (503 GB) plus 883 tmp_obj_* (169 GB), and the disk reached 99%.

    Measured cause of the 30 s overrun. Reproducing the capture by hand on the same tree:

    • GIT_INDEX_FILE=<new> git read-tree HEAD && GIT_INDEX_FILE=<new> git add -A -- . took 1 min 54 s, because a new index has no stat data, so every file is rehashed.
    • Running the same git add -A again on that index took 0.15 s.

    So keeping the checkpoint index between captures (per workspace), instead of starting from an empty one each time, would turn this into a sub-second operation for most turns. After we moved the frames out of the checkout, a cold capture took 7.6 s.

    Suggestions, in addition to those above

    1. Keep the checkpoint temp index between captures so git's stat cache works.
    2. On timeout or failure, delete tmp_pack_* / tmp_obj_* created during the attempt, or run git add with -c core.bigFileThreshold large enough to avoid streaming to packs, or skip untracked files over a size limit.
    3. Back off after repeated timeouts in the same workspace instead of retrying every turn.

    Workaround we use: a 10-minute timer that adds untracked files over 100 MB to .git/info/exclude and deletes tmp_* under .git/objects older than an hour that no process has open.
19. juliusmarminge

    added a commit that references this issue on Nov 13, 2026

    test(server): interrupted checkpoint capture must not leak batch obje…

    ...

    f8d9efd
20. lhagenmayer commented on Nov 13, 2026

    

    lhagenmayer

    on Nov 13, 2026

    More actions

    I can confirm this on macOS with an existing HEAD. When checkpoint staging reaches the 30-second timeout, Git's tmp_objdir-bulk-fsync-* directories can remain across subsequent attempts. The residual directories contained loose blobs, including objects that never reached the main object database.

    Can T3 track the temporary object directory it owns for each checkpoint and clean it up after the child exits, including on timeout, while preserving objects from concurrent Git operations?
21. spiky02plateau commented on Nov 13, 2026

    

    spiky02plateau

    on Nov 13, 2026

    Contributor

    More actions

    Another macOS data point, plus a measurement on why the capture never converges under batch mode.

    Environment

    • T3 Code Nightly 0.1.10-nightly.20261111.4529, macOS, Apple Silicon, APFS, git 2.54.0 (Apple Git-157). No core.fsync or core.fsyncMethod in the repository config.
    • A repository with about 18,000 untracked media files (1.4 GB). Nine threads ran in it as a shared checkout.

    What happened

    • In about six hours, 74 tmp_objdir-bulk-fsync-* directories built up in .git/objects, 68 MB to 605 MB each, 34 GB together. They appeared one or two per minute while threads were active.
    • git count-objects -v showed 1,102 loose objects, 83 MB, so none of those objects reached the object store. No git process was running when I looked.
    • The retained server trace covers 23:02 to 23:29. It has 52 GitVcsDriver.checkpoints.captureCheckpoint spans: 25 failed with VcsProcessTimeoutError at about 30 s, 26 succeeded, 1 was interrupted. The last timeout is at 23:29, the same minute as the newest directory.
    • After I wrote the untracked files into the object store once with git hash-object -w, no new directory appeared over the next 20 minutes with ten threads running.

    Why git leaves the directory
    tmp-objdir.c registers only an atexit cleanup and no signal handler. Any signal ends git add without removing its batch directory. In disposable repositories with T3's flags (-c core.fsync=objects,reference -c core.fsyncMethod=fsync -c core.fsyncMethod=batch add -A -- .), each run on a fresh repository with 300 MB of untracked files:

    method=batch signal=none exit=0   tmp_dirs_left=0 loose=302
    method=batch signal=TERM exit=143 tmp_dirs_left=1 loose=3
    method=batch signal=HUP  exit=129 tmp_dirs_left=1 loose=3
    method=batch signal=KILL exit=137 tmp_dirs_left=1 loose=3
    method=fsync signal=TERM exit=143 tmp_dirs_left=0 loose=42
    

    So no grace period lets git clean up.

    Convergence
    I replayed T3's capture loop: a fresh temp index each turn, SIGTERM at the timeout, SIGKILL 1 s later. The repository had 3,000 untracked 50 KB random files (152 MB), and I used a 4 s timeout in place of 30 s:

    method=batch turn=1..4  exit=143 tmp_dirs=1,2,3,4    loose=3..6
    method=fsync turn=1     exit=143 tmp_dirs=0          loose=146
    method=fsync turn=5     exit=143 tmp_dirs=0          loose=1507
    method=fsync turn=9     exit=143 tmp_dirs=0          loose=2909
    method=fsync turn=10    exit=0   tmp_dirs=0          loose=3002
    

    Under batch, a killed attempt throws away everything it wrote, so a workspace whose first capture exceeds the timeout never gets a checkpoint and leaks one directory per turn. With plain fsync, the method Linux already uses here, a killed attempt leaves no directory, and the objects it wrote count for the next attempt. The capture finished on turn 10.

    The cost is speed. A full git add -A of those 3,000 files took 10.5 s with batch and 37.2 s with per-object fsync on this machine.

    As I read #23715, a killed capture's private store is removed, which stops the leak. Its objects are removed with it, so the next attempt starts from zero. The non-convergence noted above for #29259 would remain. Keeping completed objects from a killed attempt, for example by staging without batch, or by publishing a private store's finished objects even when the add fails, would let these workspaces get a checkpoint after a few turns.

    Not checked: Windows, and whether the per-object fsync cost is acceptable for normal turns.

    Written by claude-opus-5-5 in Claude Code (T3 Code).