-
Notifications
You must be signed in to change notification settings - Fork 14.7k
[ROOT CAUSE ANALYSIS] tmp_pack Leak - Complete Code Analysis & Fix Recommendations #10034
Description
Root Cause Analysis: tmp_pack Files Leak
Related to: #8749, #5617, #6845, #6523, #8577
Impact: 137GB+ disk space leak confirmed locally, 180GB+ reported
Analysis Date: January 22, 2026
OpenCode Version: 1.1.31
Status: Root cause identified with code-level evidence
Executive Summary
I've completed a deep analysis of the tmp_pack leak bug by examining the OpenCode binary. The root cause is long-running git add . operations that fail/timeout without cleanup, combined with inadequate cleanup mechanisms that don't remove these temporary files.
Key Findings:
- ✅ Found exact code location causing the leak
- ✅ Identified missing timeout on git operations
- ✅ Confirmed cleanup function doesn't handle tmp_pack files
- ✅ Documented fix with code-level recommendations
- ✅ Verified locally: 137GB leaked across 9 files
Root Cause: Code Analysis
1. The Problematic Code
Location: Snapshot.track() function in OpenCode binary
async function track() {
if (Instance.project.vcs !== "git")
return;
const cfg = await Config.get();
if (cfg.snapshot === false)
return;
const git = gitdir(); // Returns: ~/.local/share/opencode/snapshot/<hash>
// Initialize git repo if needed
if (await fs8.mkdir(git, { recursive: true })) {
await $6`git init`.env({
...process.env,
GIT_DIR: git,
GIT_WORK_TREE: Instance.worktree
}).quiet().nothrow();
}
// ⚠️ THE LEAK SOURCE - No timeout, silent failures
await $6`git --git-dir ${git} --work-tree ${Instance.worktree} add .`
.quiet() // Suppress output
.cwd(Instance.directory)
.nothrow(); // Silent failure - CRITICAL ISSUE
const hash2 = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} write-tree`
.quiet()
.cwd(Instance.directory)
.nothrow().text();
log3.info("tracking", { hash: hash2, cwd: Instance.directory, git });
return hash2.trim();
}Why This Causes Leaks:
- No timeout - Git can run indefinitely on large workspaces
.nothrow()- Failures are silently ignored- No error handling - Exit codes never checked
- No cleanup on failure - tmp_pack files left behind
2. When This Code Runs
Snapshot.track() is called 3-5+ times per AI interaction:
- During message processing
- On step start
- On step finish
- During revert operations
- In
patch()anddiff()functions
Result: Multiple long-running git operations → multiple opportunities for failure → multiple tmp_pack files
3. Current Cleanup (Inadequate)
// Runs every 1 hour
Scheduler.register({
id: "snapshot.cleanup",
interval: 3600000, // 1 hour
run: cleanup,
scope: "instance"
});
async function cleanup() {
// ⚠️ ONLY runs git gc - doesn't remove tmp_pack files!
const result = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} gc --prune=${prune}`
.quiet()
.cwd(Instance.directory)
.nothrow();
if (result.exitCode !== 0) {
log3.warn("cleanup failed", { exitCode, stderr, stdout });
return; // Silent failure
}
log3.info("cleanup", { prune: "7.days" });
}Why This Fails:
git gc --prune=7.daysonly prunes tracked objects- tmp_pack files are untracked artifacts from failed operations
- Git gc doesn't remove tmp_pack files by design
- No explicit
rmof tmp_pack files anywhere in the code
Evidence: Local System
Before Cleanup
Location: ~/.local/share/opencode/snapshot/ed610c2ec990659cd5a93f0917806ddcc867e270/objects/pack/
9 tmp_pack files found:
-r--r--r-- 1 vincent vincent 128K Jan 20 15:51 tmp_pack_KyRuL7
-r--r--r-- 1 vincent vincent 5.5M Jan 22 14:07 tmp_pack_4jQkSi
-r--r--r-- 1 vincent vincent 888M Jan 22 13:57 tmp_pack_KmlQ1h
-r--r--r-- 1 vincent vincent 3.2G Jan 22 14:39 tmp_pack_PsoqEP ← Was actively growing
-r--r--r-- 1 vincent vincent 18G Jan 22 13:51 tmp_pack_LkPPMX
-r--r--r-- 1 vincent vincent 20G Jan 21 16:07 tmp_pack_wDyovs
-r--r--r-- 1 vincent vincent 32G Jan 22 13:05 tmp_pack_bcwOI3
-r--r--r-- 1 vincent vincent 32G Jan 20 21:54 tmp_pack_hWLfpL
-r--r--r-- 1 vincent vincent 32G Jan 22 12:10 tmp_pack_PZkeXR
Total: 137GB
Directory size: 138GBActive Process Found
$ ps aux | grep "git.*add"
vincent 38714 89.2 0.0 14620 4976 pts/1 R 14:23 10:20
/usr/bin/git --git-dir /home/vincent/.local/share/opencode/snapshot/ed610c2ec990659cd5a93f0917806ddcc867e270
--work-tree /home/vincent/dev add .
Runtime: 10+ minutes at 89% CPUAfter Cleanup
# Killed process + removed tmp_pack files
Directory size: 889MB (down from 138GB)
Recovered: 137GBRecommended Fixes
Priority 1: Add Timeout (CRITICAL)
// BEFORE (current code):
await $6`git --git-dir ${git} --work-tree ${Instance.worktree} add .`
.quiet()
.cwd(Instance.directory)
.nothrow();
// AFTER (with fix):
const addResult = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} add .`
.quiet()
.cwd(Instance.directory)
.timeout(30000) // 30 second timeout
.nothrow();
// Check result and cleanup on failure
if (addResult.exitCode !== 0) {
log3.warn("git add failed or timed out", {
exitCode: addResult.exitCode,
stderr: addResult.stderr.toString()
});
await cleanupTmpPacks(git);
return null;
}Priority 2: Enhance Cleanup (CRITICAL)
async function cleanup() {
if (Instance.project.vcs !== "git")
return;
const cfg = await Config.get();
if (cfg.snapshot === false)
return;
const git = gitdir();
const exists = await fs8.stat(git).then(() => true).catch(() => false);
if (!exists)
return;
// ========== ADD THIS: Clean tmp_pack files ==========
try {
const packDir = path.join(git, "objects", "pack");
const files = await fs8.readdir(packDir).catch(() => []);
const tmpPacks = files.filter(f => f.startsWith("tmp_pack_"));
if (tmpPacks.length > 0) {
log3.info("removing tmp_pack files", { count: tmpPacks.length });
await Promise.all(
tmpPacks.map(file =>
fs8.unlink(path.join(packDir, file))
.catch(err => log3.warn("failed to remove tmp_pack", { file, error: err.message }))
)
);
log3.info("tmp_pack cleanup complete", { removed: tmpPacks.length });
}
} catch (err) {
log3.warn("tmp_pack cleanup error", { error: err.message });
}
// ====================================================
// Existing git gc cleanup
const result = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} gc --prune=${prune}`
.quiet()
.cwd(Instance.directory)
.nothrow();
if (result.exitCode !== 0) {
log3.warn("cleanup failed", {
exitCode: result.exitCode,
stderr: result.stderr.toString(),
stdout: result.stdout.toString()
});
return;
}
log3.info("cleanup", { prune });
}
// Helper function for immediate cleanup after failures
async function cleanupTmpPacks(git) {
try {
const packDir = path.join(git, "objects", "pack");
const files = await fs8.readdir(packDir).catch(() => []);
const tmpPacks = files.filter(f => f.startsWith("tmp_pack_"));
await Promise.all(
tmpPacks.map(file => fs8.unlink(path.join(packDir, file)).catch(() => {}))
);
if (tmpPacks.length > 0) {
log3.info("cleaned tmp_pack after failure", { count: tmpPacks.length });
}
} catch (err) {
log3.warn("tmp_pack cleanup error", { error: err.message });
}
}Priority 3: Add Error Handling (HIGH)
Apply to ALL git operations in track(), patch(), and diff():
- Check exit codes
- Log failures with context
- Call
cleanupTmpPacks()on failure - Return null instead of continuing with failed state
Priority 4: Reduce Frequency (MEDIUM)
Implement throttling to avoid excessive git operations:
let lastSnapshot = null;
let lastSnapshotTime = 0;
const SNAPSHOT_THROTTLE = 5000; // 5 seconds
async function track() {
const now = Date.now();
if (now - lastSnapshotTime < SNAPSHOT_THROTTLE) {
log3.debug("throttled snapshot", { lastSnapshot });
return lastSnapshot;
}
// ... existing track() code ...
lastSnapshot = hash2.trim();
lastSnapshotTime = now;
return lastSnapshot;
}Priority 5: Configuration Options (LOW)
// User-configurable in ~/.opencode/config
{
"snapshot": {
"enabled": true,
"timeout": 30000, // NEW: Timeout in ms
"strategy": "every-step", // NEW: "on-demand" | "every-step" | "minimal"
"cleanup": {
"interval": 3600000,
"prune": "7.days",
"removeTmpPack": true // NEW: Remove tmp_pack files
}
}
}Immediate Workaround for Users
# 1. Kill any stuck git processes
pkill -f "git.*add.*opencode"
# 2. Remove all tmp_pack files
rm -f ~/.local/share/opencode/snapshot/*/objects/pack/tmp_pack_*
# 3. Verify cleanup
du -sh ~/.local/share/opencode/snapshot/*/objects/pack/Optional: Set up automated cleanup until fixed
# Add to crontab (runs every hour)
crontab -e
# Add this line:
0 * * * * find ~/.local/share/opencode/snapshot/*/objects/pack/ -name "tmp_pack_*" -deleteTesting Plan
Test Case 1: Verify tmp_pack Cleanup
- Create tmp_pack files manually in snapshot directory
- Wait for hourly cleanup or trigger manually
- Verify tmp_pack files are removed
Test Case 2: Verify Timeout Works
- Create large workspace (>10GB)
- Trigger snapshot
- Verify operation times out after 30s
- Verify no tmp_pack files left behind
Test Case 3: Verify Error Handling
- Corrupt git directory
- Trigger snapshot
- Verify error is logged
- Verify tmp_pack cleanup is called
Test Case 4: Verify Throttling
- Enable throttling
- Perform multiple operations quickly
- Verify only 1 snapshot within throttle period
Impact Analysis
Affected Users:
- All Linux/Unix users with large workspaces
- Ubuntu 22.04, 24.04, 25.10 confirmed
- NixOS confirmed
- Potentially all Unix-based systems
Severity: CRITICAL
- Disk space exhaustion (137GB-180GB+ reported)
- Can fill entire filesystem
- Degrades system performance
- No user notification
- Silent failures accumulate over time
Related Issues:
- [BUG] tmp_pack Files Consuming 180GB on Ubuntu 24.04 LTS #8749 - Ubuntu 24.04, 180GB (this analysis)
- [BUG] Ubuntu 22.04 Temp_Pack Files Using All of the Storage #5617 - Ubuntu 22.04, 79GB in 10 minutes
- HUGE snapshot folder #6845 - NixOS, 169GB snapshot folder
- opencode creates identical temp files everytime it runs #6523 - tmp_pack identical files, 60GB+
- Snapshot feature needs safeguards for large directories (home directory filled disk) #8577 - Large directories causing disk exhaustion
All share the same root cause.
Conclusion
This is a critical bug with a clear root cause and straightforward fix:
- Root cause: Long-running
git add .operations fail/timeout without cleanup - Amplified by: No timeout, silent failures, inadequate cleanup, excessive frequency
- Impact: 137GB+ wasted disk space, performance degradation, filled disks
- Fix complexity: LOW - add timeout + tmp_pack cleanup + error handling
- Priority: CRITICAL - affects all users with large workspaces
Recommended implementation order:
- ✅ Fix cleanup to remove tmp_pack files (immediate relief)
- ✅ Add timeout to git operations (prevent new leaks)
- ✅ Add error handling and logging (visibility)
- ⚙️ Optimize snapshot frequency (performance)
- ⚙️ Add configuration options (flexibility)
Full Analysis Document
I've created a complete technical analysis document with all details, code snippets, and evidence. Available at:
- Local:
/home/vincent/tmp_pack_leak_analysis.md - Can be shared if needed
Analysis performed by: @vincent-pellerin
Date: January 22, 2026
OpenCode version tested: 1.1.31
System: Ubuntu 24.04 LTS
I'm happy to provide additional details, test patches, or collaborate on implementing these fixes.