Skip to content

[ROOT CAUSE ANALYSIS] tmp_pack Leak - Complete Code Analysis & Fix Recommendations #10034

@vincent-pellerin

Description

@vincent-pellerin

Root Cause Analysis: tmp_pack Files Leak

Related to: #8749, #5617, #6845, #6523, #8577
Impact: 137GB+ disk space leak confirmed locally, 180GB+ reported
Analysis Date: January 22, 2026
OpenCode Version: 1.1.31
Status: Root cause identified with code-level evidence


Executive Summary

I've completed a deep analysis of the tmp_pack leak bug by examining the OpenCode binary. The root cause is long-running git add . operations that fail/timeout without cleanup, combined with inadequate cleanup mechanisms that don't remove these temporary files.

Key Findings:

  • ✅ Found exact code location causing the leak
  • ✅ Identified missing timeout on git operations
  • ✅ Confirmed cleanup function doesn't handle tmp_pack files
  • ✅ Documented fix with code-level recommendations
  • ✅ Verified locally: 137GB leaked across 9 files

Root Cause: Code Analysis

1. The Problematic Code

Location: Snapshot.track() function in OpenCode binary

async function track() {
  if (Instance.project.vcs !== "git")
    return;
  const cfg = await Config.get();
  if (cfg.snapshot === false)
    return;
  const git = gitdir();  // Returns: ~/.local/share/opencode/snapshot/<hash>
  
  // Initialize git repo if needed
  if (await fs8.mkdir(git, { recursive: true })) {
    await $6`git init`.env({
      ...process.env,
      GIT_DIR: git,
      GIT_WORK_TREE: Instance.worktree
    }).quiet().nothrow();
  }
  
  // ⚠️ THE LEAK SOURCE - No timeout, silent failures
  await $6`git --git-dir ${git} --work-tree ${Instance.worktree} add .`
    .quiet()              // Suppress output
    .cwd(Instance.directory)
    .nothrow();           // Silent failure - CRITICAL ISSUE
  
  const hash2 = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} write-tree`
    .quiet()
    .cwd(Instance.directory)
    .nothrow().text();
  
  log3.info("tracking", { hash: hash2, cwd: Instance.directory, git });
  return hash2.trim();
}

Why This Causes Leaks:

  1. No timeout - Git can run indefinitely on large workspaces
  2. .nothrow() - Failures are silently ignored
  3. No error handling - Exit codes never checked
  4. No cleanup on failure - tmp_pack files left behind

2. When This Code Runs

Snapshot.track() is called 3-5+ times per AI interaction:

  • During message processing
  • On step start
  • On step finish
  • During revert operations
  • In patch() and diff() functions

Result: Multiple long-running git operations → multiple opportunities for failure → multiple tmp_pack files

3. Current Cleanup (Inadequate)

// Runs every 1 hour
Scheduler.register({
  id: "snapshot.cleanup",
  interval: 3600000,  // 1 hour
  run: cleanup,
  scope: "instance"
});

async function cleanup() {
  // ⚠️ ONLY runs git gc - doesn't remove tmp_pack files!
  const result = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} gc --prune=${prune}`
    .quiet()
    .cwd(Instance.directory)
    .nothrow();
  
  if (result.exitCode !== 0) {
    log3.warn("cleanup failed", { exitCode, stderr, stdout });
    return;  // Silent failure
  }
  log3.info("cleanup", { prune: "7.days" });
}

Why This Fails:

  • git gc --prune=7.days only prunes tracked objects
  • tmp_pack files are untracked artifacts from failed operations
  • Git gc doesn't remove tmp_pack files by design
  • No explicit rm of tmp_pack files anywhere in the code

Evidence: Local System

Before Cleanup

Location: ~/.local/share/opencode/snapshot/ed610c2ec990659cd5a93f0917806ddcc867e270/objects/pack/

9 tmp_pack files found:
-r--r--r-- 1 vincent vincent  128K Jan 20 15:51 tmp_pack_KyRuL7
-r--r--r-- 1 vincent vincent  5.5M Jan 22 14:07 tmp_pack_4jQkSi
-r--r--r-- 1 vincent vincent  888M Jan 22 13:57 tmp_pack_KmlQ1h
-r--r--r-- 1 vincent vincent  3.2G Jan 22 14:39 tmp_pack_PsoqEP  ← Was actively growing
-r--r--r-- 1 vincent vincent   18G Jan 22 13:51 tmp_pack_LkPPMX
-r--r--r-- 1 vincent vincent   20G Jan 21 16:07 tmp_pack_wDyovs
-r--r--r-- 1 vincent vincent   32G Jan 22 13:05 tmp_pack_bcwOI3
-r--r--r-- 1 vincent vincent   32G Jan 20 21:54 tmp_pack_hWLfpL
-r--r--r-- 1 vincent vincent   32G Jan 22 12:10 tmp_pack_PZkeXR

Total: 137GB
Directory size: 138GB

Active Process Found

$ ps aux | grep "git.*add"
vincent  38714 89.2  0.0  14620  4976 pts/1  R  14:23  10:20 
  /usr/bin/git --git-dir /home/vincent/.local/share/opencode/snapshot/ed610c2ec990659cd5a93f0917806ddcc867e270 
  --work-tree /home/vincent/dev add .

Runtime: 10+ minutes at 89% CPU

After Cleanup

# Killed process + removed tmp_pack files
Directory size: 889MB (down from 138GB)
Recovered: 137GB

Recommended Fixes

Priority 1: Add Timeout (CRITICAL)

// BEFORE (current code):
await $6`git --git-dir ${git} --work-tree ${Instance.worktree} add .`
  .quiet()
  .cwd(Instance.directory)
  .nothrow();

// AFTER (with fix):
const addResult = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} add .`
  .quiet()
  .cwd(Instance.directory)
  .timeout(30000)  // 30 second timeout
  .nothrow();

// Check result and cleanup on failure
if (addResult.exitCode !== 0) {
  log3.warn("git add failed or timed out", {
    exitCode: addResult.exitCode,
    stderr: addResult.stderr.toString()
  });
  await cleanupTmpPacks(git);
  return null;
}

Priority 2: Enhance Cleanup (CRITICAL)

async function cleanup() {
  if (Instance.project.vcs !== "git")
    return;
  const cfg = await Config.get();
  if (cfg.snapshot === false)
    return;
  const git = gitdir();
  const exists = await fs8.stat(git).then(() => true).catch(() => false);
  if (!exists)
    return;

  // ========== ADD THIS: Clean tmp_pack files ==========
  try {
    const packDir = path.join(git, "objects", "pack");
    const files = await fs8.readdir(packDir).catch(() => []);
    const tmpPacks = files.filter(f => f.startsWith("tmp_pack_"));
    
    if (tmpPacks.length > 0) {
      log3.info("removing tmp_pack files", { count: tmpPacks.length });
      
      await Promise.all(
        tmpPacks.map(file => 
          fs8.unlink(path.join(packDir, file))
            .catch(err => log3.warn("failed to remove tmp_pack", { file, error: err.message }))
        )
      );
      
      log3.info("tmp_pack cleanup complete", { removed: tmpPacks.length });
    }
  } catch (err) {
    log3.warn("tmp_pack cleanup error", { error: err.message });
  }
  // ====================================================

  // Existing git gc cleanup
  const result = await $6`git --git-dir ${git} --work-tree ${Instance.worktree} gc --prune=${prune}`
    .quiet()
    .cwd(Instance.directory)
    .nothrow();
  
  if (result.exitCode !== 0) {
    log3.warn("cleanup failed", {
      exitCode: result.exitCode,
      stderr: result.stderr.toString(),
      stdout: result.stdout.toString()
    });
    return;
  }
  log3.info("cleanup", { prune });
}

// Helper function for immediate cleanup after failures
async function cleanupTmpPacks(git) {
  try {
    const packDir = path.join(git, "objects", "pack");
    const files = await fs8.readdir(packDir).catch(() => []);
    const tmpPacks = files.filter(f => f.startsWith("tmp_pack_"));
    
    await Promise.all(
      tmpPacks.map(file => fs8.unlink(path.join(packDir, file)).catch(() => {}))
    );
    
    if (tmpPacks.length > 0) {
      log3.info("cleaned tmp_pack after failure", { count: tmpPacks.length });
    }
  } catch (err) {
    log3.warn("tmp_pack cleanup error", { error: err.message });
  }
}

Priority 3: Add Error Handling (HIGH)

Apply to ALL git operations in track(), patch(), and diff():

  • Check exit codes
  • Log failures with context
  • Call cleanupTmpPacks() on failure
  • Return null instead of continuing with failed state

Priority 4: Reduce Frequency (MEDIUM)

Implement throttling to avoid excessive git operations:

let lastSnapshot = null;
let lastSnapshotTime = 0;
const SNAPSHOT_THROTTLE = 5000; // 5 seconds

async function track() {
  const now = Date.now();
  if (now - lastSnapshotTime < SNAPSHOT_THROTTLE) {
    log3.debug("throttled snapshot", { lastSnapshot });
    return lastSnapshot;
  }
  
  // ... existing track() code ...
  
  lastSnapshot = hash2.trim();
  lastSnapshotTime = now;
  return lastSnapshot;
}

Priority 5: Configuration Options (LOW)

// User-configurable in ~/.opencode/config
{
  "snapshot": {
    "enabled": true,
    "timeout": 30000,          // NEW: Timeout in ms
    "strategy": "every-step",  // NEW: "on-demand" | "every-step" | "minimal"
    "cleanup": {
      "interval": 3600000,
      "prune": "7.days",
      "removeTmpPack": true    // NEW: Remove tmp_pack files
    }
  }
}

Immediate Workaround for Users

# 1. Kill any stuck git processes
pkill -f "git.*add.*opencode"

# 2. Remove all tmp_pack files
rm -f ~/.local/share/opencode/snapshot/*/objects/pack/tmp_pack_*

# 3. Verify cleanup
du -sh ~/.local/share/opencode/snapshot/*/objects/pack/

Optional: Set up automated cleanup until fixed

# Add to crontab (runs every hour)
crontab -e
# Add this line:
0 * * * * find ~/.local/share/opencode/snapshot/*/objects/pack/ -name "tmp_pack_*" -delete

Testing Plan

Test Case 1: Verify tmp_pack Cleanup

  1. Create tmp_pack files manually in snapshot directory
  2. Wait for hourly cleanup or trigger manually
  3. Verify tmp_pack files are removed

Test Case 2: Verify Timeout Works

  1. Create large workspace (>10GB)
  2. Trigger snapshot
  3. Verify operation times out after 30s
  4. Verify no tmp_pack files left behind

Test Case 3: Verify Error Handling

  1. Corrupt git directory
  2. Trigger snapshot
  3. Verify error is logged
  4. Verify tmp_pack cleanup is called

Test Case 4: Verify Throttling

  1. Enable throttling
  2. Perform multiple operations quickly
  3. Verify only 1 snapshot within throttle period

Impact Analysis

Affected Users:

  • All Linux/Unix users with large workspaces
  • Ubuntu 22.04, 24.04, 25.10 confirmed
  • NixOS confirmed
  • Potentially all Unix-based systems

Severity: CRITICAL

  • Disk space exhaustion (137GB-180GB+ reported)
  • Can fill entire filesystem
  • Degrades system performance
  • No user notification
  • Silent failures accumulate over time

Related Issues:

All share the same root cause.


Conclusion

This is a critical bug with a clear root cause and straightforward fix:

  1. Root cause: Long-running git add . operations fail/timeout without cleanup
  2. Amplified by: No timeout, silent failures, inadequate cleanup, excessive frequency
  3. Impact: 137GB+ wasted disk space, performance degradation, filled disks
  4. Fix complexity: LOW - add timeout + tmp_pack cleanup + error handling
  5. Priority: CRITICAL - affects all users with large workspaces

Recommended implementation order:

  1. ✅ Fix cleanup to remove tmp_pack files (immediate relief)
  2. ✅ Add timeout to git operations (prevent new leaks)
  3. ✅ Add error handling and logging (visibility)
  4. ⚙️ Optimize snapshot frequency (performance)
  5. ⚙️ Add configuration options (flexibility)

Full Analysis Document

I've created a complete technical analysis document with all details, code snippets, and evidence. Available at:

  • Local: /home/vincent/tmp_pack_leak_analysis.md
  • Can be shared if needed

Analysis performed by: @vincent-pellerin
Date: January 22, 2026
OpenCode version tested: 1.1.31
System: Ubuntu 24.04 LTS

I'm happy to provide additional details, test patches, or collaborate on implementing these fixes.

Metadata

Metadata

Assignees

Labels

perfIndicates a performance issue or need for optimization

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions