OOM kill: message history accumulation causes 8.5GB memory usage during task execution #29730

Problem

Running tasks in tmux with oh-my-openagent plugin causes OOM kills. The opencode process consumed 8.5 GB of RAM before being killed by the Linux OOM killer.

System: 15 GB RAM, Linux, tmux session
Plugin: oh-my-openagent (oh-my-opencode v4.5.1)
Database: 74 sessions, 905 messages, 5219 parts (27 MB actual data)
OOM details from dmesg:

oom-kill:constraint=CONSTRAINT_NONE,task=opencode,pid=63414,uid=1000
Out of memory: Killed process 63414 (opencode) total-vm:88502464kB, anon-rss:8911236kB

Note: The memory explosion occurs in opencode's core message handling code, not in oh-my-openagent. However, oh-my-openagent's usage patterns (background agents, subagent spawning, context injection hooks) trigger and amplify the issue.

Root Cause

Each LLM call loads the entire conversation history into memory and creates 4 copies of it:

1. Load from SQLite → Session.messages() (session/session.ts:767-786)
2. Convert to UIMessage → toModelMessagesEffect() (session/message-v2.ts:630-913)
3. Convert to ModelMessage → AI SDK convertToModelMessages()
4. JSON stringification for token estimation (in compaction)

Memory calculation

Why oh-my-openagent amplifies this

oh-my-openagent is a plugin that heavily uses opencode's session/task APIs:

• Background agents spawn multiple subagent sessions (each with own message history)
• Context injection hooks (experimental.chat.messages.transform) add content to messages
• Tool-heavy workflows generate大量 tool outputs (shell, read, grep)
• Long-running tasks (10-20+ LLM steps) accumulate message history

While the core 4× copying issue is in opencode, oh-my-openagent's patterns make it much more likely to hit OOM:

• 3-5 concurrent subagent sessions = 3-5× message history
• 20+ tool calls per session = large tool output accumulation
• Context injection = larger prompts per LLM call

Key Code Paths (all in opencode core)

1. Full history materialization

// session/session.ts:767-786
const messages = yield* MessageV2.page({ sessionID, limit: size, before })
// Loads ALL messages + ALL parts into single array

2. Multiple copies per LLM call

// session/message-v2.ts:630-913
export function toModelMessagesEffect(messages, model) {
  // Creates UIMessage[] then ModelMessage[] — 2× memory
}

3. Deep clone in compaction

// session/compaction.ts:404
const msgs = structuredClone(selected.head)  // Full deep copy

4. Shell output accumulation

// tool/shell.ts:437-501
let full = ""  // Accumulates ALL output before truncating
full += chunk  // String concatenation

Proposed Fixes (priority order)

1. Stream messages instead of loading all (Biggest impact)

// Instead of loading all messages into memory
for await (const batch of Session.messageStream({ sessionID, pageSize: 20 })) {
  // Process batch incrementally, don't accumulate
}

2. Eliminate structuredClone in compaction (Quick win)

// Instead of deep clone
const msgs = selected.head.map(m => ({ ...m, parts: [...m.parts] }))

3. Add memory limits to shell tool

// Stream to disk from start if output > 1MB
if (Buffer.byteLength(full) > 1_000_000) {
  // Switch to disk-backed buffer immediately
}

4. Reduce SQLite page cache

// From 64MB to 16MB for constrained environments
db.run("PRAGMA cache_size = -16000")

5. Limit concurrent subagent sessions

const MAX_CONCURRENT_SESSIONS = 3

Impact

• Current: OOM kill after ~10-20 LLM steps with subagents when using oh-my-openagent
• After fix: Peak memory reduced from GB-scale to MB-scale
• Users affected: Anyone running complex tasks with subagents on systems with ≤16 GB RAM, especially when using plugins like oh-my-openagent that spawn multiple sessions

Environment

• opencode version: latest (dev branch)
• Plugin: oh-my-openagent v4.5.1
• Runtime: Bun
• OS: Linux (Arch)
• RAM: 15 GB
• tmux: yes