Skip to content

fix(oom): cap output buffers to stop 96GB memory blowups#288

Merged
subsy merged 6 commits intosubsy:mainfrom
dyxushuai:fix/oom-output-buffer-limits
Feb 13, 2026
Merged

fix(oom): cap output buffers to stop 96GB memory blowups#288
subsy merged 6 commits intosubsy:mainfrom
dyxushuai:fix/oom-output-buffer-limits

Conversation

@dyxushuai
Copy link
Contributor

@dyxushuai dyxushuai commented Feb 7, 2026

Why

We hit a real OOM during a long ralph-tui run session, even on a 96 GB machine.

The process was retaining output in several overlapping places:

  • agent stdout/stderr capture
  • engine live output buffers (for TUI/remote state)
  • in-memory iteration history

Those copies kept growing over time and eventually exhausted memory.

How

  • Added hard size caps to agent stdout/stderr in-memory buffers.
  • Added hard size caps to engine live currentOutput / currentStderr.
  • Switched iteration raw log persistence to stream from per-iteration temp stdout/stderr files, so on-disk logs keep full raw output without requiring full output in memory.
  • Truncated in-memory iteration history output while preserving tail content and completion markers.
  • Reworked truncation helpers to avoid transient current + chunk allocations under heavy streaming output.
  • Added regression tests for truncation behavior, tail preservation, and streamed raw log persistence.

Tests

  • bun test src/plugins/agents/base.memory.test.ts src/engine/memory-limits.test.ts tests/logs/persistence.test.ts
  • bun run typecheck
  • bun run build
  • ubs ... (not run in this environment: command not found)

Summary by CodeRabbit

  • New Features

    • Capture large raw stdout/stderr to temporary files and stream them into iteration logs for efficient external inspection.
    • Expose memory-safe helpers for testing and safer public interfaces to record iteration file paths.
  • Bug Fixes

    • Enforced in-memory limits on agent stdout/stderr to prevent unbounded memory growth; long outputs are truncated with a visible notice while preserving a tail.
  • Tests

    • Added tests for memory-safe output handling, truncation behaviour, file-backed streaming and related utilities.

Copilot AI review requested due to automatic review settings February 7, 2026 16:45
@vercel
Copy link

vercel bot commented Feb 7, 2026

@dyxushuai is attempting to deploy a commit to the plgeek Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 7, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds memory-bounded streaming for stdout/stderr, per-iteration raw on-disk capture and plumbing to persist raw files, and exposes test helpers; includes unit tests for buffer-limiting and file-backed log streaming.

Changes

Cohort / File(s) Summary
Engine: memory-safety & raw-output capture
src/engine/index.ts, src/engine/memory-limits.test.ts
Imported AgentExecutionResult; added appendWithCharLimit and toMemorySafeAgentResult; initialize/manage per-iteration raw output temp files, write/flush/close helpers, and propagate raw file paths into iteration payloads; exposed __test__. Added tests for memory-safe behavior.
Agent plugin: streaming limits & tests
src/plugins/agents/base.ts, src/plugins/agents/base.memory.test.ts
Introduced MAX_EXECUTION_STREAM_CHARS and STREAM_TRUNCATED_PREFIX; added wrapper appendWithCharLimit delegating to shared util; replaced direct stdout/stderr concatenation with bounded appends; exposed test-only __test__. Added unit tests for truncation scenarios.
Logs persistence: file streaming support & tests
src/logs/persistence.ts, tests/logs/persistence.test.ts
Added stream helpers (writeChunkToStream, pipeFileToStream, closeWriteStream); extended SaveIterationLogOptions with rawStdoutFilePath/rawStderrFilePath; saveIterationLog can stream raw files into persisted logs and return early. Tests added to verify file-backed streaming path.
Utilities: shared buffer limiter
src/utils/buffer-limits.ts
New appendWithCharLimit implementation that enforces a max character limit, preserves tail content and applies a truncation prefix.
Tests: misc
tests/plugins/agents/utils.test.ts
Added tests for extractErrorMessage helper covering various input shapes and edge cases.

Sequence Diagram(s)

sequenceDiagram
  participant Engine
  participant AgentPlugin
  participant Filesystem
  participant LogsPersistence

  Engine->>AgentPlugin: start iteration (capture IO)
  AgentPlugin->>AgentPlugin: onStdout/onStderr -> appendWithCharLimit
  AgentPlugin->>Filesystem: write raw stdout/stderr chunks to temp files
  AgentPlugin-->>Engine: return AgentExecutionResult (includes raw file paths)
  Engine->>LogsPersistence: saveIterationLog(..., rawStdoutFilePath, rawStderrFilePath)
  LogsPersistence->>Filesystem: stream raw files into persisted log
  LogsPersistence-->>Engine: persisted log metadata
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and specifically describes the main change: implementing memory caps on output buffers to prevent out-of-memory conditions.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into main

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

No actionable comments were generated in the recent review. 🎉

🧹 Recent nitpick comments
src/engine/index.ts (1)

1020-1031: Silent write-error disabling could lose the raw file path while the file is partially written.

When a write fails, appendRawChunk sets filePath = undefined (line 1029) and closes the handle. This means saveIterationLog won't receive the path and the partially written file will still sit in the temp directory (cleaned up later by finally). That's acceptable as best-effort, but be aware that no diagnostic is emitted—a partially written iteration will appear to have no raw log at all rather than a partial one.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link

codecov bot commented Feb 7, 2026

Codecov Report

❌ Patch coverage is 74.58194% with 76 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.47%. Comparing base (87cbcf6) to head (07be6e3).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
src/engine/index.ts 78.06% 34 Missing ⚠️
src/logs/persistence.ts 71.59% 25 Missing ⚠️
src/plugins/agents/base.ts 56.52% 10 Missing ⚠️
src/utils/buffer-limits.ts 78.78% 7 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #288      +/-   ##
==========================================
+ Coverage   44.17%   44.47%   +0.29%     
==========================================
  Files          95       96       +1     
  Lines       29839    30132     +293     
==========================================
+ Hits        13181    13400     +219     
- Misses      16658    16732      +74     
Files with missing lines Coverage Δ
src/utils/buffer-limits.ts 78.78% <78.78%> (ø)
src/plugins/agents/base.ts 38.13% <56.52%> (+0.65%) ⬆️
src/logs/persistence.ts 52.74% <71.59%> (+2.14%) ⬆️
src/engine/index.ts 50.88% <78.06%> (+2.38%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds in-memory output truncation to prevent unbounded stdout/stderr growth during long-running ralph-tui run sessions that can lead to OOMs.

Changes:

  • Cap agent execution stdout/stderr buffers held by BaseAgentPlugin.
  • Cap engine “live output” buffers and truncate per-iteration in-memory history output while preserving tail markers.
  • Add regression tests covering truncation behavior and tail preservation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/plugins/agents/base.ts Adds a capped append helper and applies it to captured agent stdout/stderr buffers.
src/plugins/agents/base.memory.test.ts Tests helper behavior for truncation + tail preservation.
src/engine/index.ts Caps engine live output buffers and truncates iteration-history agentResult stored in memory; exposes helpers for tests.
src/engine/memory-limits.test.ts Tests engine-side truncation of stored iteration history output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/engine/index.ts (1)

1243-1250: ⚠️ Potential issue | 🟡 Minor

Disk log fallback for stderr may receive a truncated buffer.

The call agentResult.stderr ?? this.state.currentStderr falls back to the engine's live buffer, which is now capped at 250 K. If the agent's own stderr field is empty while currentStderr overflowed, the on-disk iteration log will silently lose early stderr content. Given the PR's stated goal of keeping "full raw output in on-disk iteration logs", you may want to accumulate a separate unbounded (or disk-backed) stderr buffer for the log writer, or at least document this as an accepted limitation.

🧹 Nitpick comments (1)
src/plugins/agents/base.ts (1)

147-180: Duplicate implementation of appendWithCharLimit — extract to a shared utility.

This function is duplicated verbatim in src/engine/index.ts (lines 92–125). Both copies share the same algorithm and signature. If a bug is found or the truncation logic changes, both sites must be updated in lockstep.

Consider extracting it into a shared module (e.g., src/utils/string-limits.ts) and importing it in both base.ts and engine/index.ts, each supplying its own constants and prefix.

♻️ Proposed shared utility

Create a new file src/utils/string-limits.ts:

/**
 * ABOUTME: Shared helper for capping in-memory string buffers while preserving tail content.
 */

/**
 * Append chunk data while enforcing an in-memory size cap.
 * Keeps the most recent content (tail) to preserve completion markers near the end.
 */
export function appendWithCharLimit(
  current: string,
  chunk: string,
  maxChars: number,
  prefix: string
): string {
  if (!chunk) return current;
  if (maxChars <= 0) return '';

  const totalLen = current.length + chunk.length;
  if (totalLen <= maxChars) {
    return current + chunk;
  }

  if (maxChars <= prefix.length) {
    return prefix.slice(0, maxChars);
  }

  const keep = maxChars - prefix.length;
  const combinedTailStart = totalLen - keep;

  let tail: string;
  if (combinedTailStart >= current.length) {
    const startInChunk = combinedTailStart - current.length;
    tail = chunk.slice(startInChunk);
  } else {
    const tailFromCurrent = current.slice(combinedTailStart);
    const remaining = keep - tailFromCurrent.length;
    const tailFromChunk = remaining > 0 ? chunk.slice(-remaining) : '';
    tail = tailFromCurrent + tailFromChunk;
  }

  return prefix + tail;
}

Then in both base.ts and engine/index.ts:

-function appendWithCharLimit(
-  current: string,
-  chunk: string,
-  maxChars: number,
-  prefix = STREAM_TRUNCATED_PREFIX
-): string {
-  // ... duplicated body ...
-}
+import { appendWithCharLimit } from '../../utils/string-limits.js';

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@src/engine/index.ts`:
- Around line 1046-1058: The inner try/catch that creates rawOutputTempDir and
opens rawStdoutFd/rawStderrFd currently clears rawOutputTempDir on any failure
which prevents the outer finally cleanup from removing the temp dir; instead,
preserve rawOutputTempDir so the finally block can delete it: in the catch for
the mkdtemp/openSync block do not set rawOutputTempDir = undefined, only close
any successfully opened descriptors (rawStdoutFd, rawStderrFd) and clear the
corresponding file path variables (rawStdoutFilePath, rawStderrFilePath) as
needed (use closeRawOutputFiles or explicit closeSync), then rethrow or allow
the outer logic to handle the error so the finally cleanup that removes
rawOutputTempDir can run.

In `@src/logs/persistence.ts`:
- Around line 680-714: The finally block currently calls
closeWriteStream(stream) unconditionally which can throw and mask an earlier
error from writeChunkToStream or pipeFileToStream; change the pattern in the raw
file-stream branch so you capture any primary error (e.g., assign to a variable
like originalError inside the try/catch) and then in finally attempt to
closeWriteStream(stream) inside its own try/catch—if closeWriteStream throws,
rethrow the originalError if present, otherwise rethrow the close error;
reference stream, writeChunkToStream, pipeFileToStream, and closeWriteStream
when making this change so the original write/pipe errors aren’t swallowed by a
close failure.
🧹 Nitpick comments (2)
src/engine/index.ts (2)

1020-1044: A write failure on one stream disables both raw output streams.

If writeSync to stdout fails, both rawStdoutFilePath and rawStderrFilePath are set to undefined, losing the stderr raw capture as well (and vice-versa). This is probably intentional as a defensive measure (shared temp directory likely broken), but worth a brief inline comment to clarify intent.


75-129: Extract appendWithCharLimit to a shared utility module.

The function is defined identically in both src/engine/index.ts (line 96) and src/plugins/agents/base.ts (line 147), differing only in the default prefix parameter. This duplication risks divergence and violates DRY principles. Extract it into a shared utility module (e.g. src/utils/buffer.ts) and import from both locations, passing the context-specific prefix strings as needed.

@subsy
Copy link
Owner

subsy commented Feb 12, 2026

@dyxushuai thanks for reporting - will look at this tomorrow

@vercel
Copy link

vercel bot commented Feb 13, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
ralph-tui Ignored Ignored Preview Feb 13, 2026 8:49pm

Request Review

@subsy
Copy link
Owner

subsy commented Feb 13, 2026

thanks @dyxushuai - made a few final fixes, and merging this

@subsy subsy merged commit f85ce9c into subsy:main Feb 13, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments