Description

Background & Motivation

In the current implementation of LibreFang, the context size is kept in check primarily via the compaction pass, which runs periodically or when a threshold is met. However, developer tools (like cargo test, compiler builds, or dependency installers) can produce massive outputs (often exceeding 100 KB or 25,000+ tokens) in a single turn.

If an agent runs cargo test on step 2, that massive output is immediately fed into the prompt for step 3, step 4, and so on, until a compaction pass eventually aggregates it. This causes:

1. High Latency & Costs: The model immediately incurs high prompt-processing latency and cost on the very next turns.
2. Context Clutter: The model's attention is filled with hundreds of repetitive ok lines, diluting the focus on the task.

We propose introducing Active Tool Output Optimization—a hot-path mechanism that processes and minimizes tool outputs before they enter the LLM context, while preserving full diagnostic data for troubleshooting.

Proposed Architectural Options

1. Smart Output Pruning & Diagnostic Offloading (Recommended)

Instead of putting the entire stdout/stderr of a command into the conversation history, we offload the raw output to a session-specific diagnostic file.

• Mechanism:
  • The full raw output of the tool is written to a local log file, e.g., .librefang/sessions/<session_id>/tool_logs/exec_<N>.log.
  • The message appended to the LLM's context contains only a highly-compressed summary (e.g., a few lines) and a structured link pointing to the full log.
• On-Demand Inspection:
  Since the agent is already equipped with file-reading and search tools (view_file, grep_search), it can inspect the full log only when needed (e.g., if a build fails or if it needs to check a specific warning).
• Example Context Message:

  Exit code: 0
  [Full stdout/stderr (1,240 lines) offloaded to file:///home/pavver/.librefang/sessions/abc-123/tool_logs/exec_4.log]
  Use `view_file` or `grep_search` on this path if you need to read the full logs or search for specific errors.
  

2. Output Line Filtering & Smart Parsing

We can implement lightweight output processors that scan stdout and keep only the "essential" lines based on simple rules:

• For Successful Executions (exit_code == 0):
  • Scan the output and keep only summary lines (e.g., lines starting with test result:, Finished, or showing test counts), dropping individual successful test lines (test ... ok).
• For Failed Executions (exit_code != 0):
  • Keep compiler errors, panic stack traces, and lines mentioning failed tests, while stripping successful progress blocks.
• This ensures that the primary context remains highly readable and clean, containing only the actual diagnostic signals.

3. Exit Code & Stream Prioritization (Stdout vs. Stderr)

A simple, zero-parsing approach that filters streams based on command success:

• On Success: Suppress stdout entirely (or truncate it to the first/last few lines) and keep only stderr (which might contain compilation warnings or lint output).
• On Failure: Keep the end of stdout (where the failure summary lies) and the complete stderr.

4. Agent Prompt-Level Flag Guidance (Quiet/Failed Flags)

Complementing the runtime changes, we can guide the agent via the system prompt to minimize output at the source. Currently, crates/librefang-runtime/src/prompt_builder.rs contains no guidelines regarding output minimization.

We propose adding a new bullet point to OPERATIONAL_GUIDELINES (or TOOL_CALL_BEHAVIOR):

 ## Operational Guidelines
 - Prefer targeted, specific tool calls over broad ones.
+- When executing CLI tools, compilers, or test runners, prefer quiet or minimal flags (e.g., -q, --quiet) and target specific files or test suites rather than running broad workspace-wide commands. Only request verbose output when actively debugging a failure.
 - Plan your approach before executing multiple tool calls.

By adding this guideline, the LLM will naturally select cleaner execution strategies, saving tokens and compute before post-processing even occurs.

Key Benefits

1. Immediate Context Savings: Reduces prompt sizes on the very next turn after running verbose tools, rather than waiting for a compaction cycle.
2. Zero Loss of Information: Because the full logs are preserved as local files, the model retains the ability to debug complex failures on-demand.
3. No New Tools Needed: Leverages the agent's existing file-reading capabilities without adding new custom API endpoints or CLI tools.
4. Simple Implementation: Can be integrated directly into the tool execution layer (e.g., tool_runner/shell.rs) with minimal performance overhead.

Alternatives Considered

No response

Additional Context

No response