Description

Hello maintainers!

We would like to propose a set of enhancements for LibreFang's context management system. As LLMs with larger context windows (such as 1M+ tokens) and extensive reasoning capabilities (e.g., DeepSeek-R1, o1/o3-mini) become standard, managing the context window dynamically becomes critical for both latency and cost.

Currently, LibreFang uses gateway_compression.rs, context_compressor.rs, and history_fold.rs. While these are highly effective, they suffer from two issues in long-running developer chat sessions:

1. CoT Bloat: Stale reasoning steps from reasoning-centric models fill up the context with thousands of unnecessary <think> tokens.
2. Micro-Iteration Noise: Continuous developer loops (edit, run test, fail, edit, run test, pass) pollute the history with massive terminal outputs and intermediate errors that are no longer relevant to the current code state.

To address these, we propose implementing CoT-Pruning and Developer Loop Aggregation in the runtime.

1. Feature 1: Chain-of-Thought (CoT) Pruning

Problem

Reasoning models generate extensive internal reasoning traces before responding. While critical in the active turn, these past thoughts (e.g., the assistant's internal deliberations from 5 turns ago) waste significant context space.

Proposed Solution

Automatically strip reasoning traces (both <think>...</think> tags in text and raw reasoning_content API metadata) from older assistant messages.

Proposed Configuration

Add configuration settings under the [compaction] section of KernelConfig and AgentManifest:

[compaction]
# Strip reasoning content from assistant messages older than X turns. Set to 0 to disable.
strip_reasoning_after_turns = 3

Technical Design (Rust Runtime)

• Location: Integrated into crates/librefang-runtime/src/session_repair.rs or crates/librefang-runtime/src/agent_loop/prompt.rs during the message preparation phase (prepare_llm_messages).
• Algorithm:
  1. Traverse the message list backward.
  2. Count assistant turns.
  3. For assistant messages older than strip_reasoning_after_turns, prune the <think> block from the text content and clear the reasoning_content field from the provider metadata.
  4. Call session.mark_messages_mutated() and save the session.

2. Feature 2: Developer Loop Aggregation

Problem

When the agent executes autonomous loops (e.g., repeating file_write and run_tests until a compilation error is fixed), it populates the history with redundant ToolUse and ToolResult pairs. This intermediate noise is irrelevant for future turns and degrades performance.

Proposed Solution

Aggregate consecutive runs of developer tools into a single consolidated turn, stripping out intermediate terminal logs and intermediate file writes while preserving only the final success/failure outcome and final diffs.

Proposed Configuration

[compaction]
# Aggregate consecutive autonomous code/test loops
aggregate_developer_loops = true
# Maximum consecutive steps before triggering aggregation
max_loop_steps_before_aggregate = 5

Technical Design (Rust Runtime)

• Location: Implement as a new filter module or integrate into session_repair.rs.
• Algorithm:
  1. Identify contiguous runs of tool execution blocks containing developer actions (e.g., run_tests, run_command, file_write, replace_file_content, grep_search).
  2. If the run length exceeds threshold limits:
     • Retain the initial intent/call and the final outcome.
     • Replace intermediate runs with a synthesized user message indicating that compaction occurred:

       [DEVELOPER LOOP AGGREGATED] Agent executed 4 intermediate compilation and fix attempts on files X, Y. Intermediate compile errors were resolved. The final execution state is shown below.

     • Prune intermediate failing logs.

Questions for Maintainers & Design Decisions

Before starting work, we would love to hear your feedback on the overall design and discuss the following open questions:

1. Configurable Developer Tool List: What tools should be classified as "developer tools" for loop aggregation? Should the list be hardcoded in the runtime (e.g., run_tests, file_write, run_command, replace_file_content, grep_search), or should we expose it as a customizable config array (e.g., developer_tools = ["run_tests", "file_write"])?
2. CoT Deletion vs. Truncation: For CoT-Pruning, should we delete the reasoning block entirely, or truncate it (e.g., preserving only the first 100 characters for debugging purposes)?
3. Prompt Cache Impact: Since modifying past messages dynamically invalidates the provider prompt cache (specifically for Anthropic's cache control), we should run pruning only at turn boundaries or pre-compaction. What is your preferred boundary for applying these prunes to minimize cache misses?

We look forward to hearing your thoughts!

Alternatives Considered

No response

Additional Context

No response