Tool output pruning: hide comments and docstrings in source-code tool results (Python PoC: 6.1% replay saving, live A/B inconclusive; design for other languages) #191

Problem

A coding agent spends a large share of its input tokens on tool results that are source code, and 20 to 30% of a Python source view is comments and docstrings. tool_output_cap cuts by size and has no basis to decide what to cut, so a wrong cut costs the agent extra calls. We want a reduction that removes content an edit rarely needs, without touching code.

Proposed solution

A policy, tool_output_prune (opt-in), that rewrites a source-code tool result with comments and docstrings hidden:

• code lines stay verbatim and the line count is unchanged (a hidden block becomes blank lines), so cited line numbers and exact-text edits still hold;
• works from the result text alone (no command, no file name), and only acts when it can show the text is source code; otherwise it returns the text untouched;
• comments that tools read stay (# type:, # noqa, # pragma, shebang and so on);
• a one-line note tells the agent grep -n shows a hidden line;
• runs before tool_output_cap, so the cap measures the pruned size;
• safety contract: the code-lexeme sequence must be identical before and after (for Python, the syntax tree minus docstrings), otherwise no change.

A Python proof of concept exists on branch feature/tool-output-pruner (new framework/pruning.py, framework/policies/tool_output_prune.py, 18 tests, a policy page). It is not pushed yet. A design document with the full methodology and the per-language design is in the same branch at docs/design/tool-output-pruning.md.

Like #190, the shaped text reaches the agent only if the adapter applies it (the Chronicle crossing hook cannot replace a result): the policy exposes it through Governor.shape_tool_result and ApplyControls.take_shaped_result(). If #190 lands a framework-level replacement, this policy should use it.

Evidence

Benchmark: mini-swe-agent 2.4.6 on SWE-bench Verified, azure/gpt-5-mini, the PASS_E25 slice (the 25 costliest tasks the reference run resolved), 2 repeats, 50 pairs.

Live A/B (prune on vs vanilla): inconclusive.

The agent is not reproducible (no seed or temperature; reasoning model, provider-default sampling). Two vanilla runs of the same task differ with an sd of 0.46 in log cost ratio (interval 0.31 to 0.58); the prune-vs-vanilla spread is 0.53 (0.38 to 0.68). Detecting a 10% saving would take about 200 pairs, a 5% saving about 830. The extra cost is concentrated in three runs, and the same instance moved in opposite directions in its two repeats.

Deterministic replay: the saving, with behaviour held fixed. The policy only changes tool-message text, so on a recorded vanilla trajectory its cost effect is a function of recorded data. The replay sends every recorded raw_output through the real governed environment and policy, renders it with the benchmark's own template, and re-prices each call from the recorded token counts (cache hit rate kept). No model call, no noise. Validated: 1,422 of 1,422 observations re-render identically, and the modelled vanilla cost is within 1.7% of the recorded one.

Reading: the replay is an upper bound if pruning changes behaviour, and the live run suggests it might (more calls and re-reads, not provable at this size). Both instruments are needed: replay for tokens and cost, a live run for behaviour.

Languages

The PoC is Python only, which is all SWE-bench Verified contains. The design doc covers the rest: per-language comment and doc-comment rules and what must stay (Go //go:build and cgo preambles, JS/TS @ts-ignore and JSDoc types, PHPDoc annotations read by frameworks, Rust nested block comments, C/C++ raw strings, Ruby and shell heredocs, SQL hints); language detection from text alone with refusal on ambiguity; and three engine options (hand-written lexers, shared C-style and hash-style lexers behind a LanguageHandler interface, optional tree-sitter extra). Recommendation: Python stays stdlib, add shared lexers for the C-style and hash-style families, tree-sitter only as an optional extra where a lexer cannot be correct (JS/TS). A language is enabled by default only after lexeme parity on real outputs, a replay on recorded trajectories in that language, and a live run that bounds the behavioural cost.

Open questions

1. Share the result-replacement mechanism with Tool-result substitution never reaches the agent: tool_output_cap and tool_fix decide, but nothing applies the result (Chronicle crossing hook is observe-only) #190?
2. Python-only default for the first release, adding languages one at a time?
3. Keep doctests visible?
4. tree-sitter as an optional extra: acceptable for packaging?
5. What evidence bar turns it on by default, given a live A/B at this noise level needs hundreds of pairs?

Next experiments

Replay on the prune-arm trajectories to measure the behavioural cost directly; a larger live run from a random slice of ALL; replay ablations (keep the first docstring line, keep TODO/FIXME comments).

Alternatives considered

Size cap only (no basis for what to cut); a decision model scoring chunks (a call per output, own noise, not tried); schema-based field projection for MCP outputs (a different problem, separate proposal); stop-word removal and whitespace squeezing (unsafe for code).