Summary
Today reap scores load utilization (loaded vs fired). It counts whether a tool fires, but never looks at what comes back once it does. This proposes a second axis: payload-quality utilization — scoring tools that fire often but return mostly noise.
Credit: idea from @Sufficient_Roof_8240 in the r/mcp thread.
The gap
The current verdicts cover two cases:
- REAP — never fires → flagged
- MUTE — fires rarely + heavy → flagged
Neither catches a third case: a tool that fires a lot (so it reads as KEEP, healthy, green) but whose output is mostly junk every time.
Example: a fetch/read tool fires 50x/day → counts as high utilization → KEEP. But each result is mostly nav, footers, and the odd inline base64 blob instead of real content. reap marks it green; in reality it's a live-but-bloated tool burning context on every call.
Why MUTE doesn't cover it
MUTE is driven by firing rate (--mute-threshold, default 20%). It answers "is this used often enough?" The proposed metric answers a different question: "when it is used, is the payload actually useful?" Two orthogonal axes:
- Load utilization → does it fire?
- Payload utilization → when it fires, does it return signal or noise?
Why it's feasible
The data is already in the same JSONL transcripts reap parses — the tool result content sits right next to the tool_use block. No new data source needed.
Proposed behavior
- A payload-quality score per tool (e.g. ratio of useful content vs nav/boilerplate/base64 in returned results), surfaced alongside load utilization in
reap gap.
- A heuristic to classify result content as signal vs noise. Open question: a per-tool baseline may be needed, since "normal" payload shape varies by tool.
- Possibly a new verdict or reason suffix for "fires but mostly noise" (e.g. high firing rate + low payload quality).
Open questions
- What thresholds feel right? Signal/noise is fuzzier than a binary fired/not-fired.
- How to detect noise robustly across HTML pages, API JSON, file reads, etc.
- Keep it in
reap gap as a second column, or introduce a dedicated verdict?
Acceptance criteria
Summary
Today reap scores load utilization (loaded vs fired). It counts whether a tool fires, but never looks at what comes back once it does. This proposes a second axis: payload-quality utilization — scoring tools that fire often but return mostly noise.
Credit: idea from @Sufficient_Roof_8240 in the r/mcp thread.
The gap
The current verdicts cover two cases:
Neither catches a third case: a tool that fires a lot (so it reads as KEEP, healthy, green) but whose output is mostly junk every time.
Example: a
fetch/readtool fires 50x/day → counts as high utilization → KEEP. But each result is mostly nav, footers, and the odd inline base64 blob instead of real content. reap marks it green; in reality it's a live-but-bloated tool burning context on every call.Why MUTE doesn't cover it
MUTE is driven by firing rate (
--mute-threshold, default 20%). It answers "is this used often enough?" The proposed metric answers a different question: "when it is used, is the payload actually useful?" Two orthogonal axes:Why it's feasible
The data is already in the same JSONL transcripts reap parses — the tool result content sits right next to the
tool_useblock. No new data source needed.Proposed behavior
reap gap.Open questions
reap gapas a second column, or introduce a dedicated verdict?Acceptance criteria
reap gap(or a new view) reports a payload-quality metric per tool