fix: agent must not claim measurable metrics when implementation cannot observe them #230

Problem

PR #232 documentation (run-log template, execution procedure, PR description) listed tokens_per_sec as a captured metric. However, OpenAICompatibleClient.generate() returns only str — the usage field from the LLM response JSON is parsed inside generate() and then discarded. The external wrapper InstrumentedLLMClient cannot recover token counts that were already thrown away.

The result: every smoke run would report tokens_per_sec: null and total_completion_tokens: null, while the docs implied these were measured metrics.

Root Cause

1. Did not trace the data flow end-to-end: The agent saw that the LLM response includes usage and assumed it was available for capture. It did not verify that generate() actually returns or exposes this data.
2. Documented intent, not reality: The run-log template listed metrics that the implementation could not deliver. The "record as null if unavailable" fallback masked the fact that the data was structurally impossible to obtain.
3. Silent failure mode: The code recorded None for token counts without any warning or fail-fast check. A reader of the smoke output would see "not measured" and assume it was a transient issue, not a structural impossibility.

Prevention

For the agent

1. Before adding a metric to docs/templates, trace the data path from source to collection point. If the data does not flow through the instrumented boundary, either:
   • Add a minimal hook to expose it (preferred), or
   • Explicitly mark it as "structurally unavailable" in the docs and do not list it as a captured metric.
2. Fail fast on impossible metrics. If the instrumentation code is designed to capture X but X cannot flow to the collector, raise an error or log a prominent warning. Do not silently return None.
3. Distinguish "not measured this run" from "cannot be measured by this instrumentation." Use different sentinel values: null for "available but not observed" vs. "unavailable" for "structurally impossible."

For the review checklist

4. Metric traceability: Every metric in the run-log template must have a corresponding code path that can produce a non-null value. If the code path cannot produce the value, the template must not list it as a standard metric.
5. Fail-fast for zero-capture: If an instrumentation PR completes a non-trivial run with zero captured data points for a claimed metric, it must fail (not just log null).

References

• PR fix: integration test mandatory for instrumentation and cross-cutting PRs #232 review H3: https://github.com/Cor-Incorporated/persona-village-v2/pull/232#issuecomment-4525827375