fix: agent must map metric semantics to specific code paths, not instrument the nearest convenient boundary #234

Problem

PR #237 instrumented OpenAICompatibleClient.generate() to capture "persona turn latency" p50/p95. However, generate() is called by multiple code paths:

1. Persona conversation turns (inside conversation_agent.run_conversation())
2. Stance evaluation (inside stance_evaluator.evaluate_stance_change())
3. Structured output parsing (inside structured_output.parse_structured_output())

The metric persona_turn_p50_ms therefore measures all LLM generate calls during the smoke, not persona turns specifically. A reader of the smoke output would believe they are seeing conversation latency when they are actually seeing a mixed distribution.

This is the same class of error as PR #232 (instrumenting the wrong boundary), but at a finer granularity. PR #232 instrumented the wrong module; PR #237 instrumented the wrong call site within the correct module.

Root Cause

1. Did not analyze what "persona turn" means in the codebase: The term "persona turn" in Suppress guardrail gitignore prompts in team workers #215 refers to one speaker response within run_conversation() — specifically one of the 2×turns messages per pair. The agent should have traced the code path: run_conversation() → inner loop → llm_client.generate() (2 calls per iteration) and measured only those calls.

2. Chose the easiest instrumentation point over the semantically correct one: generate() is the single point where all LLM calls converge, making it easy to hook. But ease of instrumentation does not make it the correct boundary for a specific metric.

3. No "metric-to-code-path" mapping: The agent did not ask "where in the code does a persona turn happen?" before deciding where to place the instrumentation.

4. Missing "total pair latency": Suppress guardrail gitignore prompts in team workers #215 explicitly asks for "total pair latency" (time from pair start to pair completion in _run_pair_conversation). The agent did not notice this was a separate metric from per-turn latency.

Prevention

For the agent

1. When a metric has a domain-specific name (e.g., "persona turn", "pair latency"), trace its meaning to the specific code path before choosing an instrumentation point. Ask: "In the codebase, what function call corresponds to one instance of this metric?"

2. Produce a metric-to-code-path mapping table as part of the plan:

| Metric | Code Path | Instrumentation Point |
|--------|-----------|----------------------|
| Persona turn latency | `conversation_agent.run_conversation()` inner loop, per `generate()` | Inside run_conversation, timing each generate() call |
| Total pair latency | `orchestrator._run_pair_conversation()` start to return | Timing wrapper around the full call |
| Stance eval latency | `stance_evaluator.evaluate_stance_change()` | Timing wrapper or separate hook |
| All LLM calls (aggregate) | `utils/llm_client.generate()` | Hook at this level, labelled "llm_call_*" not "persona_turn_*" |

3. If a single instrumentation point captures multiple semantic categories, label it with the broadest correct name. "All LLM calls" is honest; "persona turn" when it includes stance evaluation is not.

4. If a requested metric cannot be separated at the chosen instrumentation point, flag this explicitly in the plan and propose either: (a) a more specific instrumentation point, or (b) renaming the metric to match reality.

References

• PR fix: agent must not conflate separate metrics — enumerate all requested metrics and verify none are missing #237 review finding epic: internal AI guardrails thin distribution for Cor-Incorporated #1: https://github.com/Cor-Incorporated/persona-village-v2/pull/237#issuecomment-4525916917
• Issue Suppress guardrail gitignore prompts in team workers #215 AC: "Captures p50/p95 for persona turn latency and total pair latency"