Repository navigation
fix: agent must create a traceability matrix mapping each acceptance criterion to implementation before coding #233
Description
Problem
Across three iterations (PR #232, PR #237 rework) of the same task (#215 smoke metrics instrumentation), the implementation failed to satisfy acceptance criteria that were clearly written in the issue:
| Acceptance Criterion | PR #232 | PR #237 | Root cause |
|---|---|---|---|
| Token/sec or explicit unavailable reason | null silently |
null silently |
Did not read "explicit reason" requirement |
| Persona turn latency (not all LLM calls) | N/A (broken) | Measured all generate() calls including stance evaluation |
Did not map "persona turn" to its specific code path |
| Total pair latency | Missing | Missing | Did not notice this criterion at all |
| RAG/embedding latency or "not invoked" | Missing | Missing | Did not address this criterion |
| psutil RSS when available | Used ru_maxrss only |
Used ru_maxrss only |
Did not check if psutil was a dependency |
| Completed conversation count (separate from persisted) | Combined | Combined | Treated as same as persisted log count |
RSS sampler finally cleanup |
N/A | Missing | Applied finally to scope but not to sampler |
Every one of these failures shares the same root cause: the agent did not create a traceability matrix mapping each acceptance criterion to its implementation approach before writing code.
Root Cause
-
No criterion-by-criterion verification step: The agent read the issue, formed a general plan, and started coding. It did not produce a checklist like:
- AC1: completed conversation count → where? →
orchestrator/_run_pair_conversationreturn count - AC2: token/sec or reason → how? → hook extracts
usage; if absent, output"reason": "backend_omits_usage" - AC3: persona turn p50/p95 → where? → only inside
run_conversation()per-turn loop, NOT allgenerate()calls - ...
- AC1: completed conversation count → where? →
-
"Closes #N" written before verification: The PR description claimed
Closes #215without having verified each criterion was satisfied. -
Did not check dependency availability:
psutilwas listed in the acceptance criteria but not inpyproject.toml. The agent should have flagged this as a blocker or added the dependency.
Prevention
For the agent (workflow rule)
- Before writing any implementation code for an issue with acceptance criteria, produce a traceability matrix as part of the plan:
| # | Acceptance Criterion | Implementation Location | Approach | Dependency Check |
|---|---------------------|------------------------|----------|-----------------|
| 1 | ... | file.py:class.method | ... | psutil needed? check pyproject.toml |
-
Each acceptance criterion must have a corresponding test that would fail if the criterion is not met. If a criterion is intentionally deferred, the PR must NOT claim
Closes #N. -
After implementation, verify the matrix: Walk through each row and confirm the implementation actually delivers what the criterion asks for. Do not hand-wave ("close enough").
-
Do not write
Closes #Nin a PR description unless every single acceptance criterion has a passing verification. If some criteria are deferred, usePartially addresses #Nand create follow-up issues.
For the review checklist
- Traceability gate: If a PR claims to close an issue with acceptance criteria, the reviewer should verify that the PR description or a linked document contains a row-by-row traceability matrix.
References
- Issue Suppress guardrail gitignore prompts in team workers #215 acceptance criteria: https://github.com/Cor-Incorporated/persona-village-v2/issues/215
- PR fix: agent must not conflate separate metrics — enumerate all requested metrics and verify none are missing #237 review: https://github.com/Cor-Incorporated/persona-village-v2/pull/237#issuecomment-4525916917
- Previous iteration: fix: AI agent instrumentation must use source-level hooks, not monkey-patching of already-imported names #228
Activity