Skip to content

Commit 8a4ca75

Browse files
Fix converge-coverage-denominators: add per-category denominators, key ledger, and incomplete_assessment outcome
Apply the remediation from the bug assessment on issue #3752. - Replace undenominated Step 6 summary with per-category checked/total metrics - Add coverage ledger table (one row per inventory key with status) - Add incomplete_assessment fail-closed outcome in Step 7 - Update Step 8 handoff and Step 9 outcome list - Add regression test pinning the contract Refs #3752 Assisted-by: GitHub Copilot (model: claude-sonnet-4.6, autonomous) Co-authored-by: Copilot <[email protected]>
1 parent 5e2f9bc commit 8a4ca75

2 files changed

Lines changed: 108 additions & 9 deletions

File tree

‎templates/commands/converge.md‎

Lines changed: 34 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -185,17 +185,39 @@ Before appending anything, output a compact, severity-graded summary (no file wr
185185
|----|----------|----------|--------|----------|----------------|
186186
| F1 | missing | HIGH | FR-008 | Example: no append-only guard detected in path/to/module.py when writing tasks.md | Add append-only enforcement |
187187
188-
**Summary metrics:**
188+
**Summary metrics** (all denominators derived from the inventory built in Step 3, not from model memory):
189189
190-
- Requirements / acceptance criteria checked
191-
- Plan decisions checked
192-
- Constitution principles checked (or "skipped — template")
190+
- Functional Requirements (FR-###): `<checked>/<total>` checked
191+
- Success Criteria (SC-###): `<checked>/<total>` checked
192+
- User-story acceptance scenarios (US#/AC#): `<checked>/<total>` checked
193+
- Plan decisions: `<checked>/<total>` checked
194+
- Constitution principles: `<checked>/<total>` checked (or "skipped — template")
193195
- Findings by gap type (missing / partial / contradicts / unrequested)
194196
- Findings by severity
195197
196-
### 7. Append Convergence Tasks (or report converged)
198+
**Coverage ledger** (one row per inventory key — include every key from Step 3):
197199
198-
**If there are one or more actionable findings** (`tasks_appended` outcome):
200+
| Key | Status | Finding |
201+
|-----|--------|---------|
202+
| FR-001 | ✅ satisfied | — |
203+
| SC-002 | ⚠️ gap | → F1 |
204+
| US1/AC2 | — unassessed | — |
205+
206+
Status values: `✅ satisfied`, `⚠️ gap → Finding Fxx`, `— unassessed`.
207+
208+
> **Fail-closed rule**: if any inventory key carries `— unassessed` status, the command
209+
> MUST use the `incomplete_assessment` outcome (Step 7) and MUST NOT emit `converged` or
210+
> append tasks.
211+
212+
### 7. Append Convergence Tasks (or report incomplete / converged)
213+
214+
**If any inventory key is unassessed** (`incomplete_assessment` outcome):
215+
216+
- Do **not** modify `tasks.md` at all.
217+
- Report: **"⚠️ Incomplete assessment — N inventory key(s) were not assessed. Re-run `/speckit.converge` after addressing the coverage gaps, or verify that the spec artifacts contain all required identifiers."**
218+
- List the unassessed keys by category so the operator knows exactly what was skipped.
219+
220+
**If there are one or more actionable findings and all inventory keys are assessed** (`tasks_appended` outcome):
199221
200222
Append to the **end** of `tasks.md`, per the append contract:
201223
@@ -219,14 +241,17 @@ Append to the **end** of `tasks.md`, per the append contract:
219241
4. Never reuse or renumber existing IDs. If a prior Convergence phase exists, add a new,
220242
separately-numbered one below it — do not touch the old one.
221243

222-
**If there are no actionable findings** (`converged` outcome):
244+
**If there are no actionable findings and all inventory keys are assessed** (`converged` outcome):
223245

224246
- Do **not** modify `tasks.md` at all — no empty phase header.
225247
- Report: **"✅ Converged — the implementation satisfies the spec, plan, and tasks."**
226-
- Include the summary counts of what was checked.
248+
- Include the per-category `checked/total` summary counts.
227249

228250
### 8. Provide Next Actions (Handoff)
229251

252+
- On `incomplete_assessment`: explain that N key(s) were not assessed (list them), and
253+
instruct the operator to re-run `/speckit.converge` after ensuring the agent reads all
254+
relevant source files or after resolving any ambiguities in the spec artifacts.
230255
- On `tasks_appended`: state how many tasks were appended under which phase, and recommend
231256
running `__SPECKIT_COMMAND_IMPLEMENT__` to complete them; note that a follow-up converge
232257
run will find fewer or no remaining items.
@@ -243,7 +268,7 @@ After producing the result, check if `.specify/extensions.yml` exists in the pro
243268
- For each remaining hook, do **not** attempt to interpret or evaluate hook `condition` expressions:
244269
- If the hook has no `condition` field, or it is null/empty, treat the hook as executable
245270
- If the hook defines a non-empty `condition`, skip the hook and leave condition evaluation to the HookExecutor implementation
246-
- Report the convergence outcome (`converged` or `tasks_appended`) in-session before listing
271+
- Report the convergence outcome (`converged`, `tasks_appended`, or `incomplete_assessment`) in-session before listing
247272
any hooks, so users can decide whether to run optional follow-up commands.
248273
- For each executable hook, output the following based on its `optional` flag:
249274
- **Optional hook** (`optional: true`):
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
"""Regression test for the converge.md template output contract.
2+
3+
Pins the three requirements introduced in issue #3752:
4+
1. Step 6 summary metrics use per-category denominator syntax (checked/total).
5+
2. Step 6 includes a coverage ledger table with key-level status.
6+
3. Step 7 defines an ``incomplete_assessment`` fail-closed outcome.
7+
"""
8+
9+
import re
10+
from pathlib import Path
11+
12+
REPO_ROOT = Path(__file__).parent.parent
13+
CONVERGE_TEMPLATE = REPO_ROOT / "templates" / "commands" / "converge.md"
14+
15+
16+
def _text() -> str:
17+
return CONVERGE_TEMPLATE.read_text(encoding="utf-8")
18+
19+
20+
def test_per_category_denominator_syntax_present():
21+
"""Step 6 must expose checked/total per category, not a bare aggregate."""
22+
text = _text()
23+
# The template must include the denominator pattern for each major category.
24+
assert "checked>/<total>" in text or re.search(r"checked>/<total>", text), (
25+
"converge.md Step 6 must use '<checked>/<total>' denominator syntax "
26+
"for per-category metrics."
27+
)
28+
29+
30+
def test_coverage_ledger_table_present():
31+
"""Step 6 must include a key-ledger table so reports are auditable."""
32+
text = _text()
33+
assert "Coverage ledger" in text, (
34+
"converge.md must contain a 'Coverage ledger' section in Step 6."
35+
)
36+
# The ledger table must define the three status values.
37+
assert "✅ satisfied" in text
38+
assert "⚠️ gap" in text
39+
assert "unassessed" in text
40+
41+
42+
def test_incomplete_assessment_outcome_present():
43+
"""Step 7 must define an ``incomplete_assessment`` fail-closed outcome."""
44+
text = _text()
45+
assert "incomplete_assessment" in text, (
46+
"converge.md Step 7 must define an 'incomplete_assessment' outcome "
47+
"to fail closed when inventory keys are unassessed."
48+
)
49+
50+
51+
def test_incomplete_assessment_blocks_tasks_md_write():
52+
"""The incomplete_assessment branch must NOT write tasks.md."""
53+
text = _text()
54+
# Locate the incomplete_assessment section and verify the prohibition.
55+
idx = text.find("incomplete_assessment` outcome")
56+
assert idx != -1
57+
# Within the next 500 chars after the outcome heading, the template must
58+
# say "Do not modify" tasks.md.
59+
excerpt = text[idx : idx + 500]
60+
assert "Do **not** modify" in excerpt or "do not modify" in excerpt.lower(), (
61+
"The incomplete_assessment branch must instruct the agent NOT to modify tasks.md."
62+
)
63+
64+
65+
def test_step_8_handoff_covers_incomplete_assessment():
66+
"""Step 8 must provide a next-action note for incomplete_assessment."""
67+
text = _text()
68+
# Step 8 handoff section must reference incomplete_assessment explicitly.
69+
step8_idx = text.find("### 8.")
70+
assert step8_idx != -1
71+
step8_text = text[step8_idx : step8_idx + 600]
72+
assert "incomplete_assessment" in step8_text, (
73+
"Step 8 handoff must include guidance for the incomplete_assessment outcome."
74+
)

0 commit comments

Comments
 (0)