Skip to content

Commit 362631d

Browse files
committed
fix: make converge coverage auditable
Require artifact-derived per-category coverage, expose stable inventory keys, and fail closed before appending tasks when assessment is incomplete. Closes github#3752 Assisted-by: Hermes Agent (model: gpt-5.6-terra, autonomous)
1 parent c0fe0e4 commit 362631d

3 files changed

Lines changed: 143 additions & 21 deletions

File tree

‎docs/reference/agentic-sdd.md‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,18 @@ Assesses the codebase against the feature's spec, plan, and tasks to confirm not
109109
/speckit.converge
110110
```
111111

112-
It first prints a severity-graded findings summary, then resolves to one of two outcomes:
113-
114-
- **Converged** — no gaps found. `tasks.md` is left byte-for-byte unchanged and you'll see a clean result like `✅ Converged — the implementation satisfies the spec, plan, and tasks.` You're done; proceed to review or open a PR.
115-
- **Tasks appended** — gaps found. Converge appends them as new tasks under a Convergence section in `tasks.md` and tells you how many. Run `/speckit.implement` again to complete them, then `/speckit.converge` once more. Each pass finds fewer items; repeat until it reports converged.
112+
It first prints a severity-graded findings summary, including per-category `checked/total`
113+
coverage and a compact stable-key inventory ledger. Counts are derived from the feature artifacts,
114+
not model memory. It then resolves to one of three outcomes:
115+
116+
- **Incomplete assessment** — one or more inventory keys could not be assessed. The report names
117+
those keys and their incomplete coverage category, does not modify `tasks.md`, and is **not** a
118+
clean result. Resolve the missing evidence and run `/speckit.converge` again.
119+
- **Converged** — every applicable inventory category is fully assessed and no gaps were found.
120+
`tasks.md` is left byte-for-byte unchanged and you'll see a clean result like
121+
`✅ Converged — the implementation satisfies the spec, plan, and tasks.` You're done; proceed
122+
to review or open a PR.
123+
- **Tasks appended** — every applicable inventory category is fully assessed and gaps were found.
124+
Converge appends them as new tasks under a Convergence section in `tasks.md` and tells you how
125+
many. Run `/speckit.implement` again to complete them, then `/speckit.converge` once more.
126+
Each pass finds fewer items; repeat until it reports converged.

‎templates/commands/converge.md‎

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ You **MUST** consider the user input before proceeding (if not empty).
5050
5151
Wait for the result of the hook command before proceeding to the Goal.
5252
```
53+
5354
After emitting the block above you MUST actually invoke the hook and wait for it to finish before continuing. Run it the same way you would run the command yourself in this agent/session (the invocation may differ from the literal `{command}` id shown above, e.g. a skills-mode agent runs it as `/skill:speckit-...` or `$speckit-...`). Emitting the block alone does not run the hook.
5455
5556
- If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently
@@ -132,15 +133,30 @@ Load only the minimal necessary context from each artifact:
132133
133134
### 3. Build the Intent Inventory
134135
135-
Create an internal model (do not echo raw artifacts):
136-
137-
- **Requirements inventory**: one stable key per FR-### / SC-### / user-story acceptance
138-
scenario (e.g. `US1/AC2`), plus the plan decisions and constitution principles that
139-
impose buildable obligations.
136+
Create an internal model (do not echo raw artifact prose). The report **MUST** emit the
137+
stable keys and statuses from this model in the compact inventory ledger required by Step 6:
138+
139+
- **Requirements inventory**: one stable key per buildable `FR-###`, buildable `SC-###`, and
140+
user-story acceptance scenario (e.g. `US1/AC2`), plus the plan decisions and constitution
141+
principles that impose buildable obligations.
142+
- **Per-key status**: initialize every applicable key as `unassessed`. End the assessment with
143+
exactly one status per key: `checked:no-gap`, `checked:gap(F#)`, or `unassessed` when the
144+
evidence could not be evaluated. A user story is **not** assessed on behalf of its individual
145+
acceptance scenarios; each `USn/ACn` key is counted independently. When a user story or
146+
scenario has no explicit ID, derive `US<story-position>/AC<scenario-position>` from its order
147+
under the User Stories section; do not collapse it into a prose-only row.
148+
- **Plan and constitution keys**: use an explicit label from the artifact when present (for
149+
example `plan: D1` or `Constitution II`). Otherwise, derive a stable ordinal key from its
150+
artifact order (`plan: decision-01`, `Constitution principle-01`).
151+
- **Coverage denominators**: count applicable keys from the loaded artifacts before assessing
152+
code. Keep separate totals for buildable FRs, buildable SCs, acceptance scenarios, plan
153+
decisions, and applicable Constitution MUST principles. A category with no applicable keys is
154+
reported as `0/0 (not applicable)` and does not block convergence. Do not derive, estimate, or
155+
recall a denominator from model memory.
140156
- **Code-scope map**: from the file paths named in `plan.md` and `tasks.md`, plus a keyword
141157
search for the concepts each requirement describes, derive the set of source files and
142-
components in scope for assessment. Bound the assessment to these — do **not** infer
143-
scope beyond what the artifacts define.
158+
components in scope for assessment. Bound the assessment to these — do **not** infer scope
159+
beyond what the artifacts define.
144160
145161
### 4. Assess the Codebase and Classify Findings
146162
@@ -185,17 +201,42 @@ Before appending anything, output a compact, severity-graded summary (no file wr
185201
|----|----------|----------|--------|----------|----------------|
186202
| F1 | missing | HIGH | FR-008 | Example: no append-only guard detected in path/to/module.py when writing tasks.md | Add append-only enforcement |
187203
188-
**Summary metrics:**
204+
**Coverage metrics (all denominators are derived from the loaded artifacts):**
189205
190-
- Requirements / acceptance criteria checked
191-
- Plan decisions checked
192-
- Constitution principles checked (or "skipped — template")
206+
- Buildable FRs checked: `<checked>/<total>`
207+
- Buildable SCs checked: `<checked>/<total>`
208+
- Acceptance scenarios checked: `<checked>/<total>`
209+
- Plan decisions checked: `<checked>/<total>`
210+
- Constitution MUST principles checked: `<checked>/<total>` (or `skipped — template`)
193211
- Findings by gap type (missing / partial / contradicts / unrequested)
194212
- Findings by severity
195213
196-
### 7. Append Convergence Tasks (or report converged)
214+
**Intent inventory ledger:** emit every applicable stable key in compact category-grouped lines;
215+
do not replace acceptance-scenario keys with a user-story total. Use `✓` for `checked:no-gap`,
216+
`F#` for `checked:gap(F#)`, and `unassessed` for incomplete evidence. For example:
217+
218+
```text
219+
FR: FR-001 ✓, FR-002 F1
220+
SC: SC-001 ✓
221+
Acceptance scenarios: US1/AC1 ✓, US1/AC2 F2, US3/AC4 unassessed
222+
Plan: plan: storage decision ✓
223+
Constitution: Constitution II ✓
224+
```
225+
226+
### 7. Resolve the Convergence Outcome
227+
228+
Evaluate outcomes in this exact order:
229+
230+
**If any applicable key is `unassessed`, or any coverage numerator is below its denominator**
231+
(`incomplete_assessment` outcome):
197232

198-
**If there are one or more actionable findings** (`tasks_appended` outcome):
233+
- Do **not** modify `tasks.md` at all — no Convergence phase, even if findings were already
234+
discovered for other keys.
235+
- Report the incomplete category counts and each `unassessed` key in the inventory ledger.
236+
- State that the result is not converged and recommend resolving the missing evidence, then
237+
rerunning `__SPECKIT_COMMAND_CONVERGE__`.
238+
239+
**Otherwise, if there are one or more actionable findings** (`tasks_appended` outcome):
199240

200241
Append to the **end** of `tasks.md`, per the append contract:
201242

@@ -219,14 +260,18 @@ Append to the **end** of `tasks.md`, per the append contract:
219260
4. Never reuse or renumber existing IDs. If a prior Convergence phase exists, add a new,
220261
separately-numbered one below it — do not touch the old one.
221262

222-
**If there are no actionable findings** (`converged` outcome):
263+
**Otherwise** (`converged` outcome):
223264

265+
- All applicable coverage metrics are `total/total` and there are zero actionable findings.
224266
- Do **not** modify `tasks.md` at all — no empty phase header.
225267
- Report: **"✅ Converged — the implementation satisfies the spec, plan, and tasks."**
226-
- Include the summary counts of what was checked.
268+
- Include the coverage metrics and inventory ledger.
227269

228270
### 8. Provide Next Actions (Handoff)
229271

272+
- On `incomplete_assessment`: state which keys need assessment and recommend resolving the
273+
missing evidence, then rerunning `__SPECKIT_COMMAND_CONVERGE__`. Do not recommend
274+
`__SPECKIT_COMMAND_IMPLEMENT__` from a partial assessment.
230275
- On `tasks_appended`: state how many tasks were appended under which phase, and recommend
231276
running `__SPECKIT_COMMAND_IMPLEMENT__` to complete them; note that a follow-up converge
232277
run will find fewer or no remaining items.
@@ -243,8 +288,8 @@ After producing the result, check if `.specify/extensions.yml` exists in the pro
243288
- For each remaining hook, do **not** attempt to interpret or evaluate hook `condition` expressions:
244289
- If the hook has no `condition` field, or it is null/empty, treat the hook as executable
245290
- If the hook defines a non-empty `condition`, skip the hook and leave condition evaluation to the HookExecutor implementation
246-
- Report the convergence outcome (`converged` or `tasks_appended`) in-session before listing
247-
any hooks, so users can decide whether to run optional follow-up commands.
291+
- Report the convergence outcome (`incomplete_assessment`, `converged`, or `tasks_appended`) in-session
292+
before listing any hooks, so users can decide whether to run optional follow-up commands.
248293
- For each executable hook, output the following based on its `optional` flag:
249294
- **Optional hook** (`optional: true`):
250295

@@ -268,6 +313,7 @@ After producing the result, check if `.specify/extensions.yml` exists in the pro
268313
Executing: `/{command}`
269314
EXECUTE_COMMAND: {command}
270315
```
316+
271317
After emitting the block above you MUST actually invoke the hook and wait for it to finish before continuing. Run it the same way you would run the command yourself in this agent/session (the invocation may differ from the literal `{command}` id shown above, e.g. a skills-mode agent runs it as `/skill:speckit-...` or `$speckit-...`). Emitting the block alone does not run the hook.
272318
273319
- If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
"""Regression contract for auditable ``/speckit.converge`` outcomes (#3752)."""
2+
3+
from pathlib import Path
4+
5+
6+
REPO_ROOT = Path(__file__).parent.parent
7+
CONVERGE_TEMPLATE = REPO_ROOT / "templates" / "commands" / "converge.md"
8+
9+
10+
def _template() -> str:
11+
return CONVERGE_TEMPLATE.read_text(encoding="utf-8")
12+
13+
14+
def test_converge_counts_each_acceptance_scenario_independently():
15+
content = _template()
16+
17+
assert "user-story acceptance scenario (e.g. `US1/AC2`)" in content
18+
assert "each `USn/ACn` key is counted independently" in content
19+
assert "A user story is **not** assessed on behalf" in content
20+
assert "derive `US<story-position>/AC<scenario-position>`" in content
21+
assert "derive a stable ordinal key from its" in content
22+
23+
24+
def test_converge_reports_artifact_derived_coverage_denominators():
25+
content = _template()
26+
27+
for metric in (
28+
"Buildable FRs checked: `<checked>/<total>`",
29+
"Buildable SCs checked: `<checked>/<total>`",
30+
"Acceptance scenarios checked: `<checked>/<total>`",
31+
"Plan decisions checked: `<checked>/<total>`",
32+
"Constitution MUST principles checked: `<checked>/<total>`",
33+
):
34+
assert metric in content
35+
36+
assert "does not block convergence. Do not derive" in content
37+
assert "denominator from model memory." in content
38+
39+
40+
def test_converge_emits_a_stable_key_inventory_ledger():
41+
content = _template()
42+
43+
assert "**Intent inventory ledger:** emit every applicable stable key" in content
44+
assert "US1/AC1 ✓, US1/AC2 F2, US3/AC4 unassessed" in content
45+
assert "do not replace acceptance-scenario keys with a user-story total" in content
46+
47+
48+
def test_converged_is_blocked_by_incomplete_assessment():
49+
content = _template()
50+
51+
incomplete = content.index("`incomplete_assessment` outcome")
52+
tasks_appended = content.index("`tasks_appended` outcome")
53+
converged = content.index("`converged` outcome")
54+
55+
assert incomplete < tasks_appended < converged
56+
assert "any coverage numerator is below its denominator" in content
57+
assert "Do **not** modify `tasks.md` at all — no Convergence phase" in content
58+
assert "All applicable coverage metrics are `total/total`" in content
59+
60+
61+
def test_converge_documents_all_outcomes_for_handoff_and_hooks():
62+
content = _template()
63+
64+
assert "- On `incomplete_assessment`:" in content
65+
assert "(`incomplete_assessment`, `converged`, or `tasks_appended`)" in content

0 commit comments

Comments
 (0)