Repository navigation
[Bug]: /speckit.converge can declare convergence without auditable acceptance-scenario coverage #3752
Description
Bug Description
/speckit.converge already requires one stable intent-inventory key per FR-###, buildable SC-###, and user-story acceptance scenario such as US1/AC2. However, its output contract exposes only a single undenominated line, Requirements / acceptance criteria checked.
An agent can therefore assess user stories in bulk, report a plausible aggregate (for example, 16 FR + 12 SC + 4 user stories), and still emit converged without exposing that only 4 user-story rows were checked instead of the 16 acceptance scenarios required by the inventory rule.
This is an observability/closure-contract defect rather than a missing inventory instruction: the prompt tells the agent what to assess, but the report cannot demonstrate whether it did so.
Steps to Reproduce
- Run
/speckit.convergeafter an implementation pass on a feature whosespec.mdhas multiple user-story acceptance scenarios. - Have the assessment aggregate user stories instead of evaluating each scenario key.
- Observe that the report can satisfy the current summary-metrics text while omitting the number of scenarios assessed.
- In the observed case, a second pass changed only the granularity from user stories to individual acceptance scenarios and surfaced a missing HIGH finding at
US3/AC4; a third pass at the same granularity found no remaining gaps.
Expected Behavior
The convergence report should make completion independently auditable:
- report
checked/totalseparately for buildable FRs, buildable SCs, user-story acceptance scenarios, plan decisions, and applicable Constitution MUST principles; - derive each denominator from the artifacts loaded during the run, not from model memory;
- expose the stable inventory keys with a compact checked/gap/unassessed status;
- allow
convergedonly when every applicable category is fully assessed and no findings remain; - fail closed with a distinct
incomplete_assessmentresult when any inventory key is unassessed, without appending a partial Convergence phase totasks.md.
Actual Behavior
The current template requires the fine-grained inventory internally but only asks for aggregate summary labels. As a result, a report can look complete while concealing a mismatch such as 4 assessed user stories versus 16 required acceptance scenarios. A false converged would end the implementation/convergence loop with an unassessed HIGH requirement.
Specify CLI Version
0.14.3.dev0 (main at time of report)
AI Agent
Not applicable — this is a core command-template contract affecting all agent integrations.
Operating System
Linux 5.15.147-sun60iw2
Python Version
Python 3.13.5
Error Logs
No process error: the defect is a false-completeness reporting path.
Additional Context
Proposed focused change:
- Update
templates/commands/converge.mdwith denominated per-category coverage, an auditable key ledger, and the fail-closed outcome precedenceincomplete_assessment→tasks_appended→converged. - Document the third outcome in the Agentic SDD reference.
- Add a deterministic template-contract regression test plus manual command validation.
Searches of existing upstream Issues/PRs found no duplicate for converge coverage denominators or incomplete assessment.
AI disclosure: This report was prepared by Hermes Agent (OpenAI gpt-5.6-terra) on behalf of @cadugevaerd, using Carlos's observed three-pass evidence and current upstream source inspection.
Activity
fix: make converge coverage auditable
cadugevaerd commentedon Jul 26, 2026
Implementação proposta em #3754: adiciona métricas
checked/totalderivadas dos artefatos, ledger de chaves estáveis e o desfecho fail-closedincomplete_assessmentantes detasks_appended/converged.github-actions commentedon Jul 29, 2026
Bug assessment — converge-coverage-denominators: Valid · severity high
Bug Assessment:
/speckit.convergecan declare convergence without auditable acceptance-scenario coverageconverge-coverage-denominators/speckit.convergecan declare convergence without auditable acceptance-scenario coverage #3752Report (summarized)
Reported by @cadugevaerd (via Hermes Agent / gpt-5.6-terra). The
/speckit.convergetemplate requires the agent to build an internal inventory with one stable key perFR-###,SC-###, andUS#/AC#acceptance scenario, but its Step 6 "Summary metrics" output block exposes only a single undenominated line — "Requirements / acceptance criteria checked" — with nochecked/totalper category. This allows an agent to aggregate across categories (e.g. reporting "16 FR + 12 SC + 4 user stories") and still emitconvergedeven though only 4 user-story rows were checked instead of the 16 individual acceptance scenarios required by the inventory rule. The author observed a three-pass convergence cycle where a second pass at finer granularity surfaced a missing HIGH finding atUS3/AC4that the first pass had hidden.No linked gist, log, or external URL to fetch.
Symptom
An agent running
/speckit.convergecan produce a summary that appears complete but hides a mismatch between the number of acceptance scenarios actually evaluated and the number required by the intent inventory built in Step 3. The current template has no fail-closed outcome for partial assessment, so a falseconvergedcan end the implementation loop while HIGH requirements remain unassessed.Reproduction
spec.mdcontains multiple user stories, each with multiple acceptance scenarios (e.g. 3 user stories × ~5 AC each = ~15US#/AC#keys)./speckit.convergeafter an implementation pass.convergedto be emitted because no denominator constraint is enforced in the output contract.spec.mdstructure that triggered the three-pass observation — provided as narrative but no artifact attached]Suspected Code Paths
templates/commands/converge.md:188–193— The "Summary metrics" bullet list in Step 6 is the direct site of the defect. The line "Requirements / acceptance criteria checked" is undenominated and does not requirechecked/totalbreakdowns per category (FR, SC, US/AC).templates/commands/converge.md:133–143— Step 3 "Build the Intent Inventory" correctly requires per-key inventory (US1/AC2-style keys), but there is no link back to the Step 6 output contract that enforces those keys are surfaced/counted in the report.templates/commands/converge.md:197–225— Step 7 defines only two outcomes:tasks_appendedandconverged. There is noincomplete_assessmentoutcome to fail closed when the agent has not assessed all inventory keys.Root Cause Hypothesis
The template separates assessment logic (Step 3–5, which correctly mandates per-key inventory) from output contract (Step 6, which only requests narrative aggregates). Because LLM agents interpret the Step 6 output schema as the authoritative report format, they can satisfy it by collapsing per-key results into category-level prose without violating any explicit instruction. The absence of a third outcome (
incomplete_assessment) means the only gate forconvergedis zero actionable findings — not full inventory coverage. Confidence: high — the template text directly confirms the mismatch between Steps 3 and 6.Proposed Remediation
Preferred: Update Step 6 of
templates/commands/converge.mdto replace the single "Requirements / acceptance criteria checked" line with per-categorychecked/totalmetrics derived explicitly from the inventory constructed in Step 3. Add a compact key-ledger table (one row per inventory key with status:✅ satisfied,⚠️ gap → Finding Fxx, or— unassessed) so the report is independently auditable. Add a third outcomeincomplete_assessment(between the existingtasks_appendedandconvergedbranches) in Step 7: if any inventory key carriesunassessedstatus, the command MUST halt withincomplete_assessmentand must NOT append a Convergence phase totasks.md. The author's proposed change in Additional Context aligns with this approach.Alternatives:
checked/totaldenominator per category in Step 6 without a full key ledger. This reduces verbosity but leaves gaps harder to audit (no key-level trace). Trade-off: lower agent overhead, but a sufficiently confused model could still under-count.partial_assessmentoutcome that appends tasks but flags the incomplete coverage. Trade-off: avoids hard stops but does not preventconvergedfrom being declared after a follow-up pass that also under-counts.Files likely to change:
templates/commands/converge.md— Step 6 summary metrics block and Step 7 outcome branchesTests to add or update:
converge.mdtemplate and asserts the presence of denominator syntax (e.g.checked/total) in the Step 6 output schema and theincomplete_assessmentoutcome in Step 7.Risks & Considerations
incomplete_assessmentresult and surface gaps they previously hid — this is the intended behavior.tasks.mdfiles with prior Convergence phases are unaffected — the change is to the prompt template, not totasks.mdformat.Open Questions
spec.mdstructure (number of user stories and AC) needed to reliably reproduce the aggregation collapse behavior?]converged(for auditability), or only when gaps exist?]Fix converge-coverage-denominators: add per-category denominators, ke…
github-actions commentedon Jul 30, 2026
Pull request created: #3875
github-actions commentedon Jul 30, 2026
A draft PR has been opened applying the
converge-coverage-denominatorsremediation from the bug assessment.What changed in
templates/commands/converge.md:checked/totaldenominators (FR, SC, US/AC, plan decisions, constitution) and a compact key-ledger table with✅ satisfied/⚠️ gap → Fxx/— unassessedstatus per inventory key.incomplete_assessmentfail-closed outcome: if any key is unassessed, the command halts without writingtasks.md— preventing a falseconverged.tests/test_converge_template_contract.pypins the contract so this cannot silently regress.Next: review the draft PR, then (if this repository has the
bug-testworkflow configured) apply thebug-testlabel to trigger the test-validation stage.Posted on behalf of
@BenBtgby GitHub Copilot (model: claude-sonnet-4.6, autonomous).github-actions commentedon Sep 29, 2026
This issue has been automatically marked as stale because it has had no activity for 60 days. It will be closed in 30 days unless there is a substantive update. If the problem still applies, please provide updated context such as whether it is still reproducible, the affected version or environment, its current impact, and any new reproduction details. A comment that only states that the issue is "still relevant" does not give maintainers enough information to reconsider it.
alfredoperez commentedon Nov 9, 2026
A worked example in support of this. In SpecKit Companion we keep a small map from each requirement (FR-001...) to the tasks that deliver it and the tests that check it. It only works because our task template asks each task to name its requirement in a fixed way.
If the stock spec and task templates showed one format for that (for example, each task line ending with the requirement IDs it covers), converge and analyze could trace coverage instead of guessing, and the gap in #3752 closes. Happy to share our template wording and the map format as a starting point.
AI disclosure: drafted with Claude Code (Claude Opus 5.5, default settings); I checked it before posting.