Skip to content

[bug] wiki_refine_draft sets every refined draft's confidence to 0.85, lifting it past the curator's approve threshold #578

Closed

Description

@cdeust

What happened

wiki_refine_draft (handler_refine, mcp_server/handlers/wiki_refine.py:300) overwrites the draft's confidence with a fixed 0.85 every time an LLM submits refined prose, and records the audit memo with the same 0.85 (line 315). No source backs the value: ADR-0467 preserves the file as it was during #514 and records no threshold choice.

The template synthesizer sets a draft's confidence to the mean confidence of its source claims, or 0.4 when no claim carries one (mcp_server/core/draft_synthesizer.py:271-277). The curator uses that number as a gate: below MIN_CONFIDENCE_APPROVE = 0.6 a draft gets a reason and can at best be held, below MIN_CONFIDENCE_HOLD = 0.4 it is rejected (mcp_server/core/draft_curator.py:15-16, 99-110, 138-141). wiki_curate then moves approved drafts toward wiki_compile.

So rewriting a draft's prose lifts it above the approve threshold whatever the evidence behind its claims. A draft synthesized at 0.4 from weak claims clears the confidence check of evaluate_draft once refined, although no claim changed.

Reproduction steps

  1. Run wiki_extract then wiki_synthesize on any store; pick a pending draft and note its confidence.
  2. Call wiki_refine_draft with that draft_id and a new lead.
  3. Read the draft again: confidence is 0.85, and evaluate_draft no longer reports "below approve threshold" for it.

Expected behavior

Refinement changes the wording of a draft and leaves its confidence as the source claims set it. The audit memo records the draft's confidence as it stands, not a constant.

Evidence required to close

A test on the SQLite backend that refines a draft whose confidence is below MIN_CONFIDENCE_APPROVE and asserts that its confidence is unchanged, that the refined_llm memo carries that confidence, and that evaluate_draft still reports the approve-threshold reason.

Activity

  1. cdeust commented on Sep 15, 2026

    @cdeust
    OwnerAuthor

    Correction on the reach of this defect, checked after opening the issue.

    wiki_get_draft and wiki_refine_draft are not registered as MCP tools: mcp_server/tool_registry_wiki.py registers ten wiki tools and neither of these, and git log -S wiki_refine over the tool_registry*.py files finds no commit that ever did. The only caller of wiki_refine in the tree is test_refine_lists_pending_drafts, which exercises handler_get.

    In the maintainer's PostgreSQL store (read-only query, 2026-09-16), wiki.memos holds a single refined_llm memo: id 105, 2026-04-14, synth_model test_runner_v1, confidence 0.85. The 546 drafts all carry synth_model template_v1.

    So the overwrite is latent: no draft reaches it through the MCP surface today. The fix still stands, because the handler is the documented Path B and whoever wires it would inherit the 0.85.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions