Skip to content

fix(memory): handle non-numeric confidence in bootstrap --infer - #6656

Merged
kovtcharov-amd merged 4 commits into
amd:mainfrom
mangeshraut712:fix/memory-infer-confidence-6637
Nov 4, 2026
Merged

fix(memory): handle non-numeric confidence in bootstrap --infer#6656
kovtcharov-amd merged 4 commits into
amd:mainfrom
mangeshraut712:fix/memory-infer-confidence-6637

Conversation

@mangeshraut712

@mangeshraut712 mangeshraut712 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

gaia memory bootstrap --infer no longer crashes when the model returns a word confidence such as "high". Word confidences are mapped to numbers; unusable ones are skipped with a message that names the insight.

Why

_bootstrap_infer() did float(insight.get("confidence", 0.7)) with no guard, so one "high" ended the whole review with a traceback, even after earlier insights were approved. The Agent UI's /api/memory/stream-inference caught the error but silently substituted 0.7, which goes against the no-silent-fallbacks rule.

Linked issue

Fixes #6637

Changes

  • New normalize_confidence() in src/gaia/agents/base/inferred_insights.py, used by both paths:
    • numbers and numeric strings are clamped to 0–1 (as before)
    • low / medium / high (case-insensitive, whitespace stripped) map to 0.4 / 0.7 / 0.9
    • anything else ("very sure", null, lists, bools, NaN) returns None
    • a missing confidence still defaults to 0.7
  • CLI: an insight with an unusable confidence is skipped with a printed reason; review continues with the rest.
  • UI stream: the skip is sent as a log SSE event instead of a silent 0.7, and the done total counts only insights actually sent.

Test plan

  • pytest tests/unit/test_memory_bootstrap_infer.py tests/unit/test_memory_router.py (232 passed, 2 skipped)
    • helper: "high", "HIGH ", numeric strings, clamping, and rejection of "very sure", "", None, list, dict, bool, NaN
    • CLI end to end: a reply mixing "high", "very sure" and null stores the valid insights and prints the skips; missing confidence still shows 70%
    • router: "HIGH " maps to 0.9, "very sure" is skipped with a log event, total is 2
  • pytest tests/unit -k memory (2274 passed, 105 skipped)
  • python util/lint.py --black --isort --pylint --flake8: black/isort/flake8 pass; pylint reports only the two existing E1120 errors in src/gaia/utils/terminal.py and src/gaia/utils/awake_clock.py, which are also on main

@github-actions github-actions Bot added cli CLI changes tests Test changes agents labels Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Verdict: Approve

A model that answers "high" instead of a number no longer crashes gaia memory bootstrap --infer partway through a review. The Agent UI's inference stream also stops silently swapping in 70% for an unusable confidence. Both paths now share one helper: it maps low/medium/high to numbers and skips anything else with a message that names the insight. I found no blocking issues. The fix follows the no-silent-fallbacks rule, and the tests cover the CLI loop, the SSE route and the helper's edge cases.

Real-world evidence

N/A: no evidence bundle was produced for this run. To exercise the change for real, a live model has to return a word confidence, and this runner has no inference. The verdict rests on static review plus the PR's unit tests, which drive the CLI review loop and the /api/memory/stream-inference route with a mocked LLM reply. If you have a model handy, pasting one real gaia memory bootstrap --infer run showing a Skipped: confidence … line would close the gap. That's optional.

🔍 Technical details

Issues found

🟢 Stream done.total now means "sent", not "parsed" (src/gaia/ui/routers/memory.py:1643). This is correct, and the dashboard only displays the number (MemoryDashboard.tsx:1039). If every insight is skipped, though, the user sees "0 insights" with only the per-item log lines as the reason. That's acceptable, since the skip lines are visible in the same log panel. No change needed.

🟢 Siblings checked, none affected. MemoryStore._normalize_bulk_items (memory_store.py:4889) still does a bare float(item.get("confidence", 0.7)). It is only fed by commit_inference and the seed path, which receive values this PR has already normalised to floats, so it isn't reachable with a word confidence. cli.py:8667 and memory.py:2612 pass confidences through without parsing.

🟢 Commit history: fix(memory): use math.isnan for NaN check is a follow-up to the main fix. It will disappear on squash-merge.

Known bug classes: no new waits, subprocesses or shared-file writes. On model-output handling, the parser now tolerates the word/number variants models actually emit, and raw values reach the user only via !r in the skip messages.

Docs drift: no doc states the old 0.7 substitution or describes how confidences are parsed (I grepped docs/ and hub/), so there is nothing stale.

Strengths

  • One helper (normalize_confidence) replaces two diverging inline parsers. The CLI previously crashed and the UI quietly defaulted.
  • bool, NaN, None and containers are explicitly rejected and tested. True is no longer coerced into 100%.
  • The CLI test checks stored content and printed output together, so it would catch a regression in either.

@kovtcharov-amd
fix(memory): reject infinite confidence values too
27fef43 
A model can answer "Infinity" or a float overflow; clamping that to 1.0 would
store a full-confidence insight the model never meant. Treat any non-finite
number like NaN: skip the insight and say why.
@kovtcharov-amd
Merge remote-tracking branch 'origin/main' into review/pr6656
554f67f
@kovtcharov-amd
kovtcharov-amd added this pull request to the merge queue Nov 4, 2026
Merged via the queue into amd:main with commit 978f686 Nov 4, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kovtcharov-amd kovtcharov-amd Awaiting requested review from kovtcharov-amd kovtcharov-amd is a code owner

Assignees

No one assigned

Labels

agents cli CLI changes tests Test changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

gaia memory --infer crashes on a non-numeric confidence value

2 participants

@mangeshraut712 @kovtcharov-amd