fix(memory): keep a zero confidence from scoring as the default - #5586

Fixes #5585

Why

A confidence of 0.0 is a value the storage layer validates, persists and considers legal
(0 <= confidence <= 1, deermem/core/storage.py:211-214), but two read sites rescued their
default with or 0.5, which cannot tell "explicitly zero" from "unset"
(deermem/core/retrieval.py:585, deermem/core/storage.py:1738). Reproduced as #5585:
BM25 × time_decay + confidence × 0.2 therefore added the default weight to a zero-confidence
fact, and the same stored fact scored 0.100001 when indexed through FTS5RetrievalAdapter.upsert
against 1e-06 when the identical row was indexed through FTS5Retrieval.index_fact — whose own
defaulting (fact.get("confidence", 0.5), retrieval.py:167 and :296) distinguishes the two
cases correctly. So the ranking of a discounted memory depended on which code path happened to
write its row, and the returned payload still reported "confidence": 0.0
(fact.setdefault at retrieval.py:459 never overwrites fact_json), which is why the inflation
is invisible to the caller. _compute_final_score itself is already strict in the right way: it
substitutes 0.5 only for a non-numeric or non-finite value (retrieval.py:503-507), so it
expects 0.0 to be a real weight.

What changed

• A fact persisted with confidence: 0.0 now ranks strictly below one with 0.5, through both
  the FTS5 index and the substring fallback, and matches what the same fact already scores via
  FTS5Retrieval.index_fact. Callers see one consistent ranking regardless of which write path
  produced the row.
• An absent or null confidence still means "unset" and still defaults to 0.5; the explicit
  null case keeps its existing tolerance instead of becoming a TypeError.
• confidence: 0 in the substring fallback now returns a score of 0.0 rather than 0.5, so it
  can no longer outrank a mid-confidence fact when the index is unavailable or a scope is dirty.
• Two tests appended to backend/tests/test_memory_retrieval_adapter.py, one per read path, each
  with an in-file control that pins the unchanged defaulting.

Surface area

FTS5RetrievalAdapter._document and FileMemoryStorage._search_substring. No schema change, no new
dependency, no I/O, and the stored Markdown payload is untouched — only the score derived from it.

• Frontend UI
• Backend API — memory search scores returned by search_facts
• Agents / LangGraph — DeerMem retrieval ranking
• Sandbox
• Skills
• Dependencies
• Default behavior change — for facts whose stored confidence is falsy but legal (0, 0.0), the retrieval score drops by 0.1. Every other value, including a missing key, produces byte-identical scores to before, so no currently-ranking fact changes except the ones this report calls a bug.
• Docs / tests / CI only

Bug fix verification

Both failures are the falsy collapse, verbatim:

FAILED test_indexed_zero_confidence_keeps_its_retrieval_weight  - assert 0.100001 < 0.100001
FAILED test_substring_fallback_keeps_zero_confidence_weight     - assert {'half': 0.5, 'zero': 0.5} == {'half': 0.5, 'zero': 0.0}

The controls pass on both sides, which is the point: test_indexed_zero_confidence_keeps_its_retrieval_weight
also asserts that a fact with no confidence key still scores exactly like an explicit 0.5, and the
18 pre-existing tests in the file are unaffected.

Validation

• Not run to completion: PYTHONPATH=. pytest -m "not live" --ignore=tests/blocking_io tests/ (the
  make test suite) — the run on this branch was still in progress when this PR was written, so only
  the targeted 11-file set above is reported.
• The 11 test files that import the storage/retrieval modules
  (test_memory_storage*.py, test_memory_retrieval_adapter.py, test_deermem_self_contained.py,
  test_memory_eviction.py, test_memory_fact_dedup.py, test_memory_scope_gate.py,
  test_memory_updater.py, test_updater_watermark.py, test_custom_agent.py):
  384 passed, 1 skipped on main → 386 passed, 1 skipped here (the +2 are the new tests; no
  existing test changes outcome).
• The make lint steps, run directly: ruff check . → All checks passed!, and
  ruff format --check . → 1545 files already formatted (ruff 0.15.12, backend/ruff.toml).
• Not run: make itself (no make on this Windows host, so the targets were executed as their
  underlying commands with the project venv), the live and blocking_io pytest marks, the frontend
  suites, and Docker e2e. Reported from a local source checkout, not CI.

AI assistance

Tool(s) used: Qoder (agentic CLI).

How you used it: reproduction scripting and the first draft of the two tests; the root-cause
call, the choice of None-only defaulting over the repo's .get(k, 0.5) idiom (so an explicit
null keeps its current tolerance instead of raising), and the line-by-line review of the final
diff were done here by the agent against main; no human has read the diff yet, which is why the box below is unticked.

• The account owner should tick this once they have read the diff themselves.