feat: rule-based contradiction detection (issue #27)#433
feat: rule-based contradiction detection (issue #27)#433Nitrogonza9 wants to merge 2 commits intoMemPalace:developfrom
Conversation
Implement fact_checker.py with rule-based assertion checking against the knowledge graph. Detects four types of conflicts: - Attribution: wrong person credited for a task (RED) - Tenure: incorrect duration claims vs KG start dates (YELLOW) - Role: conflicting role/title claims (RED) - Relationship: incorrect family/partner relationships (RED) Expired KG facts (valid_to set) are excluded from conflict detection. Unknown entities return GREEN (no data to contradict). Zero API calls, zero new dependencies — pure pattern matching + KG lookup. Wire into MCP server as opt-in mempalace_check_facts tool so agents can verify assertions before recording them. Addresses issue MemPalace#27 — contradiction detection referenced in README but not previously implemented. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
web3guru888
left a comment
There was a problem hiding this comment.
Review: Rule-based contradiction detection
Great addition — this fills a real gap. We run a KG with 710 entities and 1,014 triples across five domains, and contradiction detection is something we built independently in our integration. Some thoughts from that experience:
What works well here:
- The severity tiers (RED/YELLOW/GREEN) are exactly right. Attribution conflicts and role mismatches deserve different urgency than numeric drift.
- Opt-in via
mempalace_check_factsrather than auto-wiring into writes — good call. Our integration calls contradiction checks selectively during what we call the "Evaluate" phase (precision-focused retrieval), not during broad exploration. - Expired fact exclusion via
valid_tois important and well-handled. We had a bug early on where archived triples were triggering false contradictions.
Where regex hits a ceiling (from our experience with 1,014 triples):
The four regex patterns cover the clean cases well, but in practice the contradictions that hurt are softer:
-
Directional claims: "X increases Y" vs "X has no effect on Y" — common across domains. The regex approach would need a pattern for every causal verb, while semantic similarity catches these at ~0.75 cosine.
-
Cross-domain contradictions: We see these most often — an astrophysics finding about radiation effects contradicting an epidemiology claim about dose-response curves. Same underlying relationship, completely different vocabulary. Regex patterns scoped to person/role/attribution won't surface these.
-
Negation: "Alice is NOT the lead on auth migration" — the current
_ATTRIBUTION_PATTERNwould match this as a positive attribution claim for Alice. Worth adding negation handling (even a simplenot|no longer|didn't|nevercheck before the verb).
Concrete suggestion — negation guard:
In _check_attribution, after the regex match, a quick check like:
# Guard against negated claims
prefix = text[:match.start()].lower()
if any(neg in prefix[-20:] for neg in ("not ", "no longer ", "never ", "didn't ")):
return conflicts # skip — negated assertionThis is cheap and catches the most common false positive we saw.
Question: Is there a plan to compose this with min_similarity from PR #350? That combination (rule-based for structured claims + embedding similarity for soft contradictions) could cover both ends. Our integration does exactly that — regex-like checks for hard contradictions, cosine similarity for soft ones.
Test coverage is solid — 18 tests across all paths. Nice work.
|
Thanks @web3guru888 — this is exactly the kind of feedback I was hoping for from someone running a real KG at scale. Negation guard — implementing now. You're absolutely right, this is a concrete false positive. The prefix check you suggested is clean and cheap. I'll add it to Soft contradictions and the regex ceiling — agreed. The four patterns were always meant as a starting point for the structured/hard cases (wrong person, wrong tenure, wrong role). The directional claims and cross-domain contradictions you describe are genuinely harder problems. I deliberately avoided trying to solve them with more regex because that path leads to an explosion of patterns that are fragile and domain-specific. Composing with Re: cross-domain contradictions in your astrophysics/epidemiology example — that's a fascinating use case. MemPalace's wing/room structure could actually help here: if two wings have related rooms, the tunnel system already connects them. A similarity-based contradiction check scoped to tunnel-connected rooms could surface exactly those cross-domain conflicts without searching the entire KG. I'll push the negation guard fix shortly. Thanks for taking the time to review with real production context. — Gonzalo |
Check for negation words (not, no longer, never, didn't, doesn't, isn't, wasn't) both before and within the matched assertion span. Prevents "Soren did NOT finish the auth migration" from triggering a false attribution conflict. Applied to all three regex-based checkers: attribution, role, and relationship. Two new tests for negation handling. Credit: @web3guru888 for the suggestion and the prefix-check pattern. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
|
Glad the negation guard fits. Good thinking to extend it to all claim types — the same false-positive risk applies to The tunnel-scoped similarity check is a sharp observation. In our setup, astrophysics and epidemiology connect via climate (the shared node). So a claim about solar forcing in the astrophysics wing and a claim about UV-driven epidemiological risk in the epidemiology wing would have a tunnel path through climate. Scoping the contradiction check to tunnel-reachable rooms would catch exactly that category while avoiding irrelevant cross-wing noise. If you build the |
|
hey @Nitrogonza9 — thanks for putting this together, the architecture is solid and it addresses a real gap (issue #27). we're going to hold off on merging this though — v4 is bringing local NLP providers (#507) which will enable semantic contradiction detection rather than regex-based pattern matching. a semantic approach would catch things like "Alice is employed by Google" vs "Alice works at Meta" that regex can't. we don't want to ship a regex implementation now and replace it shortly after. when the NLP work lands we'll revisit contradiction detection with semantic understanding built in. appreciate the contribution — the checker design (GREEN/YELLOW/RED with reasoning) is a pattern we'll likely keep. |
Summary
Implements the missing
fact_checker.pymodule referenced in the README andknowledge_graph.py:347— addresses issue #27.New
mempalace/fact_checker.py— rule-based assertion checker with four detection types:Expired KG facts (
valid_toset) are excluded — only current facts trigger conflictsUnknown entities return GREEN (no data = no contradiction)
Zero API calls, zero new dependencies — pure regex pattern matching + KG lookup
New MCP tool
mempalace_check_facts— opt-in tool for agents to verify assertions before recording. Not auto-wired into writes to avoid being intrusive.18 tests covering all conflict types, edge cases, expired facts, and MCP result format
Design decisions
check_factsexplicitly rather than having it run on every write. This keeps writes fast and lets agents decide when to fact-check.query_entity()andquery_relationship()— no direct SQL.Test plan
pytest tests/test_fact_checker.py -v— 18 tests passpytest tests/ -v— full suite 552 passed, 0 failedruff check— no lint errors🤖 Generated with Claude Code