Skip to content

feat: rule-based contradiction detection (issue #27)#433

Closed
Nitrogonza9 wants to merge 2 commits intoMemPalace:developfrom
Nitrogonza9:feat/fact-checker
Closed

feat: rule-based contradiction detection (issue #27)#433
Nitrogonza9 wants to merge 2 commits intoMemPalace:developfrom
Nitrogonza9:feat/fact-checker

Conversation

@Nitrogonza9
Copy link
Copy Markdown

Summary

Implements the missing fact_checker.py module referenced in the README and knowledge_graph.py:347 — addresses issue #27.

  • New mempalace/fact_checker.py — rule-based assertion checker with four detection types:

    • Attribution (RED): "Soren finished the auth migration" when KG says Maya is assigned
    • Tenure (YELLOW): "Kai has been here 2 years" when KG shows 6 years (started 2020)
    • Role (RED): "Alice is a designer" when KG says engineer
    • Relationship (RED): "Max is Alice's partner" when KG says child_of
  • Expired KG facts (valid_to set) are excluded — only current facts trigger conflicts

  • Unknown entities return GREEN (no data = no contradiction)

  • Zero API calls, zero new dependencies — pure regex pattern matching + KG lookup

  • New MCP tool mempalace_check_facts — opt-in tool for agents to verify assertions before recording. Not auto-wired into writes to avoid being intrusive.

  • 18 tests covering all conflict types, edge cases, expired facts, and MCP result format

Design decisions

  • Opt-in, not auto-wire: Agents call check_facts explicitly rather than having it run on every write. This keeps writes fast and lets agents decide when to fact-check.
  • Pattern-based extraction: Uses regex to extract claims from natural language. This is intentionally simple — covers the common patterns shown in the README examples without requiring NLP dependencies.
  • Reuses KG query methods: query_entity() and query_relationship() — no direct SQL.

Test plan

  • pytest tests/test_fact_checker.py -v — 18 tests pass
  • pytest tests/ -v — full suite 552 passed, 0 failed
  • ruff check — no lint errors
  • No new dependencies
  • No API keys or network access needed

🤖 Generated with Claude Code

Implement fact_checker.py with rule-based assertion checking against
the knowledge graph. Detects four types of conflicts:
- Attribution: wrong person credited for a task (RED)
- Tenure: incorrect duration claims vs KG start dates (YELLOW)
- Role: conflicting role/title claims (RED)
- Relationship: incorrect family/partner relationships (RED)

Expired KG facts (valid_to set) are excluded from conflict detection.
Unknown entities return GREEN (no data to contradict). Zero API calls,
zero new dependencies — pure pattern matching + KG lookup.

Wire into MCP server as opt-in mempalace_check_facts tool so agents
can verify assertions before recording them.

Addresses issue MemPalace#27 — contradiction detection referenced in README
but not previously implemented.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Copy link
Copy Markdown

@web3guru888 web3guru888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Rule-based contradiction detection

Great addition — this fills a real gap. We run a KG with 710 entities and 1,014 triples across five domains, and contradiction detection is something we built independently in our integration. Some thoughts from that experience:

What works well here:

  • The severity tiers (RED/YELLOW/GREEN) are exactly right. Attribution conflicts and role mismatches deserve different urgency than numeric drift.
  • Opt-in via mempalace_check_facts rather than auto-wiring into writes — good call. Our integration calls contradiction checks selectively during what we call the "Evaluate" phase (precision-focused retrieval), not during broad exploration.
  • Expired fact exclusion via valid_to is important and well-handled. We had a bug early on where archived triples were triggering false contradictions.

Where regex hits a ceiling (from our experience with 1,014 triples):

The four regex patterns cover the clean cases well, but in practice the contradictions that hurt are softer:

  1. Directional claims: "X increases Y" vs "X has no effect on Y" — common across domains. The regex approach would need a pattern for every causal verb, while semantic similarity catches these at ~0.75 cosine.

  2. Cross-domain contradictions: We see these most often — an astrophysics finding about radiation effects contradicting an epidemiology claim about dose-response curves. Same underlying relationship, completely different vocabulary. Regex patterns scoped to person/role/attribution won't surface these.

  3. Negation: "Alice is NOT the lead on auth migration" — the current _ATTRIBUTION_PATTERN would match this as a positive attribution claim for Alice. Worth adding negation handling (even a simple not|no longer|didn't|never check before the verb).

Concrete suggestion — negation guard:

In _check_attribution, after the regex match, a quick check like:

# Guard against negated claims
prefix = text[:match.start()].lower()
if any(neg in prefix[-20:] for neg in ("not ", "no longer ", "never ", "didn't ")):
    return conflicts  # skip — negated assertion

This is cheap and catches the most common false positive we saw.

Question: Is there a plan to compose this with min_similarity from PR #350? That combination (rule-based for structured claims + embedding similarity for soft contradictions) could cover both ends. Our integration does exactly that — regex-like checks for hard contradictions, cosine similarity for soft ones.

Test coverage is solid — 18 tests across all paths. Nice work.

@Nitrogonza9
Copy link
Copy Markdown
Author

Thanks @web3guru888 — this is exactly the kind of feedback I was hoping for from someone running a real KG at scale.

Negation guard — implementing now. You're absolutely right, this is a concrete false positive. The prefix check you suggested is clean and cheap. I'll add it to _check_attribution and extend it to _check_role and _check_relationship as well — negation applies to all claim types, not just attribution.

Soft contradictions and the regex ceiling — agreed. The four patterns were always meant as a starting point for the structured/hard cases (wrong person, wrong tenure, wrong role). The directional claims and cross-domain contradictions you describe are genuinely harder problems. I deliberately avoided trying to solve them with more regex because that path leads to an explosion of patterns that are fragile and domain-specific.

Composing with min_similarity from PR #350 — yes. That's the right architecture: rule-based for hard contradictions (structured claims with clear subject/predicate/object), embedding similarity for soft ones (semantic drift, causal conflicts, cross-domain). I'd love to see the check_facts tool accept an optional use_similarity: bool flag that falls back to cosine comparison when regex finds nothing. Happy to build that if the maintainers agree on the approach.

Re: cross-domain contradictions in your astrophysics/epidemiology example — that's a fascinating use case. MemPalace's wing/room structure could actually help here: if two wings have related rooms, the tunnel system already connects them. A similarity-based contradiction check scoped to tunnel-connected rooms could surface exactly those cross-domain conflicts without searching the entire KG.

I'll push the negation guard fix shortly. Thanks for taking the time to review with real production context.

— Gonzalo

Check for negation words (not, no longer, never, didn't, doesn't,
isn't, wasn't) both before and within the matched assertion span.
Prevents "Soren did NOT finish the auth migration" from triggering
a false attribution conflict.

Applied to all three regex-based checkers: attribution, role, and
relationship. Two new tests for negation handling.

Credit: @web3guru888 for the suggestion and the prefix-check pattern.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@web3guru888
Copy link
Copy Markdown

Glad the negation guard fits. Good thinking to extend it to all claim types — the same false-positive risk applies to _check_role ("X NOT working at Y" → should not flag) and _check_relationship ("X NO LONGER manages Y").

The tunnel-scoped similarity check is a sharp observation. In our setup, astrophysics and epidemiology connect via climate (the shared node). So a claim about solar forcing in the astrophysics wing and a claim about UV-driven epidemiological risk in the epidemiology wing would have a tunnel path through climate. Scoping the contradiction check to tunnel-reachable rooms would catch exactly that category while avoiding irrelevant cross-wing noise.

If you build the use_similarity flag, one thing worth deciding early: does it check against all drawers in the reached rooms, or just the KG triples? The KG triples are more structured (easier to compare predicates), but the drawers have full context. For contradiction detection specifically, I'd lean toward the KG — you already have subject/predicate/object, so semantic drift is more detectable at the triple level than in raw drawer text.

@bensig bensig changed the base branch from main to develop April 11, 2026 22:22
@bensig
Copy link
Copy Markdown
Collaborator

bensig commented Apr 12, 2026

hey @Nitrogonza9 — thanks for putting this together, the architecture is solid and it addresses a real gap (issue #27).

we're going to hold off on merging this though — v4 is bringing local NLP providers (#507) which will enable semantic contradiction detection rather than regex-based pattern matching. a semantic approach would catch things like "Alice is employed by Google" vs "Alice works at Meta" that regex can't.

we don't want to ship a regex implementation now and replace it shortly after. when the NLP work lands we'll revisit contradiction detection with semantic understanding built in. appreciate the contribution — the checker design (GREEN/YELLOW/RED with reasoning) is a pattern we'll likely keep.

@bensig bensig closed this Apr 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants