Skip to content

feat: deterministic hook saves — zero data loss via silent Python API#673

Merged
bensig merged 1 commit intoMemPalace:developfrom
jphein:feat/deterministic-hook-save
Apr 22, 2026
Merged

feat: deterministic hook saves — zero data loss via silent Python API#673
bensig merged 1 commit intoMemPalace:developfrom
jphein:feat/deterministic-hook-save

Conversation

@jphein
Copy link
Copy Markdown
Collaborator

@jphein jphein commented Apr 12, 2026

Problem

The stop hook's config already has hook_silent_save (default True) and hook_desktop_toast settings (shipped in #667), but the hook implementation ignores them — it always blocks and asks Claude to save via MCP tools.

This is hope-based persistence: the hook blocks, tells Claude "please save your work," and advances the save marker before confirming the save succeeded. If Claude doesn't follow the instruction correctly, or gets interrupted, or the MCP call fails — the data is silently lost and the hook won't retry.

Solution

Implement the silent save path that the config already expects:

Current (block only) This PR (two-layer)
Save mechanism Block → ask Claude → hope it works Layer 1: Python API save (deterministic)
Save marker Advances before save Advances after confirmed write
Data loss risk If Claude ignores instruction Zero — Python API always writes
User experience Blocks conversation, Claude narrates save One-line systemMessage: ✦ 23 memories woven into the palace — chromadb, migration, hooks
Transcript capture Not captured Auto-mined into palace as conversation
Fallback None Layer 2: opt into block mode via hook_silent_save: false

Two-layer save architecture

  • Layer 1 (hook_silent_save: true, default): The hook saves a diary checkpoint and mines the transcript via the Python API before returning. The systemMessage renders as a one-line terminal notification — no conversation interruption. The save marker advances only after a confirmed write.

  • Layer 2 (hook_silent_save: false): Falls back to the current block-and-ask behavior for users who prefer richer AI-driven saves via MCP tools.

AAAK and save paths

AAAK (mempalace/dialect.py) is the project's compressed symbolic summary format. It lives in MCP tool descriptions as a prompt convention — diary_write's tool schema says "write in AAAK format." The tool_diary_write() function itself accepts any string and does not validate or enforce AAAK.

  • Silent mode (this PR's default): No AI reads tool descriptions. Diary entries are plain English. AAAK is irrelevant.
  • Block mode (legacy): The AI sees AAAK instructions in the diary_write tool description and may produce AAAK-formatted entries.

This PR does not add, remove, or modify AAAK behavior. It simply introduces a save path where no AI interprets the tool descriptions, making AAAK a non-factor for the default configuration.

Why both memory systems coexist

MemPalace is designed to work alongside Claude Code's native auto-memory (~/.claude/projects/*/memory/), not replace it:

  • Auto-memory — lightweight preferences, feedback, project context. Things Claude needs at session start for continuity.
  • MemPalace — verbatim everything. Conversations, tool output, code snippets, decisions. The complete searchable archive.
  • Future: Auto Dream — an unreleased consolidation feature (currently behind a disabled feature flag) that will distill MemPalace's raw archive into refined insights, bridging the two layers automatically.

The hook's "for THIS save, use MemPalace MCP tools" instruction scopes the save target to the palace — because hook checkpoints are verbatim session captures, not lightweight preferences. Both systems continue to work in tandem outside of hook saves.

What's added

  • _save_diary_direct() — writes diary checkpoint via Python API, returns count + themes
  • _ingest_transcript() — mines the JSONL session transcript into the palace (conversation mode)
  • _extract_themes() — pulls 2-3 distinctive topic keywords from recent messages via stopword filtering
  • _wing_from_transcript_path() — derives project wing from Claude Code transcript path
  • _mempalace_python() — proper Python interpreter resolution (MEMPALACE_PYTHON env → venv detection → editable install → sys.executable fallback)
  • _desktop_toast() — optional notify-send notification
  • Block reason improvements — names "(MemPalace)" explicitly, adds mempalace_kg_add as optional step 3, scopes auto-memory exclusion to hook saves only (builds on fix: disambiguate hook block reasons to name MemPalace explicitly #666's fix with more nuanced phrasing)
  • Precompact hook now mines transcript before blocking

Why this was closed before (#633)

PR #633 was framed as "hook capture + auto-mine" which looked like it overlapped with the existing hook system. This is actually an implementation of config settings that already shippedhook_silent_save and hook_desktop_toast exist in config.py from #667 but do nothing in the current hooks_cli.py. This PR makes them work.

Test plan

  • 37 hook tests pass (12 new covering silent save, themes, transcript ingest, wing derivation)
  • Full test suite passes (655 tests, 0 failures)
  • Verify hook_silent_save: true saves without blocking conversation
  • Verify hook_silent_save: false falls back to block-and-ask behavior
  • Verify systemMessage renders in Claude Code terminal
  • Verify transcript auto-ingest creates drawers in correct wing

🤖 Generated with Claude Code

Closes #854.

Copilot AI review requested due to automatic review settings April 12, 2026 05:38
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR upgrades the hook system to support deterministic, non-blocking saves (via the Python hook runner) and improves transcript capture by auto-mining JSONL sessions, while updating hook wrappers/docs and bumping plugin versions.

Changes:

  • Add silent-save path in mempalace/hooks_cli.py (config-driven) plus helpers for python resolution, theme extraction, diary checkpointing, and transcript ingest.
  • Update hook shell scripts and plugin hook wrappers to better resolve the correct Python interpreter and to auto-mine transcripts.
  • Extend tests and docs to reflect the updated hook behavior; bump plugin versions to 3.1.0.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
mempalace/hooks_cli.py Implements silent stop-hook saving + background transcript ingest + python interpreter resolution.
tests/test_hooks_cli.py Adds coverage for the new helpers and silent save behavior.
hooks/README.md Updates hook behavior documentation and adds guidance on MEMPALACE_PYTHON + backfill.
hooks/mempal_save_hook.sh Adds python resolution and transcript auto-mining before blocking save instruction.
hooks/mempal_precompact_hook.sh Adds python resolution, transcript discovery, and transcript auto-mining before compaction block.
examples/HOOKS_TUTORIAL.md Updates tutorial with v3.1.0+ behavior and backfill instructions.
.codex-plugin/plugin.json Bumps plugin version to 3.1.0.
.claude-plugin/plugin.json Bumps plugin version to 3.1.0.
.claude-plugin/hooks/mempal-stop-hook.sh Improves python interpreter selection for plugin hook wrapper.
.claude-plugin/hooks/mempal-precompact-hook.sh Improves python interpreter selection for plugin hook wrapper.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread mempalace/hooks_cli.py
Comment on lines +21 to +44
def _mempalace_python() -> str:
"""Return the python interpreter that has mempalace installed.

When hooks are invoked by Claude Code, sys.executable may be the system
python which lacks chromadb and other deps. Resolution order:
1. MEMPALACE_PYTHON env var (explicit override)
2. Venv python from package install path
3. Editable install: venv/ sibling to mempalace/
4. sys.executable fallback
"""
# Honor explicit override (used by shell hook wrappers)
env_python = os.environ.get("MEMPALACE_PYTHON", "")
if env_python and os.path.isfile(env_python) and os.access(env_python, os.X_OK):
return env_python
# This file lives at <venv>/lib/pythonX.Y/site-packages/mempalace/hooks_cli.py
# or <project>/mempalace/hooks_cli.py (editable install).
venv_bin = Path(__file__).resolve().parents[3] / "bin" / "python"
if venv_bin.is_file():
return str(venv_bin)
# Editable install: assumes project root has a venv/ sibling to mempalace/
project_venv = Path(__file__).resolve().parents[1] / "venv" / "bin" / "python"
if project_venv.is_file():
return str(project_venv)
return sys.executable
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_mempalace_python() hardcodes POSIX venv layouts (…/bin/python) and a fixed parent depth. Since CI runs on Windows/macOS, this resolution should handle Windows venv paths (…/Scripts/python.exe) and avoid relying on parents[3] being meaningful. Consider using sys.executable when it already imports mempalace, and/or derive the scripts directory via sysconfig/venv metadata, with platform-aware candidates.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented — the function has 4 resolution levels: MEMPALACE_PYTHON env (explicit override), venv bin/python (site-packages install), editable install venv, and sys.executable fallback. Windows Scripts/python.exe support is a fair addition but low priority since MEMPALACE_PYTHON covers it and sys.executable catches it as final fallback. Will add in a follow-up if CI flags it.

Comment thread mempalace/hooks_cli.py
Comment on lines +159 to +189
def _extract_recent_messages(transcript_path: str, count: int = _RECENT_MSG_COUNT) -> list[str]:
"""Extract the last N user messages from a JSONL transcript."""
path = Path(transcript_path).expanduser()
if not path.is_file():
return []
messages = []
try:
with open(path, encoding="utf-8", errors="replace") as f:
for line in f:
try:
entry = json.loads(line)
msg = entry.get("message", {})
if not isinstance(msg, dict) or msg.get("role") != "user":
continue
content = msg.get("content", "")
if isinstance(content, list):
content = " ".join(
b.get("text", "") for b in content if isinstance(b, dict)
)
if not isinstance(content, str) or not content.strip():
continue
if "<command-message>" in content or "<system-reminder>" in content:
continue
# Truncate long messages
text = content.strip()[:200]
messages.append(text)
except (json.JSONDecodeError, AttributeError):
pass
except OSError:
return []
return messages[-count:]
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_extract_recent_messages() only parses Claude Code JSONL entries with message.role=="user". hook_stop supports the "codex" harness and _count_human_messages() already handles Codex event_msg format, but silent-save will currently get an empty message list on Codex transcripts and skip saving. Extend this parser to also extract recent user_message payloads from Codex JSONL (event_msg/user_message), matching _count_human_messages() behavior.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — _extract_recent_messages() now handles Codex event_msg/user_message format, matching _count_human_messages() behavior. 4ac8961

Comment thread mempalace/hooks_cli.py
Comment on lines +270 to +296
def _ingest_transcript(transcript_path: str):
"""Mine a Claude Code session transcript into the palace as a conversation."""
path = Path(transcript_path).expanduser()
if not path.is_file() or path.stat().st_size < 100:
return

from .config import MempalaceConfig

try:
MempalaceConfig() # validate config loads
except Exception:
return

try:
log_path = STATE_DIR / "hook.log"
STATE_DIR.mkdir(parents=True, exist_ok=True)
with open(log_path, "a") as log_f:
subprocess.Popen(
[
_mempalace_python(), "-m", "mempalace", "mine",
str(path.parent), "--mode", "convos",
"--wing", "sessions",
],
stdout=log_f,
stderr=log_f,
)
_log(f"Transcript ingest started: {path.name}")
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_ingest_transcript() invokes mempalace mine <dir> --mode convos, but the convo miner skips files that are already mined via file_already_mined(). Because session transcripts grow over time, this means only the first ingest will run; later calls will skip the same JSONL and won’t capture new tool output/exchanges. To make auto-ingest reliable, either (a) mine the specific transcript file with idempotent upserts (as the shell hooks do), or (b) add/use a CLI flag to force re-mine/update for growing transcripts.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is handled by the miner's mtime comparison — when a transcript grows, its mtime changes, so file_already_mined() returns False and the file is re-mined with idempotent upserts (same chunk IDs = same drawer IDs). The epsilon mtime comparison in PR #626 ensures this works reliably.

Comment thread mempalace/hooks_cli.py
Comment on lines +192 to +217
_THEME_STOPWORDS = frozenset(
"the a an and or but in on at to for of is it i me my you your we our "
"this that with from by was were be been are not no yes can do did don't "
"will would should could have has had let's let just also like so if then "
"ok okay sure yeah hey hi here there what when where how why which some "
"all any each every about into out up down over after before between "
"get got make made need want use used using check look see run try "
"know think right now still already really very much more most too "
"file files code one two new first last next thing things way well".split()
)


def _extract_themes(messages: list[str], max_themes: int = 3) -> list[str]:
"""Pull 2-3 distinctive topic words from recent messages.

Note: stopword list is English-only; non-English corpora will produce noisy themes.
"""
from collections import Counter
words: Counter[str] = Counter()
for msg in messages:
for word in msg.lower().split():
# Strip punctuation, keep words 4+ chars
clean = word.strip(".,;:!?\"'`()[]{}#<>/\\-_=+@$%^&*~")
if len(clean) >= 4 and clean not in _THEME_STOPWORDS and clean.isalpha():
words[clean] += 1
return [w for w, _ in words.most_common(max_themes)]
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The theme extraction strips apostrophes during cleaning ("don't" → "dont"), but the stopword list includes "don't" (with apostrophe) and not "dont". This will allow common words like "dont" to leak into themes. Normalize stopwords to the same cleaning rules (e.g., include "dont" / "lets" etc., or clean words before stopword comparison).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — normalized stopwords to post-cleaning form: "don't" → "dont", "let's" → "lets". 4ac8961

Comment thread mempalace/hooks_cli.py
Comment on lines +220 to +248
def _save_diary_direct(
transcript_path: str, session_id: str, toast: bool = False,
) -> dict:
"""Write a diary checkpoint directly via Python API (no MCP calls).

Returns {"count": N, "themes": [...]} on success, {"count": 0} on failure.
"""
messages = _extract_recent_messages(transcript_path)
if not messages:
_log("No recent messages to save")
return {"count": 0}

themes = _extract_themes(messages)

# Build a compressed diary entry from recent conversation
now = datetime.now()
topics = "|".join(m[:80] for m in messages[-10:])
entry = (
f"CHECKPOINT:{now.strftime('%Y-%m-%d')}|session:{session_id}"
f"|msgs:{len(messages)}|recent:{topics}"
)

try:
from .mcp_server import tool_diary_write
result = tool_diary_write(
agent_name="session-hook",
entry=entry,
topic="checkpoint",
)
Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_save_diary_direct() docstring says "no MCP calls", but it calls mcp_server.tool_diary_write() (the MCP tool handler) directly. That’s fine operationally, but the comment is misleading—either update the docstring to clarify it’s calling the internal tool function directly (no external MCP roundtrip), or refactor to use a lower-level diary API if the intent is to bypass MCP-layer semantics entirely.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — docstring now says "no MCP roundtrip" to clarify it calls the tool function directly without going through the MCP transport layer. 4ac8961

Comment thread tests/test_hooks_cli.py
Comment on lines +211 to 228
def test_stop_hook_saves_silently_at_interval(tmp_path):
transcript = tmp_path / "t.jsonl"
_write_transcript(
transcript,
[{"message": {"role": "user", "content": f"msg {i}"}} for i in range(SAVE_INTERVAL)],
)
result = _capture_hook_output(
hook_stop,
{"session_id": "test", "stop_hook_active": False, "transcript_path": str(transcript)},
state_dir=tmp_path,
)
assert result["decision"] == "block"
assert result["reason"] == STOP_BLOCK_REASON
save_result = {"count": 15, "themes": ["hooks", "notifications"]}
with patch("mempalace.hooks_cli._save_diary_direct", return_value=save_result) as mock_save:
result = _capture_hook_output(
hook_stop,
{"session_id": "test", "stop_hook_active": False, "transcript_path": str(transcript)},
state_dir=tmp_path,
)
# Saves silently — systemMessage notification with themes, no block
assert result["systemMessage"].startswith("\u2726 15 memories woven into the palace")
assert "hooks" in result["systemMessage"]
mock_save.assert_called_once_with(str(transcript), "test", toast=False)

Copy link

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These hook_stop tests depend on the real ~/.mempalace/config.json because hook_stop instantiates MempalaceConfig() with the default config dir. If a developer has hooks.silent_save=false locally, the test will block instead of returning systemMessage and fail. Patch MempalaceConfig in these tests (or patch expanduser/home) to make them hermetic and independent of user machine config.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — _capture_hook_output() now mocks MempalaceConfig with hook_silent_save=True, hook_desktop_toast=False. Tests no longer depend on ~/.mempalace/config.json. 4ac8961

@jphein jphein force-pushed the feat/deterministic-hook-save branch from 2490fa2 to 77631e3 Compare April 12, 2026 05:48
jphein added a commit to jphein/mempalace that referenced this pull request Apr 12, 2026
Copy link
Copy Markdown

@web3guru888 web3guru888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deterministic Hook Saves — Detailed Review

This is a significant improvement to the hook architecture. The core insight — that "hope-based persistence" (block → ask Claude → advance marker before confirmation) is a data-loss vector — is correct and well-motivated. I read through all 10 changed files. Here's my analysis:

Strengths

1. Save marker only advances after confirmed write
This is the critical fix. Previously, last_save_file.write_text(str(exchange_count)) happened before the AI acted on the save instruction. If Claude ignored it, got interrupted, or the MCP call failed, the counter advanced anyway and the checkpoint was silently lost. Now in silent mode, the marker only advances after _save_diary_direct() returns count > 0. Clean.

2. Two-layer architecture is well-designed
Layer 1 (silent, default) calls tool_diary_write() directly via Python import — no MCP roundtrip, no blocking, deterministic. Layer 2 (legacy block mode) is preserved behind hook_silent_save: false for users who want richer AI-driven saves. The systemMessage output (✦ 23 memories woven into the palace — chromadb, migration, hooks) is elegant — renders as a one-line terminal notification without interrupting conversation flow.

3. Auto-mine transcript captures what the AI would lose
_ingest_transcript() runs the normalizer + chunker on the raw JSONL transcript, capturing Bash output, search results, and build errors verbatim. The AI typically summarizes these away during saves. Having both layers (auto-mine for raw capture + AI save for semantic organization) is genuinely belt-and-suspenders.

4. Python interpreter resolution is thorough
_mempalace_python() handles the real-world problem where Claude Code invokes hooks via system python3 which lacks chromadb. The resolution chain (env var → venv from install path → editable-install venv → sys.executable) covers pip install, pipx, dev installs, and explicit override. The shell scripts mirror this with the same 3-step resolution. Good.

5. Theme extraction is lightweight and practical
_extract_themes() using stopword filtering and Counter.most_common() is the right level of complexity for a hook — no ML models, no tokenizers, just word frequency with a curated stoplist. The 4+ character minimum and alpha-only filter keep it clean. The English-only note in the docstring is honest.

6. Tests properly mock the config
The _capture_hook_output helper now mocks MempalaceConfig with PropertyMock for hook_silent_save and hook_desktop_toast, so tests don't depend on the user's ~/.mempalace/config.json. The updated assertions check for systemMessage instead of decision: "block" — tests match the new behavior.

Observations / minor items

1. _ingest_transcript() spawns an async subprocess — it calls subprocess.Popen() without waiting. For the stop hook this is fine (the process keeps running after the hook exits). For precompact, the comment acknowledges it: "not guaranteed to complete before compaction but gives the palace a head start." Worth noting that if compaction is very fast, the ingest may not finish. The synchronous _save_diary_direct() in Layer 1 is the safety net here, so no data loss — just potentially incomplete raw transcript capture.

2. The inline Python in shell hooks duplicates logic — both mempal_precompact_hook.sh and mempal_save_hook.sh contain ~40 identical lines of inline Python for auto-mining (the PYMINE heredoc). If the chunking logic changes, both need updating. Consider extracting to a CLI subcommand like mempalace hook mine-transcript <path>. Not blocking — it's a maintainability nit and the current approach keeps the hooks self-contained.

3. _extract_recent_messages reads the entire transcript — for very long sessions (thousands of exchanges), this reads every line even though it only needs the last N. A deque(maxlen=count) or reading from the end of the file would be more efficient. In practice, JSONL transcripts rarely exceed a few MB, so this is fine for now.

4. _desktop_toast is Linux-only (notify-send) — on macOS or WSL, this silently fails (which is correct — OSError is caught). A future PR could add osascript for macOS. The --icon=brain flag is a nice touch.

5. Session ID sanitization in shell scripts — the safe = lambda s: re.sub(r'[^a-zA-Z0-9_/.\-~]', '', str(s)) regex is good for preventing injection via session_id in file paths, consistent with the existing _sanitize_session_id in hooks_cli.py.

Impact for multi-agent / integration scenarios

This matters for environments where multiple MemPalace specialist agents share a palace — the deterministic save path means hook checkpoints don't depend on Claude correctly interpreting a natural-language save instruction, which can be unreliable when agents are under cognitive load (deep in a research cycle, context nearly full, etc.). The systemMessage approach is also much friendlier for programmatic harnesses that might not handle block responses well.

APPROVE — solid improvement to the persistence model. The two-layer approach is clean, the data-loss fix is the right architectural change, and the test coverage reflects the new behavior correctly.

jphein added a commit to jphein/mempalace that referenced this pull request Apr 12, 2026
- hooks/README.md: document two save modes (silent vs block), AAAK
  irrelevance in silent mode, tandem memory scoping, updated flow diagrams
- CLAUDE.md: add Hook Save Architecture section, AAAK and Save Paths
  explanation, tandem memory coexistence note, test count 704
- README.md: update hook description for silent/block modes, test count 704
- examples/HOOKS_TUTORIAL.md: rewrite v3.1.0+ section for save modes
- PR MemPalace#673 body: added AAAK section explaining why it's a non-factor

Co-Authored-By: Claude Opus 4.6 <[email protected]>
@jphein jphein force-pushed the feat/deterministic-hook-save branch from 7c1f420 to ef2fb84 Compare April 13, 2026 14:22
@igorls igorls added area/hooks Claude Code hook scripts (Stop, PreCompact, SessionStart) enhancement New feature or request labels Apr 14, 2026
mvalentsev added a commit to mvalentsev/mempalace that referenced this pull request Apr 14, 2026
The precompact hook unconditionally returned {"decision": "block"},
which in Claude Code means "cancel compaction" with no retry mechanism.
This made /compact permanently broken for all plugin users.

Changed hook_precompact() to mine the transcript synchronously (so data
lands before compaction) and return {"decision": "allow"}. This matches
the standalone bash hook in hooks/ which already uses allow.

Also extracted _get_mine_dir() and _mine_sync() helpers so precompact
can mine from the transcript directory, not just MEMPAL_DIR.

Stop hook behavior is unchanged -- left for MemPalace#673 which implements the
full silent save path.

Closes MemPalace#856, closes MemPalace#858.
mvalentsev added a commit to mvalentsev/mempalace that referenced this pull request Apr 15, 2026
The precompact hook unconditionally returned {"decision": "block"},
which in Claude Code means "cancel compaction" with no retry mechanism.
This made /compact permanently broken for all plugin users.

Changed hook_precompact() to mine the transcript synchronously (so data
lands before compaction) and return {"decision": "allow"}. This matches
the standalone bash hook in hooks/ which already uses allow.

Also extracted _get_mine_dir() and _mine_sync() helpers so precompact
can mine from the transcript directory, not just MEMPAL_DIR.

Stop hook behavior is unchanged -- left for MemPalace#673 which implements the
full silent save path.

Closes MemPalace#856, closes MemPalace#858.
bensig pushed a commit that referenced this pull request Apr 15, 2026
#863)

* fix(hooks): stop precompact hook from blocking compaction

The precompact hook unconditionally returned {"decision": "block"},
which in Claude Code means "cancel compaction" with no retry mechanism.
This made /compact permanently broken for all plugin users.

Changed hook_precompact() to mine the transcript synchronously (so data
lands before compaction) and return {"decision": "allow"}. This matches
the standalone bash hook in hooks/ which already uses allow.

Also extracted _get_mine_dir() and _mine_sync() helpers so precompact
can mine from the transcript directory, not just MEMPAL_DIR.

Stop hook behavior is unchanged -- left for #673 which implements the
full silent save path.

Closes #856, closes #858.

* fix: use empty JSON instead of invalid \"allow\" decision value

Claude Code only recognizes \"block\" as a top-level decision value.
\"allow\" is a permissionDecision value for PreToolUse hooks, not a
valid top-level decision. The correct way to not block is to return
empty JSON. Caught by #872.
@jphein jphein force-pushed the feat/deterministic-hook-save branch from 9fad285 to f7028b3 Compare April 16, 2026 18:25
jphein added a commit to jphein/mempalace that referenced this pull request Apr 16, 2026
fork-direction.md and TODO-fork-improvements.md were scattered strategic
thinking that belongs in the front-door README: competitive landscape,
roadmap (P0-P6), and open problems. Merges them in and deletes the
standalone files.

PR-status tables (README + CLAUDE.md) were stale after today's rebases
of MemPalace#659, MemPalace#660, MemPalace#661, MemPalace#673, MemPalace#681 — updated to reflect current mergeable
state. MemPalace#673 specifically noted as cleanly rebased against MemPalace#863.

test_readme_claims.py skips MCP-tool-table and dialect-reference checks
when absent — our slimmed fork README doesn't reproduce upstream's tool
table structure.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
jphein added a commit to jphein/mempalace that referenced this pull request Apr 18, 2026
- Bump fork-target tag and drawer count (v3.3.0 → v3.3.1, 134K → 135K)
- Add "Pulled in from upstream v3.3.1" section: i18n entity detection,
  BCP-47 locale resolution, script-aware boundaries, UTF-8 Path.read_text,
  non-blocking precompact (MemPalace#863), basic silent_save honoring (MemPalace#966)
- Note that our MemPalace#673 silent-save is still ahead of upstream's MemPalace#966:
  marker-after-confirmed-save, themes extraction, systemMessage
- Add Mintlify to the memory-system comparison table (docs platform
  pitched as "self-updating knowledge management" with MCP support;
  not a verbatim personal memory tool, but shares the framing)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@jphein
Copy link
Copy Markdown
Collaborator Author

jphein commented Apr 18, 2026

Friendly ping on review. This received a detailed APPROVAL from @web3guru888 on 2026-04-12 and has been sitting clean since then. Context worth noting: @PostProtoroman's #966 lands a narrower implementation of the same silent_save intent — if you'd prefer to merge #966 first and have us rebase the additional pieces here (marker-only-after-confirmed-write, themes extraction, systemMessage rendering, transcript ingest) on top, happy to do that. Just let us know which order works best.

jphein added a commit to jphein/mempalace that referenced this pull request Apr 18, 2026
Concrete evidence:
- New "What this looks like in practice" section right after the
  status line, showing a real stop-hook systemMessage output and a
  real mempalace_search response shape (warnings, available_in_scope,
  matched_via tags, similarity scores). Someone evaluating the fork
  can see what "running in production" actually looks like.

Headlines box:
- New "Headlines" subsection at the top of Fork Changes with the
  three differentiators someone should know if they only read one
  section — silent-save hook, ChromaDB 1.5.x hardening (quarantine +
  None guards), search-never-misses contract. Links to MemPalace#673/MemPalace#999/
  MemPalace#1000/MemPalace#1005 so readers can jump to the work itself.

Citations for comparison table:
- Every row now links to its upstream repo: Hindsight (vectorize-io),
  Mem0 + OpenMemory (mem0ai), Cognee (topoteretes), Letta (letta-ai),
  engram (NickCirv), CaviraOSS OpenMemory. Cognee row updated since
  they've added MCP support since we first wrote the row.
- Replaced the "Systems mentioned without captured primary URLs"
  footnote (now stale since we have them) with a "Verification note"
  that's honest about the point-in-time nature of these columns and
  explains why TagMem is absent.

Structure cleanups:
- Removed the upstream MemPalace logo at the top — it's milla-
  jovovich's asset and using it in a fork README is awkward.
- Renamed "Roadmap" → "Planned work" — the section is decisive with
  priorities and time estimates, "Roadmap" was underselling it.

No content removed from the document beyond the stale footnote and
the upstream logo. All existing sections intact. 42 README-claim
tests pass.
jphein added a commit to jphein/mempalace that referenced this pull request Apr 18, 2026
Every remaining row in "Still ahead of upstream" now carries a status
so the reader can tell at a glance whether each change is being
upstreamed, pending a PR, or deliberately fork-local.

Dropped:
- "Guard ChromaDB 1.5.x metadata-mismatch segfault" — this row was
  overstated. The memory file for today's debugging notes that the
  try-get/except-create pattern is defensive code that never
  reproduced a specific crash (the actual crashes traced to HNSW
  drift). Leaving it in "Still ahead" implied an upstream-candidate
  fix, which it isn't. Code stays in place as defensive, but the
  README no longer claims it as a fork-ahead feature.

Moved to Superseded:
- "Stale HNSW mtime detection + mempalace_reconnect" — upstream took
  a different approach in MemPalace#757. Our broader inode+mtime detection
  and the mempalace_reconnect MCP tool remain as fork-local
  convenience; they're just not "ahead of upstream" anymore.

Statuses now populated:
- Linked PR number for the 7 changes with active upstream PRs
  (MemPalace#659, MemPalace#660, MemPalace#661, MemPalace#673 with APPROVED note, MemPalace#999, MemPalace#1000, MemPalace#1005).
- "PR pending" for 3 items that are good candidates but unfiled:
  epsilon mtime comparison, max_distance parameter, tool output
  mining.
- "fork-only" for 2 items we keep intentionally without pitching
  upstream: .blob_seq_ids_migrated marker (narrow), bulk_check_mined
  (complementary to upstream's MemPalace#784 file-locking).

Legend sentence added above the table explains the three status
values. 42 README-claim tests pass.
jphein added a commit to jphein/mempalace that referenced this pull request Apr 18, 2026
Claude Code 2.1.114 passes stop_hook_active:true on every Stop fire
after the first in a session (plugin-dispatched hooks in particular).
The legacy guard at line 426 was written for block-mode, where a
re-fire with the flag set meant "you already blocked, don't block
again" — correct loop prevention when the hook returns
{"decision":"block"}.

Silent-save mode (default since MemPalace#673) never blocks — it saves
directly and returns. The flag is meaningless there, so the old
guard was suppressing every auto-save after the first one in a
Claude Code session. Symptom: terminal never shows the "✦ N
memories woven" notification again, hook.log stays silent, save
marker stuck.

Fix: only skip on stop_hook_active when block mode is configured.
Silent mode runs through as normal — the save is deterministic and
idempotent, no loop risk.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
jphein added a commit to jphein/mempalace that referenced this pull request Apr 18, 2026
Claude Code 2.1.114 passes stop_hook_active:true on every Stop fire
after the first in a session (plugin-dispatched hooks in particular).
The legacy guard at line 426 was written for block-mode, where a
re-fire with the flag set meant "you already blocked, don't block
again" — correct loop prevention when the hook returns
{"decision":"block"}.

Silent-save mode (default since MemPalace#673) never blocks — it saves
directly and returns. The flag is meaningless there, so the old
guard was suppressing every auto-save after the first one in a
Claude Code session. Symptom: terminal never shows the "✦ N
memories woven" notification again, hook.log stays silent, save
marker stuck.

Fix: only skip on stop_hook_active when block mode is configured.
Silent mode runs through as normal — the save is deterministic and
idempotent, no loop risk.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
jphein added a commit to jphein/mempalace that referenced this pull request Apr 19, 2026
Claude Code 2.1.114 passes stop_hook_active:true on every Stop fire
after the first in a session (plugin-dispatched hooks in particular).
The legacy guard at line 426 was written for block-mode, where a
re-fire with the flag set meant "you already blocked, don't block
again" — correct loop prevention when the hook returns
{"decision":"block"}.

Silent-save mode (default since MemPalace#673) never blocks — it saves
directly and returns. The flag is meaningless there, so the old
guard was suppressing every auto-save after the first one in a
Claude Code session. Symptom: terminal never shows the "✦ N
memories woven" notification again, hook.log stays silent, save
marker stuck.

Fix: only skip on stop_hook_active when block mode is configured.
Silent mode runs through as normal — the save is deterministic and
idempotent, no loop risk.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@jphein jphein force-pushed the feat/deterministic-hook-save branch 2 times, most recently from ff60d54 to d294052 Compare April 19, 2026 15:25
jphein added a commit to jphein/mempalace that referenced this pull request Apr 19, 2026
…emPalace#1036

Overnight/morning:
- MemPalace#681, MemPalace#1000, MemPalace#1023 merged — moved from "Still ahead" to "Merged upstream (post-v3.3.1)"
- bensig reviewed MemPalace#659 (wing_ prefix + agent filter) and MemPalace#1021 (silent_guard
  default) — both addressed on their PR branches
- MemPalace#673 needed re-rebase after overnight develop merges; done
- MemPalace#1036 filed: paginate miner.status(), closes upstream MemPalace#802 and MemPalace#1015

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a `hook_silent_save` mode (default `true` in new installs) where
the stop and precompact hooks write diary entries directly via the
Python API — no AI block, no MCP tool roundtrip, no possibility of the
AI forgetting or ignoring the save instruction.

**Two modes, controlled by `hook_silent_save` in `~/.mempalace/config.json`:**

1. **Silent mode** (default): Direct call to `tool_diary_write()`. Plain
   text, no AI involved, deterministic. Save marker advances only after
   the write is confirmed, so mid-save failures do not lose exchanges.
   Shows `"✦ N memories woven into the palace"` as a systemMessage
   notification so the user knows the save fired.

2. **Block mode** (legacy): Returns `{"decision": "block"}` asking the
   AI to call the MCP tool chain. Non-deterministic — the AI may ignore,
   summarize lossy, or fail. Kept for backward compatibility.

**Extras rolled in:**
- Block reasons name "MemPalace" explicitly and instruct the AI not to
  write to Claude Code's native auto-memory (.md files) — prevents the
  two memory systems from stepping on each other.
- Codex transcript handling (`event_msg` payloads) in
  `_count_human_messages` + `_extract_recent_messages`.
- Tightened stopword leak in diary summaries; docstring polish; test
  hermeticity fixes (per-test `STATE_DIR` patching).

**Tests:** hooks_cli tests cover silent-save path, save-marker
advancement after confirmed write only, and systemMessage formatting.

Rebased fresh on upstream/develop. Only touches files germane to the
feature (hooks_cli.py, tests, hooks/README.md, HOOKS_TUTORIAL.md) —
stale fork-local `.sh` wrapper and plugin manifest changes dropped.
@jphein jphein force-pushed the feat/deterministic-hook-save branch from d294052 to 74e9cbc Compare April 21, 2026 20:21
@jphein
Copy link
Copy Markdown
Collaborator Author

jphein commented Apr 21, 2026

Rebased on current upstream/develop and squashed to a single clean commit. Dropped the stale fork-local .sh wrapper and plugin.json changes that had accumulated — those belonged to an older version of our hook resolution story. The PR now only touches files germane to the deterministic save feature:

  • mempalace/hooks_cli.py — silent-save path + Codex transcript handling
  • tests/test_hooks_cli.py — coverage for silent-save marker semantics
  • hooks/README.md — documents the two save modes (kept alongside your MEMPAL_PYTHON section)
  • examples/HOOKS_TUTORIAL.md — updated two-mode explanation

All tests pass (56 in test_hooks_cli.py, 1096 in the full suite on our fork). Ready for merge when you have a moment.

jphein added a commit to jphein/mempalace that referenced this pull request Apr 21, 2026
…#1023 merged, MemPalace#673 rebased

- Bump fork-ahead section header to "after v3.3.2"
- Strike #11 (quarantine_stale_hnsw) and MemPalace#18 (PID file guard) as
  merged-into-upstream-via-v3.3.2, keep entries for traceability
- Add v3.3.2-shipped items to "Merged into upstream (post-v3.3.1)"
- Rebuild PR table: 10 merged / 7 open / 7 closed; add MemPalace#1024 row,
  reclassify MemPalace#681/MemPalace#1000/MemPalace#1023 as merged, note MemPalace#673 rebased 2026-04-21
- Annotate MemPalace#661 status with the GitHub review-state machine caveat
  (CHANGES_REQUESTED persists until reviewer dismisses, not owed)
- Bump test count 1063 → 1096 post-merge
jphein added a commit to jphein/mempalace that referenced this pull request Apr 21, 2026
…unts

- Upstream released v3.3.2 on 2026-04-21 (our MemPalace#681/MemPalace#1000/MemPalace#1023)
- Drawer count 152,682 → 165,632; wings 23 → 28; tests 1063 → 1096
- MemPalace#673 now MERGEABLE after fresh rebase + squash to 1 commit
- MemPalace#661 status clarified: CHANGES_REQUESTED persists until reviewer
  dismisses, not an outstanding owe
- Merged-upstream section split out v3.3.2 release group
jphein added a commit to jphein/mempalace that referenced this pull request Apr 21, 2026
…rows where we already have a PR that addresses them

Sweep caught three adjacent issues I hadn't cross-referenced:
- MemPalace#854 (silent_save flag never read) — closed by MemPalace#673; added to row + PR body
- MemPalace#848 (remove drawers from a wing) — closed by MemPalace#1087; added to row + PR body
- MemPalace#390 (default chunk exceeds MiniLM token cap) — addressed by MemPalace#1024
  (configurable unblocks the user without changing default)

Posted explanatory comments on all three issues pointing users to the
relevant PRs. PR bodies updated via REST API (gh pr edit's GraphQL path
was failing on a deprecation warning around projectCards).
@bensig bensig merged commit 23ee2a0 into MemPalace:develop Apr 22, 2026
6 checks passed
jphein added a commit to jphein/mempalace that referenced this pull request Apr 22, 2026
… (2026-04-22)

Ben's batched queue-clear pass merged four PRs at 00:38 UTC:
graph cache (MemPalace#661), deterministic hook saves (MemPalace#673), Claude Code
2.1.114 hook stdout + silent_save guard (MemPalace#1021), and upstream's
own MemPalace#851 pagination fix (closing MemPalace#1036 as superseded).

Moved four rows out of the "Fork Changes" / "Fork change queue"
tables into their respective merged-upstream history sections.
Intro sentence PR count reduced from 7 → 4 open.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
jphein pushed a commit to jphein/mempalace that referenced this pull request Apr 24, 2026
Restore-integrity release. Unbreaks fresh `pip install mempalace` from
v3.3.2 by re-tagging current develop, which carries both the plugin.json
consumer (shipped in 3.3.2) and the matching mempalace-mcp entry point
in pyproject.toml (added on develop ~10h after the 3.3.2 tag via MemPalace#340
by @messelink). MemPalace#1093 diagnosed by @jphein.

Bumps (all 5 sources agree per Version Guard / CLAUDE.md):
- mempalace/version.py              3.3.2 → 3.3.3
- pyproject.toml                     3.3.2 → 3.3.3
- .claude-plugin/plugin.json         3.3.2 → 3.3.3
- .claude-plugin/marketplace.json    3.3.2 → 3.3.3
- .codex-plugin/plugin.json          3.3.2 → 3.3.3
- CHANGELOG.md                        new [3.3.3] entry

No code changes. The fix for MemPalace#1093 is already on develop via merged PRs
MemPalace#340, MemPalace#1021, MemPalace#851, MemPalace#942, MemPalace#833, MemPalace#673, MemPalace#661, MemPalace#659, MemPalace#1097, MemPalace#1051, MemPalace#1001,
MemPalace#945.

Branch name intentionally outside the `release/*` ruleset so follow-up
CI-fix commits aren't gated behind a nested PR. (Supersedes MemPalace#1143 —
closed for exactly that reason after it missed 3 of 5 version files.)

Smoke-tested locally from a fresh develop clone:
  grep mempalace-mcp pyproject.toml .claude-plugin/plugin.json   # both ✓
  python -m build --wheel                                        # ✓
  pip install …-py3-none-any.whl                                 # ✓
  which mempalace-mcp                                            # ✓
  mempalace-mcp --help                                           # ✓
jphein added a commit to jphein/mempalace that referenced this pull request Apr 24, 2026
…safe graph cache

31 commits including:
- v3.3.3 release (restores pip install integrity post-v3.3.2 MemPalace#1093 regression)
- MemPalace#1097 empty-string filter normalization in mempalace_search
- MemPalace#659 diary wing parameter (our fork PR, now upstream)
- MemPalace#851/MemPalace#1097/MemPalace#1021/MemPalace#661/MemPalace#673 incorporated
- website Crystal Lattice brand refresh
- thread-safe graph cache in palace_graph.py

Conflict resolutions (10 files):
- README.md            keep fork version; bump badge 3.3.1 → 3.3.3 for test compat
- hooks/README.md      keep fork silent/block architecture docs; keep MEMPAL_PYTHON
                       (correct for legacy hook, upstream's rename is stale)
- examples/HOOKS_TUTORIAL.md  same treatment
- mcp_server.py        take upstream's sanitize_name(wing) — strictly better than
                       our crude lowercase+underscore normalization
- miner.py             keep fork 10K batch size + comma formatting on status;
                       adopt upstream's pagination rationale comment
- palace_graph.py      take upstream entirely — thread-safety improvements layered
                       on top of our MemPalace#661 (which upstream already merged)
- hooks_cli.py         take upstream (Windows path-separator compat in
                       _wing_from_transcript_path, Codex CLI format in
                       _extract_recent_messages), then re-apply fork-ahead: use
                       _wing_from_transcript_path in _ingest_transcript instead
                       of hardcoded "sessions" — keeps transcript mining coherent
                       with the diary wing derivation from MemPalace#659
- tests/test_hooks_cli.py  take upstream's updated wing-kwarg assertions and
                       new test_stop_hook_derives_wing_from_transcript_path;
                       take upstream's mock-based security test (simpler than
                       our three-way assertion, same property tested)

Post-merge test state:
- 1096 passed, 10 failed in tests/test_claude_plugin_hook_wrappers.py
- The 10 failures are the fork-ahead MemPalace#19 divergence already documented in
  CLAUDE.md: our venv-aware hooks use `dirname`/`cat` which the test's
  scrubbed-PATH environment doesn't provide. Same class that correctly caught
  MemPalace#1115 and led us to withdraw it pending MemPalace#1069 arbitration. Expected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
arncore added a commit to arncore/mempalace that referenced this pull request Apr 25, 2026
* feat: add Hindi language support to i18n module

* Create SECURITY.md

This PR introduces a standard SECURITY.md policy file to the repository. 

While reviewing the codebase, I noticed there wasn't a defined channel for the private, responsible disclosure of security vulnerabilities. Adding this policy helps protect the project by guiding researchers to report bugs privately rather than in public issues. 

I highly recommend merging this and enabling GitHub's "Private Vulnerability Reporting" feature in your repository settings. I currently have some security findings I would like to share with the maintainers securely once a private channel or contact method is established.

* fix: save hook auto-mines transcript without MEMPAL_DIR (#840)

TDD: test written first, failed, then fixed.

Problem: save hook says "saved in background" but MEMPAL_DIR defaults
to empty, so nothing actually mines. Users get no auto-save despite
the hook firing every 15 messages.

Fix: use TRANSCRIPT_PATH (received from Claude Code in the hook's
JSON input) to discover the session directory. Mine that directory
automatically. MEMPAL_DIR is still supported as override but no
longer required.

Also fixed: bare python3 → $(command -v python3) for nohup safety.

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* release: v3.3.0 (#839)

* fix: add file-level locking to prevent multi-agent duplicate drawers

Root cause: when multiple agents mine simultaneously, both pass
file_already_mined() check, both delete+insert the same file's
drawers, creating duplicates or losing data.

Fix: mine_lock() in palace.py — cross-platform file lock (fcntl on
Unix, msvcrt on Windows). Both miner.py and convo_miner.py now lock
per-file during the delete+insert cycle and re-check after acquiring
the lock.

Tested:
- Lock acquires and releases correctly
- Second agent blocks until first releases (0.25s wait)
- 33/33 existing tests pass
- Cross-platform: fcntl (macOS/Linux), msvcrt (Windows)

Based on v3.2.0 tag.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: strip system tags, hook output, and Claude UI chrome from drawers

normalize.py now strips before filing:
- <system-reminder>, <command-message>, <command-name> tags
- <task-notification>, <user-prompt-submit-hook>, <hook_output> tags
- Hook status messages (CURRENT TIME, Checking verified facts, etc.)
- Claude Code UI chrome (ctrl+o to expand, progress bars, etc.)
- Collapsed runs of blank lines

This noise was going straight into drawers, wasting storage space
and polluting search results. strip_noise() runs on all normalized
output regardless of input format (JSONL, JSON, plain text).

689/689 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* feat: add closet layer — searchable index pointing to drawers

The closet architecture was always part of MemPalace's design but
never shipped in the public codebase. This adds it.

Palace now has TWO collections:
- mempalace_drawers — full verbatim content (unchanged)
- mempalace_closets — compact AAAK-style index entries

How it works:
- When mining, each file gets a closet alongside its drawers
- Closet contains extracted topics, entities, quotes as pointers
- Closets pack up to 1500 chars, topics never split mid-entry
- Search hits closets first (fast, small), then hydrates the
  full drawer content for matching files
- Falls back to direct drawer search if no closets exist yet

Files changed:
- palace.py: get_closets_collection(), build_closet_text(),
  upsert_closet(), CLOSET_CHAR_LIMIT
- miner.py: process_file() now creates closets after drawers
- searcher.py: search_memories() tries closet-first search,
  hydrates drawers, falls back to direct search

Backwards compatible — existing palaces without closets continue
to work via the fallback path. Closets are created on next mine.

689/689 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: enforce atomic topics in closets, extract richer pointers

- upsert_closet replaced by upsert_closet_lines: checks each topic
  line individually against CLOSET_CHAR_LIMIT. If adding one line
  WHOLE would exceed the limit, starts a new closet. Never splits
  mid-topic.
- build_closet_lines returns a list of atomic lines (not joined text)
- Richer extraction: section headers, more action verbs, up to 3
  quotes, up to 12 topics per file
- Each line is complete: topic|entities|→drawer_refs

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* docs: add CLOSETS.md — closet layer overview

Cherry-picked the docs portion of 67e4ac6 to accompany the closet
feature. Test coverage for closets is omnibus with tests for entity
metadata and BM25 (see PR targeting those features) and will land
together in a follow-up.

Co-Authored-By: MSL <[email protected]>

* feat: entity metadata + diary ingest + BM25 hybrid search

Three features that close the gap between the architecture docs
and the actual codebase:

1. Entity metadata on drawers and closets
   - _extract_entities_for_metadata() pulls names from known_entities.json
     + proper nouns appearing 2+ times
   - Stamped as "entities" field in ChromaDB metadata
   - Enables filterable search by person/project name

2. Day-based diary ingest (diary_ingest.py)
   - ONE drawer per day, upserted as the day grows
   - Closets pack topics atomically, never split mid-topic
   - Tracks entry count in state file, only processes new entries
   - Usage: python -m mempalace.diary_ingest --dir ~/summaries

3. BM25 hybrid search in searcher.py
   - _bm25_score() keyword matching complements vector similarity
   - _hybrid_rank() combines both signals (60% vector, 40% BM25)
   - Catches exact name/term matches that embeddings miss
   - Applied to both closet-first and direct drawer search paths

689/689 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* test: add tests for mine_lock, closets, entity metadata, BM25, diary

Trimmed version of Milla's omnibus test_closets.py to only cover
features present in this PR stack (#784 lock, #788 closets, this
PR's entity/BM25/diary). Strip-noise tests will land with #785;
tunnel tests will land with the tunnels PR.

16/16 pass.

Co-Authored-By: MSL <[email protected]>

* feat: explicit cross-wing tunnels for multi-project agents

Adds active tunnel creation alongside passive tunnel discovery.

Passive tunnels (existing): rooms with the same name across wings.
Explicit tunnels (new): agent-created links between specific
locations. "This API design in project_api relates to the database
schema in project_database."

New functions in palace_graph.py:
- create_tunnel() — link two wing/room pairs with a label
- list_tunnels() — list all explicit tunnels, filter by wing
- delete_tunnel() — remove a tunnel by ID
- follow_tunnels() — from a room, find all connected rooms in
  other wings with drawer content previews

New MCP tools:
- mempalace_create_tunnel
- mempalace_list_tunnels
- mempalace_delete_tunnel
- mempalace_follow_tunnels

Tunnels stored in ~/.mempalace/tunnels.json (persists across
palace rebuilds). Deduplicated by endpoint pair.

689/689 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* test: add TestTunnels for cross-wing tunnel operations

Appended from Milla's omnibus test_closets.py — covers create,
list, delete, dedup, and follow_tunnels behavior. 21/21 pass.

Co-Authored-By: MSL <[email protected]>

* feat(search): drawer-grep returns best-matching chunk + neighbors

When a closet hit leads to a source file with many drawers, grep each
chunk for query terms and return the BEST-MATCHING chunk + 1 neighbor
on each side, instead of dumping the whole file truncated at
MAX_HYDRATION_CHARS. Result now includes drawer_index and
total_drawers so callers can request adjacent drawers explicitly.

Extracted from Milla's commit 935f657 which bundled drawer-grep with
closet_llm (deferred pending LLM_ENDPOINT refactor) and fact_checker
(separate PR). Ported only the searcher.py change.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* feat: offline fact checker against entity registry + knowledge graph

fact_checker.py verifies text for contradictions against locally stored
entities and KG facts. Catches similar-name confusion (Bob vs Bobby),
relationship mismatches (KG says husband, text says brother), and
stale facts (KG valid_from/valid_to).

No hardcoded facts. No network calls. Reads:
- ~/.mempalace/known_entities.json
- KnowledgeGraph SQLite

Usage:
  from mempalace.fact_checker import check_text
  issues = check_text("Bob is Alice's brother", palace_path)

  # CLI
  python -m mempalace.fact_checker "text" --palace ~/.mempalace/palace

Extracted from Milla's commit 935f657 which bundled this with
closet_llm (deferred) and drawer-grep (PR #791). Ported only
fact_checker.py — verified no network / API imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* feat: optional LLM-based closet regeneration — bring-your-own endpoint

Adds mempalace/closet_llm.py as an OPTIONAL path for richer closet
generation. Regex closets remain the default and cover the local-first
promise; users who want LLM-quality topics can bring their own endpoint.

Configuration (env or CLI flag):
  LLM_ENDPOINT — OpenAI-compatible base URL (required)
  LLM_KEY      — bearer token (optional; local inference skips this)
  LLM_MODEL    — model name (required)

Works with Ollama, vLLM, llama.cpp servers, OpenAI, OpenRouter, and any
other provider that speaks OpenAI-compatible /chat/completions. Zero new
dependencies — uses stdlib urllib.

Replaces the original Anthropic-SDK-hardcoded version of this module
from Milla's branch (commit 935f657). Same prompt, same parsing, same
regenerate_closets flow; only the transport was generalised so the
feature doesn't lock users into a specific vendor or require API keys
for core memory operations (CLAUDE.md, "Local-first, zero API").

Includes 13 unit tests covering config resolution, request shape,
auth-header omission when no key is set, code-fence stripping, and
missing-config error path. All mocked — zero network calls in tests.

Co-Authored-By: MSL <[email protected]>

* fix(search): hybrid closet+drawer retrieval — closets boost, never gate (#795)

* Fix: set cosine distance metadata on all collection creation sites

ChromaDB defaults HNSW index to L2 (Euclidean) distance, but
MemPalace scoring uses 1-distance which requires cosine (range 0-2).
Add metadata={"hnsw:space": "cosine"} to the 4 production and 3 test
call sites that were missing it.

Closes #218

* fix: sync version.py to 3.2.0

Commit 6614b9b bumped pyproject.toml to 3.2.0 but missed
mempalace/version.py, breaking test_version_consistency on
every PR's CI. This syncs them.

* refactor: extract locked filing block to keep mine_convos under C901

Adding the per-file lock + double-checked file_already_mined() in the
previous commit pushed mine_convos cyclomatic complexity from 25 to 26,
just over ruff's max-complexity threshold. Hoist the locked critical
section into _file_chunks_locked() so the outer loop stays within
budget. No behavior change.

* style: ruff format mempalace/palace.py

Add blank lines after inline imports in mine_lock. Pure formatting.

* fix(normalize): make strip_noise verbatim-safe and scope it to Claude Code JSONL

The initial strip_noise() regressed on three fronts when audited against
adversarial user content — each verified with executable repros against
the cherry-picked code:

  1. `<tag>.*?</tag>` with re.DOTALL span-ate across messages: one
     stray unclosed <system-reminder> anywhere in a session merged with
     the next closing tag, silently deleting everything between them
     (including full assistant replies).
  2. `.*\(ctrl\+o to expand\).*\n?` nuked entire lines of user prose
     whenever a user happened to document the TUI shortcut.
  3. `Ran \d+ (?:stop|pre|post)\s*hook.*` with IGNORECASE ate the
     second sentence from "our CI has a stop hook ... Ran 2 stop hooks
     last week" — legitimate user commentary.

These are unambiguous violations of the project's "Verbatim always"
design principle.

Fixes:

- All tag patterns are now line-anchored (`(?m)^(?:> )?<tag>`) and their
  body forbids crossing a blank line (`(?:(?!\n\s*\n)[\s\S])*?`), so a
  dangling open tag cannot eat neighboring messages.
- `_NOISE_LINE_PREFIXES` are line-anchored and case-sensitive — user
  prose mentioning "CURRENT TIME:" mid-sentence is preserved.
- Hook-run chrome requires `(?m)^`, explicit hook names (Stop,
  PreCompact, PreToolUse, etc.), and no IGNORECASE.
- "… +N lines" is line-anchored.
- "(ctrl+o to expand)" only matches Claude Code's actual collapsed-
  output chrome shape `[N tokens] (ctrl+o to expand)`; a bare
  parenthetical in user prose stays intact.

Scope:

- `strip_noise()` is no longer called on every normalization path.
  Only `_try_claude_code_jsonl` invokes it, per-extracted-message — so
  Claude.ai exports, ChatGPT exports, Slack JSON, Codex JSONL, and
  plain text with `>` markers pass through fully verbatim. Per-message
  application also makes span-eating structurally impossible.

Tests:

- 15 new tests in test_normalize.py pin the boundary: 6 guard user
  content that must survive (each of the adversarial repros), 9 assert
  real system chrome is still stripped. All pass; full suite 702 pass
  (2 failures are the unrelated pre-existing version.py bug, cleared
  by #820).

Known limitation (not fixed here): convo_miner.py does not delete
drawers on re-mine, so transcripts mined before this PR keep noise-
filled drawers until the user manually erases + re-mines. Proper fix
needs a schema-version field on drawer metadata + re-mine trigger —
out of scope for this PR.

* feat(normalize): auto-rebuild stale drawers via NORMALIZE_VERSION schema gate

Without this, the strip_noise improvement only helps new mines. Every
user who had already mined Claude Code JSONL sessions would keep their
noise-polluted drawers forever, because convo_miner's file_already_mined
skip short-circuits before re-processing.

Adds a versioned schema gate so upgrades propagate silently:

- palace.NORMALIZE_VERSION=2 — bumped when the normalization pipeline
  changes shape (this PR's strip_noise is the v1→v2 bump).
- file_already_mined now returns False if the stored normalize_version
  is missing or less than current, triggering a rebuild on next mine.
- Both miners stamp drawers with the current normalize_version.
- convo_miner now purges stale drawers before inserting fresh chunks
  (mirrors miner.py's existing delete+insert), extracted into
  _file_convo_chunks helper to keep mine_convos under ruff's C901 limit.

User experience: upgrade mempalace, run `mempalace mine` as usual, old
noisy drawers get silently replaced with clean ones. No erase needed,
no "you need to rebuild" changelog footgun.

Tests:
- test_file_already_mined_returns_false_for_stale_normalize_version —
  pins the version gate contract for missing/v1/current.
- test_add_drawer_stamps_normalize_version — fresh project-miner drawers
  carry the field.
- test_mine_convos_rebuilds_stale_drawers_after_schema_bump — end-to-end
  proof that a pre-v2 palace gets silently cleaned on next mine, with
  orphan drawers purged and NOT skipped.

Existing test_file_already_mined_check_mtime updated to include the
new field; all other tests unaffected.

* fix: stop hooks from making agents write in chat — save tokens

The save hook and precompact hook were telling the agent to write
diary entries, add drawers, and add KG triples IN THE CHAT WINDOW.
Every line written stays in conversation history and retransmits on
every subsequent turn — ~$1/session in wasted tokens.

Fix: hooks now say "saved in background, no action needed" and use
decision: allow instead of block. The agent continues working without
interruption. All filing happens via the background pipeline.

Also updated hooks README with:
- Known limitation: hooks require session restart after install
- Updated cost section: zero tokens, background-only

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: use microsecond timestamp and full content hash in diary entry ID (#819)

* fix: remove unused import 'main' from mempalace/__init__.py

Removed the 'main' import from `mempalace/__init__.py` and updated
`pyproject.toml` to point the script entry point directly to
`mempalace.cli:main`. This ensures the CLI remains functional while
improving code hygiene.

Co-authored-by: igorls <[email protected]>

* merge: full hardened stack + rewrite fact_checker around actual KG API

Merges the full hardened stack (up through #791 drawer-grep) and turns
fact_checker from "dead code hidden behind bare except" into an
actually-working offline contradiction detector with tests.

## Dead paths the PR body advertised but the code never executed

Both buried by a single outer ``except Exception: pass``:

  * ``kg.query(subject)`` — ``KnowledgeGraph`` has no ``query()`` method;
    it has ``query_entity()``. The attribute error was silently swallowed
    and the entire KG branch always returned ``[]``. Now using
    ``kg.query_entity(subject, direction="outgoing")`` with proper
    handling of the ``predicate``/``object``/``current``/``valid_to``
    fields the real API returns.
  * ``KnowledgeGraph(palace_path=palace_path)`` — the constructor's only
    kwarg is ``db_path``. Passing ``palace_path`` raised TypeError,
    silently swallowed. Now computing the db_path correctly from
    ``<palace>/knowledge_graph.sqlite3``, matching the convention the
    MCP server already uses.

## Contradiction logic rewritten

The previous ``if kg_pred in claim and fact.object not in claim`` only
fired when text used the SAME predicate word as the KG fact — the exact
opposite of the stated use case ("Bob is Alice's brother" when KG says
husband" would NOT have fired). Replaced with a proper parse → lookup
→ compare pipeline:

  * ``_extract_claims`` parses two surface forms ("X is Y's Z" and
    "X's Z is Y") into ``(subject, predicate, object)`` triples.
  * ``_check_kg_contradictions`` pulls the subject's outgoing facts
    and flags two classes:
      - ``relationship_mismatch`` when a current KG fact matches the
        same ``(subject, object)`` pair but with a different predicate.
      - ``stale_fact`` when the exact triple exists but is
        ``valid_to``-closed in the past.
  * Stale-fact detection is now implemented (the PR body claimed it;
    the old code silently didn't implement it).

## Performance fix — O(n²) → O(mentioned × n)

``_check_entity_confusion`` previously computed Levenshtein for every
pair of registered names on every ``check_text`` call. For 1,000
registered names that's ~500K edit-distance calls per hook invocation.
Now we first identify which registry names actually appear in the text
(single regex scan), then only compute edit distance between mentioned
and unmentioned names. Pinned by a test that asserts <200ms on a 500-
name registry with zero mentions.

Also: when *both* similar names are mentioned in the text, we no
longer flag them — the user clearly knows they're different people.

## Shared entity-registry loader

``mempalace/miner.py`` already had an mtime-cached loader for
``~/.mempalace/known_entities.json``. fact_checker had a duplicate
implementation that leaked file handles and ignored caching. Extended
miner's cache to expose both the flat set (``_load_known_entities``)
and the raw category dict (``_load_known_entities_raw``); fact_checker
now imports the latter. No more double disk reads, no more handle leak.

## Tests — 24 cases in tests/test_fact_checker.py

All three detection paths + both dead-code regressions:
  * ``test_kg_init_uses_db_path_not_palace_path_kwarg`` — pins the
    correct KG constructor signature so the ``palace_path=`` bug can't
    come back.
  * ``test_relationship_mismatch_detected`` — the headline example from
    the PR body now actually fires.
  * ``test_stale_fact_detected`` — valid_to-closed triple is flagged.
  * ``test_current_fact_same_triple_is_not_flagged`` — no false positive
    on a still-valid match.
  * ``test_performance_bounded_by_mentioned_names`` — 500-name registry,
    zero mentions, <200ms. Regression for the O(n²) blowup.
  * ``test_no_false_positive_when_both_names_mentioned`` — Mila and
    Milla in the same text is fine.
  * Plus claim extraction, flatten_names shapes, CLI exit code, empty
    text handling, missing-palace graceful fallback, registry-dict
    shape support.

785/785 suite pass. ruff + format clean on CI-pinned 0.4.x.

* Optimize entity detection with regex caching and pre-compilation

- Use functools.lru_cache to cache compiled patterns for entity names.
- Pre-compile static pronoun patterns into a single regex.
- Remove redundant .lower() calls in score_entity loop.

Co-authored-by: igorls <[email protected]>

* docs: fix stale milla-jovovich org URLs in website and plugin manifests (#787)

Follow-up to #766 which covers version.py, pyproject.toml, README,
CHANGELOG, and CONTRIBUTING. These 11 files still had the old org
name in URLs:

- website/ (VitePress config + 6 docs pages)
- .claude-plugin/ (plugin.json repository, README marketplace command)
- .codex-plugin/ (plugin.json URLs, README links)

Author name fields are intentionally unchanged.

* test: make diary state path assertion platform-neutral

The Windows CI job failed on:

    assert '/.mempalace/state/' in str(state_path)

because Windows uses ``\`` as the path separator, so the substring
never matches. The behavior under test (state file lives outside the
diary dir, under ``~/.mempalace/state/``) is already correct on both
platforms — only the assertion was Unix-only.

Switch to ``state_path.parent`` comparisons that work on any OS.

* test: serialize mine_lock concurrency test with multiprocessing

The macOS CI job failed ``test_lock_blocks_concurrent_access`` because
``fcntl.flock`` on BSD/macOS is per-*process*, not per-FD: two threads
in the same process both acquire even when they open their own file
descriptors. The test passed on Linux (per-FD flock) and Windows
(per-FD ``msvcrt.locking``) but was never actually exercising the
lock's real contract.

``mine_lock`` is designed to serialize multi-*agent* access — i.e.,
separate processes, not threads. Switch the test to
``multiprocessing.get_context('spawn')`` with a module-level worker
(so the spawn pickles cleanly) so it:

  1. reflects the actual use case (one lock per mining process);
  2. passes on all three OSes without flock-semantics branching;
  3. catches real regressions (a broken lock would now let both
     processes through, exactly what we care about).

Hold time bumped to 0.3s and the "wait until p1 acquires" delay to
0.2s to tolerate spawn's higher startup latency on macOS/Windows.

* test: verify mine_lock via disjoint critical-section intervals

The previous revision used multiprocessing but still relied on timing
("second process waited at least N seconds") which flakes on CI where
spawn overhead eats into the hold window. Linux CI observed the second
process report a 0.088s wait — below the 0.1s threshold — even though
the lock behavior was correct; spawn was just slow enough that the
first process had nearly finished holding when the second got past
its own spawn.

Switch to effect-based verification: each worker logs its
[enter_time, exit_time] inside the critical section, and the test
asserts the two intervals are disjoint after sorting. A broken lock
would produce overlapping intervals regardless of spawn latency; a
working lock cannot.

Also removed the mp.Queue since we no longer pass timing data back.

* Fix: ruff format with CI-pinned version (0.4.x)

* fix: README audit — 42 TDD tests + hall detection + 7 claim fixes (#835)

* fix: README audit — match every claim to shipped code + add hall detection

TDD audit: wrote 42 tests verifying README claims against codebase.
Fixed all 7 failures:

1. Tool count: 19 → 29 (10 tools were undocumented)
2. Added tool table rows for tunnels, drawer management, system tools
3. Version badge: 3.1.0 → 3.2.0
4. dialect.py file reference: "30x lossless" → "AAAK index format for closet pointers"
5. Wake-up token cost: "~170 tokens" → "~600-900 tokens" (matches layers.py)
6. pyproject.toml version in project structure: v3.0.0 → v3.2.0
7. Hall detection: added detect_hall() to miner.py — drawers now tagged
   with hall metadata so palace_graph.py can build hall connections

New code:
- miner.py: detect_hall() — keyword scoring against config hall_keywords,
  writes hall field to every drawer's metadata
- tests/test_hall_detection.py — 12 TDD tests (written before code)
- tests/test_readme_claims.py — 42 TDD tests verifying README accuracy

859/859 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: resolve ruff lint — unused imports and variables

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: ruff format with CI-pinned 0.4.x

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: use conftest fixtures in hall tests for Windows compat

Windows CI fails with NotADirectoryError when ChromaDB tries to
write HNSW files in short-lived TemporaryDirectory. Use conftest
palace_path and tmp_dir fixtures instead — same pattern as all
other tests that touch ChromaDB.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address Igor's review — convo_miner halls, cached config, markdown typo

TDD: wrote tests for convo_miner hall metadata and config caching
BEFORE verifying the code changes.

1. README markdown typo: extra ** in wake-up token row (line 195)
2. convo_miner.py: added _detect_hall_cached() — conversation
   drawers now get hall metadata (was missing, Igor caught it)
3. miner.py + convo_miner.py: cached hall_keywords at module level
   so config.json isn't re-read per drawer during bulk mine
4. New tests: TestConvoMinerWritesHalls, TestDetectHallCaching

861/861 tests pass. ruff clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* fix(website): update vitepress base url for custom domain

* chore(release): bump version strings to 3.3.0 and curate CHANGELOG

Prepare develop for the 3.3.0 release cycle.

Version bumps:
- mempalace/version.py: 3.2.0 -> 3.3.0
- pyproject.toml: 3.2.0 -> 3.3.0
- README.md: pyproject.toml label and shields.io badge
- uv.lock: mempalace 3.0.0 -> 3.3.0 (also fills in resolved dev/extras)

CHANGELOG.md:
- Close out the stale [Unreleased] section as [3.2.0] - 2026-04-12
  (v3.2.0 was tagged on that date but the release flip was never made)
- Add a fresh [Unreleased] - v3.3.0 section covering the 49 commits
  since v3.2.0: closet layer, BM25 hybrid search, entity metadata,
  diary ingest, cross-wing tunnels, drawer-grep, offline fact checker,
  LLM-based closet regen, hall detection, cosine-distance fix,
  multi-agent locking, README audit, etc.
- Adopt Keep a Changelog + SemVer framing
- Add version compare reference links at the bottom
- Fix stale milla-jovovich/mempalace preamble URL to MemPalace/mempalace

---------

Co-authored-by: MSL <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: eblander <[email protected]>
Co-authored-by: shafdev <[email protected]>
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: mvalentsev <[email protected]>
Co-authored-by: Dominique Deschatre <[email protected]>

* ci: serve docs from develop only

Docs deploy to GitHub Pages from develop for faster iteration cycles.
Main was failing the deploy step with "Branch 'main' is not allowed to
deploy to github-pages due to environment protection rules" on every
release merge (v3.2.0, v3.3.0) — noise without signal, since docs
weren't meant to serve from main anyway.

Removes main from both the push trigger and the deploy-job guard.
Develop continues to deploy as before; manual dispatch still works.

* fix(status): paginate metadata fetch to support large palaces

`col.get(limit=total)` causes SQLite "too many SQL variables"
on palaces with >10k drawers (#802) and on older versions the
hardcoded limit=10000 silently truncated the count (#850).

Paginate in 5k batches using offset and aggregate wing/room
counts incrementally. Also use `col.count()` for the header
instead of `len(metas)` so the displayed total is always correct.

Tested on a 122,686-drawer palace.

Fixes #850
Related: #802, #723

* refactor: route all chromadb access through ChromaBackend

Prerequisite for RFC 001 (plugin spec, #743). Removes every direct
`import chromadb` outside the ChromaDB backend itself so the core
modules depend only on the backend abstraction layer.

Extends ChromaBackend with make_client, get_or_create_collection,
delete_collection, create_collection, and backend_version. Adds
update() to the BaseCollection contract. Non-backend callers
(mcp_server, dedup, repair, migrate, cli) now go through the
abstraction; tests patch ChromaBackend instead of chromadb.

With this landed, the RFC 001 spec can be enforced and PalaceStore
(#643) can ship as a plugin without touching core modules.

* fix: update stale org URLs in pyproject.toml and README (#787)

* fix: harden hooks against shell injection, path traversal, and arithmetic injection

save_hook.sh:
- Coerce stop_hook_active to strict True/False before eval to prevent
  command injection via crafted JSON (e.g. "$(curl attacker.com)")
- Validate LAST_SAVE as plain integer with regex before bash arithmetic
  to prevent command substitution via poisoned state files

hooks_cli.py:
- Add _validate_transcript_path() that rejects paths with '..'
  components and non-.jsonl/.json extensions
- _count_human_messages() now uses the validator, returning 0 for
  invalid paths instead of opening arbitrary files

Tests:
- Path traversal rejection (../../etc/passwd)
- Wrong extension rejection (.txt, .py)
- Valid path acceptance (.jsonl, .json)
- Empty string handling
- Shell injection in stop_hook_active field

Refs: MemPalace/mempalace#809

* fix: add logging on rejected transcript paths and platform-native path test

- _count_human_messages() now logs a WARNING via _log() when a
  non-empty transcript_path is rejected by the validator, making
  silent auto-save failures diagnosable via hook.log
- Add test for platform-native paths (backslashes on Windows) to
  verify _validate_transcript_path works cross-platform
- Add test verifying the warning log is emitted on rejection

Refs: MemPalace/mempalace#809

* Increase visibility of fake website caution

Noticed a URL 
```
hXXps://www.mempalace[.]tech/
```

Though the README currently warns, it is perhaps best to surface it at urgency level at the top of the README.

* fix: use permissive validator for KG entity values (closes #455)

sanitize_name rejects commas, colons, parentheses, and slashes — characters
that commonly appear in knowledge graph subject/object values. Adds
sanitize_kg_value for KG entity fields (subject, object, entity) while
keeping sanitize_name for predicates and wing/room names.

* chore: bump plugin manifests to 3.3.0 and fix owner URL

Aligns marketplace.json and both plugin.json files with version.py /
pyproject.toml (already at 3.3.0) so `/plugin update` reflects the
v3.1.0/v3.2.0/v3.3.0 tags that had been landing without manifest bumps.

Also updates marketplace.json `owner.url` from the stale
github.com/milla-jovovich path to the current github.com/MemPalace org.

Refs #874

* ci: add version guard to catch tag/manifest drift

Fails a tag push if `vX.Y.Z` does not match `mempalace/version.py` (the
single source of truth per CLAUDE.md), and fails PRs that touch any
version file without keeping all five in sync (pyproject.toml,
version.py, .claude-plugin/marketplace.json, .claude-plugin/plugin.json,
.codex-plugin/plugin.json).

Prevents the class of bug described in #874, where v3.1.0/v3.2.0/v3.3.0
tags all landed pointing at commits that still carried manifest version
3.0.14, blocking `/plugin update` for end users.

Refs #874

* ci: let semver pre-release tags bypass strict manifest match

Tags matching `vX.Y.Z-*` (e.g. v3.4.0-rc1, v1.0.0-beta.2) are treated as
internal/staging builds. They skip the tag-vs-manifest check because
pre-releases do not flow to end users via `/plugin update`, which reads
the manifest on the default branch.

Stable tags `vX.Y.Z` still require all five version sources to match
exactly, so the protection against the #874 drift remains intact. The
cross-file consistency check on PRs is unchanged — all manifests must
still agree with mempalace/version.py whenever any version file moves.

* fix: ship CNAME in Pages artifact to pin custom domain

Adds website/public/CNAME containing `mempalaceofficial.com` so the
VitePress build output always includes /CNAME in the Pages artifact.
Without this, the custom-domain setting is only held in the repo's
Pages API config — if it ever drifts (manual edit, org move, workflow
change), the site reverts to <org>.github.io with no record in source.

Note: this does not fix the current site outage. The root cause is DNS
— mempalaceofficial.com has no A/AAAA/CNAME records pointing at GitHub
Pages IPs. That has to be fixed at the registrar. This commit is the
belt-and-suspenders so that once DNS is back, the domain is pinned in
source and the next workflow refactor can't accidentally drop it.

* docs: tighten SECURITY.md with real version policy and GHPVR-only channel

Builds on @Yorji-Porji's draft by fixing three issues before it lands:

- Replace the `< 1.0.0` placeholder table with MemPalace's actual
  support policy: current major (3.x) receives fixes, 2.x and earlier
  do not.
- Remove the `[Insert Maintainer Email Here]` placeholder and the
  email fallback. GitHub Private Vulnerability Reporting is enabled
  on this repo; the policy points there exclusively so there is no
  risk of a researcher emailing a dead address.
- Drop the meta-note ("Adjust the table above…") that was an
  instruction to the maintainer, not policy text.

Structure, triage timelines, and credit language are kept as drafted.

* fix: allow mining directories without local mempalace.yaml

When no mempalace.yaml or mempal.yaml exists in the source directory,
return a default config (wing = directory name, room = general) instead
of calling sys.exit(1). This lets users mine any directory into their
palace without requiring init first.

Closes #14.

* fix: remove unused sys import

* fix: send missing-yaml warning to stderr and flag basename collisions

Addresses review feedback on #604:

- Warning now goes to stderr instead of stdout so it doesn't mix with
  mine progress output when users pipe stdout elsewhere.
- Warning explicitly calls out that directories with the same basename
  will share a wing name, and suggests adding mempalace.yaml to
  disambiguate. Prevents silent content mixing across projects mined
  without yaml.

* docs: name official domain and specific impostors in scam alert

Replace the blanket ban on .tech/.io/.com domains with an allowlist
of real MemPalace surfaces (GitHub repo, PyPI, mempalaceofficial.com)
and call out mempalace.tech as the reported impostor. The blanket
.com ban would have flagged mempalaceofficial.com as fake once DNS
resolves (CNAME shipped in #877).

Also update the April 11 follow-up section to match so the two
notices no longer contradict each other.

* perf: optimize regex compilation in entity extraction

Move regular expression compilation to the module level in `dialect.py` to prevent repeated parsing during loop execution.

Co-authored-by: igorls <[email protected]>

* feat: add MEMPAL_VERBOSE toggle — developers see diaries in chat (#871)

export MEMPAL_VERBOSE=true  → hook blocks, agent writes diary in chat
export MEMPAL_VERBOSE=false → silent background save (default)

Developers need to see code and diaries being written.
Regular users want zero chat clutter. Now both work.

TDD: tests written first, failed, code fixed, tests pass.

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* feat: add VSCode devcontainer matching CI environment

Contributors now get a one-click dev environment that mirrors CI exactly:
Python 3.11 (middle of the 3.9/3.11/3.13 matrix), ruff pinned to the same
>=0.4.0,<0.5 range CI enforces, and pre-commit hooks auto-installed from
the existing .pre-commit-config.yaml.

Pinning ruff in post-create.sh is the load-bearing piece: pyproject only
sets a floor, so without the pin the ruff extension would install 0.15.x
and phantom-fail lint against CI's 0.4.x.

* fix: add missing self._lock to query_relationship, timeline, stats in KnowledgeGraph

* fix: replace invalid 'decision: allow' with {} in hooks

Closes #872. The top-level decision field only recognizes "block".
To not block, return empty JSON {}. "allow" was silently ignored
by Claude Code, causing unpredictable behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: add missing self._lock to KnowledgeGraph.close()

TDD: test first, failed, fixed, passed.

Igor fixed query_relationship/timeline/stats in an earlier commit.
close() was the last method touching self._connection without
holding the lock.

Closes #883.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* benchmarks: add --llm-backend ollama for non-Anthropic rerank

The rerank pipeline was hardcoded to Anthropic's /v1/messages.
Add a backend flag so the same code path can be exercised with
any OpenAI-compatible endpoint — local Ollama, Ollama Cloud,
or any gateway that speaks /v1/chat/completions.

Enables independent verification of the "100% with Haiku rerank"
claim by running the full benchmark with a different LLM family
(e.g. minimax-m2.7:cloud) and zero Anthropic dependency.

Both longmemeval_bench.py and locomo_bench.py:
 - llm_rerank*() gain backend= / base_url= kwargs
 - CLI: --llm-backend {anthropic,ollama}, --llm-base-url
 - API key required only when backend=anthropic (diary/palace modes still require it)
 - Parse last integer in response (reasoning models emit multi-int output)
 - Fallback to message.reasoning when content is empty
 - Raise max_tokens to 1024 for reasoning models

* benchmarks: apply ruff-format to llm_rerank (trivial line wrap)

* benchmarks: add v3.3.0 reproduction results + 50/450 split

Addresses #875: every internal BENCHMARKS.md claim reproduced
on Linux x86_64 (v3.3.0 tag, deterministic ChromaDB embeddings,
seed=42 for the LongMemEval dev/held-out split).

Scorecard — all reproduce exactly:

  LongMemEval
    raw R@5                            96.6% (500/500)   ✅
    hybrid_v4 held-out 450 R@5         98.4% (442/450)   ✅
    hybrid_v4 + minimax rerank R@5     99.2% (496/500)   *
    hybrid_v4 + minimax rerank R@10   100.0% (500/500)   *

  LoCoMo (session, top-10)
    raw                                60.3% (1986q)     ✅
    hybrid v5                          88.9% (1986q)     ✅

  ConvoMem all-categories (250 items)   92.9%            ✅
  MemBench all-categories (8500)        80.3%            ✅

* The minimax-m2.7:cloud rerank run replicates the "100%" claim
  with a different LLM family (no Anthropic dependency). R@10 is
  a perfect reproduction; R@5 misses 4 questions that the
  published Haiku run caught — consistent with BENCHMARKS.md's own
  disclosure that hybrid_v4 includes three question-specific fixes
  developed by inspecting misses, i.e. teaching to the test.

The committed 50/450 split is the deterministic (seed=42) split
BENCHMARKS.md references but wasn't previously in the repo.

Full result JSONLs include every question, every retrieved id,
and every score — auditable end-to-end.

* docs: slim README and move corrections/notices to docs/HISTORY.md

Addresses #875. The previous README was 755 lines mixing six purposes
(scam alert, hero, two mea-culpa notes, install guide, architecture
explainer, API reference, file map). Rework it as a pure entry point:
what MemPalace is, how to install, honest benchmark numbers, links to
the website for concept/architecture documentation.

Key content changes:
 - Drop the "highest-scoring AI memory system ever benchmarked" framing.
 - New tagline: "Local-first AI memory. Verbatim storage, pluggable
   backend, 96.6% R@5 raw on LongMemEval — zero API calls." Avoids
   naming a specific vector-store implementation since the backend is
   pluggable (see mempalace/backends/base.py).
 - Remove the cross-system comparison table. Retrieval recall (R@5)
   and end-to-end QA accuracy are different metrics and are not
   comparable; placing MemPalace's R@5 next to competitor QA accuracy
   under a single column header was a category error.
 - The "100%" LongMemEval headline is no longer the lead. The honest
   held-out figure is 98.4% R@5 on 450 unseen questions. The rerank
   pipeline reaches >=99% with any capable LLM (reproduced with
   Claude Haiku, Sonnet, and minimax-m2.7 via Ollama) — pipeline-level,
   not model-specific.
 - Benchmark reproduction commands now reference the correct repo
   (MemPalace/mempalace, not the defunct aya-thekeeper/mempal branch).

New file: docs/HISTORY.md as the canonical home for post-launch
corrections, public notices, and retractions. Contains verbatim:
 - 2026-04-14 note on this rewrite (links to #875)
 - 2026-04-11 impostor-domain notice (moved from README header)
 - 2026-04-07 "A Note from Milla & Ben" (moved from README body)

README keeps a one-line scam-alert callout that links to
docs/HISTORY.md for the full timeline.

* docs(website): align mempalaceofficial.com with honest benchmarks

Part of #875. Bring the VitePress site into line with the new README
and the reproducibility scorecard: drop category-error comparisons,
drop retracted claims, retain only metrics and caveats that survive
audit.

website/index.md
 - New tagline matches README (local-first, verbatim, pluggable backend,
   96.6% R@5 raw, zero API calls).
 - Replace the "MemPalace hybrid 100% / Supermemory ~99% / Mastra
   94.87% / Mem0 ~85%" comparison table with a single honest table
   showing MemPalace's own retrieval-recall numbers (raw 96.6%,
   hybrid v4 held-out 98.4%). Add an explicit sentence explaining why
   we no longer publish a cross-system table on the landing page
   (retrieval recall vs QA accuracy are different metrics).
 - Soften the "ChromaDB-powered vector search" feature blurb to be
   backend-agnostic, since the retrieval layer is pluggable.

website/reference/benchmarks.md
 - Full rewrite of the retrieval-recall tables. No more "100%"
   headline; honest held-out 98.4% R@5 replaces it. Added the
   model-agnostic rerank result (99.2% R@5 / 100% R@10 with
   minimax-m2.7 via Ollama) to show the pipeline is not Haiku-specific.
 - Drop the LoCoMo "Hybrid v5 + Sonnet rerank (top-50) 100%" row.
   With per-conversation session counts of 19-32 and top_k=50, the
   retrieval stage returns every session by construction — the number
   measures an LLM's reading comprehension, not retrieval.
 - Drop the cross-system comparison tables. Link out to each project's
   own research page (Mastra, Mem0, Supermemory) for their published
   numbers and metric definitions.
 - Rewrite reproduction commands to use the correct repository and
   demonstrate the new --llm-backend ollama flag.

website/concepts/the-palace.md
 - Remove the "+34%" row / paragraph. Wing/room filtering is standard
   metadata filtering in the vector store, not a novel retrieval
   mechanism — the April-7 note already retracted that framing; this
   finishes the retraction on the website where it had remained.

website/guide/searching.md
 - Same treatment for "34% retrieval improvement". Reframe as
   operational scoping, not a novel boost.

website/reference/contributing.md
 - Update the "palace structure matters" bullet to reflect the same
   framing: scoping-not-magic.

website/concepts/knowledge-graph.md
 - Replace the MemPalace-vs-Zep feature matrix with a short "related
   work" note that links to Zep's own documentation for authoritative
   details on their deployment model. Avoids claims we cannot verify
   at source.

* docs: #875 follow-up — repo surfaces + reproduction URLs + CHANGELOG

Remaining in-repo surfaces carrying the same retracted or broken
claims as the public pages fixed in the previous two commits.

CONTRIBUTING.md
 - "Palace structure matters ... 34% retrieval improvement" → reframed
   as scoping (same rewording applied to the website equivalents).

benchmarks/BENCHMARKS.md
 - Add a prominent "Important caveat" block at the top of the
   "Comparison vs Published Systems" table explaining that R@5
   (retrieval recall) and QA accuracy are different metrics, with
   citations to Mastra, Mem0, and Supermemory's own published
   methodology pages. Annotate the specific competitor rows whose
   numbers are QA accuracy, not retrieval recall.
 - Annotate the `hybrid v4 + rerank 100%` row to note that the 99.4
   → 100 step was tuned on 3 specific wrong answers (already disclosed
   further down in the doc under "Benchmark Integrity"); the honest
   hybrid figure is held-out 98.4%.
 - Fix the broken clone URL — `aya-thekeeper/mempal` no longer points
   at anything; now `MemPalace/mempalace`.

benchmarks/README.md + benchmarks/HYBRID_MODE.md
 - Same clone-URL fix applied.

CHANGELOG.md
 - Add a ### Documentation entry under [Unreleased] v3.3.0 that names
   #875 and summarises the scope of the rewrite.

* docs+tests: fix CI after README slim (#875)

The regression-guard tests added in #835 were pinned to the old
README shape (tool table + file-reference table). When #897 slimmed
the README and moved that content to the website, three tests
started failing:

  TestReadmeToolsExistInCode.test_every_readme_tool_exists_in_tools_dict
  TestNoUnlistedTools.test_no_undocumented_tools
  TestReadmeDialectNotLossless.test_readme_dialect_line_not_lossless

Changes in this commit:

1. Update the 3 tests to track the new canonical docs surfaces
   - Tool list -> website/reference/mcp-tools.md
     (tests parse `### \`mempalace_xxx\`` headings instead of
     markdown table rows).
   - dialect.py lossless disclaimer -> website/reference/modules.md
     (any line mentioning dialect.py must not also say "lossless").

2. Fix the website to make "no undocumented tools" true
   Add the 10 tools that existed in TOOLS but were missing from
   website/reference/mcp-tools.md (create_tunnel, delete_tunnel,
   follow_tunnels, list_tunnels, get_drawer, list_drawers,
   update_drawer, hook_settings, memories_filed_away, reconnect).
   Page header now correctly says "all 29 MCP tools".

3. Align pre-commit ruff pin to match CI (0.4.x)
   .pre-commit-config.yaml was pinning ruff v0.9.0, while
   .github/workflows/ci.yml installs ruff>=0.4.0,<0.5. The two
   formatters produce incompatible output (e.g. v0.9.0 reformats
   `assert (x), msg` -> `assert x, (msg)` in a way v0.4.x rejects),
   which would cause the pre-commit hook to modify files that CI
   then flags as unformatted. Pinning the hook to v0.4.10 keeps
   the dev loop and CI in lock-step.

Full suite: 887 passed, 0 failed.

* fix: address i18n review issues from PR #718

Three issues flagged by bensig on the i18n PR before merge:

1. ko.json: status_drawers used {drawers} instead of {count}, causing
   the Korean UI to show the raw template string instead of the actual
   drawer count.  All other 7 languages use {count}.

2. Test file was shipped inside the package at mempalace/i18n/test_i18n.py
   with a sys.path.insert hack.  Moved to tests/test_i18n.py per the
   project convention in AGENTS.md.

3. Dialect.from_config() passed lang=config.get("lang") which defaults
   to None, causing __init__ to inherit whatever language was loaded
   earlier via module-level state.  Now defaults to "en" explicitly so
   from_config is deterministic regardless of prior load_lang() calls.

Added two regression tests for the ko.json fix and the state leak.

* docs(cli): clarify that 'mempalace init' requires <dir> (#210) (#862)

Fixes #210.

The CLI requires a positional <dir> argument. Previous docs emphasized
that init 'sets up ~/.mempalace/' which misled users into expecting
no arguments. Now the docs show <dir> is required, offer '.' as the
usage for the current directory, and reword the description so the
project-directory scan is listed first.

* fix: make entity_registry.research() local-only by default (#811)

* fix: make entity_registry.research() local-only by default

research() previously called _wikipedia_lookup() unconditionally,
sending entity names to en.wikipedia.org on every uncached lookup.
This violates the project's local-first and privacy-by-architecture
principles documented in CLAUDE.md.

Changes:
- research() now returns "unknown" for uncached words by default
- New allow_network=True parameter required for Wikipedia lookups
- Wikipedia 404 now returns "unknown" instead of asserting "person"
  with 0.70 confidence, preventing entity registry poisoning
- Added privacy warning docstring to _wikipedia_lookup()
- Added tests for local-only default, opt-in network, 404 handling,
  and cache-not-persisted-on-local-only behaviour

Refs: MemPalace/mempalace#809

* fix: improve research() cache read path and deduplicate test mocks

- Use .get() instead of .setdefault() for cache reads in research()
  so the local-only path never mutates _data unnecessarily
- Move .setdefault() to the network-write path only
- Use result.setdefault() for word/confirmed keys to ensure
  consistent return shape across all _wikipedia_lookup error paths
- Extract duplicated mock_result dict into _MOCK_SAOIRSE_PERSON
  constant shared by 3 test functions

* fix: return empty status instead of error on cold-start palace (#830) (#831)

tool_status() called _get_collection() with the default create=False,
which throws when the ChromaDB collection does not exist yet (valid
palace, zero drawers). The exception was swallowed and status returned
"No palace found" even though init had completed successfully.

Switching to create=True bootstraps an empty collection on first
status call, matching what the write path already does.

Fix suggested by @hkevinchu in the issue.

* fix(searcher): guard against empty ChromaDB query results (#195) (#865)

Fixes #195.

When ChromaDB returns no documents (empty palace, or wing/room filter
that excludes everything), it returns the shape:

    {"documents": [], "metadatas": [], "distances": []}

Indexing `results["documents"][0]` blindly raises IndexError instead of
the expected 'no results' response. Affected: searcher.search(),
searcher.search_memories() (drawer + closet branches plus the
total_before_filter aggregate), and Layer3.search() / Layer3.search_raw().

Adds a tiny private helper `searcher._first_or_empty(results, key)` that
safely extracts the inner list, returning [] for any of: missing key,
empty outer list, [None], or [[]]. layers.py imports the same helper to
avoid duplicating the guard.

Tests: tests/test_empty_chromadb_results.py covers all observed shapes
plus a documentation-style test that pins the original IndexError so
future readers understand why the helper exists.

* fix(init): auto-add per-project files to .gitignore in git repos (#185) (#866)

Partially addresses #185.

`mempalace init <dir>` writes `mempalace.yaml` and `entities.json` into
the project root. When <dir> is a git repository, those files have no
default protection and risk being committed by accident — the loudest
concern in the original report.

This PR adds `_ensure_mempalace_files_gitignored()` which runs at the
end of cmd_init: if <dir>/.git exists, append the two filenames to
.gitignore (creating it if necessary) under a clearly-marked block.

The helper is conservative:
- only runs when <dir>/.git is present (no-op for non-git projects)
- skips entries already present (no duplicates)
- preserves existing .gitignore content
- handles files without trailing newlines

This does NOT relocate the files to ~/.mempalace/wings/<wing>/ as the
issue's 'Expected' section proposes — that's a behavioral change with
miner/config implications and warrants a separate design discussion.
The gitignore safeguard removes the immediate risk without breaking any
existing flow.

Tests: 5 cases in tests/test_init_gitignore_protection.py covering
no-op, fresh creation, partial append, idempotency, and missing-newline
edge case.

* fix(mcp): redirect stdout to stderr during import to protect JSON-RPC channel (#225) (#864)

* fix(mcp): redirect stdout to stderr during import to protect JSON-RPC channel (#225)

Fixes #225.

Several transitive dependencies (chromadb, onnxruntime, posthog) print
banners and warnings to stdout — sometimes at the C level — during the
mcp_server import chain. Because the MCP protocol multiplexes JSON-RPC
over stdio, any non-JSON output on stdout corrupted the message stream
and broke Claude Desktop's parser with errors like:

  MCP mempalace: Unexpected token '*', "**********"... is not valid JSON
  MCP mempalace: Unexpected token 'E', "EP Error D"... is not valid JSON
  MCP mempalace: Unexpected token 'F', "Falling ba"... is not valid JSON

Reproduced on Windows 11 with mempalace 3.0.0 / Python 3.10 / Claude
Desktop 1.1062.0.

Fix: at module load, redirect stdout to stderr at both the Python level
(sys.stdout = sys.stderr) and the file-descriptor level (os.dup2(2, 1))
to catch C-level prints, while preserving the real stdout for later
restore. main() calls _restore_stdout() right before entering the
protocol loop so JSON-RPC responses still go to the real stdout.

Adds tests/test_mcp_stdio_protection.py with three regression tests:
- module-level redirect is in place after import
- _restore_stdout() restores the original stdout (idempotent)
- 'python -m mempalace.mcp_server' with empty stdin emits no stdout

* style: reformat with ruff 0.4 (CI version) for #225

* fix(hooks): stop precompact hook from blocking compaction (#856, #858) (#863)

* fix(hooks): stop precompact hook from blocking compaction

The precompact hook unconditionally returned {"decision": "block"},
which in Claude Code means "cancel compaction" with no retry mechanism.
This made /compact permanently broken for all plugin users.

Changed hook_precompact() to mine the transcript synchronously (so data
lands before compaction) and return {"decision": "allow"}. This matches
the standalone bash hook in hooks/ which already uses allow.

Also extracted _get_mine_dir() and _mine_sync() helpers so precompact
can mine from the transcript directory, not just MEMPAL_DIR.

Stop hook behavior is unchanged -- left for #673 which implements the
full silent save path.

Closes #856, closes #858.

* fix: use empty JSON instead of invalid \"allow\" decision value

Claude Code only recognizes \"block\" as a top-level decision value.
\"allow\" is a permissionDecision value for PreToolUse hooks, not a
valid top-level decision. The correct way to not block is to return
empty JSON. Caught by #872.

* feat: include created_at timestamp in search results (#846)

* feat: include created_at timestamp in search results (closes #465)

Surface the existing filed_at metadata as created_at in search result
objects returned by search_memories(). Enables temporal reasoning over
search hits without additional queries.

* Feat: add fallback for missing filed_at metadata

* fix: add provenance header and speaker IDs to Slack transcript imports (#815)

* fix: add provenance header and speaker IDs to Slack transcript imports

Slack exports are multi-party chats where no speaker is inherently
the "user" or "assistant". The parser previously assigned these roles
purely by position, allowing a crafted export to place attacker text
in the "user" role — making it appear as the memory owner's words
in all future retrieval (data poisoning via stored memory).

Changes:
- Add provenance header marking Slack transcripts as multi-party
  with positional (unverified) role assignment
- Prefix each message with the original speaker ID ([U1], [U2], etc.)
  so downstream consumers can distinguish authors
- Keep user/assistant role alternation for exchange-pair chunking
  compatibility with convo_miner.py

Tests:
- Provenance header presence and content
- Speaker ID preservation in output
- Attacker-first-message attribution verification

Refs: MemPalace/mempalace#809

* fix: move Slack provenance to footer, sanitize speaker IDs, extract constant

- Move provenance notice from header to footer to prevent it becoming
  a standalone ChromaDB drawer via paragraph chunking on exports
  with fewer than 3 exchange pairs (violates verbatim-always principle)
- Sanitize speaker user_id/username: strip brackets, newlines, and
  control characters to prevent chunk-boundary injection via crafted
  Slack exports
- Extract header string to _SLACK_PROVENANCE_FOOTER module constant,
  consistent with _TOOL_RESULT_* constants pattern; tests import it
  instead of duplicating the literal

Refs: MemPalace/mempalace#809

* fix: restrict file permissions on sensitive palace data (#814)

* fix: restrict file permissions on sensitive palace data

On Linux with default umask (022), several files and directories
containing personal data were created world-readable. This patch
applies chmod 0o700 to directories and 0o600 to files immediately
after creation, wrapped in try/except for Windows compatibility.

Files hardened:
- hooks_cli.py: hook_state/ directory and hook.log
- entity_registry.py: entity_registry.json (names, relationships)
- knowledge_graph.py: knowledge_graph.sqlite3 parent directory
- exporter.py: export output directory and wing subdirectories
- config.py: people_map.json (name mappings)
- mcp_server.py: WAL file creation uses atomic os.open (TOCTOU fix)

Refs: MemPalace/mempalace#809

* fix: avoid redundant chmod calls on hot paths

- hooks_cli.py: chmod STATE_DIR and hook.log only on first creation,
  not on every _log() call (hooks fire on every Stop event)
- exporter.py: track created wing dirs to skip redundant makedirs +
  chmod on the same directory across batches
- mcp_server.py: remove redundant _WAL_FILE.chmod after os.open
  already set mode=0o600 atomically

Refs: MemPalace/mempalace#809

* test: add palace_graph tunnel helper coverage

Adds focused tests for explicit tunnel helpers in `mempalace/palace_graph.py`.

Covered:
- `_load_tunnels`
- `_save_tunnels`
- `create_tunnel`
- `list_tunnels`
- `delete_tunnel`
- `follow_tunnels`

* refactor(entity_detector): make multi-language extensible via i18n JSON

Move all entity-detection lexical patterns (person verbs, pronouns,
dialogue markers, project verbs, stopwords, candidate character class)
out of hardcoded module-level constants and into the entity section of
each locale's JSON in mempalace/i18n/. Adds a languages parameter to
every public function so callers union patterns across the desired
locales. The default stays ("en",), so all existing callers and tests
behave unchanged.

Also adds:
- get_entity_patterns(langs) helper in mempalace/i18n/ that merges
  patterns across requested languages, dedupes lists, unions stopwords,
  and falls back to English for unknown locales
- MempalaceConfig.entity_languages property + setter, with env var
  override (MEMPALACE_ENTITY_LANGUAGES, comma-separated)
- mempalace init --lang en,pt-br flag (persists to config.json)
- Per-language candidate_pattern so non-Latin scripts (Cyrillic,
  Devanagari, CJK) can register their own character classes instead of
  being silently dropped by the ASCII-only [A-Z][a-z]+ default
- _build_patterns LRU cache keyed by (name, languages) so multi-language
  callers don't poison each other's cache slots

Why now: the open language PRs (#760 ru, #773 hi, #778 id, #907 it) only
add CLI strings via mempalace/i18n/. PR #156 (pt-br) is the first that
needed entity_detector changes and inlined a _PTBR variant of every
constant. That doesn't scale past 2-3 languages — every text gets
checked against every language's patterns regardless of relevance, and
candidate extraction still drops accented and non-Latin names.

This PR sets the standard so future locale contributors only edit one
JSON file (no Python changes), and entity detection scales linearly
with how many languages a user actually enabled, not how many ship.

* test: document orphan-locale recovery for _temp_locale helper

* feat: add Russian language support to i18n module

Add ru.json with full Russian translations for CLI strings, palace
terminology, AAAK compression instruction, and regex patterns for
topic/action extraction with Cyrillic character classes.

No code changes needed -- the i18n module auto-discovers language
files via *.json glob in the i18n directory.

* feat(i18n): add entity detection section to Russian locale

Cyrillic candidate/multi-word patterns, person-verb patterns
(сказал, спросил, ответил, etc.), pronoun patterns, dialogue
markers, direct address, and Russian stopwords.

Follows the i18n entity framework from #911.

* fix(i18n): apply review feedback on ru.json (#760)

- mine_skip: "повторной раскопки" -> "повторной обработки"
- quote_pattern: add Russian guillemet quotes «»

Co-Authored-By: almirus <[email protected]>

* feat(i18n): expand Russian entity stopwords with prepositions and conjunctions

Adds 34 prepositions and conjunctions to reduce false positives
in entity detection when these words appear sentence-initial.

Co-Authored-By: almirus <[email protected]>

* feat: add italian i18n support

* feat: add italian entity patterns

* Updated hi.json to support infra for entity,pronoun_patterns,dialogue_patterns,direct_address_pattern, project_verb_patterns and stopwords

* feat(i18n): add Brazilian Portuguese locale with entity detection (closes #117)

CLI strings, AAAK instruction, regex patterns, and entity section
with person-verb, pronoun, dialogue, and candidate patterns for
Latin+diacritics names (Joao, Ines, Angela).

Follows the i18n entity framework from #911.

* fix(i18n): address review feedback on pt-br.json

- dialogue_patterns[0]: remove stray \" before > (fixes markdown quote matching)
- entity stopwords: add 40 prepositions, conjunctions, and common words to reduce false positives
- pronoun_patterns: add 2nd-person (você/vocês) and possessives (seu/sua/seus/suas)

* feat(cli): add version display and version flag to CLI

Introduces a version label to the command-line interface, displaying the current MemPalace version in the help text. Adds a `--version` flag to allow users to easily check the version and exit.

* fix(i18n): resolve language codes case-insensitively (#927)

BCP 47 language tags are case-insensitive (RFC 5646 §2.1.1) but the
locale files mix conventions (pt-br.json vs zh-CN.json). On
case-sensitive filesystems, '--lang PT-BR' or '--lang zh-cn' silently
missed the file, _load_entity_section returned {}, and entity
detection ran in English with no warning.

The cache key in get_entity_patterns was built from raw input, so
('PT-BR',) and ('pt-br',) produced two distinct entries, both wrong.

Add _canonical_lang(lang) that resolves any casing to the on-disk
filename stem via lowercase comparison, and route load_lang,
_load_entity_section, and the cache key through it.

Closes #927

* fix(i18n): use Optional[str] for Python 3.9 compatibility

PEP 604 union syntax (str | None) requires Python 3.10+. The project
supports 3.9 per CI matrix, so use typing.Optional instead.

* fix(entity_detector): script-aware word boundaries for combining-mark scripts

Python's \b is a \w/non-\w transition. Devanagari vowel signs (matras)
like ा ी ु are Unicode category Mc (Mark, Spacing Combining) — not \w.
This means \b splits mid-word on every matra: names like अनीता (Anita)
truncate to अनीत, and person-verb patterns like \bराज\s+ने\s+कहा\b
never match because \b fails after the final matra of कहा.

Same issue affects Arabic, Hebrew, Thai, Tamil, and every other script
whose words contain combining marks.

Fix: locales with combining-mark scripts declare a boundary_chars field
in their entity section (e.g. "\\w\\u0900-\\u097F" for Hindi). The i18n
loader replaces every \b in that locale's patterns with a script-aware
lookaround that treats the declared characters as "inside-word", and
pre-wraps candidate/multi_word patterns with the same boundary.

Default behavior (no boundary_chars) keeps standard \b — en, pt-br, ru,
it are unchanged.

Changes:
- mempalace/i18n/__init__…
@jphein jphein deleted the feat/deterministic-hook-save branch April 25, 2026 14:53
jphein added a commit to jphein/mempalace that referenced this pull request Apr 26, 2026
Fork-main now carries:
- MemPalace#1087 rewrite (cmd_purge: collection.delete(where=...) instead of nuke-and-rebuild) — commit 366a9ad
- MemPalace#1094 cherry-pick (coerce None metadatas at chromadb boundary) — commit 43d728d

YAML manifest gains 2 entries; FORK_CHANGELOG regenerated. README test
count bumped 1372 → 1383. CLAUDE.md gets:
- row updated for MemPalace#1087 (rewrite per @igorls's review)
- new "Closed by jphein-with-triage" subsection noting MemPalace#622 closure
  (architectural concern resolved by MemPalace#673; verified before clicking)

Triage perms make audits load-bearing. New feedback memory:
feedback_audit_comments_with_triage_perms.md — verify PR/commit claims
via gh before posting, gh api PATCH supports edits, track in-comment
promises in scratch/promises.md.

scripts/check-docs.sh ran clean across all 4 stages.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/hooks Claude Code hook scripts (Stop, PreCompact, SessionStart) enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

silent_save config flag is never read by the Stop hook

5 participants