Copilot review on #5 caught that the README still
documented session_recovery_read as current, which would leave users
calling a tool that returns "Unknown tool" after this PR ships.

Updates the four user-facing sections that described the recovery
collection as a current shipping feature:

* "What just shipped" — adds 2026-05-05 update note explaining the
  split was retired, with the generalizable lesson preserved (a side
  collection without a semantic-search MCP read tool is invisible).
* "Verbatim vs derivative" principle — reframes the recovery
  collection as the failed first attempt rather than a current
  example; pattern stays valid but each future sibling needs its
  own read surface.
* "Structural retrieval fixes" bullet — describes verbatim-only
  end-state, links to the new spec, retains historical context for
  Apr 25 → May 5.
* "Deterministic hook saves" bullet — updates to the new ingest-only
  save path, removes "recovery-collection marker" mention.
* "What it looks like in production" code block — updates the
  systemMessage example to the new "Transcript ingest triggered
  (wing=...)" shape.
* "P8" architectural principle — keeps it but reframes as on-hold
  with a precondition (read-surface parity) drawn from the cycle.
* Fork-ahead tracking table — replaces the old multi-collection-split
  row with a verbatim-only retirement row pointing at the four PRs;
  also retires references to session_recovery_read in the drawer_id
  surfacing row; adds new row for the mining management CLI (#4).

42 README meta-tests pass (test_readme_claims.py).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Four findings from Copilot's review pass on the mining-management CLI:

1. **--wing filter pushed into the query** (Copilot cli.py:800).
   Previously cmd_mined fetched every drawer in the palace and filtered
   in Python, doing O(total) work even for "mempalace mined --wing X"
   on a 150K-drawer palace where X has 12K. Pre-fetch IDs via
   col.get(where=...) to size the loop, then pass where= on each
   batch get(). On the canonical palace, --wing filters now scan
   ~10x less data.

2. **Reject negative --limit at parse time** (Copilot cli.py:818).
   argparse type=int silently accepted -1, producing nonsensical
   "... -2 more" output. Add a typed _nonneg_int validator that
   raises ArgumentTypeError on negatives. argparse turns that into
   a clean exit code 2 with usage error message.

3. **Update module __doc__** (Copilot cli.py:1495). main() uses the
   module docstring as the help epilog. The hard-coded command list
   stopped at "status" — added "mined" and "purge --source-file"
   so `mempalace --help` reflects the new commands.

4. **Dispatch test for `mempalace mined`** (Copilot cli.py:1542).
   Sibling commands have main()-level dispatch tests; mined didn't.
   A typo or registration regression would have slipped through unit
   coverage. Added test_main_mined_dispatches and a paired
   test_main_mined_rejects_negative_limit (verifies #2).

Tests: 68 cli tests pass; full suite green at 1566 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(cli): mempalace mined + purge --source-file

Adds two ergonomic gaps in the mining-management surface JP asked for:

1. **mempalace purge --source-file <path>** — extends the existing
   purge command with a third filter alongside --wing and --room.
   Single-filter or combined ($and) usage works the same way wing/room
   do; the filter accepts a metadata.source_file exact-match. End-to-
   end test exercises a real palace and confirms only the matching
   drawers are deleted, leaving siblings intact.

2. **mempalace mined** — companion to `mempalace status`, but groups
   by wing × source_file rather than wing × room. Answers "which
   files have I mined into this wing?" so an operator can pick
   targets for the new --source-file purge. Skips drawers without a
   source_file metadata key (diary entries, kg drawers, etc.).
   Honors --wing and --limit (default 50 sources per wing, --limit 0
   to show all). Uses the same paginated col.get() pattern as
   miner.status() to handle 100K+ drawer palaces without tripping
   sqlite's max-variable limit.

Together these close the "removing manually mined data" half of JP's
ask. The "automining when files change" half is queued for a separate
file-watcher daemon PR.

Pre-existing `mempalace mine <dir>` already covers "adding" (and
re-mines on mtime change via bulk_check_mined dedup), so no command
needed for refresh.

Tests: 8 new test cases (3 for purge --source-file with mock + real
palace; 3 for cmd_mined including end-to-end groupBy + wing filter +
no-palace error path; 2 updated to add the new arg to the test
namespace builder). Full suite: 1564 passed, 1 skipped, lint clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(cli): address Copilot review on #4

Four findings from Copilot's review pass on the mining-management CLI:

1. **--wing filter pushed into the query** (Copilot cli.py:800).
   Previously cmd_mined fetched every drawer in the palace and filtered
   in Python, doing O(total) work even for "mempalace mined --wing X"
   on a 150K-drawer palace where X has 12K. Pre-fetch IDs via
   col.get(where=...) to size the loop, then pass where= on each
   batch get(). On the canonical palace, --wing filters now scan
   ~10x less data.

2. **Reject negative --limit at parse time** (Copilot cli.py:818).
   argparse type=int silently accepted -1, producing nonsensical
   "... -2 more" output. Add a typed _nonneg_int validator that
   raises ArgumentTypeError on negatives. argparse turns that into
   a clean exit code 2 with usage error message.

3. **Update module __doc__** (Copilot cli.py:1495). main() uses the
   module docstring as the help epilog. The hard-coded command list
   stopped at "status" — added "mined" and "purge --source-file"
   so `mempalace --help` reflects the new commands.

4. **Dispatch test for `mempalace mined`** (Copilot cli.py:1542).
   Sibling commands have main()-level dispatch tests; mined didn't.
   A typo or registration regression would have slipped through unit
   coverage. Added test_main_mined_dispatches and a paired
   test_main_mined_rejects_negative_limit (verifies #2).

Tests: 68 cli tests pass; full suite green at 1566 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Copilot review on #5 caught that the README still
documented session_recovery_read as current, which would leave users
calling a tool that returns "Unknown tool" after this PR ships.

Updates the four user-facing sections that described the recovery
collection as a current shipping feature:

* "What just shipped" — adds 2026-05-05 update note explaining the
  split was retired, with the generalizable lesson preserved (a side
  collection without a semantic-search MCP read tool is invisible).
* "Verbatim vs derivative" principle — reframes the recovery
  collection as the failed first attempt rather than a current
  example; pattern stays valid but each future sibling needs its
  own read surface.
* "Structural retrieval fixes" bullet — describes verbatim-only
  end-state, links to the new spec, retains historical context for
  Apr 25 → May 5.
* "Deterministic hook saves" bullet — updates to the new ingest-only
  save path, removes "recovery-collection marker" mention.
* "What it looks like in production" code block — updates the
  systemMessage example to the new "Transcript ingest triggered
  (wing=...)" shape.
* "P8" architectural principle — keeps it but reframes as on-hold
  with a precondition (read-surface parity) drawn from the cycle.
* Fork-ahead tracking table — replaces the old multi-collection-split
  row with a verbatim-only retirement row pointing at the four PRs;
  also retires references to session_recovery_read in the drawer_id
  surfacing row; adds new row for the mining management CLI (#4).

42 README meta-tests pass (test_readme_claims.py).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

…ol (#8)

* refactor(mcp): retire mempalace_session_recovery collection + read tool

Follow-up to fix/drop-checkpoint-write-path. With nothing writing to
the recovery collection anymore (the hooks moved to verbatim-only on
the parent branch), the read paths and the migration that fed the
collection are dead code. Delete them.

Removed in mempalace/:
- palace.py: _SESSION_RECOVERY_COLLECTION constant,
  get_session_recovery_collection(), _CHECKPOINT_TOPICS tuple
- mcp_server.py: _get_session_recovery_collection(),
  _recovery_collection_cache global, the topic-routing branch in
  tool_diary_write (now always lands in mempalace_drawers),
  tool_session_recovery_read() and its TOOLS dict registration
- migrate.py: migrate_checkpoints_to_recovery() and the dependent
  _CHECKPOINT_TOPICS import
- cli.py: cmd_repair --mode reorganize handler + the choice flag

Removed in tests/:
- tests/test_session_recovery.py — entire file (recovery-collection
  module test suite)
- tests/test_migrate.py: TestMigrateCheckpointsToRecovery class
- tests/test_mcp_server.py: TestCheckpointRouting and
  TestSessionRecoveryRead classes

Removed in docs/:
- website/reference/mcp-tools.md: mempalace_session_recovery_read
  section (caught by the readme-claims meta-test before deploy)

Production data on disks still has the mempalace_session_recovery
collection with its 763 archived entries — this PR's code change
makes the collection unreachable through any MCP/CLI path, but does
NOT delete the on-disk data. A separate one-shot script
(scripts/phase2_purge_recovery.py per the spec) handles the
collection-level delete after deploy. JP signed off on hard-delete
in the spec ack.

Net diff: 12 files, −1300 lines / +30 lines. 1540 tests pass, 1
skipped. Ruff lint + format clean.

Stacked on #3 (which stops new writes); will
auto-rebase against main when that merges.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs(README): align with verbatim-only retirement

Copilot review on #5 caught that the README still
documented session_recovery_read as current, which would leave users
calling a tool that returns "Unknown tool" after this PR ships.

Updates the four user-facing sections that described the recovery
collection as a current shipping feature:

* "What just shipped" — adds 2026-05-05 update note explaining the
  split was retired, with the generalizable lesson preserved (a side
  collection without a semantic-search MCP read tool is invisible).
* "Verbatim vs derivative" principle — reframes the recovery
  collection as the failed first attempt rather than a current
  example; pattern stays valid but each future sibling needs its
  own read surface.
* "Structural retrieval fixes" bullet — describes verbatim-only
  end-state, links to the new spec, retains historical context for
  Apr 25 → May 5.
* "Deterministic hook saves" bullet — updates to the new ingest-only
  save path, removes "recovery-collection marker" mention.
* "What it looks like in production" code block — updates the
  systemMessage example to the new "Transcript ingest triggered
  (wing=...)" shape.
* "P8" architectural principle — keeps it but reframes as on-hold
  with a precondition (read-surface parity) drawn from the cycle.
* Fork-ahead tracking table — replaces the old multi-collection-split
  row with a verbatim-only retirement row pointing at the four PRs;
  also retires references to session_recovery_read in the drawer_id
  surfacing row; adds new row for the mining management CLI (#4).

42 README meta-tests pass (test_readme_claims.py).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Closes the "automining when files are added or updated" half of JP's
ask. The other half (CLI add/remove of mined data) ships in
jphein/mempalace#4 as `mempalace mined` + `mempalace purge --source-file`.

Adds a watchdog-based file watcher that runs inside the daemon's
lifespan. Configured via `PALACE_WATCH_DIRS` env var:

    PALACE_WATCH_DIRS="/home/jp/Projects/realmwatch=wing_realmwatch,
                       /home/jp/Projects/oracle=wing_oracle"

Each entry is `path` or `path=wing`. Bare path derives wing via
`mempalace.config.normalize_wing_name(basename)` to match the
local-spawn miner default. Non-existent paths are warned-and-skipped,
not fatal — a misconfigured env var doesn't kill startup.

Architecture:

* watcher.py — pure-Python module: parse_watch_dirs(),
  _DebouncedMineHandler (subclass of FileSystemEventHandler with a
  per-target debounce timer), WatcherService (lifecycle wrapper around
  watchdog.Observer).
* main.py lifespan — instantiate WatcherService, schedule one
  recursive watch per target, register stop() on shutdown. Failures
  are non-fatal; the daemon serves traffic regardless.
* main.py — `_internal_mine()` async closure runs the same
  `mempalace mine ... --mode projects --wing X` argv as the existing
  /mine endpoint, gated by the same `_mine_sem`.
* main.py — new `GET /watch` endpoint surfaces the running target
  list. No POST/DELETE — runtime add/remove requires daemon restart
  (operator just edits the systemd unit env and restarts).

Debounce: events on the same target collapse via a 2-second per-target
timer. Catches editor write+rename storms and *.swp / *.lock churn.
Also pre-filters by extension allowlist (29 extensions: code, configs,
markdown). Inotify fires for everything; the handler discards
non-watched extensions before the timer schedules.

Path translation: when a watch path lives in a Syncthing-replicated
directory, the daemon-side path is translated through
`PALACE_DAEMON_PATH_MAP` before the mine subprocess runs (same
mechanism used by /mine).

Tests: 11 new unittest cases in tests/test_watcher.py:
- TestParseWatchDirs (7 cases): empty, single-with-derived-wing,
  explicit-wing, multiple, skips-nonexistent, skips-files, env-fallback
- TestDebouncedMineHandler (4 cases): single-event-fires,
  burst-collapses-to-one, skips-directory-events, skips-bad-extensions

Stacked on #1 (path-translation). When that
merges, this PR auto-rebases against main; reviewers see only the
new commits in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(watcher): file-watcher service for auto-mining on file change

Closes the "automining when files are added or updated" half of JP's
ask. The other half (CLI add/remove of mined data) ships in
jphein/mempalace#4 as `mempalace mined` + `mempalace purge --source-file`.

Adds a watchdog-based file watcher that runs inside the daemon's
lifespan. Configured via `PALACE_WATCH_DIRS` env var:

    PALACE_WATCH_DIRS="/home/jp/Projects/realmwatch=wing_realmwatch,
                       /home/jp/Projects/oracle=wing_oracle"

Each entry is `path` or `path=wing`. Bare path derives wing via
`mempalace.config.normalize_wing_name(basename)` to match the
local-spawn miner default. Non-existent paths are warned-and-skipped,
not fatal — a misconfigured env var doesn't kill startup.

Architecture:

* watcher.py — pure-Python module: parse_watch_dirs(),
  _DebouncedMineHandler (subclass of FileSystemEventHandler with a
  per-target debounce timer), WatcherService (lifecycle wrapper around
  watchdog.Observer).
* main.py lifespan — instantiate WatcherService, schedule one
  recursive watch per target, register stop() on shutdown. Failures
  are non-fatal; the daemon serves traffic regardless.
* main.py — `_internal_mine()` async closure runs the same
  `mempalace mine ... --mode projects --wing X` argv as the existing
  /mine endpoint, gated by the same `_mine_sem`.
* main.py — new `GET /watch` endpoint surfaces the running target
  list. No POST/DELETE — runtime add/remove requires daemon restart
  (operator just edits the systemd unit env and restarts).

Debounce: events on the same target collapse via a 2-second per-target
timer. Catches editor write+rename storms and *.swp / *.lock churn.
Also pre-filters by extension allowlist (29 extensions: code, configs,
markdown). Inotify fires for everything; the handler discards
non-watched extensions before the timer schedules.

Path translation: when a watch path lives in a Syncthing-replicated
directory, the daemon-side path is translated through
`PALACE_DAEMON_PATH_MAP` before the mine subprocess runs (same
mechanism used by /mine).

Tests: 11 new unittest cases in tests/test_watcher.py:
- TestParseWatchDirs (7 cases): empty, single-with-derived-wing,
  explicit-wing, multiple, skips-nonexistent, skips-files, env-fallback
- TestDebouncedMineHandler (4 cases): single-event-fires,
  burst-collapses-to-one, skips-directory-events, skips-bad-extensions

Stacked on #1 (path-translation). When that
merges, this PR auto-rebases against main; reviewers see only the
new commits in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(watcher): address Copilot review on #2

Seven findings, all addressed:

1. **Path translation before is_dir check** (Copilot watcher.py:133).
   parse_watch_dirs() validated against the daemon filesystem before
   any path translation, so a PALACE_WATCH_DIRS value written in the
   client namespace was always rejected as "not a directory". Add a
   translator= kwarg; main.py now passes _translate_client_path so
   client-namespace entries reach the daemon's view first.

2. **Drop on_any_event in favor of specific subscriptions** (Copilot
   watcher.py:160). watchdog 3.x emits opened/closed events on plain
   file reads on Linux; routing those through the debounce would
   re-mine the project on every file open. Replaced on_any_event
   with explicit on_created / on_modified / on_moved / on_deleted.

3. **Check dest_path on FileMovedEvent** (Copilot watcher.py:171).
   Editor save-via-rename writes a temp file then renames to the
   real filename. The src_path on the move event is the temp; only
   dest_path has the real extension. Extracted _has_watchable_extension()
   helper; on_moved() checks both sides.

4. **Per-target schedule() failure isolation** (Copilot watcher.py:219).
   A bare for-loop let an exception in observer.schedule() (e.g.
   inotify watch limit on a large repo) abort the entire service.
   Wrap each schedule() in try/except; one failed target no longer
   disables the others. Logged at warning level.

5. **Cancel debounce timers on stop** (Copilot watcher.py:224). stop()
   tore down the observer but left armed threading.Timer objects to
   fire mid-teardown. Add cancel_pending() on the handler; service
   stop() walks all handlers first.

6. **Surface watcher-coroutine errors** (Copilot watcher.py:246).
   asyncio.run_coroutine_threadsafe() returns a Future the caller
   must observe; dropping it swallowed exceptions silently. Add a
   _log_future_exception done-callback.

7. **/watch reports running state accurately** (Copilot main.py:558).
   app.state.watcher was set unconditionally even when start()
   returned early (empty env, missing dep, all targets failed). Now
   only published to app.state when watcher.is_running. GET /watch
   reports running: false for idle/disabled instead of misleading
   running: true.

Tests: 28 pass (was 24). New cases:
- test_rename_to_watched_extension_fires (covers #3)
- test_cancel_pending_drops_armed_timer (covers #5)
- test_one_failed_target_doesnt_disable_others (covers #4)
- test_translator_runs_before_is_dir (covers #1)
Existing handler tests retargeted from on_any_event to on_modified
since on_any_event no longer exists in this revision.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>