Skip to content

fix: memory_dirs editable widget with per-dir index status#295

Merged
memtomem merged 2 commits intomainfrom
fix/memory-dirs-ui-and-scan
Apr 19, 2026
Merged

fix: memory_dirs editable widget with per-dir index status#295
memtomem merged 2 commits intomainfrom
fix/memory-dirs-ui-and-scan

Conversation

@memtomem
Copy link
Copy Markdown
Owner

@memtomem memtomem commented Apr 19, 2026

Summary

  • Web Config tab now shows indexing.memory_dirs as a categorized, editable widget (User / Claude projects / Claude plans / Codex collapsible groups) with per-item remove + reindex + inline add, replacing the previous 30-path comma-joined read-only line.
  • Each dir gets a "(N chunks)" / "(not indexed)" badge fed by a new GET /api/memory-dirs/status endpoint, so users can see at a glance which dirs still need a manual reindex.
  • 7 new unit tests for the aggregation helper; 18 new i18n keys with en/ko parity.

Why

After v0.1.12 (#292) moved provider memory dirs to opt-in explicit entries, a typical post-migration install ends up with ~25 ~/.claude/projects/*/memory paths added to config.json. Two unrelated code paths then failed the same user expectation at once:

  • settings-config.js rendered memory_dirs via _READONLY_FIELDS.indexing + val.join(', ') — a flat, uneditable line. POST/POST /api/memory-dirs/{add,remove} existed as APIs but the UI never surfaced them.
  • lifespan.app_lifespan only started the FileWatcher (indexing/watcher.py:85-96), which schedules Observer listeners but never walks existing files. Files that pre-existed the migration were invisible until the user manually ran mem_index. So "I added the dirs, why isn't my memory there?" was structurally unanswerable from the UI alone.

Both problems share the same trigger but are independent bugs — one UI, one startup lifecycle. Bundling them keeps the user-facing story coherent (the Config tab now tells the truth: what's watched, how to prune it, how to kick a reindex — and which dirs are still empty).

Why pull-based instead of gated startup scan

The first commit on this branch (f611d47) tried to close the indexing gap with a background asyncio.Task that scanned every memory_dir with zero chunks at startup. Local smoke on a real post-migration config (29 dirs, ~2000 .md files) made that design untenable:

  • ONNX embedding the full backlog ran for minutes and left the UI unresponsive.
  • Every per-file upsert_chunks failure (e.g., pre-existing schema drift in the user's DB) flooded the console with identical ERROR lines — ~200 copies of the same message.
  • Users had no way to see what was being indexed or how long it would take.

The second commit (918c105) pivots: no automatic scan. Startup is back to "watcher only". The widget fetches per-dir status and displays (N chunks) or (not indexed) badges; the user clicks [↻] when they want to pay the reindex cost. This makes the "needs work" state visible without imposing a blind cost.

Changes

Part A — memory_dirs becomes an editable widget

  • settings-config.js: remove memory_dirs from _READONLY_FIELDS.indexing. Introduce _NO_RESET_FIELDS to suppress the button for widgets that persist each action immediately (reset-to-default is semantically meaningless here). Register _buildMemoryDirsWidget in _CONFIG_CUSTOM_WIDGETS.
  • The widget groups paths client-side via _categorizeMemoryDir, mirroring _detect_provider_dirs in config.py. Categories: user, claude-memory, claude-plans, codex. user is open by default; others start collapsed (Claude projects can be 20+ entries).
  • Each row has [↻] (POST /api/index) and [✕] (POST /api/memory-dirs/remove). Each group summary has [↻] (loops over category). A top-level [↻ Reindex all] calls POST /api/reindex. Bottom input calls POST /api/memory-dirs/add.
  • The widget lives outside the per-section Save/dirty flow by design: memory_dirs mutations carry side effects (watcher schedule, DB chunk state) that must land atomically on the server, not batched via PATCH /api/config.
  • Confirmation dialog for removal uses showConfirm; all toasts go through t('toast.memory_dir.*', {...}). Covered by the existing i18n regression guards in test_i18n.py (no template-literal toasts, no English-literal titles).
  • style.css: .memory-dirs-* classes. Claude-memory group gets max-height: 220px; overflow-y: auto so 25+ entries don't blow out the card.

Part B — per-dir index status

  • indexing/engine.py: new memory_dir_stats(storage, memory_dirs) helper returning [{path, chunk_count, source_file_count, exists}] for each configured dir. Aggregates via a single storage.get_source_files_with_counts() call bucketed by normalised-path prefix — avoids N LIKE queries for large dir lists.
  • web/routes/system.py: new GET /api/memory-dirs/status endpoint returning the helper's output.
  • settings-config.js widget: fetches status after first paint (non-blocking); re-fetches after add / remove / reindex. Per-item badge renders {count} chunks, greys out the path text for 0-chunk dirs, and strikes through + italicises it for dirs missing on disk. Group summary shows {files} files · {chunks} chunks aggregate.
  • 4 new i18n keys (settings.memory_dirs.status_{empty,missing,chunks,group}) with Korean translations and placeholder parity.

Not changed (out of scope)

  • Startup auto-scan: attempted in commit 1, reverted in commit 2. Pull-based is what ended up shipping.
  • _canonical_provider_dirs() scope narrowing — the migration already ran; retroactively trimming would surprise users who rely on the dirs.
  • Promoting memory_dirs management to its own top-level Sources tab — discussed in review; worth a follow-up once the widget's current layout is proven in practice.
  • Cloud-sync mount handling (GoogleDrive etc.) — separate tracked issue.
  • _resolve_namespace behaviour where parent == memory_dir_root + enable_auto_ns=true collapses to default — pre-existing design gap, will file a follow-up issue.

Tests

  • New (7): TestMemoryDirStats in test_indexing_engine.py — empty input, missing dirs, empty dirs, aggregation, per-dir bucketing, nested-file coverage, input-order preservation.
  • Existing: full pytest -m 'not ollama' suite green (1845 pass, 46 ollama-deselected). Relevant sub-suites exercised specifically: test_web_routes.py, test_web_hot_reload.py, test_web_routes_extended.py, test_web_exclude_guard.py, test_i18n.py. No regressions.
  • Lint: uv run ruff check + uv run ruff format --check pass. uv run mypy clean on the four touched Python files.
  • JS: node -c settings-config.js clean (no syntax errors).
  • Manual smoke (see Test plan): mm web against a post-migration 29-dir config — widget renders all 4 categories correctly, per-dir badges populate after initial status fetch, [↻] triggers reindex and badges refresh.

Test plan

  • Restart mm web on a post-v0.1.12 config.json (many Claude-project dirs). Confirm Config tab → Indexing → Memory Dirs renders 4 collapsible groups with correct counts; User open, others collapsed.
  • Within ~200ms of paint, each dir item shows a (N chunks) or (not indexed) badge. Group summaries show {files} files · {chunks} chunks.
  • Click [✕] on one Claude-project entry → confirm dialog → accept → verify entry disappears + ~/.memtomem/config.json no longer contains it.
  • Use [+ Add] to add /tmp/test-memdir-$$ → verify new User-group entry + badge shows (not indexed) (chunk_count=0).
  • Click per-row [↻] on an empty dir → toast "Reindexing …" → on completion the badge flips to {N} chunks.
  • Click group-level [↻] on claude-projects → sequential reindex across all dirs in the group; badges refresh after each.
  • Click top-level [↻ Reindex all] → POST /api/reindex; badges refresh with aggregated totals.
  • Check logs: no "Initial scan: indexing N memory_dir(s)" message at startup (confirms pull-based design).
  • Delete a dir from disk (rm -rf /tmp/test-memdir-$$) without removing it from config → reload page → badge shows missing (strikethrough + italic).
  • Sanity-check Korean locale renders the 4 category labels and all badge messages correctly.

🤖 Generated with Claude Code

pandas-studio and others added 2 commits April 19, 2026 23:06
The v0.1.12 ``auto_discover`` migration appends every canonical Claude
project memory dir to ``config.json`` (~25 paths on a dev box), which
exposed two gaps:

1. **UI**: ``settings-config.js`` rendered ``indexing.memory_dirs`` as a
   read-only, comma-joined flat line. Post-migration users saw a 30-path
   wall of text with no way to prune entries except editing
   ``config.json`` by hand or hitting the (undocumented in the card)
   POST/DELETE ``/api/memory-dirs`` endpoints.

2. **Startup**: ``lifespan.app_lifespan`` only started the ``FileWatcher``
   and never walked existing files. The watcher reacts to
   ``on_modified/on_created/on_moved`` only, so pre-existing files in
   any newly-added memory_dir stayed invisible to search until the user
   ran a manual ``mem_index``. This made the migration look like it did
   nothing.

Fix A — editable widget: remove ``memory_dirs`` from ``_READONLY_FIELDS``
and register a custom widget that groups dirs into four collapsible
categories (User / Claude projects / Claude plans / Codex, matching
``_detect_provider_dirs``) with per-item reindex/remove buttons, a
per-category reindex, a top-level reindex-all, and an inline add input.
Each mutation hits the existing ``/api/memory-dirs/{add,remove}`` and
``/api/{index,reindex}`` endpoints directly — the widget sits outside
the section's Save/dirty flow because these actions carry side effects
(watcher schedule, DB chunk state) that must land atomically on the
server. A new ``_NO_RESET_FIELDS`` suppresses the ↺ button for such
widgets so users don't see a permanently-disabled icon.

Fix B — gated startup scan: add ``dirs_needing_initial_scan`` to
``indexing/engine.py`` (returns dirs whose ``source_file`` prefix has
zero rows in ``chunks``) and wire ``_initial_scan_missing_dirs`` into
both the MCP and web lifespans as a background ``asyncio.Task``,
cancelled on shutdown. The gate keeps restart cost bounded — dirs
already indexed are skipped — so this doesn't regress warm-start
latency for existing installs.

Tests: 6 new unit tests for the gating helper covering empty inputs,
missing dirs, nested ``source_file`` coverage, and mixed indexed /
unindexed sets. Full ``pytest -m 'not ollama'`` suite (1844 tests)
green; ruff + mypy clean on touched files; i18n parity preserved (18
new keys added to both en.json and ko.json with matching
``{placeholder}`` sets).

Co-Authored-By: Claude <[email protected]>
Replace the gated startup scan with an on-demand indexing model. Smoke
testing on a real post-migration config (29 memory_dirs, ~2000 .md
files) made the first design untenable: ONNX-embedding the entire
backlog at startup ran for minutes and left the UI unresponsive while
every per-file failure flooded the console. The user-visible "I added
dirs but can't see my memory" gap survives, but the fix lives in the UI
instead — the widget now shows a "(N chunks)" / "(not indexed)" badge
per dir and the user clicks ↻ when they want to pay the reindex cost.

Changes:

- ``server/lifespan.py`` + ``web/app.py``: drop the
  ``_initial_scan_missing_dirs`` background task, its cancel helper,
  and the ``asyncio`` / ``dirs_needing_initial_scan`` imports. Startup
  goes back to "watcher only".
- ``indexing/engine.py``: replace ``dirs_needing_initial_scan`` (which
  only answered a binary question) with ``memory_dir_stats``, which
  returns ``[{path, chunk_count, source_file_count, exists}]`` — the
  widget needs the counts for its badges anyway, so this subsumes the
  old helper. Aggregates via a single
  ``storage.get_source_files_with_counts()`` call bucketed by
  normalised-path prefix.
- ``web/routes/system.py``: new ``GET /api/memory-dirs/status``
  endpoint returning the helper's output.
- ``web/static/settings-config.js``: widget fetches status after first
  paint (non-blocking) and re-fetches after add / remove / reindex
  actions. Per-item badge renders the chunk count, greys out the path
  for 0-chunk dirs, and strikes through + italicises it for dirs
  missing on disk. Group summary shows aggregate
  ``"{files} files · {chunks} chunks"``.
- Tests: ``TestDirsNeedingInitialScan`` → ``TestMemoryDirStats``
  (7 tests covering empty input, missing dirs, aggregation, per-dir
  bucketing, nested-file coverage, input-order preservation).

i18n: 4 new settings keys + matching Korean translations for
``status_empty`` / ``status_missing`` / ``status_chunks`` /
``status_group``; existing ``confirm.*`` and ``toast.*`` keys keep
their placeholder parity.

Net: -157 / +250 lines vs. main. The lifespan/web-app diffs shrink,
the widget grows by the status-fetch path, and the helper gets richer.

Co-Authored-By: Claude <[email protected]>
@memtomem memtomem changed the title fix: memory_dirs editable widget + gated startup scan fix: memory_dirs editable widget with per-dir index status Apr 19, 2026
@memtomem memtomem merged commit 11f4395 into main Apr 19, 2026
7 checks passed
@memtomem memtomem deleted the fix/memory-dirs-ui-and-scan branch April 19, 2026 14:48
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants