fix: memory_dirs editable widget with per-dir index status#295
Merged
fix: memory_dirs editable widget with per-dir index status#295
Conversation
The v0.1.12 ``auto_discover`` migration appends every canonical Claude
project memory dir to ``config.json`` (~25 paths on a dev box), which
exposed two gaps:
1. **UI**: ``settings-config.js`` rendered ``indexing.memory_dirs`` as a
read-only, comma-joined flat line. Post-migration users saw a 30-path
wall of text with no way to prune entries except editing
``config.json`` by hand or hitting the (undocumented in the card)
POST/DELETE ``/api/memory-dirs`` endpoints.
2. **Startup**: ``lifespan.app_lifespan`` only started the ``FileWatcher``
and never walked existing files. The watcher reacts to
``on_modified/on_created/on_moved`` only, so pre-existing files in
any newly-added memory_dir stayed invisible to search until the user
ran a manual ``mem_index``. This made the migration look like it did
nothing.
Fix A — editable widget: remove ``memory_dirs`` from ``_READONLY_FIELDS``
and register a custom widget that groups dirs into four collapsible
categories (User / Claude projects / Claude plans / Codex, matching
``_detect_provider_dirs``) with per-item reindex/remove buttons, a
per-category reindex, a top-level reindex-all, and an inline add input.
Each mutation hits the existing ``/api/memory-dirs/{add,remove}`` and
``/api/{index,reindex}`` endpoints directly — the widget sits outside
the section's Save/dirty flow because these actions carry side effects
(watcher schedule, DB chunk state) that must land atomically on the
server. A new ``_NO_RESET_FIELDS`` suppresses the ↺ button for such
widgets so users don't see a permanently-disabled icon.
Fix B — gated startup scan: add ``dirs_needing_initial_scan`` to
``indexing/engine.py`` (returns dirs whose ``source_file`` prefix has
zero rows in ``chunks``) and wire ``_initial_scan_missing_dirs`` into
both the MCP and web lifespans as a background ``asyncio.Task``,
cancelled on shutdown. The gate keeps restart cost bounded — dirs
already indexed are skipped — so this doesn't regress warm-start
latency for existing installs.
Tests: 6 new unit tests for the gating helper covering empty inputs,
missing dirs, nested ``source_file`` coverage, and mixed indexed /
unindexed sets. Full ``pytest -m 'not ollama'`` suite (1844 tests)
green; ruff + mypy clean on touched files; i18n parity preserved (18
new keys added to both en.json and ko.json with matching
``{placeholder}`` sets).
Co-Authored-By: Claude <[email protected]>
Replace the gated startup scan with an on-demand indexing model. Smoke
testing on a real post-migration config (29 memory_dirs, ~2000 .md
files) made the first design untenable: ONNX-embedding the entire
backlog at startup ran for minutes and left the UI unresponsive while
every per-file failure flooded the console. The user-visible "I added
dirs but can't see my memory" gap survives, but the fix lives in the UI
instead — the widget now shows a "(N chunks)" / "(not indexed)" badge
per dir and the user clicks ↻ when they want to pay the reindex cost.
Changes:
- ``server/lifespan.py`` + ``web/app.py``: drop the
``_initial_scan_missing_dirs`` background task, its cancel helper,
and the ``asyncio`` / ``dirs_needing_initial_scan`` imports. Startup
goes back to "watcher only".
- ``indexing/engine.py``: replace ``dirs_needing_initial_scan`` (which
only answered a binary question) with ``memory_dir_stats``, which
returns ``[{path, chunk_count, source_file_count, exists}]`` — the
widget needs the counts for its badges anyway, so this subsumes the
old helper. Aggregates via a single
``storage.get_source_files_with_counts()`` call bucketed by
normalised-path prefix.
- ``web/routes/system.py``: new ``GET /api/memory-dirs/status``
endpoint returning the helper's output.
- ``web/static/settings-config.js``: widget fetches status after first
paint (non-blocking) and re-fetches after add / remove / reindex
actions. Per-item badge renders the chunk count, greys out the path
for 0-chunk dirs, and strikes through + italicises it for dirs
missing on disk. Group summary shows aggregate
``"{files} files · {chunks} chunks"``.
- Tests: ``TestDirsNeedingInitialScan`` → ``TestMemoryDirStats``
(7 tests covering empty input, missing dirs, aggregation, per-dir
bucketing, nested-file coverage, input-order preservation).
i18n: 4 new settings keys + matching Korean translations for
``status_empty`` / ``status_missing`` / ``status_chunks`` /
``status_group``; existing ``confirm.*`` and ``toast.*`` keys keep
their placeholder parity.
Net: -157 / +250 lines vs. main. The lifespan/web-app diffs shrink,
the widget grows by the status-fetch path, and the helper gets richer.
Co-Authored-By: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
indexing.memory_dirsas a categorized, editable widget (User / Claude projects / Claude plans / Codex collapsible groups) with per-item remove + reindex + inline add, replacing the previous 30-path comma-joined read-only line.GET /api/memory-dirs/statusendpoint, so users can see at a glance which dirs still need a manual reindex.Why
After v0.1.12 (#292) moved provider memory dirs to opt-in explicit entries, a typical post-migration install ends up with ~25
~/.claude/projects/*/memorypaths added toconfig.json. Two unrelated code paths then failed the same user expectation at once:settings-config.jsrenderedmemory_dirsvia_READONLY_FIELDS.indexing+val.join(', ')— a flat, uneditable line.POST/POST /api/memory-dirs/{add,remove}existed as APIs but the UI never surfaced them.lifespan.app_lifespanonly started theFileWatcher(indexing/watcher.py:85-96), which schedulesObserverlisteners but never walks existing files. Files that pre-existed the migration were invisible until the user manually ranmem_index. So "I added the dirs, why isn't my memory there?" was structurally unanswerable from the UI alone.Both problems share the same trigger but are independent bugs — one UI, one startup lifecycle. Bundling them keeps the user-facing story coherent (the Config tab now tells the truth: what's watched, how to prune it, how to kick a reindex — and which dirs are still empty).
Why pull-based instead of gated startup scan
The first commit on this branch (
f611d47) tried to close the indexing gap with a backgroundasyncio.Taskthat scanned everymemory_dirwith zero chunks at startup. Local smoke on a real post-migration config (29 dirs, ~2000 .md files) made that design untenable:upsert_chunksfailure (e.g., pre-existing schema drift in the user's DB) flooded the console with identical ERROR lines — ~200 copies of the same message.The second commit (
918c105) pivots: no automatic scan. Startup is back to "watcher only". The widget fetches per-dir status and displays(N chunks)or(not indexed)badges; the user clicks[↻]when they want to pay the reindex cost. This makes the "needs work" state visible without imposing a blind cost.Changes
Part A —
memory_dirsbecomes an editable widgetsettings-config.js: removememory_dirsfrom_READONLY_FIELDS.indexing. Introduce_NO_RESET_FIELDSto suppress the↺button for widgets that persist each action immediately (reset-to-default is semantically meaningless here). Register_buildMemoryDirsWidgetin_CONFIG_CUSTOM_WIDGETS._categorizeMemoryDir, mirroring_detect_provider_dirsinconfig.py. Categories:user,claude-memory,claude-plans,codex.useris open by default; others start collapsed (Claude projects can be 20+ entries).[↻](POST/api/index) and[✕](POST/api/memory-dirs/remove). Each group summary has[↻](loops over category). A top-level[↻ Reindex all]calls POST/api/reindex. Bottom input calls POST/api/memory-dirs/add.memory_dirsmutations carry side effects (watcher schedule, DB chunk state) that must land atomically on the server, not batched viaPATCH /api/config.showConfirm; all toasts go throught('toast.memory_dir.*', {...}). Covered by the existing i18n regression guards intest_i18n.py(no template-literal toasts, no English-literal titles).style.css:.memory-dirs-*classes. Claude-memory group getsmax-height: 220px; overflow-y: autoso 25+ entries don't blow out the card.Part B — per-dir index status
indexing/engine.py: newmemory_dir_stats(storage, memory_dirs)helper returning[{path, chunk_count, source_file_count, exists}]for each configured dir. Aggregates via a singlestorage.get_source_files_with_counts()call bucketed by normalised-path prefix — avoids NLIKEqueries for large dir lists.web/routes/system.py: newGET /api/memory-dirs/statusendpoint returning the helper's output.settings-config.jswidget: fetches status after first paint (non-blocking); re-fetches after add / remove / reindex. Per-item badge renders{count} chunks, greys out the path text for 0-chunk dirs, and strikes through + italicises it for dirs missing on disk. Group summary shows{files} files · {chunks} chunksaggregate.settings.memory_dirs.status_{empty,missing,chunks,group}) with Korean translations and placeholder parity.Not changed (out of scope)
_canonical_provider_dirs()scope narrowing — the migration already ran; retroactively trimming would surprise users who rely on the dirs.memory_dirsmanagement to its own top-level Sources tab — discussed in review; worth a follow-up once the widget's current layout is proven in practice._resolve_namespacebehaviour whereparent == memory_dir_root+enable_auto_ns=truecollapses todefault— pre-existing design gap, will file a follow-up issue.Tests
TestMemoryDirStatsintest_indexing_engine.py— empty input, missing dirs, empty dirs, aggregation, per-dir bucketing, nested-file coverage, input-order preservation.pytest -m 'not ollama'suite green (1845 pass, 46 ollama-deselected). Relevant sub-suites exercised specifically:test_web_routes.py,test_web_hot_reload.py,test_web_routes_extended.py,test_web_exclude_guard.py,test_i18n.py. No regressions.uv run ruff check+uv run ruff format --checkpass.uv run mypyclean on the four touched Python files.node -c settings-config.jsclean (no syntax errors).mm webagainst a post-migration 29-dir config — widget renders all 4 categories correctly, per-dir badges populate after initial status fetch,[↻]triggers reindex and badges refresh.Test plan
mm webon a post-v0.1.12config.json(many Claude-project dirs). Confirm Config tab → Indexing → Memory Dirs renders 4 collapsible groups with correct counts; User open, others collapsed.(N chunks)or(not indexed)badge. Group summaries show{files} files · {chunks} chunks.[✕]on one Claude-project entry → confirm dialog → accept → verify entry disappears +~/.memtomem/config.jsonno longer contains it.[+ Add]to add/tmp/test-memdir-$$→ verify new User-group entry + badge shows(not indexed)(chunk_count=0).[↻]on an empty dir → toast "Reindexing …" → on completion the badge flips to{N} chunks.[↻]on claude-projects → sequential reindex across all dirs in the group; badges refresh after each.[↻ Reindex all]→ POST/api/reindex; badges refresh with aggregated totals.rm -rf /tmp/test-memdir-$$) without removing it from config → reload page → badge showsmissing(strikethrough + italic).🤖 Generated with Claude Code