Skip to content

Investigation: /api/indexing/active endpoint reports active indefinitely after watcher / SSE indexing #640

@memtomem

Description

@memtomem

Symptom

While running `mm web` and copying files into a configured `memory_dir`, the
header indicator stayed in the "indexing" state for many minutes after the
copy. Server log showed continuous client polling:

```
INFO: 127.0.0.1:53468 - "GET /api/indexing/active HTTP/1.1" 200 OK
INFO: 127.0.0.1:53468 - "GET /api/indexing/active HTTP/1.1" 200 OK
... (repeating every few seconds)
```

The endpoint was answering `{"active": true}` long past the point where any
single-file index should have completed (single `.md` file, watcher path).

Suspicion

`IndexEngine._active_runs` is a manual counter incremented at the entry point
of `index_path`, `index_file`, and `index_path_stream` and decremented in
`finally`. The first two are lock-protected and unlikely to leak, but
`index_path_stream` runs outside `_index_lock` and is an async generator —
if the FastAPI `StreamingResponse` is cancelled (client disconnects, network
flap, browser tab close mid-run, anyio cancellation), the generator's
`finally` block doesn't always execute promptly, and could in pathological
cases not run at all. That would leave `_active_runs > 0` permanently until
the server restarts.

What to verify

  1. Reproduce by starting an SSE `/api/index/stream` run, killing the
    client connection mid-stream, and curling `/api/indexing/active`. If
    it answers `true` indefinitely, the leak is confirmed.
  2. Inspect whether `_active_runs` decrement guarantees survive
    `asyncio.CancelledError` paths in async generators consumed by
    `StreamingResponse`.
  3. Check whether watcher-triggered `_reindex` paths (`watcher.py`) are
    counted (they go through `index_file`, which IS counted) — and
    whether nested concurrent runs ever overlap and one finally is
    skipped.

Possible fix shapes

  • Replace the manual counter with an `asyncio.Lock` count derived from
    active task tracking (`set[asyncio.Task]`), so cancelled tasks self-prune.
  • Add a watchdog: if no run has registered/unregistered for N minutes,
    reset the counter on the next poll (last-resort).
  • Wrap the SSE `async for` body in `try/finally` outside the existing
    one to make the decrement re-entrant against double cancellation.

Context

Surfaced during diagnosis of #639 (Sources tab orphan render). The
indexing-active polling itself isn't a critical bug — UI just shows a
spinner — but if confirmed it explains the "indexing forever" reports
that have been hard to reproduce.

Acceptance

  • A test that exercises SSE stream cancellation mid-run and asserts
    `engine.is_active === False` after the cancellation propagates.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions