…ry and non-destructive removal (#590)

* Parallelize ingest planning and size it to the container CPU budget

The discovery/hash planning pass ran single-threaded, so a multi-million-file
corpus stalled one core while the GPUs sat idle. Fan the independent per-file
classification across a thread pool and assemble the plan in sorted order so the
result is identical to the serial pass. Size the pool to available_cpu_count(),
which folds in the cgroup CFS quota and CPU affinity instead of trusting
os.cpu_count() (which reports the host, not the pod). New ingest_workers config
(0 = auto) and add --max-cpus cap the pool per run.

* Ingest by symlink and make removal non-destructive with folder and glob patterns

add now symlinks a source into the documents dir instead of copying it. A
prepared corpus already lives on local disk, so linking it in place removes the
single-threaded byte copy and the doubled disk footprint that a large ingest
paid before the GPUs got any work. Discovery follows these top-level links,
keying each linked source under its label so every downstream consumer stays
documents_dir-relative; the symlink-escape guard is relaxed only for these
self-describing linked roots, and a nested link that points outside them is
still skipped.

remove no longer deletes files on disk. Source bytes are the user's, and a
linked-in corpus must never be removed, so the delete_files parameter and the
--delete flag are gone from every surface (store, CLI, MCP, HTTP, the Python
API, and the skill docs). Removal is index-only plus a durable skip-marker;
removing a top-level linked root instead unlinks that symlink. Matching now
expands beyond an exact name: a folder removes every source indexed beneath it
(whole-segment boundary), and a glob (a name with * ? or []) removes every
fnmatch. The CLI confirms before a multi-document removal.

* Surface ingest_workers in the INGEST settings group

* Type the discovery walk helper, document the cgroup read, refresh docs

Tighten _walk_into's ignore_dirs annotation to frozenset[str], note why the
cgroup CPU quota is read by hand (os.process_cpu_count is 3.13+, floor is 3.11),
and update the usage/tutorial docs for symlink-based add and folder/glob remove.

* Detect moved sources by content hash and relocate them instead of re-ingesting

Moving indexed files used to cost a full re-ingest: the old paths were removed
and the new paths embedded from scratch, even though the content was identical.

Sync now hash-indexes the sources whose files just disappeared and matches them
against the new files in the same pass. A match is a move, so the source is
re-keyed in place (chunks, page texts, concepts, entities, and citations all
repoint to the new name, the sources row keeps its hash and vectors) rather than
deleted and re-embedded. Pairing is one-to-one and deterministic when hashes
collide. The result carries a `relocated` count across the CLI, TUI, HTTP, and
MCP surfaces.

add stops demanding --force when a source moved: a dangling link (its old target
gone) auto-relinks to the new path, and the TUI reports "already indexed,
location changed" instead of treating it as new.

* Make symlink ingest cross-platform: junction and hard-link fallbacks, then copy

Symlink creation needs privilege on Windows (Developer Mode or admin), so a
symlink-only add would fail for an unprivileged Windows user. Degrade instead of
breaking, preferring each option that still avoids a byte copy:

- symlink where the platform allows it (POSIX, or privileged Windows);
- a directory junction (no privilege) for a directory on Windows, created via
  _winapi.CreateJunction and verified;
- a same-volume file hard link for a file on Windows;
- a plain copy as the last resort (cross-volume, UNC, or any failure).

A junction is not a symlink, so link detection is unified on os.readlink (which
recognizes both and works on the 3.11 floor, where os.path.isjunction is absent),
discovery follows junctions and prunes the linked roots from its main walk so a
junction tree is not walked twice, and a link is removed with rmdir for a
junction or unlink for a symlink. Capability is probed once and cached.

* Detach junctions instead of rmtree'ing through them in reset and add-cleanup

Two cleanup paths matched links with is_symlink() and rmtree'd anything else that
was a directory. A Windows junction is not a symlink but is a directory, so both
would have recursed through it and deleted the real linked-in corpus: the
knowledge-base reset (app.reset._clear_dir) and the /add cancel/failure cleanup
(remove_linked_sources).

Both now detect a link with is_link() (symlink or junction) and detach it with
remove_link() (rmdir for a junction, unlink for a symlink), never following it to
the target. The detach primitive moves next to is_link in core.system so every
consumer shares one junction-aware implementation.

* Address ultracode review: prompt cancel in the parallel plan, and cleanups

Parallel planning now cancels queued-but-unstarted work when cancel trips
mid-pass, restoring the prompt cancellation the serial path had; the docstring no
longer overclaims the plan is identical under cancel. relocate_sources opens each
table once instead of per move (LanceDB's update SQL has no CASE, so a single
batched statement is not available, and a delete+re-add across the vector tables
is not worth its risk for a rare mass relabel).

Smaller fixes: the remove confirmation counts only names that actually exist; the
CLI expands remove targets once and passes them through; expand_remove_targets
and remove_documents_durably take the pre-read known set / targets to avoid a
second sources read; --max-cpus is available on sync and rebuild, not just add;
the relocated count now shows in the background/auto-sync summary line;
get_services moves to app.ingest's module top (AGENTS forbids lazy-importing it);
a stale copy_result name and a narrating discovery comment are cleaned up; and the
relocate test now covers page_texts and citations so dropping a re-keyed table
fails.

* Replace the symlink ingest layer with path-tracking

Register external source roots in config (label -> absolute path) and index
their files where they live: no copy, no symlink, no junction. discover_files
walks documents_dir plus each registered root; resolve_source_path maps a stored
key back to its file for every consumer (search, citations, wiki, HTTP serve,
durable remove).

A vanished file is no longer removed on sync: its chunks stay searchable and the
dead path surfaces only when something tries to open it. Move detection is
preserved, now pairing a reappeared identical file to an absent source and
relocating it in place. remove un-registers a root instead of unlinking; the TUI
add-cancel path un-registers too.

Deletes the whole symlink/junction/hardlink/copy shim (symlinks_supported,
_place_source, _create_junction, _hardlink, is_link, remove_link) and the
discovery symlink-escape guard; cross-platform reduces to abspaths + os.walk.

* Update tests for path-tracking ingest

Rework discovery, registration, removal, move-detection, and cancel-cleanup
tests for the registered-roots model: a vanished file stays indexed, add
registers a root instead of copying/symlinking, remove un-registers it, and the
wiki source-resolution helpers resolve through resolve_source_path. Swap the
mock targets (register_sources / unregister_added_roots) and drop the deleted
symlink/junction/copy coverage. Also wrap pre-existing over-length test lines so
ruff on tests is clean.

* Restore the citation traversal guard via the containment-checked resolver

A crafted citation key that climbs out of every allowed root is rejected before
any file is read; registered roots (legitimately outside documents_dir) pass.
Fix the format-source test to inject the resolver failure at its new site.

* Update MCP add tests for source-root registration

Registration records a root instead of copying into documents_dir, so the tests
assert the registry (linked_roots) and exercise root-vs-root collision. Set
data_root in the fixture so the registry persists to a temp config.toml.

* Apply ruff formatting to the path-tracking changes

* Finish ruff formatting of the path-tracking changes

* Address the path-tracking audit findings

- Reject registering a root that nests under/over an existing root or is an
  ancestor of documents_dir, which would index the same file under two keys.
- Never let a registered label shadow an owned documents_dir entry, even under
  --force, so resolve_source_path can't diverge from how discovery keyed it.
- Make register/unregister an atomic read-modify-write of the persisted registry
  under the config lock (settings.mutate_value), so two processes registering
  roots concurrently can't lose each other's entry.
- Replace a tautological reindex test with one asserting the store's recorded
  hash reflects the edit; add nesting/ancestor/owned-shadow/merge coverage.
- Fix stale symlink-era wording in api.py and docs, and fold function-local
  app.ingest re-imports into the existing top-level import.

* Reframe the server force-add test for path-tracking; isolate data_root in its fixture

Force now re-points a registered root instead of overwriting an owned
documents_dir file (a label never shadows an owned entry). Also set data_root in
the server test fixture so registration persists to a temp config.toml, not the
real one.

* Rename test_add_copies_and_syncs for path-tracking; assert no copy into documents_dir

* Document that a registered label owns its key namespace over a later owned subtree

* Apply ruff formatting to a nested-def blank line in pipeline.py

* Add direct unit tests for settings.mutate_value (atomic RMW primitive)

* Cover the coverage gaps: resolve_source_path branches, source_label_taken, empty no-ops, remove confirm

* Cover remaining branch-diff gaps: --max-cpus, SyncResult relocated render, cgroup-v2 malformed, TUI relocated notify

* Keep the embed fleet warm during ingest, and make its saturation knobs tunable (#596)

* Keep the embed fleet warm during ingest, and make its saturation knobs tunable

bb-opgxa: a bulk ingest fans embed requests unevenly across replicas, so a
briefly-idle replica hit engine_idle_ttl_minutes, unloaded, and reloaded cold
mid-run — a connection-refused that snowballed into a fleet collapse. sync() now
holds the whole fleet resident (ttl 0) for the batch via a ContextVar the fleet
spawn reads (propagates into the ingest thread pool), so no replica idle-unloads
while ingest is running.

bb-emkt0: the two hardcoded knobs that starve a multi-GPU embed fleet are now
config fields — embed_batch_sequences (was a fixed 64; larger batches keep a
card's continuous-batching slots full) and ingest_max_inflight (0 = the CPU-bound
auto cap, or an explicit override so more files run their compute phase and every
least-loaded replica stays fed). Defaults preserve current behavior; the values
that lift multi-GPU utilization are set from fleet measurement.

* Make ingest_max_inflight lift the worker-pool ceiling so it works in adaptive mode

The admission gate's permit_max is max_workers() in adaptive mode, so a raised
ingest_max_inflight alone was clamped back to 32 and a multi-GPU fleet stayed
starved. Size the pool to the override too, so the single knob lifts admission
in both static and adaptive concurrency.

* Auto-scale ingest admission with the detected embed fleet (no manual cap)

The auto admission cap was CPU-bound (max(cores//2, replicas), then clamped to
32 by the worker pool), so a many-core multi-GPU box pinned ~4 files per card
and starved the fleet. Now, when ingest_max_inflight is unset, admission and the
worker pool scale with the detected embed replicas (replicas x per-replica
in-flight), so an 8-GPU box is kept fed automatically. The config/env override
stays for expert tuning; the single-card path is unchanged (CPU sizing fits).

* Spread embed dispatch evenly across replicas, and make the plan pass observable (#599)

* Reserve the picked embed replica atomically to stop the dispatch thundering herd

The fleet router picks the least-in-flight replica but only bumps that replica's
counter once the HTTP call starts, so under a bulk ingest many threads select the
same momentarily-idlest replica at once and pile onto it -- leaving the other
cards idle (measured 10-98% per-card util on 8x A100, capping throughput well
short of linear). Reserve the slot inside the same critical section as the pick
so concurrent selectors see the assignment and spread across replicas; release it
when the call resolves. Applies to the embed/rerank failover path.

* Log plan-pass progress so a large piped ingest is not silent

The plan pass (walk + stat + hash of every file) can run for tens of minutes on a multi-million-file corpus, while the Rich progress bar renders nothing without a TTY and piped stdout is block-buffered, so a headless run looks hung. It now logs periodic progress for both the discovery walk and the classify/hash pass: files examined, rate, and (once the total is known) an ETA. The one-shot line reporting the auto-chosen ingest concurrency moves to the same level for the same reason.

The lines go out at WARNING, not INFO. The default LILBEE_LOG_LEVEL is WARNING, so an INFO line would be filtered before reaching any handler and a headless sync would stay exactly as silent, which is the failure this exists to fix. The same reasoning already governs the sync-time hint in wiki/ingest.py. Both heartbeats are interval-gated, so a short sync stays quiet.

* Fix the admission-scaling test and cover the fleet-probe fallback

The auto-scaling admission change (#596) raised the no-vision concurrency to roughly eight in-flight per embed replica, but its unit test still asserted the old one-per-replica value and the fleet-probe except path had no coverage, so the 100% gate was red on both. Update the assertion to the per-replica target and add a test that forces the replica probe to raise and asserts the fallback to zero.