fix(core): let update_index choose passage IDs for unclaimed chunks - #428

What

add_text() without metadata["id"] stamps chunk["id"] = str(len(self.chunks)) — this builder's
insertion position — and update_index() compared that stamp against the target index. So the
first appended chunk was always rejected as a duplicate of passage 0:

ValueError: Passage ID '0' already exists in the index.

and the whole append aborted.

Fix

Treat an ID that came from the sequential scheme rather than from the caller as unclaimed, and resolve
those chunks against the index being updated (skipping IDs already taken there). Caller-chosen
duplicates still raise; content-hash IDs stay content-hash.

Why it matters

examples/dynamic_update_no_recompute.py only works today because it hands every chunk a globally
unique id. A caller who lets the builder choose IDs cannot append to an existing index at all.

Tests

New tests/test_update_index_auto_passage_ids.py (5 tests): 3 failed / 2 passed on the base commit,
every failure the ValueError above; 5 passed with the fix (4 of them run everywhere; the end-to-end
HNSW one is skipped on Intel macOS runners, see below).

The IVF cases assert the IDs update_index() hands to add_vectors() / remove_ids(), with those
two calls standing in for the backend, because they go straight to a FAISS IndexIVF — and the FAISS
bindings the installed HNSW wheel brings into a full-suite environment do not expose the NumPy-friendly
add_with_ids() / remove_ids() overloads, so the append dies in the binding instead of in the code
under test. The passage-ID decision this change fixes happens before any backend runs. The last test
keeps a real end-to-end append through the non-compact HNSW path, so the seam is not the only thing
covered. That last test is skipped on darwin + x86_64: there update_index()'s native
index.add() call (api.py:1194) aborts inside the FAISS bindings (Fatal Python error: Aborted,
exit code 134) and takes the whole pytest session with it. It passes on the arm64 macOS, Ubuntu and
Windows runners, so this is the Intel build of the bindings rather than the code under test -- worth a
separate look, and out of scope here.

pre-commit (pinned ruff / ruff-format) and ty==0.0.17 are green on both files.

Scope

This change leaves the non-compact HNSW branch's own ID assignment alone: after this stage it
re-derives each appended ID as str(index.ntotal + offset) (api.py:1125-1129), because update_index
does not extend <index>.ids.txt — the label→ID map the HNSW searcher reads, which falls back to the
raw label once the map runs out (leann_backend_hnsw/hnsw_backend.py:274-279). So on that path:

• an index whose IDs are the contiguous 0..n-1 that the builder itself produces (the case the new
  tests cover) now appends instead of aborting;
• an index built with non-contiguous IDs still collides: building with IDs 0, 1, 3 and appending one
  chunk stores it as 3, which was already taken, so the offset map keeps a single entry for 3 and
  the earlier passage loses its own — and since .ids.txt still has 3 lines, a hit on that earlier
  vector resolves to the newly appended passage. Making a chosen ID survive that update means
  extending .ids.txt rather than renumbering, which is a separate change; flagging it here so the
  boundary of this fix is explicit.

AI assistance was used for this change.