fix(rag): count identical chunks once in a search and cite every copy - #7131

Summary

A document indexed twice under different names no longer fills the model's context with copies of itself. Identical chunks now count once in a search, so the model gets k different chunks. Each collapsed chunk lists every file that holds it, and the agent cites all of them instead of whichever copy was indexed first.

Why

The exact_duplicate hard case in #4733 found duplicates taking 40% of the top 5 chunks given to the model. As the bot noted on the issue, collapsing after FAISS's top-k cut would only trade crowding for a short context, so the search fetches more candidates until it has k distinct ones.

Linked issue

Closes #4740

Changes

• The SDK search collapses identical chunks before cutting to k. It asks FAISS for more results until it has k distinct chunks or has seen the whole index, and records where each chunk's copies sit. chunk_metadata gains source_files / source_paths (every file holding the text), and query_metadata["source_files"] includes them. source_file / source_path stay as they were.
• query_documents cites every file holding the text: source_file is the first, and also_in lists the rest. It resolves positions in one pass instead of rag.chunks.index() per chunk.
• query_specific_file looks a chunk up only in the requested file, so a twin in another document can't supply the page number.
• The RAG SDK guide says identical chunks count once, and fixes its example's meta['source'], a key that doesn't exist.

Test plan

Run in a python:3.12 container:

• pytest tests/unit/rag/test_duplicate_chunks.py: 7 passed. All 7 fail on main (a copy takes a place, a missing source, a citation from the wrong file).
• pytest tests/unit/rag tests/test_rag.py tests/unit/test_chat_rag_budget.py tests/unit/chat/ui/test_sse_handler.py: 1,341 passed, 35 skipped.
• Full pytest tests/unit: 38,405 passed, 584 skipped, 15 failed, of 38,908 collected, with no crashed workers. The 15 fail the same way on main in this container: 14 shell-guardrail and skill-grant tests refuse gh and npm because neither is installed there, and one expects a folder it can't create, but the container runs as root. 44 tests have :: inside their parameters, so they were rerun by file to account for them.
• python util/lint.py --all: passes. Pylint crashed in its parallel workers, so it was rerun alone with --jobs 1, which exits 0.

The tool result the agent reads gains also_in, so rag_quality is the eval category to watch on the next RC.

Evidence

The new tests index one policy under policy.txt and policy (copy).txt beside three other documents, in a real FAISS index:

# main
AssertionError: a copy took one of the 3 places
AssertionError: assert '.../policy.txt' == '.../policy (copy).txt'   # page lookup used the other file

# this PR
7 passed

• Agent exposed in the Agent UI: N/A, no visible change.
• MCP tools / servers: N/A
• CLI: N/A
• HTTP API / REST: N/A

Checklist

• I have linked a GitHub issue above (Closes #N / Fixes #N / Refs #N).
• I have described why this change is being made, not just what changed.
• I have run linting and tests locally (python util/lint.py --all, pytest tests/unit/).
• I have attached real-world evidence matched to the surface I changed (see Evidence above), or marked each surface N/A.
• I have updated documentation if user-visible behavior changed (see CONTRIBUTING.md).