fix: restore exact in-memory document store snapshots by remote-controlled-man · Pull Request #13696 · deepset-ai/haystack

remote-controlled-man · 2026-11-11T15:07:35Z

Related Issues

No existing issue found for this snapshot-restoration path.

Proposed Changes:

InMemoryDocumentStore.load_from_disk() used OVERWRITE when reconnecting to a saved shared index. Existing IDs were restored, but documents added after the snapshot remained searchable, including when loading an empty snapshot. BM25 statistics could therefore describe a mixture of saved and later data.

Reconstruct the saved documents before touching the index, then clear and rebuild documents and BM25 statistics under the existing per-index lock. Preserve the saved index identity and sharing configuration. With shared=True, other instances using that index observe the restored snapshot. The docstring states this shared-store effect, and the release note records a fix to undocumented merging behavior.

The restoration regression uses shared=True explicitly and only parameterizes empty/non-empty snapshots. The invalid-document preservation test also specifies shared=True. Short inline comments explain why documents are parsed before index mutation and why restoration holds the index lock.

How did you test it?

The two narrowed shared-store regressions fail against the original production source at bbe4efe50c716d6d05a0637980db3b641236e613; the invalid-document preservation control passes. With the fix, all 7 load-from-disk cases pass, including four existing save/load tests covering binary blobs and sparse embeddings.

Checks on final commit ad8f65c7fa7826707d6664041d6c5326ddb4f236, Python 3.12.14 and Hatch 1.18.2:

hatch run test:unit test/document_stores/test_in_memory.py --basetemp=.prflow-cache/pytest: 420 passed, 8 skipped. Existing skips cover unsupported recreate_index / recreate_collection options.
hatch run test:types haystack/document_stores/in_memory/document_store.py test/document_stores/test_in_memory.py: passed.
hatch run fmt-check for both Python files: passed.
hatch run pre-commit run --files for the three changed files: passed, including YAML and release-note checks.

Remote checks for this revision remain pending at publication.

Notes for the reviewer

This PR was fully generated with an AI assistant (OpenAI Codex), including independent agent code review and automated verification. No independent human rerun is claimed. No live model or external document service was used. The full project suite and live-service integrations were not run; local validation targets the affected document-store module.

Checklist

I have read the contributors guidelines and code of conduct.
I added unit tests and updated the docstring.
The PR title follows conventional commits.
I added one fix release note describing the shared-index behavior.
I have completed the final local checks and pre-commit hooks.

vercel · 2026-11-11T15:07:43Z

@remote-controlled-man is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

github-actions · 2026-11-11T15:17:02Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/document_stores/in_memory
document_store.py
Project Total

_{This report was generated by python-coverage-comment-action}

sjrl · 2026-11-12T10:26:27Z

+            documents = [Document.from_dict(doc) for doc in data.pop("documents")]
            cls_object = default_from_dict(cls, data)
-            cls_object.write_documents(
-                documents=[Document.from_dict(doc) for doc in documents], policy=DuplicatePolicy.OVERWRITE
-            )
+            with cls_object._lock:
+                cls_object.delete_all_documents()
+                cls_object.write_documents(documents=documents, policy=DuplicatePolicy.OVERWRITE)


Could we add a short comment for each step here?

Suggested change

documents = [Document.from_dict(doc) for doc in data.pop("documents")]

cls_object = default_from_dict(cls, data)

cls_object.write_documents(

documents=[Document.from_dict(doc) for doc in documents], policy=DuplicatePolicy.OVERWRITE

)

with cls_object._lock:

cls_object.delete_all_documents()

cls_object.write_documents(documents=documents, policy=DuplicatePolicy.OVERWRITE)

# Parse the documents first so an invalid snapshot leaves the index untouched

documents = [Document.from_dict(doc) for doc in data.pop("documents")]

cls_object = default_from_dict(cls, data)

# Hold the index lock so other instances sharing it never see a partially restored index

with cls_object._lock:

cls_object.delete_all_documents()

cls_object.write_documents(documents=documents, policy=DuplicatePolicy.OVERWRITE)

Added both short comments in ad8f65c7fa7826707d6664041d6c5326ddb4f236: why parsing documents first preserves the index on an invalid snapshot, and why restoration holds the index lock. The second comment is scoped to synchronized readers to match the existing locking contract. This follow-up only adds comments; the Python AST is unchanged. The affected module passed again with 420 passed, 8 skipped; type, formatting and pre-commit checks passed. CI for the new commit is pending.

sjrl

Thanks!

fix: restore exact in-memory document store snapshots

2bbe2aa

remote-controlled-man requested a review from a team as a code owner November 11, 2026 15:07

remote-controlled-man requested review from sjrl and a balanced review from Copilot and removed request for a team November 11, 2026 15:07

Copilot AI reviewed Nov 11, 2026

github-actions Bot added topic:tests type:documentation Improvements on the docs labels Nov 11, 2026

sjrl self-assigned this Nov 11, 2026