Skip to content

fix!: pgvector - an empty filter must not delete every document - #4006

Merged
sjrl merged 6 commits into
deepset-ai:mainfrom
ericdelorefice:fix/pgvector-empty-filter-must-not-delete-everything
Sep 29, 2026
Merged

sjrl merged 6 commits into
deepset-ai:mainfrom
ericdelorefice:fix/pgvector-empty-filter-must-not-delete-everything

Conversation

@ericdelorefice

@ericdelorefice ericdelorefice commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Reopening the work from #4000, which was closed automatically by the first-PR limit before it could be read. That limit is satisfied now that #3997 is merged, and GitHub declined to reopen the original, so this is the same branch as a fresh pull request.

It also carries the two review standards from #3997: a method-specific :raises: line, and the removal of a branch my own change had made unreachable.

Problem

delete_by_filter and delete_by_filter_async start with _validate_filters(filters), which begins if filters: and therefore validates nothing when the dict is empty. The methods then skip the WHERE clause entirely and execute DELETE FROM <schema>.<table>, emptying the table and returning the row count as though the filter had matched those rows.

The reference implementation disagrees. InMemoryDocumentStore.delete_by_filter raises FilterError for the same input, because document_matches_filter({}) does:

document_matches_filter(filters={})         -> FilterError: 'operator' key missing in {}
InMemoryDocumentStore.delete_by_filter({})  -> FilterError: 'operator' key missing in {}
PgvectorDocumentStore.delete_by_filter({})  -> DELETE FROM <table>, every row, returns the count

A caller that builds filters dynamically and ends up with {} - a metadata key that turned out to be absent, a condition list that came back empty - empties the store and is told how many documents it "matched".

delete_all_documents() already exists two methods above, so the intentional case has its own entry point.

Reproduction

No database needed. _execute_sql is captured so the statement that would have been sent is visible:

from unittest.mock import MagicMock, patch
import os
os.environ["PG_CONN_STR"] = "postgresql://x/y"
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore

ds = PgvectorDocumentStore(table_name="my_table", schema_name="public")
cur = MagicMock()
cur.rowcount = 7
captured = {}

def exec_sql(sql_query=None, params=None, error_msg="", cursor=None, **kw):
    captured["sql"] = sql_query.as_string(None)
    captured["params"] = params
    return cur

with patch.object(ds, "_execute_sql", side_effect=exec_sql), \
     patch.object(ds, "_ensure_db_setup"):
    ds._cursor = cur
    print("RETURNED:", ds.delete_by_filter({}))
print("SQL SENT:", captured["sql"])
print("PARAMS  :", captured["params"])

On main:

RETURNED: 7
SQL SENT: DELETE FROM "public"."my_table"
PARAMS  : ()

No WHERE clause, no parameters, and the row count is returned as though a filtered delete had run.
_validate_filters({}) does not reject an empty dict, so nothing upstream of this stops it.

Fix

Raise FilterError when the filter is empty, matching what the in-memory store does with the same input, and point at delete_all_documents() in the message.

The guard is local to the delete paths on purpose. An empty filter is legitimate for reads - filter_documents(filters=None) returns everything - so _validate_filters is left alone rather than tightened for every caller.

Verified

  • pytest -m "not integration" tests: 100 passed, no pre-existing failures. (The integration class needs a live Postgres; these are unit tests and need none - the guard runs before _ensure_db_setup.)
  • Two tests added, one per path. Both fail on main and pass with this change.
  • ruff check and ruff format --check on src and tests: clean.
  • mypy -p haystack_integrations.document_stores.pgvector: no issues.

Separately, and not in this PR

delete_documents builds its id list with an f-string and wraps the result in psycopg.sql.SQL(...), which marks it as trusted SQL rather than escaping it:

document_ids_str = ", ".join(f"'{document_id}'" for document_id in document_ids)

Identifier() is used correctly for the schema and table on the lines above. The practical effect is that an id containing an apostrophe - one derived from a filename or a title, rather than the default hash - breaks the statement. WHERE id = ANY(%s) with the list passed as a parameter would fix it. Happy to send that as its own PR if you would like it; it is a different concern from this one and I did not want to bundle them.

delete_by_filter and delete_by_filter_async run _validate_filters, which begins
with `if filters:` and so validates nothing when the dict is empty. The methods
then skip the WHERE clause and execute DELETE FROM <schema>.<table>, emptying
the table and returning the row count as though the filter had matched.

The reference implementation disagrees. haystack core's InMemoryDocumentStore
raises FilterError for the same input, because document_matches_filter({}) does:

    document_matches_filter(filters={})        -> FilterError: 'operator' key missing
    InMemoryDocumentStore.delete_by_filter({}) -> FilterError: 'operator' key missing
    PgvectorDocumentStore.delete_by_filter({}) -> DELETE FROM <table>, every row

delete_all_documents() already exists two methods above, so the intentional case
has its own entry point and does not need this one.

The guard is local to the delete paths on purpose. An empty filter is legitimate
for reads - filter_documents(filters=None) returns everything - so _validate_filters
is left alone.

Two unit tests added, one per path. Both fail on main and pass here. 100 unit
tests pass, ruff check and format clean, mypy clean.
Two things @anakin87 corrected on deepset-ai#3997, carried over here rather than waiting to
be told twice.

He added a method-specific :raises: line to every method he changed. This file
already has eleven of them, so it is house style and I had missed it; both
delete_by_filter methods now document the FilterError and point at
delete_all_documents.

He also removed an import my own change had orphaned. The same shape was here: the
guard raises when `filters` is falsy, so the `if filters:` below it can no longer
be false. Removed in both methods.

Not applicable here, but worth recording as the reason I checked: his substantive
correction was dedenting my delete_all_documents handler out of the `else:` branch,
because the recreate_index=True path still swallowed. Verified that this guard sits
above every branch in both methods, so no path reaches the unqualified DELETE
without passing it.

100 unit tests pass, both guards still fail against current main, ruff and mypy
clean at the package's pins.
@ericdelorefice
ericdelorefice requested a review from a team as a code owner September 25, 2026 14:46
@ericdelorefice
ericdelorefice requested review from sjrl and removed request for a team September 25, 2026 14:46
@github-actions github-actions Bot added integration:pgvector type:documentation Improvements or additions to documentation labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report (pgvector)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/pgvector/src/haystack_integrations/document_stores/pgvector
  document_store.py 1090, 1138
Project Total  

This report was generated by python-coverage-comment-action

The two tests added in this PR were inserted directly beneath the
@pytest.mark.usefixtures("patches_for_unit_tests") line that decorated
test_init, so the decorator bound to the first new test and test_init
was left without it. Nothing went red - test_init passes either way -
which is why it survived review: the decorator line never moves, so the
diff shows it as unchanged context and only what it binds to changes.

Each of the two new tests now carries the decorator explicitly and
test_init has its own back. The diff against main for this file is now
22 insertions and 0 deletions, so no existing test is touched.
@sjrl sjrl self-assigned this Sep 28, 2026
@sjrl

sjrl commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Since delete_by_filter({}) now raises instead of deleting everything, could we mark this as breaking (fix!: in the title) so the next pgvector release is a major one?

@sjrl is right. delete_by_filter_async raised the sync message, so a
caller on the async path was told to use delete_all_documents() - a method
that exists, is not the one they want, and is not async. Copy-paste on my
part.

The test could not catch it. It matched "non-empty filter", which is true
of either message, so it passed while the text was wrong. It now pins both
async names, and reverting the message fails it.

Comments shortened to the suggested line in both methods; the rationale is
in the PR description and does not need repeating twice in the source.
@ericdelorefice ericdelorefice changed the title fix(pgvector): an empty filter must not delete every document fix!: pgvector - an empty filter must not delete every document Sep 28, 2026
@ericdelorefice

Copy link
Copy Markdown
Contributor Author

All three taken, in fa177c9. Thank you for the async catch in particular.

The async message. You are right and it was copy-paste. delete_by_filter_async raised the sync text, so a caller on the async path was pointed at delete_all_documents() - a method that exists, is not async, and is not the one they wanted. Both names are correct now.

Worth saying why it survived my own test, because the test was the real failure. It matched on "non-empty filter", which is true of either message, so it passed while the text was wrong. The names are the part a caller acts on, so the test now pins delete_by_filter_async and delete_all_documents_async directly. Reverting the message to the sync one fails it.

The comments. Shortened to your line in both methods. The rationale belongs in the description rather than twice in the source.

Breaking. Agreed, and retitled to fix!: pgvector - .... It is a real behaviour change rather than a defensive addition: delete_by_filter({}) currently empties the table and returns a row count, so anyone relying on that to wipe a store gets a FilterError after this. delete_all_documents() is the replacement and the message names it.

ruff check and ruff format --check clean, and the pgvector unit suite is 37 passed.

@sjrl again. I fixed the message last round and left the docstring one
line above it still pointing at delete_all_documents(). Same defect, same
method, and I had corrected the instance rather than the method.

Swept every async method in the file for references to a sync twin's name
rather than fixing this one by hand. Twenty-one methods have an _async
twin; the only remaining mentions are connection.close(), which is
psycopg's, not the store's.

Test takes the suggested single check. The combined regex fails if either
name is wrong, which is what the two separate assertions were doing more
noisily, and the history belongs in the commit rather than the docstring.
@ericdelorefice

Copy link
Copy Markdown
Contributor Author

Both taken, in 5547c3c.

The :raises: line. You are right, and it is the same defect as last round sitting one line above the thing I fixed. I corrected the instance and not the method.

So this time I swept the file rather than fixing it by hand: twenty-one methods here have an _async twin, and the only remaining mentions of a sync name inside an async body are connection.close(), which is psycopg's and not this store's. Nothing else to find.

The test. Simplified to your single check. The combined regex fails if either name is wrong, which is what the two assertions were doing more noisily, and the history belongs in the commit message rather than the docstring.

ruff check and ruff format --check clean, 37 passed. Reverting the message to the sync text still fails the test.

@ericdelorefice

Copy link
Copy Markdown
Contributor Author

Flagging the red on Python 3.14 on Linux so it does not hold you up: it is a PyPI timeout, not the patch.

The job failed in Install Hatch, before any test ran:

ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)"): /simple/click/
ERROR: ResolutionImpossible

pip retried five times for click, and with an incomplete index it fell back to trying every hatch version and gave up on resolution. Python 3.10 on Linux passed on the same commit, and the pgvector suite is 37 passed locally.

A re-run should clear it. I have not pushed anything to re-trigger, since a no-op commit seemed worse than a note.

@sjrl on both. The sync test now takes `mock_store` instead of setting
PG_CONN_STR and building the store by hand, and the async one moves to
test_document_store_async.py beside
test_count_unique_metadata_by_filter_async_rejects_empty_fields.

`mock_store` already requests `patches_for_unit_tests`, so adopting it
removes the decorator I had been carrying - and with it the whole
orphaning hazard from the first round, where inserting a test beneath that
decorator silently detached it from test_init.

Controlled after moving: removing both guards fails both tests, one in
each file. The first attempt at that control was invalid - the regex left
`if not filters:` with no body, so it failed on an IndentationError rather
than on anything the tests detected.
@ericdelorefice

Copy link
Copy Markdown
Contributor Author

Both done, in 9bff436.

mock_store. Adopted in the sync test. Worth noting it also removed something: mock_store already requests patches_for_unit_tests, so the decorator I had been carrying is gone entirely - and with it the hazard from the first round, where inserting a test directly beneath that decorator silently detached it from test_init. The fixture makes that class of mistake unavailable, which is better than me remembering not to make it.

The async test moved to test_document_store_async.py, beside test_count_unique_metadata_by_filter_async_rejects_empty_fields and in the same shape.

Controlled after the move, since a test that has changed file and fixture is worth re-proving: removing both guards fails both tests, one in each file. ruff check and ruff format --check clean.

One note on the earlier Python 3.14 red - it was a PyPI timeout in Install Hatch and this push should give CI a clean run at it.

@sjrl sjrl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@sjrl
sjrl merged commit 8060fb6 into deepset-ai:main Sep 29, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change integration:pgvector type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants