Repository navigation
Conversation
delete_by_filter and delete_by_filter_async run _validate_filters, which begins
with `if filters:` and so validates nothing when the dict is empty. The methods
then skip the WHERE clause and execute DELETE FROM <schema>.<table>, emptying
the table and returning the row count as though the filter had matched.
The reference implementation disagrees. haystack core's InMemoryDocumentStore
raises FilterError for the same input, because document_matches_filter({}) does:
document_matches_filter(filters={}) -> FilterError: 'operator' key missing
InMemoryDocumentStore.delete_by_filter({}) -> FilterError: 'operator' key missing
PgvectorDocumentStore.delete_by_filter({}) -> DELETE FROM <table>, every row
delete_all_documents() already exists two methods above, so the intentional case
has its own entry point and does not need this one.
The guard is local to the delete paths on purpose. An empty filter is legitimate
for reads - filter_documents(filters=None) returns everything - so _validate_filters
is left alone.
Two unit tests added, one per path. Both fail on main and pass here. 100 unit
tests pass, ruff check and format clean, mypy clean.
Two things @anakin87 corrected on deepset-ai#3997, carried over here rather than waiting to be told twice. He added a method-specific :raises: line to every method he changed. This file already has eleven of them, so it is house style and I had missed it; both delete_by_filter methods now document the FilterError and point at delete_all_documents. He also removed an import my own change had orphaned. The same shape was here: the guard raises when `filters` is falsy, so the `if filters:` below it can no longer be false. Removed in both methods. Not applicable here, but worth recording as the reason I checked: his substantive correction was dedenting my delete_all_documents handler out of the `else:` branch, because the recreate_index=True path still swallowed. Verified that this guard sits above every branch in both methods, so no path reaches the unqualified DELETE without passing it. 100 unit tests pass, both guards still fail against current main, ruff and mypy clean at the package's pins.
Coverage report (pgvector)Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
The two tests added in this PR were inserted directly beneath the
@pytest.mark.usefixtures("patches_for_unit_tests") line that decorated
test_init, so the decorator bound to the first new test and test_init
was left without it. Nothing went red - test_init passes either way -
which is why it survived review: the decorator line never moves, so the
diff shows it as unchanged context and only what it binds to changes.
Each of the two new tests now carries the decorator explicitly and
test_init has its own back. The diff against main for this file is now
22 insertions and 0 deletions, so no existing test is touched.
|
Since |
@sjrl is right. delete_by_filter_async raised the sync message, so a caller on the async path was told to use delete_all_documents() - a method that exists, is not the one they want, and is not async. Copy-paste on my part. The test could not catch it. It matched "non-empty filter", which is true of either message, so it passed while the text was wrong. It now pins both async names, and reverting the message fails it. Comments shortened to the suggested line in both methods; the rationale is in the PR description and does not need repeating twice in the source.
|
All three taken, in The async message. You are right and it was copy-paste. Worth saying why it survived my own test, because the test was the real failure. It matched on The comments. Shortened to your line in both methods. The rationale belongs in the description rather than twice in the source. Breaking. Agreed, and retitled to
|
@sjrl again. I fixed the message last round and left the docstring one line above it still pointing at delete_all_documents(). Same defect, same method, and I had corrected the instance rather than the method. Swept every async method in the file for references to a sync twin's name rather than fixing this one by hand. Twenty-one methods have an _async twin; the only remaining mentions are connection.close(), which is psycopg's, not the store's. Test takes the suggested single check. The combined regex fails if either name is wrong, which is what the two separate assertions were doing more noisily, and the history belongs in the commit rather than the docstring.
|
Both taken, in The So this time I swept the file rather than fixing it by hand: twenty-one methods here have an The test. Simplified to your single check. The combined regex fails if either name is wrong, which is what the two assertions were doing more noisily, and the history belongs in the commit message rather than the docstring.
|
|
Flagging the red on The job failed in pip retried five times for A re-run should clear it. I have not pushed anything to re-trigger, since a no-op commit seemed worse than a note. |
@sjrl on both. The sync test now takes `mock_store` instead of setting PG_CONN_STR and building the store by hand, and the async one moves to test_document_store_async.py beside test_count_unique_metadata_by_filter_async_rejects_empty_fields. `mock_store` already requests `patches_for_unit_tests`, so adopting it removes the decorator I had been carrying - and with it the whole orphaning hazard from the first round, where inserting a test beneath that decorator silently detached it from test_init. Controlled after moving: removing both guards fails both tests, one in each file. The first attempt at that control was invalid - the regex left `if not filters:` with no body, so it failed on an IndentationError rather than on anything the tests detected.
|
Both done, in
The async test moved to Controlled after the move, since a test that has changed file and fixture is worth re-proving: removing both guards fails both tests, one in each file. One note on the earlier |
Reopening the work from #4000, which was closed automatically by the first-PR limit before it could be read. That limit is satisfied now that #3997 is merged, and GitHub declined to reopen the original, so this is the same branch as a fresh pull request.
It also carries the two review standards from #3997: a method-specific
:raises:line, and the removal of a branch my own change had made unreachable.Problem
delete_by_filteranddelete_by_filter_asyncstart with_validate_filters(filters), which beginsif filters:and therefore validates nothing when the dict is empty. The methods then skip the WHERE clause entirely and executeDELETE FROM <schema>.<table>, emptying the table and returning the row count as though the filter had matched those rows.The reference implementation disagrees.
InMemoryDocumentStore.delete_by_filterraisesFilterErrorfor the same input, becausedocument_matches_filter({})does:A caller that builds filters dynamically and ends up with
{}- a metadata key that turned out to be absent, a condition list that came back empty - empties the store and is told how many documents it "matched".delete_all_documents()already exists two methods above, so the intentional case has its own entry point.Reproduction
No database needed.
_execute_sqlis captured so the statement that would have been sent is visible:On
main:No WHERE clause, no parameters, and the row count is returned as though a filtered delete had run.
_validate_filters({})does not reject an empty dict, so nothing upstream of this stops it.Fix
Raise
FilterErrorwhen the filter is empty, matching what the in-memory store does with the same input, and point atdelete_all_documents()in the message.The guard is local to the delete paths on purpose. An empty filter is legitimate for reads -
filter_documents(filters=None)returns everything - so_validate_filtersis left alone rather than tightened for every caller.Verified
pytest -m "not integration" tests: 100 passed, no pre-existing failures. (The integration class needs a live Postgres; these are unit tests and need none - the guard runs before_ensure_db_setup.)mainand pass with this change.ruff checkandruff format --checkonsrcandtests: clean.mypy -p haystack_integrations.document_stores.pgvector: no issues.Separately, and not in this PR
delete_documentsbuilds its id list with an f-string and wraps the result inpsycopg.sql.SQL(...), which marks it as trusted SQL rather than escaping it:Identifier()is used correctly for the schema and table on the lines above. The practical effect is that an id containing an apostrophe - one derived from a filename or a title, rather than the default hash - breaks the statement.WHERE id = ANY(%s)with the list passed as a parameter would fix it. Happy to send that as its own PR if you would like it; it is a different concern from this one and I did not want to bundle them.