Skip to content

[8.6] MOD-14065: Fix property order discrepancy in FILTER#8660

Merged
raz-mon merged 4 commits into8.6from
backport-MOD-14065-to-8.6
Mar 30, 2026
Merged

[8.6] MOD-14065: Fix property order discrepancy in FILTER#8660
raz-mon merged 4 commits into8.6from
backport-MOD-14065-to-8.6

Conversation

@raz-mon
Copy link
Copy Markdown
Collaborator

@raz-mon raz-mon commented Mar 11, 2026

Describe the changes in the pull request

Backport of #8548 to 8.6 - fixes property order discrepancy in FILTER.

  1. Current: During indexing, if a FILTER expression references properties in a certain order, documents may be incorrectly excluded when properties appear in a different order than in the filter expression.
  2. Change: Introduces EVAL_MODE_INDEX and EVAL_MODE_QUERY to distinguish between indexing and query-time evaluation. In EVAL_MODE_INDEX, missing properties no longer cause errors but return undefined values.
  3. Outcome: Consistent behavior regardless of property order in filter expressions during indexing.

Which additional issues this PR fixes

  1. MOD-14332
  2. Includes fix from Wait for index to avoid flakiness in new filter tests #8627: Added waitForIndex calls to prevent test flakiness

Main objects this PR modified

  1. src/aggregate/expr/expression.c - Core evaluation logic with eval mode distinction
  2. tests/pytests/test_filter.py - Added regression tests
  3. tests/cpptests/test_cpp_expr.cpp - Added C++ unit tests

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Release Notes

  • This PR requires release notes
  • This PR does not require release notes

8.6-specific adaptations

  • Kept QueryError_Default() removal in EvalCtx_Create (8.6 differs slightly from master)
  • Used 8.6 error codes where needed

Note

Medium Risk
Changes core expression-evaluation semantics for missing fields by introducing mode-dependent behavior, which can affect which documents are indexed under FILTER rules. Covered by extensive new C++ and Python regression tests, but still touches critical indexing logic.

Overview
Fixes a long-standing inconsistency where indexing-time FILTER expressions could yield different results depending on property order when some fields are missing.

This introduces EVAL_MODE_INDEX vs EVAL_MODE_QUERY in the expression evaluator so indexing treats missing properties as non-matching (false) without raising errors, while query-time evaluation remains strict and still errors on missing properties where expected.

Updates rule/spec filter evaluation to use index mode, keeps FT.ADD ... IF evaluation in query mode, and adds regression coverage for missing-field AND/OR and comparison/type-mismatch cases (C++ unit tests + Python integration tests).

Written by Cursor Bugbot for commit 0caa589. This will update automatically on new commits. Configure here.

@jit-ci
Copy link
Copy Markdown

jit-ci Bot commented Mar 11, 2026

🛡️ Jit Security Scan Results

CRITICAL HIGH MEDIUM

✅ No security findings were detected in this PR


Security scan by Jit

Comment thread src/aggregate/expr/expression.c
@raz-mon raz-mon force-pushed the backport-MOD-14065-to-8.6 branch 2 times, most recently from 35cced7 to 3229aa1 Compare March 12, 2026 10:59
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.87%. Comparing base (221d600) to head (0caa589).
⚠️ Report is 16 commits behind head on 8.6.

Additional details and impacted files
@@            Coverage Diff             @@
##              8.6    #8660      +/-   ##
==========================================
+ Coverage   83.84%   83.87%   +0.03%     
==========================================
  Files         367      367              
  Lines       55653    55656       +3     
  Branches    14321    14321              
==========================================
+ Hits        46664    46684      +20     
+ Misses       8828     8811      -17     
  Partials      161      161              
Flag Coverage Δ
flow 84.88% <97.61%> (+0.10%) ⬆️
unit 50.64% <69.04%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

* WIP make the order of the properties in the FILTER expression not matter for evaluation

* New approach

* Handle missing fields comparison

* Enhance readability

* Enhance tests

* Fix initialization and cleanup

* fix tests

* Remove unused APIs

* Fix leak in test

* Address review

* Explicitly set mode of query evaluators

* Fix leak in other test

* Address review

* small proposal to PR (#8587)

* Revert accidental code removal

---------

Co-authored-by: Joan Fontanals <[email protected]>
@raz-mon raz-mon force-pushed the backport-MOD-14065-to-8.6 branch from 3229aa1 to eec2548 Compare March 12, 2026 15:01
Comment thread src/aggregate/expr/expression.c
raz-mon added 2 commits March 15, 2026 17:36
- Add QueryError_ClearError to properly clean up error messages
- Revert accidental submodule updates to match 8.6 branch
- Revert EvalCtx_Eval return codes back to REDISMODULE_ERR (not EXPR_EVAL_ERR)
- Remove testEvalCtxEvalExprNullExpr and testEvalCtxEvalExprUnknownProperty tests
- Remove QueryError_ClearError from EvalCtx_Destroy (explicit calls remain in callers)
- Update QueryError_ClearError calls in rules.c and spec.c
@raz-mon
Copy link
Copy Markdown
Collaborator Author

raz-mon commented Mar 15, 2026

Approved by me.

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread src/rules.c
Add QueryError_ClearError(&r->status) before EvalCtx_Destroy(r) to prevent
leaking error messages that may be set during filter evaluation (e.g., type
mismatches in RSValue_Cmp).
@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
33.8% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

Comment thread src/document.c
}

ExprEval evaluator = {.err = status, .lookup = &lookup_s, .res = NULL, .srcrow = &row, .root = e};
// In query mode missing properties are errors, as expected for `FT.ADD`.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FT.ADD?

@raz-mon raz-mon added this pull request to the merge queue Mar 30, 2026
Merged via the queue into 8.6 with commit 3904869 Mar 30, 2026
48 of 49 checks passed
@raz-mon raz-mon deleted the backport-MOD-14065-to-8.6 branch March 30, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants