Repository navigation
[fix](cache) Share cache eligibility for nondeterministic expressions - #68672
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review Please review correctness, performance, reuse and minimality of the shared query/condition cache eligibility check. Focus on scan predicate and virtual-column coverage (including file/TVF paths), the optional Thrift flag during rolling upgrades, seeded shuffle volatility and expression identity, and query-date-dependent TIME casts including SQL cache folding. Verify that deterministic scans remain cacheable. |
There was a problem hiding this comment.
Review complete at head 07f16ef8ca737471c82cdc643ca69942325789b0 (three review rounds; all final reviewers found no new valuable issue). Four distinct findings are inline: two P2 test defects and two P3 cacheability cases. There were no existing inline comments or blocking findings to carry forward. This is a static review; builds and tests were prohibited and were not run.
Goal and proof: The new shared expression check excludes volatile scan predicates and virtual columns from condition cache and extends the query/SQL cache checks. The FE tests check the serialized scan flag and query-cache eligibility; the new regression compares result counts. Those tests do not prove the BE cache gate rejects volatile scans or still hits for deterministic scans (P2). The seeded-shuffle expected count is also dependent on a supported scan batch size (P2). Deterministic new-FE/new-BE scans are enabled by the translator and BE gate in the static trace, but the positive runtime path is not established by the regression.
Scope, reuse, and parallel paths: Moving the prior query-cache exclusion logic into ExpressionUtils.containsNonCacheableExpression is a focused reuse. The changed scan path checks attached filters and OLAP virtual columns, including lazy OLAP translation. File and TVF scanner cache predicates derive from the scan conjuncts; a filter retained above a scan does not enter its cache key. Query-cache aggregate/filter/project checks and the SQL-cache post-fold check were traced separately. The two typed-NULL P3 cases show the new exclusion is broader than needed. Seeded shuffle's volatile identity is appropriate for non-NULL arrays: the BE generator is reseeded per input block, and UniqueFunction preserves identity through rewrites. Non-NULL TIME-to-date casts read the query timestamp and must remain volatile.
Concurrency, lifecycle, and memory: The new per-scan flag is selected during FE planning, serialized into the plan, and read as BE operator state before opening the scan. No new shared mutable state, lock, thread entry, static initializer, ownership transfer, or significant allocation was added; the existing scanner and cache lifecycles are unchanged. No new error/status handling or silent-failure path was found.
Configuration and compatibility: No new configuration item or persistent format was added. The flag is an optional Thrift field with default false, so an old FE paired with a new BE safely disables condition cache. The documented Doris upgrade order upgrades all BEs before FEs. A new FE paired with an old BE would ignore this flag, but that is outside the documented rolling-upgrade sequence. Deterministic scans on a new FE/new BE can still enable the gate under the existing session switch. FE-to-BE field setting, copying, serialization, and BE consumption were checked.
Conditions, tests, and observability: The scan gate combines FE eligibility, the existing query digest, and the TopN restriction; the condition matches the cache's scan-local predicate inputs. FE unit cases cover volatile and deterministic flags, union branches, shuffle identity, TIME casts, and SQL-cache eligibility. The BE already exposes ConditionCacheHit, but the new regression never asserts it or lookup absence (P2). The hardcoded shuffle result is not valid for all supported batch sizes (P2). No runtime test result was independently verified. Existing counters are sufficient to investigate a hit, while a per-scan lookup assertion or isolated global search measurement is needed to prove volatile scans never search.
Data and operations: This PR changes no write, transaction, visible-version, delete-bitmap, EditLog, persistence, or crash-recovery path. No additional data-correctness, error-handling, resource, or performance issue was substantiated in the final sweep of all 14 changed files. The remaining performance issue is safe cache reuse being denied for the two always-NULL expressions (P3).
| order_qt_deterministic_first """ | ||
| SELECT count(*) FROM condition_cache_eligibility WHERE k % 10000 = 0 | ||
| """ | ||
| order_qt_deterministic_second """ |
There was a problem hiding this comment.
[P2] Assert BE cache behavior in both halves of this regression. These queries compare only counts: the volatile rand/shuffle/TIME counts can match even if condition-cache lookup or reuse occurs, while both deterministic k % 10000 = 0 counts stay 10 even if the BE gate disables caching entirely. The DUP-key fixture's complex predicates do reach the BE common-expression cache path. Assert no lookups for the volatile scans using a per-scan counter or isolated search metric, and assert ConditionCacheHit > 0 on the second deterministic scan; the FE test only checks the serialized flag.
| 11 | ||
|
|
||
| -- !shuffle_baseline -- | ||
| 13 |
There was a problem hiding this comment.
[P2] Avoid a fixed seeded-shuffle count here. array_shuffle(a, 1) creates a fresh seeded generator for each BE input block, so the result depends on block boundaries. This suite does not pin batch_size; with the supported batch_size=1, every one of its 100000 identical arrays gets the same permutation and this count is either 0 or 100000, never 13. Assert cache behavior with profile counters, or control the scan block shape before recording a fixed result.
| * with 1 or 2 arguments : array_shuffle(arr) or array_shuffle(arr, seed) | ||
| */ | ||
| public class ArrayShuffle extends ScalarFunction | ||
| public class ArrayShuffle extends UniqueFunction |
There was a problem hiding this comment.
[P3] Preserve NULL propagation for shuffle. array_shuffle(CAST(NULL AS ARRAY<INT>), 1) always returns NULL, but changing this class to UniqueFunction makes foldable() false. FoldConstantRuleOnFE.preProcess exits before its PropagateNullable NULL branch, leaving a volatile expression that disables SQL and query cache reuse for a stable result. Fold the known-NULL case before applying volatility; keep non-NULL shuffle calls volatile.
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: The new volatility checks prevent known-NULL TIME casts and array_shuffle calls from folding, unnecessarily excluding stable queries from SQL/query caches. Allow these NULL cases through the existing folding and type legality checks while retaining volatility for non-NULL inputs. The regression previously compared counts without proving cache behavior and recorded a seeded-shuffle count that depends on input block boundaries. Check isolated BE lookup metrics for volatile predicates at batch sizes 1 and 1024, and assert a profile cache hit on the second deterministic execution. Temporarily disable the global cache default so internal statistics queries cannot pollute the metric; restore it after the test. Generate only stable expected results. ### Release note Preserve constant folding and cache reuse for known-NULL TIME conversions and array shuffle expressions. ### Check List (For Author) - Test: 93 FE unit tests passed; FE build and Checkstyle passed. Three regression suites passed: condition_cache_eligibility (generated output and independent verification), test_cast_time_to_datetime, test_array_functions_by_literal. FE-only build used a temporary BE Rust dependency preflight workaround; the build script was restored. Runtime tests used the existing ASAN BE with FE be_exec_version=13; no BE source changed in this follow-up. - Behavior changed: Yes. Known-NULL expressions remain foldable and cacheable. - Does this need documentation: No
|
/review Please re-review the updated head
Validation: FE build/Checkstyle, 93 FE unit tests, and three regression suites passed. The second deterministic scan recorded ConditionCacheHit=1 and ConditionCacheFilteredRows=79872. Build/runtime environment details are documented in the PR description. Please also check the metric isolation and NULL-folding edge cases for any regressions. |
There was a problem hiding this comment.
Review complete at 6228602 (static inspection only). I request changes for three distinct P1 correctness gaps: a new FE can still send an unsafe condition-cache digest to an older BE; a container cast with nested TIME-to-date conversion is classified as deterministic; and a query-clock cast in a hash-join runtime-filter probe can bypass the per-scan eligibility gate. All three have concrete cache-reuse paths in the inline comments. There are no unresolved candidates and no already-reported P0/P1 comments on this head.
Goal and scope: The new BE scan gate and shared FE expression check protect ordinary volatile scan filters on updated FE/BE pairs, and the change is focused on cache eligibility. The three cases above prevent the broader data-correctness goal from being fully met. I traced FE filter/virtual-column translation, Thrift serialization, BE scan digests and segment/file cache paths, SQL/query-cache folding, and the affected tests. No new data-write, transaction, persistence, visible-version, or MoW path is modified; no discarded Status, untracked large allocation, or cross-TU initialization change was found.
Concurrency and lifecycle: The new flag is per scan and read during BE scan open. Runtime filters may arrive before open and enter the digest/segment common-expression path; that lifecycle makes the third finding reachable. No new lock or lock-order issue was found. The existing condition-cache session switch is dynamically sent in query options; no new configuration item is added. Old FE to new BE defaults safely to disabled, while new FE to old BE has the rolling-upgrade gap in the first finding. The changed field is the only new FE-to-BE value; no storage-format or function-symbol change was found.
Focus and tests: The four earlier P2/P3 threads appear addressed on this head: known-NULL TIME casts and shuffle arguments reach NULL propagation after cast legality checks; non-NULL shuffle remains volatile; the fixed seeded-shuffle count is gone; and the OLAP regression now checks zero BE lookup deltas at batch sizes 1 and 1024 plus a positive second-run ConditionCacheHit. Its temporary global default is restored in finally, and the generated output counts are consistent with the fixture. The FE tests cover these scalar cases and ordinary scan eligibility, but do not cover the three inline cases. The author reports FE and regression validation; I did not run builds or tests because the review instructions prohibit them.
Parallel paths, performance, and observability: The zero digest propagates to OLAP and external file readers for directly flagged scans; the OLAP lookup metric and per-query profile give useful evidence for the added fixture. The metric does not cover external file lookups, so I treat that fixture as OLAP-only evidence. I found no separate hot-path performance regression or observability blocker. No documentation or release-note issue beyond the correctness gaps was substantiated.
| @Override | ||
| public boolean isDeterministic() { | ||
| // TIME has no date. These casts supply the date from the current query's clock. | ||
| return child().isNullLiteral() || !(child().getDataType().isTimeType() && targetType.isDateLikeType()); |
There was a problem hiding this comment.
[P1] Classify nested TIME-to-date casts as volatile. CAST(ARRAY(TIMEDIFF(dt1,dt2)) AS ARRAY<DATETIME>) passes the recursive cast check, and BE's array element cast uses the query timestamp to supply each element's date. Here the source and target are ARRAY types, so isDeterministic() returns true and SQL/query cache can reuse yesterday's result for unchanged rows. The same path exists for MAP and STRUCT members; check nested source/target type pairs before declaring this cast cacheable.
There was a problem hiding this comment.
Fixed in d78bb71. Cast.isDeterministic() now recursively checks source/target ARRAY elements, MAP keys and values, and STRUCT fields, including deeper nesting. Known-NULL inputs still reach existing NULL folding and legality checks; stable nested conversions remain cacheable.
Added expression and SQL/query/condition-cache planner tests. The new tests reproduced the gap before the fix. All 102 targeted FE tests and the FE build/Checkstyle pass. Runtime regression verifies zero BE condition-cache lookups for ARRAY/MAP/STRUCT TIME casts at batch sizes 1 and 1024; generated output was independently verified.
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: TIME-to-date conversions inside ARRAY, MAP and STRUCT casts use the query date, but the outer container cast was classified as deterministic. SQL, query and condition caches could therefore reuse results across dates. Runtime-filter probe expressions such as CAST(CAST(p.s AS TIME) AS DATETIME) also bypassed scan predicate eligibility checks and could reuse stale all-false ranges. Recursively inspect source/target container types and apply shared expression eligibility to singleton and grouped runtime-filter targets, including implicit probe casts. Preserve known-NULL folding, cast legality checks and cache use for deterministic expressions. Add planner, expression and translator unit coverage and verify zero BE cache lookups for the affected scans. ### Release note Prevent incorrect cached results for nested TIME-to-date conversions and query-date-dependent runtime-filter probes. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 102 FE unit tests across seven targeted classes passed. - FE build with Checkstyle passed. - condition_cache_eligibility output generation and verification passed. - test_cast_time_to_datetime and test_array_functions_by_literal passed. - Existing ASAN BE, rebuilt FE; deterministic ConditionCacheHit=1 and ConditionCacheFilteredRows=79872, volatile scans have zero lookup deltas. - Behavior changed: Yes, unsafe nested casts and runtime-filter probes bypass caching. - Does this need documentation: No
|
/review Please re-review the updated head The two remaining supported-path P1 findings are fixed:
For the upgrade finding: the owner confirmed new FE/old BE does not occur under the supported BE-before-FE upgrade procedure. The direct reply is #68672 (comment) . Old FE/new BE continues to default the optional flag to false. Please assess compatibility against this supported sequence. Validation: both bugs reproduced in added tests before the fix; 102 FE unit tests and FE build/Checkstyle passed. The condition-cache regression passed output generation and independent verification, including zero BE lookups for nested casts and repeated RF-probe joins. Deterministic ConditionCacheHit=1, ConditionCacheFilteredRows=79872. TIME-cast and array-literal regression suites also passed. Runtime/build environment details are in the updated PR description. |
There was a problem hiding this comment.
Review complete for head d78bb71e44213a9341a1d4d50afa15d2e32db99b after two convergence rounds and a final sweep of all 16 changed files. I found one distinct P1 correctness issue, posted inline: the new eligibility policy permits deterministic virtual-column scan predicates even though their definitions are absent from the BE condition-cache key. The digest weakness predates this PR; this head explicitly retains eligibility for the affected scans. The earlier nested-cast and runtime-filter P1s are fixed on this head. No prior P0/P1 inline finding remains applicable under the supported upgrade sequence.
User focus: recursive ARRAY/MAP/STRUCT TIME-to-date casts are now classified volatile, while known-NULL casts still pass legality checking and fold. Singleton and grouped runtime-filter targets, including implicit probe casts, downgrade scan eligibility before serialization. The added regression checks zero volatile cache lookups and a deterministic second-scan hit; its expected outputs and test logic were inspected. The Doris upgrade guide requires BE upgrades before FE upgrades, and the optional flag defaults false for old-FE/new-BE operation. No additional issue was substantiated in those focused paths.
Critical checkpoints:
- Goal, design, tests: Shared expression screening and the per-scan FE-to-BE flag address volatile predicates, with FE unit and regression coverage. M1 leaves a wrong-row cache collision for virtual predicates. The changes are otherwise focused; the missing digest identity needs a two-query regression.
- Concurrency and lifecycle: FE sets the plan flag before serializing the scan; BE copies it into the operator and checks it at open. The shared condition cache is used across queries. This PR adds no locks, async owners, static initialization dependencies, or new lifecycle cleanup. M1 is cross-query key reuse, not a newly introduced lock-order problem.
- Configuration and compatibility: No new configuration item was added. Existing session switches control condition cache and virtual-slot CSE. The optional Thrift flag is sent on scan nodes and defaults false when absent; the documented BE-before-FE upgrade order makes the supported mixed-version phase safe.
- Parallel paths and conditions: OLAP, file, TVF, and lazy-scan setup, physical filters, singleton/grouped runtime filters, and virtual columns were traced. RF downgrades are monotone and cover implicit casts. The deterministic virtual-column check at the inline location lacks a cache-key identity check.
- Coverage and expected results: The added FE tests cover nested casts, NULL folding, shuffle volatility, scan flags, and RF targets. The regression asserts lookup counts and a positive cache hit, and its generated
.outvalues match the stated fixtures by static inspection. No test covers M1. Author-reported builds and test runs were not independently repeated: this review environment prohibits builds and tests. - Observability, persistence, and writes: Existing condition-cache lookup, hit, and filtered-row counters support diagnosis. No new transaction, EditLog, data-write, version-visibility, delete-bitmap, or persistent-format path is changed. No new error-handling or memory-ownership defect was substantiated.
- Performance and remaining issues: The per-expression eligibility checks are bounded by plan size; deterministic scans remain eligible. The final changed-file and candidate sweep found no other unresolved point beyond the inline P1.
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: VirtualSlotRef inherits the generic expression digest, which hashes the node type and opcode but omits the separately stored virtual-column definition. Different deterministic virtual predicates can therefore share a condition-cache key. On a DUP table with k=0..99999, first evaluating abs(k + 1) + abs(k + 1) = 0 causes a later predicate abs(k - 9999) + abs(k - 9999) = 0 to miss the matching k=9999 row. Return zero from VirtualSlotRef::get_digest so the existing propagation disables condition-cache reads and writes. Add constructor/parent/scanner digest unit coverage and a regression comparing both queries with uncached base-column predicates while requiring zero cache lookups. ### Release note Disable condition cache for virtual-column predicates to prevent incorrect cache reuse between different virtual-column definitions. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Added regression reproduced the missing k=9999 row on the existing pre-fix ASAN BE. - clang-format 16.0.6, header hygiene, and git diff --check passed. - clang-tidy passed for the changed header and unit-test translation unit using generated UT compiler flags without stale PCH reuse. - BE unit-test build was blocked by missing thirdparty libpaimon_c.a; post-fix runtime validation was not completed. - Behavior changed: Yes. Virtual-column predicates bypass condition cache. - Does this need documentation: No
|
/review Please re-review the updated head
Added BE unit coverage and a regression with two different deterministic virtual-column definitions. EXPLAIN checks ensure CSE actually produces virtual slots. The final regression reproduced missing row Validation: clang-format/check-format 16.0.6, header hygiene, git diff --check, and clang-tidy for the changed header and unit-test translation unit passed. |
There was a problem hiding this comment.
Complete static review of head 9180e96509b5b7b4816cde183bcc667b5c18326d. Two full coverage passes and a separate risk-focused pass returned NO_NEW_VALUABLE_FINDINGS; the final sweep covered all 18 changed files and left no unresolved candidate. No new inline comments are proposed.
User focus and data correctness. VirtualSlotRef::get_digest() returns zero for both construction paths. VExpr, CAST, and runtime-filter digest parents preserve zero; scan-open computation stops on zero before predicate normalization. Scanner refresh returns zero when the local scan digest is zero and rebuilds from its current conjuncts otherwise. OLAP and file condition-cache lookup and insertion require a nonzero digest. This statically addresses the virtual-definition collision in existing P1 4164802395. The regression asserts virtual-slot plans, compares both definitions with uncached reference queries, requires zero lookups, and retains a deterministic second-query cache-hit check. The original pre-fix missing k=9999 result is author-reported; the new BE was not run here.
Critical checkpoints. Goal and tests: the code and added unit/regression cases cover the reported unsafe reuse, subject to the execution limit below. Scope and clarity: the zero sentinel reuses the existing digest failure convention, and the FE eligibility checks share one expression helper; no broader change was needed for this path. Concurrency: late runtime filters are added under the existing conjunct lock, scanners clone a snapshot, and the changed code adds no shared mutable cache state or lock inversion. Lifecycle/static initialization: per-reader cache context is cleared before refresh, and no new global initializer or retained owner appears. Configuration: no new setting is introduced; existing condition-cache and virtual-slot switches are used. Protocol/compatibility: FE serializes the scan flag through optional Thrift field 108, BE reads it before enabling the digest, and an absent old-FE field defaults false. The new-FE/old-BE scenario in P1 4144328662 is outside the documented BE-before-FE upgrade order.
Parallel paths and conditions: OLAP, file, TVF, scan filters, and singleton/grouped runtime-filter targets were checked; eligibility is only downgraded after scan creation, including the probe's implicit cast. Nested TIME-to-date casts, seeded/unseeded shuffle, and typed-NULL folding were traced through SQL, query, and condition-cache classification; the earlier P1s 4144328668 and 4144328681 no longer apply to this head. Test results and observability: the generated .out counts and test assertions are consistent with the fixtures, and existing lookup metrics/profile counters expose disabled cache use and positive deterministic hits. Transaction, persistence, and data writes: none are modified; the material risk was read-side omission of committed rows, addressed by disabling unsafe cache reuse. FE-to-BE variables and performance: the only new transmitted flag reaches scan-open, while ordinary deterministic scans retain the cache path. No additional substantiated correctness or performance issue was found.
Validation limit. This invocation performed static inspection only, as required by the review prompt. The author reports formatting, header hygiene, diff checks, clang-tidy, FE tests, and prior regressions, but those are not independently executed results here. The BE unit test and post-fix runtime behavior remain pending CI because the local dependency set lacks thirdparty/installed/lib64/libpaimon_c.a.
|
Completed post-fix validation on
The PR description now records these results. No code changes since the completed review that found no new issues. |
|
run buildall |
TPC-H: Total hot run time: 29402 ms |
TPC-DS: Total hot run time: 152194 ms |
ClickBench: Total hot run time: 25.58 s |
FE Regression Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 29195 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-DS: Total hot run time: 152275 ms |
ClickBench: Total hot run time: 24.48 s |
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: The new volatility checks prevent known-NULL TIME casts and array_shuffle calls from folding, unnecessarily excluding stable queries from SQL/query caches. Allow these NULL cases through the existing folding and type legality checks while retaining volatility for non-NULL inputs. The regression previously compared counts without proving cache behavior and recorded a seeded-shuffle count that depends on input block boundaries. Check isolated BE lookup metrics for volatile predicates at batch sizes 1 and 1024, and assert a profile cache hit on the second deterministic execution. Temporarily disable the global cache default so internal statistics queries cannot pollute the metric; restore it after the test. Generate only stable expected results. ### Release note Preserve constant folding and cache reuse for known-NULL TIME conversions and array shuffle expressions. ### Check List (For Author) - Test: 93 FE unit tests passed; FE build and Checkstyle passed. Three regression suites passed: condition_cache_eligibility (generated output and independent verification), test_cast_time_to_datetime, test_array_functions_by_literal. FE-only build used a temporary BE Rust dependency preflight workaround; the build script was restored. Runtime tests used the existing ASAN BE with FE be_exec_version=13; no BE source changed in this follow-up. - Behavior changed: Yes. Known-NULL expressions remain foldable and cacheable. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: TIME-to-date conversions inside ARRAY, MAP and STRUCT casts use the query date, but the outer container cast was classified as deterministic. SQL, query and condition caches could therefore reuse results across dates. Runtime-filter probe expressions such as CAST(CAST(p.s AS TIME) AS DATETIME) also bypassed scan predicate eligibility checks and could reuse stale all-false ranges. Recursively inspect source/target container types and apply shared expression eligibility to singleton and grouped runtime-filter targets, including implicit probe casts. Preserve known-NULL folding, cast legality checks and cache use for deterministic expressions. Add planner, expression and translator unit coverage and verify zero BE cache lookups for the affected scans. ### Release note Prevent incorrect cached results for nested TIME-to-date conversions and query-date-dependent runtime-filter probes. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 102 FE unit tests across seven targeted classes passed. - FE build with Checkstyle passed. - condition_cache_eligibility output generation and verification passed. - test_cast_time_to_datetime and test_array_functions_by_literal passed. - Existing ASAN BE, rebuilt FE; deterministic ConditionCacheHit=1 and ConditionCacheFilteredRows=79872, volatile scans have zero lookup deltas. - Behavior changed: Yes, unsafe nested casts and runtime-filter probes bypass caching. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: VirtualSlotRef inherits the generic expression digest, which hashes the node type and opcode but omits the separately stored virtual-column definition. Different deterministic virtual predicates can therefore share a condition-cache key. On a DUP table with k=0..99999, first evaluating abs(k + 1) + abs(k + 1) = 0 causes a later predicate abs(k - 9999) + abs(k - 9999) = 0 to miss the matching k=9999 row. Return zero from VirtualSlotRef::get_digest so the existing propagation disables condition-cache reads and writes. Add constructor/parent/scanner digest unit coverage and a regression comparing both queries with uncached base-column predicates while requiring zero cache lookups. ### Release note Disable condition cache for virtual-column predicates to prevent incorrect cache reuse between different virtual-column definitions. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Added regression reproduced the missing k=9999 row on the existing pre-fix ASAN BE. - clang-format 16.0.6, header hygiene, and git diff --check passed. - clang-tidy passed for the changed header and unit-test translation unit using generated UT compiler flags without stale PCH reuse. - BE unit-test build was blocked by missing thirdparty libpaimon_c.a; post-fix runtime validation was not completed. - Behavior changed: Yes. Virtual-column predicates bypass condition cache. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: The eligibility regression compares BE-wide condition-cache lookup counters. Concurrent suites and internal statistics queries can increment these counters, and fuzzy session initialization can override its global-disable attempt. The assertion can therefore fail even when the tested scan bypasses the cache. Count actual segment lookup attempts, including misses, in OLAP reader statistics and expose ConditionCacheLookup in the scan profile. Identify each regression execution with a unique SQL comment, wait for its completed profile, and require every lookup counter to be zero. Preserve result comparisons and positive deterministic lookup/hit coverage. Remove global metric sampling, global session changes and the nonConcurrent restriction. ### Release note Expose ConditionCacheLookup in OLAP scan profiles. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - ASAN BE build passed using build.sh --be -j 48 with ENABLE_PCH=OFF. - All 37 targeted SegmentIteratorCandidatePushdownTest and VirtualSlotRefTest tests passed using run-be-ut.sh. - condition_cache_eligibility passed with fuzzy session initialization and concurrent cache-eligible queries. The BE-wide metric increased from 36 to 2952 while all 45 ineligible query profiles reported zero lookups; deterministic lookup/hit checks passed. - clang-format/check-format 16.0.6, header hygiene, Groovy compilation, git diff checks and clang-tidy changed-line checks passed. - Behavior changed: Yes. OLAP profiles expose lookup attempts and the regression uses query-scoped counters. Cache eligibility and query results are unchanged. - Does this need documentation: No
06fd0b7 to
5108dd2
Compare
|
/review Please re-review the rebased head The original five commits are retained. Review the integration with master's #68438 (ArrayShuffle UniqueFunction identity and non-constant seed rejection) while preserving NULL-only folding and cache eligibility. Also verify that master's grouped non-blocking runtime-filter behavior is preserved alongside singleton/grouped RF cache eligibility checks. Please review correctness, performance, reuse and minimality, including nested TIME-to-date casts, VirtualSlotRef zero-digest propagation, per-query ConditionCacheLookup assertions, and the supported BE-before-FE rolling upgrade. |
|
run buildall |
There was a problem hiding this comment.
Review complete for PR #68672 at 5108dd2 against a53b182. The five retained commits and all 24 changed paths were reviewed. No new substantiated inline finding remains. This was static inspection only: no build, unit test, or regression suite was run in this review.
Critical checkpoints:
- Goal, size, and reuse: the per-scan FE eligibility flag gates BE condition-cache use; a shared expression classifier covers scan filters, virtual columns, runtime-filter probes, and query-cache checks. The change is focused on cache correctness, with NULL-only folding retained for shuffle and TIME casts. Inspected FE/BE unit tests and regression cases support the intended behavior, but their execution is not independently verified here.
- Concurrency and lifecycle: no new locks, threads, or cross-translation-unit static dependencies. The lookup counter is per reader and reaches the OLAP scan profile when its scanner closes. Cache digest selection occurs before scanner preparation; virtual-slot zero digests propagate through parent expressions and scan conjuncts.
- Conditions, errors, and parallel paths: volatile scan filters and RF probes, including implicit casts and singleton/grouped RF targets, downgrade the scan flag. Grouped RF translation still uses any non-blocking member to keep the merged filter non-blocking. Invalid or ignored RF targets do not assign a filter. The existing cast-legality and nonconstant shuffle-seed rejection paths remain in place.
- Compatibility and configuration: no new operator configuration or persistent format. The new optional FE-to-BE Thrift field defaults false, so an old FE disables the cache on a new BE during the documented BE-before-FE rolling upgrade. The reverse version order is outside that procedure. No transaction, EditLog, or data-write path changes.
- Performance and observability: stable scans remain eligible, while unsafe predicates avoid cache lookups. The ConditionCacheLookup counter counts attempted OLAP segment lookups and the regression selects each query by a unique profile marker; the deterministic control checks a warmed hit. The new SQL-cache plan traversal is limited to the SQL-cache rewrite path. No additional performance defect was substantiated.
- Tests and results: inspected both BE unit tests, all three FE test paths, the two regression suites, and the generated expected output. The regression uses query-scoped lookup assertions for unsafe OLAP scans and a positive deterministic hit. No test result from this review is claimed.
Focus response: the rebased ArrayShuffle retains UniqueFunction identity and rejects nonconstant seeds while NULL-only results fold; nested TIME-to-date casts are classified through ARRAY, MAP, and STRUCT types; virtual-slot digests disable condition-cache use; per-query ConditionCacheLookup assertions cover the OLAP fixture; singleton/grouped RF probes are checked without losing grouped non-blocking behavior; and the optional Thrift default protects the supported BE-before-FE upgrade. Existing P1 threads on virtual slots, RF probes, and nested casts are addressed at this head. The older new-FE/old-BE thread describes an unsupported upgrade order, so no existing P0/P1 inline comment remains applicable.
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28912 ms |
TPC-DS: Total hot run time: 151251 ms |
ClickBench: Total hot run time: 25.47 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
already check
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
### What problem does this PR solve? Issue Number: None Related PR: apache#68672 Problem Summary: The condition-cache eligibility regression backport calls the master-only four-argument quickRunTest API. On branch-4.1 this raises a MissingMethodException before the deterministic cache-hit checks execute in both P0 and Cloud P0. Put the unique profile marker in a SQL comment and use the three-argument API, preserving profile selection and the generated count output. ### Release note None ### Check List (For Author) - Test: Regression framework compilation using bash run-regression-test.sh --compile; 35 framework unit tests passed; Groovy syntax and branch-4.1 API dispatch checked. Full SQL regression is to be run by the PR pipeline. - Behavior changed: No. Test-only compatibility fix. - Does this need documentation: No.
…69292 (#69562) ### What problem does this PR solve? Related PR: #69292 Problem Summary: Backport #69292 to `branch-4.2`. It contains the commit of #69560, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69560 first. `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and `PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating `Score()`, and MATCH projections do not add to the score. See #69292 for the supported and rejected uses. #### Branch adaptations - The `SegmentIterator` of the branch collects the columns of pushed-down expressions in `_extract_common_expr_columns` instead of per-column states, so the score column is marked there, and the new BE test checks `_common_expr_columns` and `_is_common_expr_column`. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read. - When `limit + offset` overflows, `PushDownScoreTopNIntoOlapScan` keeps the behavior of the branch and skips the push-down instead of failing. - `SearchDslParser` is imported from its package on the branch. - The branch does not have #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, `test_score_filter_matrix` lost the rows of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that the branch can run. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and unsupported uses of `score()` fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs. - Does this need documentation? - [x] Yes. The `score()` documentation should describe where `score()` can be used. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.2` at ecbc5fe), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest`, `PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280 tests), and FE Checkstyle. - PASS: `./build.sh --fe --be`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 23 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`, with unchanged `.out` files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
…69292 (#69565) ### What problem does this PR solve? Related PR: #69292 Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit of #69563, the backport of #69399, which it needs: binding runs before CTE inlining, which copies scans with virtual columns. Merge #69563 first. `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to the scan of its SELECT at the start of rewrite and rejects the uses it cannot bind. The minimum score stays a predicate of the scan, and `PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with the minimum. In the BE, an expression on the score reads the score column instead of evaluating `Score()`, and MATCH projections do not add to the score. See #69292 for the supported and rejected uses. #### Branch adaptations - The `SegmentIterator` of the branch collects the columns of pushed-down expressions in `_extract_common_expr_columns` instead of per-column states, so the score column is marked there, and the new BE test checks `_common_expr_columns` and `_is_common_expr_column`. The branch reads the columns collected there before it evaluates the expressions, also virtual ones, and a virtual slot uses a column that is already read. - The branch evaluates MATCH projections with `evaluate_inverted_index` directly, and that loop runs without the score collector. - `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch for `limit + offset`, which fails like master does. - `SearchDslParser` and `CollectionSimilarity` are included from their packages on the branch. - The branch has no SNII storage format, so `BindScoreToOlapScanTest` and `test_score_filter_matrix` cover V2 and V3, and the `.out` of `test_score_filter_matrix` is regenerated without the SNII cases. - The branch does not have #68672, which keeps predicates on virtual columns out of the condition cache. The minimum score is now a predicate of the scan, and a score depends on every rowset, so a cached result goes stale when a rowset is added: without the change, `test_score_filter_matrix` lost the rows of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that the branch can run. - `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column` set `enable_common_expr_pushdown = true`, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and `score()` is 0. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and unsupported uses of `score()` fail planning instead of evaluating it as 0. A minimum score without a TopN needs the BE change, so BEs must be upgraded before FEs. - Does this need documentation? - [x] Yes. The `score()` documentation should describe where `score()` can be used. ### Validation Validated together with the other two backports (#69399, #69419 and #69292 on `branch-4.1` at 3f911d0), in a local ASAN build: - PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`, `InferPredicateByReplaceTest`, `InferPredicatesTest`, `UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`, `TopNRuntimeFilterTest` and `MaterializedViewUtilsTest` (228 tests), and FE Checkstyle. - PASS: `./build.sh --be` and `./build.sh --fe`. - PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`, `SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests); clang-format 16 on the changed BE files. - PASS: the 16 regression suites of the branch that use `score()`, including `test_cte_inline_virtual_column`, `infer_match_predicate`, `test_score_filter_usage` and `test_score_filter_matrix`. The framework regenerated the `.out` of `test_score_filter_matrix` on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other `.out` files are unchanged. - PASS: with the fuzzy session variables of CI (`use_fuzzy_session_variable = true` and its pull request number), the score suites failed in sessions that drew `enable_common_expr_pushdown = false`; with the setting pinned, four runs of `test_score_filter_usage`, `test_score_filter_matrix` and `test_cte_inline_virtual_column`, and four of `infer_match_predicate`, passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Condition cache can incorrectly reuse false ranges for expressions whose results vary across executions, and for virtual-column predicates whose definitions are omitted from the digest. For example, running
abs(k + 1) + abs(k + 1) = 0beforeabs(k - 9999) + abs(k - 9999) = 0can incorrectly omitk = 9999.Share expression eligibility with query cache, inspect scan predicates, virtual columns and runtime-filter probes (including implicit casts), and pass scan eligibility to BE before computing the cache digest. Recognize scalar and ARRAY/MAP/STRUCT TIME-to-date conversions while preserving supported NULL folding. VirtualSlotRef returns the zero-digest sentinel to disable reuse when its definition is absent from the inherited digest.
Add per-query ConditionCacheLookup counters for hits and misses. Regression coverage identifies individual execution profiles, checks that unsafe scans perform zero lookups, and retains positive checks for deterministic cache hits.
Rebased onto master
a53b182f810be2b3731c7dfdf6f40ff5a0ad63bd. Preserve master's #68438 shuffle identity and constant-seed validation while allowing NULL-only folding, and retain master's grouped non-blocking runtime-filter test alongside cache-eligibility coverage. All five original commits are retained.Release note
Prevent incorrect cached results for volatile predicates, query-date-dependent TIME casts, and virtual-column predicates. Preserve supported NULL folding for TIME casts and shuffle calls. Expose ConditionCacheLookup in OLAP scan profiles.
Rolling upgrade: upgrade BE before FE. The optional plan flag defaults false, so a new BE disables condition cache for old FE plans; an old BE cannot enforce the new eligibility flag.
Check List (For Author)
FE_UT_PARALLEL=48 bash run-fe-ut.sh --run 'org.apache.doris.nereids.util.CacheExpressionTest,org.apache.doris.planner.ConditionCacheEligibilityTest,org.apache.doris.nereids.glue.translator.RuntimeFilterTranslatorBucketPruneTest,org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunctionTest'(8 + 6 + 7 + 18).Check List (For Reviewer who merge this PR)