Skip to content

[fix](cache) Share cache eligibility for nondeterministic expressions - #68672

Merged
yiguolei merged 5 commits into
apache:masterfrom
HappenLee:fix-condition-cache-shared-eligibility
Oct 10, 2026
Merged

yiguolei merged 5 commits into
apache:masterfrom
HappenLee:fix-condition-cache-shared-eligibility

Conversation

@HappenLee

@HappenLee HappenLee commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Condition cache can incorrectly reuse false ranges for expressions whose results vary across executions, and for virtual-column predicates whose definitions are omitted from the digest. For example, running abs(k + 1) + abs(k + 1) = 0 before abs(k - 9999) + abs(k - 9999) = 0 can incorrectly omit k = 9999.

Share expression eligibility with query cache, inspect scan predicates, virtual columns and runtime-filter probes (including implicit casts), and pass scan eligibility to BE before computing the cache digest. Recognize scalar and ARRAY/MAP/STRUCT TIME-to-date conversions while preserving supported NULL folding. VirtualSlotRef returns the zero-digest sentinel to disable reuse when its definition is absent from the inherited digest.

Add per-query ConditionCacheLookup counters for hits and misses. Regression coverage identifies individual execution profiles, checks that unsafe scans perform zero lookups, and retains positive checks for deterministic cache hits.

Rebased onto master a53b182f810be2b3731c7dfdf6f40ff5a0ad63bd. Preserve master's #68438 shuffle identity and constant-seed validation while allowing NULL-only folding, and retain master's grouped non-blocking runtime-filter test alongside cache-eligibility coverage. All five original commits are retained.

Release note

Prevent incorrect cached results for volatile predicates, query-date-dependent TIME casts, and virtual-column predicates. Preserve supported NULL folding for TIME casts and shuffle calls. Expose ConditionCacheLookup in OLAP scan profiles.

Rolling upgrade: upgrade BE before FE. The optional plan flag defaults false, so a new BE disables condition cache for old FE plans; an old BE cannot enforce the new eligibility flag.

Check List (For Author)

  • Test:
    • Unit Test: 39 targeted FE tests passed after rebase using FE_UT_PARALLEL=48 bash run-fe-ut.sh --run 'org.apache.doris.nereids.util.CacheExpressionTest,org.apache.doris.planner.ConditionCacheEligibilityTest,org.apache.doris.nereids.glue.translator.RuntimeFilterTranslatorBucketPruneTest,org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunctionTest' (8 + 6 + 7 + 18).
    • Manual test: FE Checkstyle passed with zero violations; repository check-format.sh with clang-format 16.0.6 and BE build hygiene passed. Original patch comparison confirms identical added/removed lines for 23 files; ArrayShuffle adapts to the equivalent functionality now in master.
    • Regression test: BE tests and regression suites were not rerun locally for this rebase; existing coverage is retained and run buildall will verify integration.
  • Behavior changed:
    • Yes. Unsafe scans bypass condition cache; deterministic scans retain eligibility. No additional behavior change introduced by rebase.
  • Does this need documentation?
    • No. Correctness fix using existing cache switches; upgrade considerations are stated above.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee

Copy link
Copy Markdown
Contributor Author

/review Please review correctness, performance, reuse and minimality of the shared query/condition cache eligibility check. Focus on scan predicate and virtual-column coverage (including file/TVF paths), the optional Thrift flag during rolling upgrades, seeded shuffle volatility and expression identity, and query-date-dependent TIME casts including SQL cache folding. Verify that deterministic scans remain cacheable.

@morrySnow morrySnow changed the title [fix](fe) Share cache eligibility for nondeterministic expressions [fix](cache) Share cache eligibility for nondeterministic expressions Sep 30, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at head 07f16ef8ca737471c82cdc643ca69942325789b0 (three review rounds; all final reviewers found no new valuable issue). Four distinct findings are inline: two P2 test defects and two P3 cacheability cases. There were no existing inline comments or blocking findings to carry forward. This is a static review; builds and tests were prohibited and were not run.

Goal and proof: The new shared expression check excludes volatile scan predicates and virtual columns from condition cache and extends the query/SQL cache checks. The FE tests check the serialized scan flag and query-cache eligibility; the new regression compares result counts. Those tests do not prove the BE cache gate rejects volatile scans or still hits for deterministic scans (P2). The seeded-shuffle expected count is also dependent on a supported scan batch size (P2). Deterministic new-FE/new-BE scans are enabled by the translator and BE gate in the static trace, but the positive runtime path is not established by the regression.

Scope, reuse, and parallel paths: Moving the prior query-cache exclusion logic into ExpressionUtils.containsNonCacheableExpression is a focused reuse. The changed scan path checks attached filters and OLAP virtual columns, including lazy OLAP translation. File and TVF scanner cache predicates derive from the scan conjuncts; a filter retained above a scan does not enter its cache key. Query-cache aggregate/filter/project checks and the SQL-cache post-fold check were traced separately. The two typed-NULL P3 cases show the new exclusion is broader than needed. Seeded shuffle's volatile identity is appropriate for non-NULL arrays: the BE generator is reseeded per input block, and UniqueFunction preserves identity through rewrites. Non-NULL TIME-to-date casts read the query timestamp and must remain volatile.

Concurrency, lifecycle, and memory: The new per-scan flag is selected during FE planning, serialized into the plan, and read as BE operator state before opening the scan. No new shared mutable state, lock, thread entry, static initializer, ownership transfer, or significant allocation was added; the existing scanner and cache lifecycles are unchanged. No new error/status handling or silent-failure path was found.

Configuration and compatibility: No new configuration item or persistent format was added. The flag is an optional Thrift field with default false, so an old FE paired with a new BE safely disables condition cache. The documented Doris upgrade order upgrades all BEs before FEs. A new FE paired with an old BE would ignore this flag, but that is outside the documented rolling-upgrade sequence. Deterministic scans on a new FE/new BE can still enable the gate under the existing session switch. FE-to-BE field setting, copying, serialization, and BE consumption were checked.

Conditions, tests, and observability: The scan gate combines FE eligibility, the existing query digest, and the TopN restriction; the condition matches the cache's scan-local predicate inputs. FE unit cases cover volatile and deterministic flags, union branches, shuffle identity, TIME casts, and SQL-cache eligibility. The BE already exposes ConditionCacheHit, but the new regression never asserts it or lookup absence (P2). The hardcoded shuffle result is not valid for all supported batch sizes (P2). No runtime test result was independently verified. Existing counters are sufficient to investigate a hit, while a per-scan lookup assertion or isolated global search measurement is needed to prove volatile scans never search.

Data and operations: This PR changes no write, transaction, visible-version, delete-bitmap, EditLog, persistence, or crash-recovery path. No additional data-correctness, error-handling, resource, or performance issue was substantiated in the final sweep of all 14 changed files. The remaining performance issue is safe cache reuse being denied for the two always-NULL expressions (P3).

order_qt_deterministic_first """
SELECT count(*) FROM condition_cache_eligibility WHERE k % 10000 = 0
"""
order_qt_deterministic_second """

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Assert BE cache behavior in both halves of this regression. These queries compare only counts: the volatile rand/shuffle/TIME counts can match even if condition-cache lookup or reuse occurs, while both deterministic k % 10000 = 0 counts stay 10 even if the BE gate disables caching entirely. The DUP-key fixture's complex predicates do reach the BE common-expression cache path. Assert no lookups for the volatile scans using a per-scan counter or isolated search metric, and assert ConditionCacheHit > 0 on the second deterministic scan; the FE test only checks the serialized flag.

11

-- !shuffle_baseline --
13

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid a fixed seeded-shuffle count here. array_shuffle(a, 1) creates a fresh seeded generator for each BE input block, so the result depends on block boundaries. This suite does not pin batch_size; with the supported batch_size=1, every one of its 100000 identical arrays gets the same permutation and this count is either 0 or 100000, never 13. Assert cache behavior with profile counters, or control the scan block shape before recording a fixed result.

* with 1 or 2 arguments : array_shuffle(arr) or array_shuffle(arr, seed)
*/
public class ArrayShuffle extends ScalarFunction
public class ArrayShuffle extends UniqueFunction

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Preserve NULL propagation for shuffle. array_shuffle(CAST(NULL AS ARRAY<INT>), 1) always returns NULL, but changing this class to UniqueFunction makes foldable() false. FoldConstantRuleOnFE.preProcess exits before its PropagateNullable NULL branch, leaving a volatile expression that disables SQL and query cache reuse for a stable result. Fold the known-NULL case before applying volatility; keep non-NULL shuffle calls volatile.

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Sep 30, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: The new volatility checks prevent known-NULL TIME casts and
array_shuffle calls from folding, unnecessarily excluding stable queries from
SQL/query caches. Allow these NULL cases through the existing folding and type
legality checks while retaining volatility for non-NULL inputs.

The regression previously compared counts without proving cache behavior and
recorded a seeded-shuffle count that depends on input block boundaries. Check
isolated BE lookup metrics for volatile predicates at batch sizes 1 and 1024,
and assert a profile cache hit on the second deterministic execution. Temporarily
disable the global cache default so internal statistics queries cannot pollute
the metric; restore it after the test. Generate only stable expected results.

### Release note

Preserve constant folding and cache reuse for known-NULL TIME conversions and
array shuffle expressions.

### Check List (For Author)

- Test: 93 FE unit tests passed; FE build and Checkstyle passed. Three regression
  suites passed: condition_cache_eligibility (generated output and independent
  verification), test_cast_time_to_datetime, test_array_functions_by_literal.
  FE-only build used a temporary BE Rust dependency preflight workaround; the
  build script was restored. Runtime tests used the existing ASAN BE with FE
  be_exec_version=13; no BE source changed in this follow-up.
- Behavior changed: Yes. Known-NULL expressions remain foldable and cacheable.
- Does this need documentation: No
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review Please re-review the updated head 6228602b195dc000f25bb371fe2ea5d6661bb1ba and verify the four prior inline findings are addressed.

  • Known-NULL TIME casts and array_shuffle/shuffle arguments now reach existing NULL folding; non-NULL calls remain volatile. Unsupported cast pairs still retain their legality checks.
  • The regression checks zero BE cache lookups for volatile scans at batch sizes 1 and 1024, plus a positive ConditionCacheHit on the second deterministic execution. It isolates global metrics from concurrent suites and internal statistics sessions, restoring the global setting afterward.
  • Removed the fixed seeded-shuffle count; expected output was generated and independently verified.

Validation: FE build/Checkstyle, 93 FE unit tests, and three regression suites passed. The second deterministic scan recorded ConditionCacheHit=1 and ConditionCacheFilteredRows=79872. Build/runtime environment details are documented in the PR description. Please also check the metric isolation and NULL-folding edge cases for any regressions.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at 6228602 (static inspection only). I request changes for three distinct P1 correctness gaps: a new FE can still send an unsafe condition-cache digest to an older BE; a container cast with nested TIME-to-date conversion is classified as deterministic; and a query-clock cast in a hash-join runtime-filter probe can bypass the per-scan eligibility gate. All three have concrete cache-reuse paths in the inline comments. There are no unresolved candidates and no already-reported P0/P1 comments on this head.

Goal and scope: The new BE scan gate and shared FE expression check protect ordinary volatile scan filters on updated FE/BE pairs, and the change is focused on cache eligibility. The three cases above prevent the broader data-correctness goal from being fully met. I traced FE filter/virtual-column translation, Thrift serialization, BE scan digests and segment/file cache paths, SQL/query-cache folding, and the affected tests. No new data-write, transaction, persistence, visible-version, or MoW path is modified; no discarded Status, untracked large allocation, or cross-TU initialization change was found.

Concurrency and lifecycle: The new flag is per scan and read during BE scan open. Runtime filters may arrive before open and enter the digest/segment common-expression path; that lifecycle makes the third finding reachable. No new lock or lock-order issue was found. The existing condition-cache session switch is dynamically sent in query options; no new configuration item is added. Old FE to new BE defaults safely to disabled, while new FE to old BE has the rolling-upgrade gap in the first finding. The changed field is the only new FE-to-BE value; no storage-format or function-symbol change was found.

Focus and tests: The four earlier P2/P3 threads appear addressed on this head: known-NULL TIME casts and shuffle arguments reach NULL propagation after cast legality checks; non-NULL shuffle remains volatile; the fixed seeded-shuffle count is gone; and the OLAP regression now checks zero BE lookup deltas at batch sizes 1 and 1024 plus a positive second-run ConditionCacheHit. Its temporary global default is restored in finally, and the generated output counts are consistent with the fixture. The FE tests cover these scalar cases and ordinary scan eligibility, but do not cover the three inline cases. The author reports FE and regression validation; I did not run builds or tests because the review instructions prohibit them.

Parallel paths, performance, and observability: The zero digest propagates to OLAP and external file readers for directly flagged scans; the OLAP lookup metric and per-query profile give useful evidence for the added fixture. The metric does not cover external file lookups, so I treat that fixture as OLAP-only evidence. I found no separate hot-path performance regression or observability blocker. No documentation or release-note issue beyond the correctness gaps was substantiated.

Comment thread gensrc/thrift/PlanNodes.thrift Resolved
@Override
public boolean isDeterministic() {
// TIME has no date. These casts supply the date from the current query's clock.
return child().isNullLiteral() || !(child().getDataType().isTimeType() && targetType.isDateLikeType());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Classify nested TIME-to-date casts as volatile. CAST(ARRAY(TIMEDIFF(dt1,dt2)) AS ARRAY<DATETIME>) passes the recursive cast check, and BE's array element cast uses the query timestamp to supply each element's date. Here the source and target are ARRAY types, so isDeterministic() returns true and SQL/query cache can reuse yesterday's result for unchanged rows. The same path exists for MAP and STRUCT members; check nested source/target type pairs before declaring this cast cacheable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d78bb71. Cast.isDeterministic() now recursively checks source/target ARRAY elements, MAP keys and values, and STRUCT fields, including deeper nesting. Known-NULL inputs still reach existing NULL folding and legality checks; stable nested conversions remain cacheable.

Added expression and SQL/query/condition-cache planner tests. The new tests reproduced the gap before the fix. All 102 targeted FE tests and the FE build/Checkstyle pass. Runtime regression verifies zero BE condition-cache lookups for ARRAY/MAP/STRUCT TIME casts at batch sizes 1 and 1024; generated output was independently verified.

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Oct 2, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: TIME-to-date conversions inside ARRAY, MAP and STRUCT casts
use the query date, but the outer container cast was classified as
deterministic. SQL, query and condition caches could therefore reuse results
across dates. Runtime-filter probe expressions such as
CAST(CAST(p.s AS TIME) AS DATETIME) also bypassed scan predicate eligibility
checks and could reuse stale all-false ranges.

Recursively inspect source/target container types and apply shared expression
eligibility to singleton and grouped runtime-filter targets, including implicit
probe casts. Preserve known-NULL folding, cast legality checks and cache use
for deterministic expressions. Add planner, expression and translator unit
coverage and verify zero BE cache lookups for the affected scans.

### Release note

Prevent incorrect cached results for nested TIME-to-date conversions and
query-date-dependent runtime-filter probes.

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - 102 FE unit tests across seven targeted classes passed.
    - FE build with Checkstyle passed.
    - condition_cache_eligibility output generation and verification passed.
    - test_cast_time_to_datetime and test_array_functions_by_literal passed.
    - Existing ASAN BE, rebuilt FE; deterministic ConditionCacheHit=1 and
      ConditionCacheFilteredRows=79872, volatile scans have zero lookup deltas.
- Behavior changed: Yes, unsafe nested casts and runtime-filter probes bypass caching.
- Does this need documentation: No
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review Please re-review the updated head d78bb71e44213a9341a1d4d50afa15d2e32db99b.

The two remaining supported-path P1 findings are fixed:

  • Recursively classify TIME-to-date conversions in ARRAY/MAP/STRUCT casts, preserving known-NULL folding and cast legality checks.
  • Check runtime-filter target expressions and implicit probe casts in both singleton and grouped translation, including grouped targets with differing expressions. Never reenable a previously ineligible scan.

For the upgrade finding: the owner confirmed new FE/old BE does not occur under the supported BE-before-FE upgrade procedure. The direct reply is #68672 (comment) . Old FE/new BE continues to default the optional flag to false. Please assess compatibility against this supported sequence.

Validation: both bugs reproduced in added tests before the fix; 102 FE unit tests and FE build/Checkstyle passed. The condition-cache regression passed output generation and independent verification, including zero BE lookups for nested casts and repeated RF-probe joins. Deterministic ConditionCacheHit=1, ConditionCacheFilteredRows=79872. TIME-cast and array-literal regression suites also passed. Runtime/build environment details are in the updated PR description.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete for head d78bb71e44213a9341a1d4d50afa15d2e32db99b after two convergence rounds and a final sweep of all 16 changed files. I found one distinct P1 correctness issue, posted inline: the new eligibility policy permits deterministic virtual-column scan predicates even though their definitions are absent from the BE condition-cache key. The digest weakness predates this PR; this head explicitly retains eligibility for the affected scans. The earlier nested-cast and runtime-filter P1s are fixed on this head. No prior P0/P1 inline finding remains applicable under the supported upgrade sequence.

User focus: recursive ARRAY/MAP/STRUCT TIME-to-date casts are now classified volatile, while known-NULL casts still pass legality checking and fold. Singleton and grouped runtime-filter targets, including implicit probe casts, downgrade scan eligibility before serialization. The added regression checks zero volatile cache lookups and a deterministic second-scan hit; its expected outputs and test logic were inspected. The Doris upgrade guide requires BE upgrades before FE upgrades, and the optional flag defaults false for old-FE/new-BE operation. No additional issue was substantiated in those focused paths.

Critical checkpoints:

  • Goal, design, tests: Shared expression screening and the per-scan FE-to-BE flag address volatile predicates, with FE unit and regression coverage. M1 leaves a wrong-row cache collision for virtual predicates. The changes are otherwise focused; the missing digest identity needs a two-query regression.
  • Concurrency and lifecycle: FE sets the plan flag before serializing the scan; BE copies it into the operator and checks it at open. The shared condition cache is used across queries. This PR adds no locks, async owners, static initialization dependencies, or new lifecycle cleanup. M1 is cross-query key reuse, not a newly introduced lock-order problem.
  • Configuration and compatibility: No new configuration item was added. Existing session switches control condition cache and virtual-slot CSE. The optional Thrift flag is sent on scan nodes and defaults false when absent; the documented BE-before-FE upgrade order makes the supported mixed-version phase safe.
  • Parallel paths and conditions: OLAP, file, TVF, and lazy-scan setup, physical filters, singleton/grouped runtime filters, and virtual columns were traced. RF downgrades are monotone and cover implicit casts. The deterministic virtual-column check at the inline location lacks a cache-key identity check.
  • Coverage and expected results: The added FE tests cover nested casts, NULL folding, shuffle volatility, scan flags, and RF targets. The regression asserts lookup counts and a positive cache hit, and its generated .out values match the stated fixtures by static inspection. No test covers M1. Author-reported builds and test runs were not independently repeated: this review environment prohibits builds and tests.
  • Observability, persistence, and writes: Existing condition-cache lookup, hit, and filtered-row counters support diagnosis. No new transaction, EditLog, data-write, version-visibility, delete-bitmap, or persistent-format path is changed. No new error-handling or memory-ownership defect was substantiated.
  • Performance and remaining issues: The per-expression eligibility checks are bounded by plan size; deterministic scans remain eligible. The final changed-file and candidate sweep found no other unresolved point beyond the inline P1.

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Oct 4, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: VirtualSlotRef inherits the generic expression digest, which
hashes the node type and opcode but omits the separately stored virtual-column
definition. Different deterministic virtual predicates can therefore share a
condition-cache key. On a DUP table with k=0..99999, first evaluating
abs(k + 1) + abs(k + 1) = 0 causes a later predicate
abs(k - 9999) + abs(k - 9999) = 0 to miss the matching k=9999 row.
Return zero from VirtualSlotRef::get_digest so the existing propagation disables
condition-cache reads and writes. Add constructor/parent/scanner digest unit
coverage and a regression comparing both queries with uncached base-column
predicates while requiring zero cache lookups.

### Release note

Disable condition cache for virtual-column predicates to prevent incorrect
cache reuse between different virtual-column definitions.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - Added regression reproduced the missing k=9999 row on the existing pre-fix ASAN BE.
    - clang-format 16.0.6, header hygiene, and git diff --check passed.
    - clang-tidy passed for the changed header and unit-test translation unit using generated UT compiler flags without stale PCH reuse.
    - BE unit-test build was blocked by missing thirdparty libpaimon_c.a; post-fix runtime validation was not completed.
- Behavior changed: Yes. Virtual-column predicates bypass condition cache.
- Does this need documentation: No
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review Please re-review the updated head 9180e96509b5b7b4816cde183bcc667b5c18326d, focusing on the remaining virtual-column condition-cache P1.

VirtualSlotRef::get_digest() now returns the existing zero-digest sentinel. Verify its propagation through parent expressions, scan conjuncts and scanner refresh, and that zero disables both cache lookup and insertion. Ordinary deterministic predicates retain the existing positive cache-hit regression.

Added BE unit coverage and a regression with two different deterministic virtual-column definitions. EXPLAIN checks ensure CSE actually produces virtual slots. The final regression reproduced missing row k=9999 on the existing pre-fix ASAN BE when the second query followed the first; uncached reference queries return the correct result. The original review's redundant AND comparisons are simplified away, so the regression uses repeated addition instead.

Validation: clang-format/check-format 16.0.6, header hygiene, git diff --check, and clang-tidy for the changed header and unit-test translation unit passed. run-be-ut.sh --run -j 48 --filter=VirtualSlotRefTest.ConditionCacheDigest* is blocked before compilation by missing thirdparty/installed/lib64/libpaimon_c.a. The new BE was not built, so post-fix runtime validation remains pending CI. Please assess the fix with that explicit limitation.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete static review of head 9180e96509b5b7b4816cde183bcc667b5c18326d. Two full coverage passes and a separate risk-focused pass returned NO_NEW_VALUABLE_FINDINGS; the final sweep covered all 18 changed files and left no unresolved candidate. No new inline comments are proposed.

User focus and data correctness. VirtualSlotRef::get_digest() returns zero for both construction paths. VExpr, CAST, and runtime-filter digest parents preserve zero; scan-open computation stops on zero before predicate normalization. Scanner refresh returns zero when the local scan digest is zero and rebuilds from its current conjuncts otherwise. OLAP and file condition-cache lookup and insertion require a nonzero digest. This statically addresses the virtual-definition collision in existing P1 4164802395. The regression asserts virtual-slot plans, compares both definitions with uncached reference queries, requires zero lookups, and retains a deterministic second-query cache-hit check. The original pre-fix missing k=9999 result is author-reported; the new BE was not run here.

Critical checkpoints. Goal and tests: the code and added unit/regression cases cover the reported unsafe reuse, subject to the execution limit below. Scope and clarity: the zero sentinel reuses the existing digest failure convention, and the FE eligibility checks share one expression helper; no broader change was needed for this path. Concurrency: late runtime filters are added under the existing conjunct lock, scanners clone a snapshot, and the changed code adds no shared mutable cache state or lock inversion. Lifecycle/static initialization: per-reader cache context is cleared before refresh, and no new global initializer or retained owner appears. Configuration: no new setting is introduced; existing condition-cache and virtual-slot switches are used. Protocol/compatibility: FE serializes the scan flag through optional Thrift field 108, BE reads it before enabling the digest, and an absent old-FE field defaults false. The new-FE/old-BE scenario in P1 4144328662 is outside the documented BE-before-FE upgrade order.

Parallel paths and conditions: OLAP, file, TVF, scan filters, and singleton/grouped runtime-filter targets were checked; eligibility is only downgraded after scan creation, including the probe's implicit cast. Nested TIME-to-date casts, seeded/unseeded shuffle, and typed-NULL folding were traced through SQL, query, and condition-cache classification; the earlier P1s 4144328668 and 4144328681 no longer apply to this head. Test results and observability: the generated .out counts and test assertions are consistent with the fixtures, and existing lookup metrics/profile counters expose disabled cache use and positive deterministic hits. Transaction, persistence, and data writes: none are modified; the material risk was read-side omission of committed rows, addressed by disabling unsafe cache reuse. FE-to-BE variables and performance: the only new transmitted flag reaches scan-open, while ordinary deterministic scans retain the cache path. No additional substantiated correctness or performance issue was found.

Validation limit. This invocation performed static inspection only, as required by the review prompt. The author reports formatting, header hygiene, diff checks, clang-tidy, FE tests, and prior regressions, but those are not independently executed results here. The BE unit test and post-fix runtime behavior remain pending CI because the local dependency set lacks thirdparty/installed/lib64/libpaimon_c.a.

@HappenLee

Copy link
Copy Markdown
Contributor Author

Completed post-fix validation on 9180e96509b; the missing-dependency limitation in the previous review is now resolved.

  • Built libpaimon_c.a and liblance_c.a with Rust 1.94.0, plus the repository-pinned simdutf and AWS SDK dependencies, locally in the worktree.
  • ASAN BE build passed with ENABLE_PCH=OFF ./build.sh --be -j 48. PCH was disabled only for an existing wide::integer include-order error; no unrelated source changes were made.
  • All 18 selected BE unit tests passed: VirtualSlotRefTest.* and FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload.
  • condition_cache_eligibility and virtual_slot_ref_basic passed against the rebuilt BE. The previously missing 9999 row is returned correctly, virtual-column predicates perform zero condition-cache lookups, and ordinary deterministic predicates still hit the cache.
  • Post-build clang-tidy passed for the modified header.

The PR description now records these results. No code changes since the completed review that found no new issues.

@HappenLee

Copy link
Copy Markdown
Contributor Author

run buildall

924060929
924060929 previously approved these changes Oct 8, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29402 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9180e96509b5b7b4816cde183bcc667b5c18326d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17607	5662	5588	5588
q2	2202	315	260	260
q3	10146	1454	868	868
q4	4679	486	355	355
q5	7456	815	547	547
q6	186	194	154	154
q7	831	820	620	620
q8	9241	1384	1056	1056
q9	5755	4562	4488	4488
q10	6838	1293	1053	1053
q11	434	270	237	237
q12	635	411	295	295
q13	18077	3118	2404	2404
q14	277	264	247	247
q15	q16	758	743	687	687
q17	1210	743	566	566
q18	7818	6271	6215	6215
q19	1124	965	600	600
q20	387	342	230	230
q21	4972	2634	2627	2627
q22	408	368	305	305
Total cold run time: 101041 ms
Total hot run time: 29402 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6729	6643	6781	6643
q2	751	612	606	606
q3	5576	5673	5381	5381
q4	2051	2221	1599	1599
q5	5808	5718	5778	5718
q6	240	185	147	147
q7	2285	1993	1884	1884
q8	3177	2794	2776	2776
q9	8269	8166	7932	7932
q10	3646	3604	3250	3250
q11	608	407	370	370
q12	668	699	505	505
q13	2750	3059	2379	2379
q14	281	286	258	258
q15	q16	679	683	615	615
q17	7653	7089	6931	6931
q18	13133	12301	13029	12301
q19	920	804	793	793
q20	2203	2182	1959	1959
q21	5820	4719	4875	4719
q22	547	467	418	418
Total cold run time: 73794 ms
Total hot run time: 67184 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152194 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9180e96509b5b7b4816cde183bcc667b5c18326d, data reload: false

query5	4308	596	455	455
query6	422	240	203	203
query7	4813	444	245	245
query8	325	187	173	173
query9	8776	3957	3966	3957
query10	478	319	255	255
query11	6009	3403	3188	3188
query12	144	102	85	85
query13	1276	446	316	316
query14	6534	4814	4517	4517
query14_1	4247	4253	4180	4180
query15	207	198	183	183
query16	960	449	424	424
query17	870	661	533	533
query18	2426	428	339	339
query19	198	161	138	138
query20	81	80	81	80
query21	214	131	114	114
query22	12958	13118	12833	12833
query23	13220	12449	12119	12119
query23_1	12204	12310	12324	12310
query24	7133	830	471	471
query24_1	482	475	473	473
query25	540	425	371	371
query26	1260	286	137	137
query27	2745	427	260	260
query28	4634	1921	1917	1917
query29	1535	581	443	443
query30	301	218	186	186
query31	884	764	647	647
query32	148	96	86	86
query33	493	300	240	240
query34	898	846	490	490
query35	737	754	628	628
query36	856	812	739	739
query37	129	98	88	88
query38	1805	1735	1687	1687
query39	716	687	689	687
query39_1	679	683	677	677
query40	223	117	98	98
query41	65	63	61	61
query42	92	80	81	80
query43	365	372	322	322
query44	1292	681	679	679
query45	174	178	163	163
query46	846	952	584	584
query47	3030	2974	2967	2967
query48	296	303	211	211
query49	563	387	294	294
query50	682	272	203	203
query51	10587	10340	10277	10277
query52	80	81	71	71
query53	192	207	151	151
query54	231	184	186	184
query55	72	67	68	67
query56	232	212	226	212
query57	1583	1595	1537	1537
query58	289	266	251	251
query59	2377	2396	2132	2132
query60	281	236	207	207
query61	147	151	147	147
query62	398	339	283	283
query63	200	158	176	158
query64	2744	971	890	890
query65	3434	3375	3364	3364
query66	1799	423	316	316
query67	20423	20464	20170	20170
query68	3028	974	617	617
query69	398	293	260	260
query70	933	866	849	849
query71	307	229	213	213
query72	2837	2650	2247	2247
query73	527	533	295	295
query74	4597	4483	4293	4293
query75	2314	2280	1960	1960
query76	2304	1019	584	584
query77	353	393	300	300
query78	9181	9031	8524	8524
query79	976	837	510	510
query80	1234	441	363	363
query81	616	318	278	278
query82	739	135	111	111
query83	305	197	176	176
query84	316	131	101	101
query85	881	451	371	371
query86	408	240	225	225
query87	2009	1954	1822	1822
query88	3609	2696	2668	2668
query89	345	301	256	256
query90	1934	188	183	183
query91	163	159	129	129
query92	99	85	85	85
query93	1019	972	557	557
query94	641	303	285	285
query95	576	349	395	349
query96	678	514	229	229
query97	2483	2429	2308	2308
query98	149	148	142	142
query99	771	766	659	659
Total cold run time: 235837 ms
Total hot run time: 152194 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9180e96509b5b7b4816cde183bcc667b5c18326d, data reload: false

query1	0.01	0.00	0.01
query2	0.17	0.09	0.09
query3	0.42	0.20	0.21
query4	1.62	0.22	0.22
query5	0.32	0.30	0.31
query6	1.15	0.67	0.66
query7	0.04	0.01	0.00
query8	0.09	0.08	0.07
query9	0.55	0.40	0.40
query10	0.63	0.60	0.58
query11	0.33	0.20	0.18
query12	0.33	0.20	0.19
query13	0.51	0.52	0.52
query14	0.91	0.89	0.88
query15	0.68	0.59	0.59
query16	0.37	0.38	0.38
query17	1.05	1.00	1.03
query18	0.34	0.31	0.31
query19	1.92	1.84	1.80
query20	0.02	0.01	0.01
query21	15.46	0.35	0.30
query22	4.84	0.13	0.13
query23	15.85	0.50	0.30
query24	2.54	0.53	0.39
query25	0.16	0.10	0.10
query26	0.76	0.27	0.21
query27	0.10	0.09	0.09
query28	3.62	0.86	0.44
query29	12.52	4.30	3.33
query30	0.45	0.22	0.22
query31	2.77	0.61	0.35
query32	3.24	0.63	0.51
query33	3.04	2.98	2.98
query34	15.84	4.06	3.43
query35	3.37	3.40	3.38
query36	0.61	0.51	0.50
query37	0.13	0.09	0.10
query38	0.08	0.06	0.06
query39	0.06	0.07	0.06
query40	0.19	0.17	0.17
query41	0.11	0.05	0.06
query42	0.07	0.06	0.06
query43	0.07	0.06	0.05
Total cold run time: 97.34 s
Total hot run time: 25.58 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 16.87% (42/249) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29195 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9180e96509b5b7b4816cde183bcc667b5c18326d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17399	5661	5536	5536
q2	2130	317	256	256
q3	10244	1400	853	853
q4	4683	479	354	354
q5	7456	819	543	543
q6	190	187	154	154
q7	794	795	607	607
q8	9264	1379	1090	1090
q9	5697	4474	4466	4466
q10	6822	1297	1039	1039
q11	448	258	237	237
q12	630	403	286	286
q13	18081	3066	2406	2406
q14	283	269	244	244
q15	q16	740	728	685	685
q17	1216	749	571	571
q18	7160	6278	6218	6218
q19	1136	976	594	594
q20	402	337	225	225
q21	5607	2519	2618	2519
q22	388	372	312	312
Total cold run time: 100770 ms
Total hot run time: 29195 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6639	6653	6737	6653
q2	758	626	587	587
q3	5618	5689	5362	5362
q4	2059	2156	1590	1590
q5	5698	5616	5794	5616
q6	244	187	142	142
q7	2262	1996	1849	1849
q8	3131	2859	2780	2780
q9	8277	8244	8213	8213
q10	3669	3604	3284	3284
q11	614	406	359	359
q12	665	694	498	498
q13	2734	3081	2402	2402
q14	281	280	265	265
q15	q16	666	689	622	622
q17	7797	7015	6883	6883
q18	13151	12305	12999	12305
q19	934	786	846	786
q20	2195	2170	1930	1930
q21	5775	4699	4805	4699
q22	502	435	400	400
Total cold run time: 73669 ms
Total hot run time: 67225 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 50.00% (2/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 65.15% (30803/47282)
Line Coverage 49.96% (323963/648423)
Region Coverage 45.41% (261708/576329)
Branch Coverage 47.09% (122843/260854)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.75% (35121/45760)
Line Coverage 62.00% (397435/641070)
Region Coverage 58.48% (335625/573942)
Branch Coverage 59.35% (154342/260051)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152275 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9180e96509b5b7b4816cde183bcc667b5c18326d, data reload: false

query5	4315	603	458	458
query6	428	217	199	199
query7	4831	443	219	219
query8	352	180	173	173
query9	8736	4032	4026	4026
query10	470	309	261	261
query11	6021	3431	3202	3202
query12	140	87	82	82
query13	1248	452	324	324
query14	6528	4758	4570	4570
query14_1	4335	4284	4275	4275
query15	205	212	183	183
query16	945	437	426	426
query17	865	646	547	547
query18	2419	428	315	315
query19	201	168	164	164
query20	84	78	80	78
query21	214	139	114	114
query22	13001	13055	12870	12870
query23	13249	12635	12125	12125
query23_1	12346	12302	12279	12279
query24	7060	862	453	453
query24_1	465	454	450	450
query25	527	396	348	348
query26	1261	252	138	138
query27	2768	444	268	268
query28	4604	1936	1942	1936
query29	1542	598	442	442
query30	294	220	184	184
query31	895	768	643	643
query32	133	91	84	84
query33	505	298	233	233
query34	922	875	500	500
query35	753	758	651	651
query36	817	817	728	728
query37	133	103	92	92
query38	1798	1733	1695	1695
query39	709	722	681	681
query39_1	654	672	646	646
query40	216	115	94	94
query41	69	68	66	66
query42	90	81	81	81
query43	369	391	331	331
query44	1343	695	708	695
query45	187	175	161	161
query46	841	959	588	588
query47	2889	2990	2850	2850
query48	293	307	226	226
query49	600	394	302	302
query50	711	285	213	213
query51	10271	10449	10056	10056
query52	85	79	71	71
query53	194	214	162	162
query54	230	192	177	177
query55	88	79	64	64
query56	224	238	201	201
query57	1600	1578	1588	1578
query58	274	249	247	247
query59	2377	2404	2130	2130
query60	283	231	220	220
query61	147	143	138	138
query62	389	346	284	284
query63	207	168	169	168
query64	2720	923	761	761
query65	3479	3392	3376	3376
query66	1775	423	302	302
query67	20312	20270	20205	20205
query68	2917	1045	648	648
query69	396	287	241	241
query70	943	859	891	859
query71	293	227	209	209
query72	2760	2600	2316	2316
query73	520	576	301	301
query74	4610	4497	4302	4302
query75	2358	2289	1956	1956
query76	2272	1068	640	640
query77	366	398	309	309
query78	8985	8987	8453	8453
query79	886	867	535	535
query80	474	422	375	375
query81	523	333	279	279
query82	203	137	113	113
query83	201	203	179	179
query84	296	125	104	104
query85	777	422	371	371
query86	253	240	230	230
query87	1981	1969	1784	1784
query88	3634	2729	2689	2689
query89	333	310	271	271
query90	2079	184	173	173
query91	161	145	122	122
query92	93	90	90	90
query93	1000	1016	558	558
query94	469	337	296	296
query95	584	351	313	313
query96	676	530	241	241
query97	2477	2434	2321	2321
query98	149	163	154	154
query99	775	746	657	657
Total cold run time: 233195 ms
Total hot run time: 152275 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9180e96509b5b7b4816cde183bcc667b5c18326d, data reload: false

query1	0.00	0.00	0.01
query2	0.10	0.05	0.04
query3	0.27	0.12	0.12
query4	1.61	0.12	0.12
query5	0.30	0.26	0.23
query6	1.16	0.68	0.69
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.43	0.36	0.37
query10	0.61	0.59	0.57
query11	0.21	0.14	0.13
query12	0.18	0.15	0.14
query13	0.51	0.51	0.49
query14	1.01	0.96	0.98
query15	0.65	0.61	0.61
query16	0.31	0.36	0.36
query17	1.16	1.14	1.14
query18	0.23	0.21	0.22
query19	2.06	1.94	2.03
query20	0.02	0.01	0.02
query21	15.45	0.24	0.14
query22	4.51	0.06	0.05
query23	16.23	0.32	0.12
query24	3.15	0.56	0.44
query25	0.11	0.06	0.04
query26	1.18	0.25	0.14
query27	0.06	0.04	0.04
query28	1.80	0.64	0.41
query29	12.54	4.17	3.28
query30	0.30	0.14	0.13
query31	2.77	0.60	0.34
query32	3.22	0.62	0.52
query33	3.15	3.15	3.22
query34	15.38	4.06	3.38
query35	3.36	3.38	3.34
query36	0.56	0.45	0.45
query37	0.10	0.06	0.06
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 95.21 s
Total hot run time: 24.48 s

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: The new volatility checks prevent known-NULL TIME casts and
array_shuffle calls from folding, unnecessarily excluding stable queries from
SQL/query caches. Allow these NULL cases through the existing folding and type
legality checks while retaining volatility for non-NULL inputs.

The regression previously compared counts without proving cache behavior and
recorded a seeded-shuffle count that depends on input block boundaries. Check
isolated BE lookup metrics for volatile predicates at batch sizes 1 and 1024,
and assert a profile cache hit on the second deterministic execution. Temporarily
disable the global cache default so internal statistics queries cannot pollute
the metric; restore it after the test. Generate only stable expected results.

### Release note

Preserve constant folding and cache reuse for known-NULL TIME conversions and
array shuffle expressions.

### Check List (For Author)

- Test: 93 FE unit tests passed; FE build and Checkstyle passed. Three regression
  suites passed: condition_cache_eligibility (generated output and independent
  verification), test_cast_time_to_datetime, test_array_functions_by_literal.
  FE-only build used a temporary BE Rust dependency preflight workaround; the
  build script was restored. Runtime tests used the existing ASAN BE with FE
  be_exec_version=13; no BE source changed in this follow-up.
- Behavior changed: Yes. Known-NULL expressions remain foldable and cacheable.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: TIME-to-date conversions inside ARRAY, MAP and STRUCT casts
use the query date, but the outer container cast was classified as
deterministic. SQL, query and condition caches could therefore reuse results
across dates. Runtime-filter probe expressions such as
CAST(CAST(p.s AS TIME) AS DATETIME) also bypassed scan predicate eligibility
checks and could reuse stale all-false ranges.

Recursively inspect source/target container types and apply shared expression
eligibility to singleton and grouped runtime-filter targets, including implicit
probe casts. Preserve known-NULL folding, cast legality checks and cache use
for deterministic expressions. Add planner, expression and translator unit
coverage and verify zero BE cache lookups for the affected scans.

### Release note

Prevent incorrect cached results for nested TIME-to-date conversions and
query-date-dependent runtime-filter probes.

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - 102 FE unit tests across seven targeted classes passed.
    - FE build with Checkstyle passed.
    - condition_cache_eligibility output generation and verification passed.
    - test_cast_time_to_datetime and test_array_functions_by_literal passed.
    - Existing ASAN BE, rebuilt FE; deterministic ConditionCacheHit=1 and
      ConditionCacheFilteredRows=79872, volatile scans have zero lookup deltas.
- Behavior changed: Yes, unsafe nested casts and runtime-filter probes bypass caching.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: VirtualSlotRef inherits the generic expression digest, which
hashes the node type and opcode but omits the separately stored virtual-column
definition. Different deterministic virtual predicates can therefore share a
condition-cache key. On a DUP table with k=0..99999, first evaluating
abs(k + 1) + abs(k + 1) = 0 causes a later predicate
abs(k - 9999) + abs(k - 9999) = 0 to miss the matching k=9999 row.
Return zero from VirtualSlotRef::get_digest so the existing propagation disables
condition-cache reads and writes. Add constructor/parent/scanner digest unit
coverage and a regression comparing both queries with uncached base-column
predicates while requiring zero cache lookups.

### Release note

Disable condition cache for virtual-column predicates to prevent incorrect
cache reuse between different virtual-column definitions.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - Added regression reproduced the missing k=9999 row on the existing pre-fix ASAN BE.
    - clang-format 16.0.6, header hygiene, and git diff --check passed.
    - clang-tidy passed for the changed header and unit-test translation unit using generated UT compiler flags without stale PCH reuse.
    - BE unit-test build was blocked by missing thirdparty libpaimon_c.a; post-fix runtime validation was not completed.
- Behavior changed: Yes. Virtual-column predicates bypass condition cache.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary:
The eligibility regression compares BE-wide condition-cache lookup counters.
Concurrent suites and internal statistics queries can increment these counters,
and fuzzy session initialization can override its global-disable attempt.
The assertion can therefore fail even when the tested scan bypasses the cache.

Count actual segment lookup attempts, including misses, in OLAP reader
statistics and expose ConditionCacheLookup in the scan profile. Identify each
regression execution with a unique SQL comment, wait for its completed profile,
and require every lookup counter to be zero. Preserve result comparisons and
positive deterministic lookup/hit coverage. Remove global metric sampling,
global session changes and the nonConcurrent restriction.

### Release note

Expose ConditionCacheLookup in OLAP scan profiles.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - ASAN BE build passed using build.sh --be -j 48 with ENABLE_PCH=OFF.
    - All 37 targeted SegmentIteratorCandidatePushdownTest and VirtualSlotRefTest tests passed using run-be-ut.sh.
    - condition_cache_eligibility passed with fuzzy session initialization and concurrent cache-eligible queries. The BE-wide metric increased from 36 to 2952 while all 45 ineligible query profiles reported zero lookups; deterministic lookup/hit checks passed.
    - clang-format/check-format 16.0.6, header hygiene, Groovy compilation, git diff checks and clang-tidy changed-line checks passed.
- Behavior changed: Yes. OLAP profiles expose lookup attempts and the regression uses query-scoped counters. Cache eligibility and query results are unchanged.
- Does this need documentation: No
@HappenLee
HappenLee force-pushed the fix-condition-cache-shared-eligibility branch from 06fd0b7 to 5108dd2 Compare October 9, 2026 10:27
@HappenLee

Copy link
Copy Markdown
Contributor Author

/review Please re-review the rebased head 5108dd2fef09332bb3683f0346a88613c0477bb2 against master a53b182f810be2b3731c7dfdf6f40ff5a0ad63bd.

The original five commits are retained. Review the integration with master's #68438 (ArrayShuffle UniqueFunction identity and non-constant seed rejection) while preserving NULL-only folding and cache eligibility. Also verify that master's grouped non-blocking runtime-filter behavior is preserved alongside singleton/grouped RF cache eligibility checks.

Please review correctness, performance, reuse and minimality, including nested TIME-to-date casts, VirtualSlotRef zero-digest propagation, per-query ConditionCacheLookup assertions, and the supported BE-before-FE rolling upgrade.

@HappenLee

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete for PR #68672 at 5108dd2 against a53b182. The five retained commits and all 24 changed paths were reviewed. No new substantiated inline finding remains. This was static inspection only: no build, unit test, or regression suite was run in this review.

Critical checkpoints:

  • Goal, size, and reuse: the per-scan FE eligibility flag gates BE condition-cache use; a shared expression classifier covers scan filters, virtual columns, runtime-filter probes, and query-cache checks. The change is focused on cache correctness, with NULL-only folding retained for shuffle and TIME casts. Inspected FE/BE unit tests and regression cases support the intended behavior, but their execution is not independently verified here.
  • Concurrency and lifecycle: no new locks, threads, or cross-translation-unit static dependencies. The lookup counter is per reader and reaches the OLAP scan profile when its scanner closes. Cache digest selection occurs before scanner preparation; virtual-slot zero digests propagate through parent expressions and scan conjuncts.
  • Conditions, errors, and parallel paths: volatile scan filters and RF probes, including implicit casts and singleton/grouped RF targets, downgrade the scan flag. Grouped RF translation still uses any non-blocking member to keep the merged filter non-blocking. Invalid or ignored RF targets do not assign a filter. The existing cast-legality and nonconstant shuffle-seed rejection paths remain in place.
  • Compatibility and configuration: no new operator configuration or persistent format. The new optional FE-to-BE Thrift field defaults false, so an old FE disables the cache on a new BE during the documented BE-before-FE rolling upgrade. The reverse version order is outside that procedure. No transaction, EditLog, or data-write path changes.
  • Performance and observability: stable scans remain eligible, while unsafe predicates avoid cache lookups. The ConditionCacheLookup counter counts attempted OLAP segment lookups and the regression selects each query by a unique profile marker; the deterministic control checks a warmed hit. The new SQL-cache plan traversal is limited to the SQL-cache rewrite path. No additional performance defect was substantiated.
  • Tests and results: inspected both BE unit tests, all three FE test paths, the two regression suites, and the generated expected output. The regression uses query-scoped lookup assertions for unsafe OLAP scans and a positive deterministic hit. No test result from this review is claimed.

Focus response: the rebased ArrayShuffle retains UniqueFunction identity and rejects nonconstant seeds while NULL-only results fold; nested TIME-to-date casts are classified through ARRAY, MAP, and STRUCT types; virtual-slot digests disable condition-cache use; per-query ConditionCacheLookup assertions cover the OLAP fixture; singleton/grouped RF probes are checked without losing grouped non-blocking behavior; and the optional Thrift default protects the supported BE-before-FE upgrade. Existing P1 threads on virtual slots, RF probes, and nested casts are addressed at this head. The older new-FE/old-BE thread describes an unsupported upgrade order, so no existing P0/P1 inline comment remains applicable.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 50.00% (2/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 65.23% (30941/47431)
Line Coverage 50.04% (325402/650342)
Region Coverage 45.47% (263031/578453)
Branch Coverage 47.14% (123424/261812)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28912 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5108dd2fef09332bb3683f0346a88613c0477bb2, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17600	5728	5621	5621
q2	2118	311	268	268
q3	10173	1401	843	843
q4	4681	472	359	359
q5	7444	793	543	543
q6	185	188	158	158
q7	774	812	631	631
q8	9245	1377	1069	1069
q9	5711	4445	4472	4445
q10	6880	1297	1030	1030
q11	453	254	240	240
q12	633	417	286	286
q13	18048	3072	2407	2407
q14	283	266	243	243
q15	q16	746	721	678	678
q17	1195	744	577	577
q18	7112	6249	6183	6183
q19	1111	970	596	596
q20	387	334	228	228
q21	5066	2239	2427	2239
q22	362	313	268	268
Total cold run time: 100207 ms
Total hot run time: 28912 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6154	6133	6128	6128
q2	672	584	534	534
q3	5273	5440	4936	4936
q4	2019	2151	1530	1530
q5	5252	5181	5989	5181
q6	251	194	150	150
q7	2300	2103	1829	1829
q8	3178	2781	2784	2781
q9	8352	8164	8286	8164
q10	3811	3711	3293	3293
q11	627	455	411	411
q12	689	730	559	559
q13	2946	3349	2493	2493
q14	316	324	278	278
q15	q16	698	710	646	646
q17	8357	7665	7241	7241
q18	13095	12226	13007	12226
q19	933	777	785	777
q20	2171	2172	1927	1927
q21	5728	4640	4797	4640
q22	493	472	398	398
Total cold run time: 73315 ms
Total hot run time: 66122 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151251 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5108dd2fef09332bb3683f0346a88613c0477bb2, data reload: false

query5	4301	594	465	465
query6	417	218	197	197
query7	4802	445	222	222
query8	321	182	178	178
query9	8734	4007	4022	4007
query10	467	317	262	262
query11	5873	3437	3163	3163
query12	146	89	84	84
query13	1256	419	326	326
query14	6546	4794	4441	4441
query14_1	4248	4248	4254	4248
query15	207	204	177	177
query16	937	435	460	435
query17	844	675	520	520
query18	2404	409	297	297
query19	189	149	124	124
query20	76	84	73	73
query21	204	132	105	105
query22	12992	13007	12841	12841
query23	13120	12546	12010	12010
query23_1	12177	12163	12119	12119
query24	7060	815	456	456
query24_1	452	450	451	450
query25	495	390	322	322
query26	1259	248	130	130
query27	2784	427	258	258
query28	4643	1915	1909	1909
query29	1536	568	429	429
query30	284	213	185	185
query31	875	780	639	639
query32	141	85	91	85
query33	498	297	234	234
query34	919	843	492	492
query35	741	733	646	646
query36	790	832	765	765
query37	130	97	83	83
query38	1803	1736	1692	1692
query39	711	692	685	685
query39_1	659	674	675	674
query40	215	114	90	90
query41	67	62	66	62
query42	88	85	84	84
query43	368	380	325	325
query44	1303	689	673	673
query45	177	176	162	162
query46	849	963	559	559
query47	3028	3001	2932	2932
query48	316	314	223	223
query49	600	386	292	292
query50	678	266	206	206
query51	10474	10617	10106	10106
query52	75	78	77	77
query53	184	202	151	151
query54	220	184	170	170
query55	68	68	62	62
query56	229	207	210	207
query57	1616	1594	1586	1586
query58	281	256	249	249
query59	2338	2382	2140	2140
query60	264	230	217	217
query61	139	148	145	145
query62	385	342	282	282
query63	191	159	173	159
query64	2710	911	771	771
query65	3432	3392	3375	3375
query66	1767	407	302	302
query67	20325	20255	20062	20062
query68	2962	992	598	598
query69	380	280	248	248
query70	925	855	847	847
query71	299	223	213	213
query72	2837	2722	2382	2382
query73	522	533	288	288
query74	4598	4533	4309	4309
query75	2392	2300	1940	1940
query76	2287	1008	594	594
query77	377	408	309	309
query78	9207	9032	8432	8432
query79	962	835	501	501
query80	1250	524	361	361
query81	577	323	276	276
query82	581	135	107	107
query83	291	190	171	171
query84	306	119	102	102
query85	838	406	355	355
query86	396	235	231	231
query87	1983	1983	1807	1807
query88	3566	2663	2648	2648
query89	331	288	262	262
query90	1874	178	172	172
query91	154	138	116	116
query92	95	84	74	74
query93	982	982	547	547
query94	637	308	282	282
query95	581	395	314	314
query96	663	511	236	236
query97	2454	2419	2314	2314
query98	160	145	144	144
query99	788	759	650	650
Total cold run time: 234413 ms
Total hot run time: 151251 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5108dd2fef09332bb3683f0346a88613c0477bb2, data reload: false

query1	0.01	0.01	0.00
query2	0.15	0.09	0.09
query3	0.37	0.21	0.21
query4	1.62	0.21	0.21
query5	0.31	0.29	0.29
query6	1.16	0.65	0.65
query7	0.04	0.01	0.01
query8	0.09	0.08	0.07
query9	0.49	0.39	0.39
query10	0.59	0.57	0.58
query11	0.33	0.19	0.17
query12	0.32	0.19	0.19
query13	0.52	0.52	0.51
query14	0.92	0.88	0.88
query15	0.66	0.58	0.58
query16	0.36	0.37	0.36
query17	1.00	1.03	1.03
query18	0.30	0.29	0.29
query19	1.95	1.87	1.82
query20	0.02	0.01	0.02
query21	15.41	0.35	0.30
query22	4.80	0.12	0.12
query23	15.86	0.48	0.28
query24	2.40	0.57	0.42
query25	0.16	0.11	0.11
query26	0.91	0.26	0.22
query27	0.11	0.09	0.10
query28	3.43	0.80	0.41
query29	12.44	4.26	3.32
query30	0.37	0.22	0.22
query31	2.76	0.60	0.34
query32	3.22	0.63	0.50
query33	3.02	2.98	2.96
query34	15.76	4.12	3.44
query35	3.38	3.38	3.38
query36	0.57	0.50	0.49
query37	0.14	0.09	0.10
query38	0.07	0.07	0.06
query39	0.06	0.06	0.06
query40	0.19	0.18	0.17
query41	0.11	0.06	0.05
query42	0.06	0.06	0.05
query43	0.06	0.06	0.06
Total cold run time: 96.5 s
Total hot run time: 25.47 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.75% (35237/45910)
Line Coverage 61.95% (398330/643009)
Region Coverage 58.37% (336273/576073)
Branch Coverage 59.22% (154578/261013)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 28.57% (52/182) 🎉
Increment coverage report
Complete coverage report

@HappenLee
HappenLee dismissed stale reviews from github-actions[bot] and github-actions[bot] October 10, 2026 03:31

already check

@yiguolei
yiguolei merged commit 326c136 into apache:master Oct 10, 2026
45 of 46 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Oct 14, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68672

Problem Summary: The condition-cache eligibility regression backport calls the
master-only four-argument quickRunTest API. On branch-4.1 this raises a
MissingMethodException before the deterministic cache-hit checks execute in both
P0 and Cloud P0. Put the unique profile marker in a SQL comment and use the
three-argument API, preserving profile selection and the generated count output.

### Release note

None

### Check List (For Author)

- Test: Regression framework compilation using bash run-regression-test.sh --compile;
    35 framework unit tests passed; Groovy syntax and branch-4.1 API dispatch checked.
    Full SQL regression is to be run by the PR pipeline.
- Behavior changed: No. Test-only compatibility fix.
- Does this need documentation: No.
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
…69292 (#69562)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.2`. It contains the commit
of #69560, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69560
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- When `limit + offset` overflows, `PushDownScoreTopNIntoOlapScan` keeps
the behavior of the branch and skips the push-down instead of failing.
- `SearchDslParser` is imported from its package on the branch.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.2` at ecbc5fe), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest`,
`PullUpProjectExprUnderTopNTest` and `MaterializedViewUtilsTest` (280
tests), and FE Checkstyle.
- PASS: `./build.sh --fe --be`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 23 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`, with unchanged
`.out` files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 26, 2026
…69292 (#69565)

### What problem does this PR solve?

Related PR: #69292

Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit
of #69563, the backport of #69399, which it needs: binding runs before
CTE inlining, which copies scans with virtual columns. Merge #69563
first.

`score()` only worked as `ORDER BY score() LIMIT n` directly above the
scan whose MATCH defines it. A minimum score filter without that TopN
evaluated the unbound `score()` placeholder as zero and silently dropped
every row:

```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;  -- returned nothing
```

`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()`
to the scan of its SELECT at the start of rewrite and rejects the uses
it cannot bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score
together with the minimum. In the BE, an expression on the score reads
the score column instead of evaluating `Score()`, and MATCH projections
do not add to the score. See #69292 for the supported and rejected uses.

#### Branch adaptations

- The `SegmentIterator` of the branch collects the columns of
pushed-down expressions in `_extract_common_expr_columns` instead of
per-column states, so the score column is marked there, and the new BE
test checks `_common_expr_columns` and `_is_common_expr_column`. The
branch reads the columns collected there before it evaluates the
expressions, also virtual ones, and a virtual slot uses a column that is
already read.
- The branch evaluates MATCH projections with `evaluate_inverted_index`
directly, and that loop runs without the score collector.
- `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch
for `limit + offset`, which fails like master does.
- `SearchDslParser` and `CollectionSimilarity` are included from their
packages on the branch.
- The branch has no SNII storage format, so `BindScoreToOlapScanTest`
and `test_score_filter_matrix` cover V2 and V3, and the `.out` of
`test_score_filter_matrix` is regenerated without the SNII cases.
- The branch does not have #68672, which keeps predicates on virtual
columns out of the condition cache. The minimum score is now a predicate
of the scan, and a score depends on every rowset, so a cached result
goes stale when a rowset is added: without the change,
`test_score_filter_matrix` lost the rows of the first rowset.
`VirtualSlotRef::get_digest` returns 0 as on master, the part of #68672
that this needs, with the `VirtualSlotRef` tests of #68672 that the
branch can run.
- `test_score_filter_usage`, `test_score_filter_matrix` and
`test_cte_inline_virtual_column` set `enable_common_expr_pushdown =
true`, as the score suites of the branch do: CI on branch-4.1 draws it
at random for each session, and without it MATCH is not evaluated with
the index in the scan and `score()` is 0.

### Release note

`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an
optional minimum score `score() > x` or `score() >= x`, without `ORDER
BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set
operation still needs `ORDER BY score() LIMIT`. Other uses of `score()`
are rejected instead of evaluating it as 0.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as
0. A minimum score without a TopN needs the BE change, so BEs must be
upgraded before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.

### Validation

Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d0), in a local ASAN build:

- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.x dev/4.1.x-conflict reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants