Skip to content

branch-4.2 [feature](runtime filter) Add decoupled runtime filter support - #68212

Merged
yiguolei merged 5 commits into
apache:branch-4.2from
englefly:pick-62737-4.2
Oct 13, 2026
Merged

branch-4.2 [feature](runtime filter) Add decoupled runtime filter support#68212
yiguolei merged 5 commits into
apache:branch-4.2from
englefly:pick-62737-4.2

Conversation

@englefly

@englefly englefly commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Cherry-pick of #62737 to branch-4.2

[feature](runtime filter) Add decoupled runtime filter support

Adaptations for branch-4.2:

  • RuntimeFilterConsumer keeps branch-4.2's wait_infinitely computation (no BITMAP_FILTER term) and wraps it in the new desc->__isset.wait_time_ms branch.
  • RuntimeFilterTranslator/generator call branch-4.2's PushDownContext.createPushDownContextForHashJoin factory instead of master's unified createPushDownContext.
  • The three new session variables use branch-4.2's @VariableMgr.VarAttr; runtimeFilterWaitInfinitely becomes public as upstream requires.

Rebased onto the latest branch-4.2 (Variant relational-ops RF guards merged into RuntimeFilterPushDownVisitor, new Variant RF unit tests preserved in RuntimeFilterTest).

CI fixes:

  • shape_check baselines regenerated with -forceGenOut for every case that failed in P0 regression: tpcds_sf100 4/11/31/32/74/92 (shape, no_stats_shape, noStatsRfPrune, rf_prune), tpcds_sf1000 4/11/24/30/32/81/92 (shape, hint), tpcds_sf1000_constraints 4/11/24/30/81, tpcds_sf10t_orc 4/11/18/31/32/54/74/92, tpch_sf1000 q2/q8/q9/q17/q18 (shape, shape_no_stats, nostats_rf_prune, rf_prune), tpch_sf1000_constraints q9 and the dphyper variants tpcds_sf1000 query54/query72 and tpch_sf1000 q8/q9/q18. Every regenerated file differs from the branch-4.2 baseline only in runtime-filter annotations (build RF target lists and applied RFs); plan shapes, join distributions and predicates are untouched. For the dphyper files the regenerated lines match the mismatches reported by CI one by one.
  • RuntimeFilterTest#testDecoupledRuntimeFilter now asserts the branch-4.2 observable behaviour instead of a separate exprOrder == -1 filter. On branch-4.2 RuntimeFilterContext#getRuntimeFilterBySrcAndType identifies a filter by (srcExpr, type, builderNode) only, so the decoupled RF is merged into the standard RF of the builder join (master dedups per target scan including exprOrder and keeps them apart). The test checks, with and without the feature, that the join2 RF additionally targets supplier.s_suppkey and that the join1 RF becomes non-blocking when the decoupled RF is preferred.

Testing:

  • run-fe-ut.sh --run org.apache.doris.nereids.postprocess.RuntimeFilterTest -> Tests run: 33, Failures: 0, Errors: 0.
  • regression suite nereids_syntax_p0/decoupled_runtime_filter passes.
  • every shape_check suite touching a regenerated baseline passes locally (tpcds_sf100/sf1000/sf1000_constraints/sf10t_orc, tpch_sf1000/tpch_sf1000_constraints, including the dphyper suites). The decoupled RFs are reproduced because the local cluster has the same loaded tpcds/tpch data as CI.
  • Unrelated red checks from the previous run: BE UT failures were FileCacheProfileReporterTest (x3) and RegexGramRecallTest.DynamicPatternRegexExecution, which are known flaky and are muted in TeamCity (same tests muted on other PR builds), and the NonConcurrent FE IMAGE GENERATED FAIL / CHECKPOINT FAIL failure is infra-wide (identical failure on branch-4.2: [Improvement](scan) Push down LIMIT to segment iterator (#61713 #62222 #64285 #64479 #68804) #68788).

@englefly
englefly requested a review from yiguolei as a code owner September 18, 2026 16:35
@englefly

englefly commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

hello-stephen commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.27% (1826/2492)
Line Coverage 61.07% (32853/53800)
Region Coverage 48.70% (31141/63943)
Branch Coverage 51.58% (9656/18720)

@hello-stephen

hello-stephen commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 56.10% (115/205) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 96.23% (51/53) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.39% (31447/42274)
Line Coverage 58.69% (350844/597755)
Region Coverage 55.47% (293090/528420)
Branch Coverage 56.29% (132181/234838)

@englefly

englefly commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

Gabriel39 pushed a commit that referenced this pull request Oct 12, 2026
@suxiaogang223
[fix](regression) Move unstable cross-cloud storage suites to p2 (#68866
358e87c 
)

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Cross-cloud COS/OBS access repeatedly times out in regression runs for
unrelated PRs, including `curlCode: 28` during uploads and
object-storage connection timeouts. Examples: [External
#103511](http://43.132.222.7:8111/viewLog.html?buildId=1065598), [P0
#127021](http://43.132.222.7:8111/viewLog.html?buildId=1065768), and
[Cloud P0
#68212](http://43.132.222.7:8111/viewLog.html?buildId=1065729).

Move the following suites to p2 directories and change their tier tags
from `p0` to `p2`, preserving existing `external` tags:

- `refactor_storage_param_s3_load`
- `test_s3_tvf_s3_storage`
- `test_outfile_s3_storage`
- `paimon_base_filesystem`
- `test_file_tvf_s3`
- `test_outfile_with_different_s3`

Move the four existing golden files alongside their suites without
changing their contents. P0, Cloud P0, and External pipelines exclude
these suites, while their coverage remains available in p2. Suite bodies
are unchanged.

### Release note

None

### Check List (For Author)

- Test:
- [x] Manual static validation: suite selection, unchanged suite bodies
and golden files, and `git diff --check` passed. Live cloud regression
was not run.
- Behavior changed:
    - [x] Yes. Regression suite scheduling only.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
@hello-stephen

hello-stephen commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 55.61% (114/205) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 72.68% (149/205) 🎉
Increment coverage report
Complete coverage report

apupier and others added 5 commits October 13, 2026 11:23
@apupier @englefly
[fix](ci) Update github host from macos-13 to macos-15 (apache#67747)
88e7d2d 
macos-13 is no more available

https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

it is causing a lot of PR jobs to wait in queue for a long time when
there is a PR on 4.2 branch

it has been already upgraded on main branch

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Signed-off-by: Aurélien Pupier <[email protected]>
@englefly
branch-4.2 [feature](runtime filter) Add decoupled runtime filter sup…
9ccabbc 
…port (apache#62737)

support decoupled rf.
(A join B) join (C join D)
decoupled rf: B->C and B->D

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: Copilot <[email protected]>
@englefly
branch-4.2 [fix](regression) Update shape_check expected outputs for …
79f5cc5 
…decoupled runtime filter

Regenerated with -forceGenOut on branch-4.2 + apache#62737 after rebasing onto the
latest branch-4.2: tpcds_sf100 queries 4/11/31/32/74/92 (shape, no_stats_shape,
noStatsRfPrune, rf_prune), tpcds_sf1000 queries 4/11/24/30/32/81/92 (shape, hint)
and query4/11/24/30/81 in shape_check/tpcds_sf1000_constraints, tpcds_sf10t_orc
queries 4/11/18/31/32/54/74/92, tpch_sf1000 q8/q9 plus q2/q17/q18 in
shape/no_stats/nostats_rf_prune/rf_prune and q9 in tpch_sf1000_constraints.

Only runtime-filter annotations change (build RF target lists and applied RFs);
plan shapes, join distributions and predicates are unchanged.
@englefly
branch-4.2 [fix](fe-ut) Fix RuntimeFilterTest#testDecoupledRuntimeFilter
9a240df 
The test asserted a separate decoupled RF with exprOrder == -1, which is how the
feature behaves on master. On branch-4.2 the decoupled RF is merged into the
standard RF of the builder join instead: RuntimeFilterContext#getRuntimeFilterBySrcAndType
identifies a filter by (srcExpr, type, builderNode) only, and the decoupled RF
source expression is the build-side slot of that same join, i.e. exactly the
standard RF source. Upstream master dedups per target scan including exprOrder,
which is what keeps the two filters apart there.

Assert the branch-4.2 observable behaviour instead, with and without the feature:
* with enable_decoupled_runtime_filter=false the join2 RF targets only
  lineorder.lo_custkey and the join1 RF stays blocking;
* with the feature enabled the same RF additionally targets supplier.s_suppkey
  (the decoupled RF reaches the build-side scan of join1) and the join1 RF is
  marked non-blocking because the decoupled RF is preferred.

RuntimeFilterTest passes: Tests run: 29, Failures: 0, Errors: 0.
@englefly
branch-4.2 [fix](regression) Refresh dphyper shape_check baselines fo…
90e593f 
…r decoupled runtime filter

Regenerated with -forceGenOut for the five dphyper cases reported by P0
regression: shape_check/tpcds_sf1000/dphyper queries 54/72 and
shape_check/tpch_sf1000/dphyper q8/q9/q18. Every changed line matches the
mismatch reported by CI and only runtime-filter annotations change.
@englefly

englefly commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 72.68% (149/205) 🎉
Increment coverage report
Complete coverage report

@englefly

englefly commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

run cloudut

@hello-stephen

hello-stephen commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.52% (1963/2500)
Line Coverage 65.52% (35414/54048)
Region Coverage 53.10% (34040/64102)
Branch Coverage 55.87% (10485/18767)

@yiguolei
yiguolei merged commit f318170 into apache:branch-4.2 Oct 13, 2026
32 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@englefly @hello-stephen @yiguolei @apupier