Skip to content

Commit ded3ddf

Browse files
morningmanclaude
andcommitted
[fix](test) pin parallel_pipeline_task_num=1 to keep bucket shuffle in prune_bucket_with_bucket_shuffle_join
Root cause: the second extractFragment asserts RIGHT OUTER JOIN(BUCKET_SHUFFLE) under the new distribute planner, but the bucket-shuffle downgrade heuristic ChildrenPropertiesRegulator.isBucketShuffleDownGrade (totalBucketNum < backEndNum*paraNum*0.8) fires on the CloudP0 agent: the left side prunes to 1 partition x 10 buckets = 10, backEndNum=1, and with the default parallel_pipeline_task_num=0 paraNum auto-derives to 16 on the 32-core BE, so 10 < 1*16*0.8 = 12.8 -> downgrade -> PARTITIONED. The join is then labeled PARTITIONED, never BUCKET_SHUFFLE, so all 120 retries fail (~128s, muted). This became reachable after apache#56279 (302ba2e) replaced the old Math.min(10, backEndNum*paraNum) cap with the *0.8 formula; the var is also fuzzy, an additional flake source. Solution: set parallel_pipeline_task_num=1 in the second-phase multi_sql so paraNum=1 and the threshold becomes 10 < 0.8 (false) -> no downgrade -> BUCKET_SHUFFLE kept deterministically, independent of BE core count and fuzz. Bucket pruning (tablet<20), exchangeNum==1 and result correctness are unaffected. Tests: existing suite is the test (CloudP0 e2e, CI-gated). TEST-only, no product change; apache#56279's formula is intentional and left as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 139d919 commit ded3ddf

1 file changed

Lines changed: 4 additions & 0 deletions

File tree

‎regression-test/suites/nereids_syntax_p0/distribute/prune_bucket_with_bucket_shuffle_join.groovy‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,14 @@ suite("prune_bucket_with_bucket_shuffle_join") {
7676
assertTrue(exchangeNum > 1)
7777
}
7878

79+
// Pin parallelism so the bucket-shuffle downgrade heuristic
80+
// (totalBucketNum < backEndNum*paraNum*0.8 in ChildrenPropertiesRegulator) cannot fire on
81+
// high-core or multi-BE hosts; with paraNum=1 the 10-bucket left side keeps BUCKET_SHUFFLE.
7982
multi_sql """
8083
set enable_nereids_distribute_planner=true;
8184
set enable_pipeline_x_engine=true;
8285
set disable_join_reorder=true;
86+
set parallel_pipeline_task_num=1;
8387
"""
8488

8589
extractFragment(sqlStr, "RIGHT OUTER JOIN(BUCKET_SHUFFLE)") { exchangeNum ->

0 commit comments

Comments
 (0)