Skip to content

Commit 097b15b

Browse files
morningmanclaude
andcommitted
[fix](fe) Forward four local exchange session variables to the master
### What problem does this PR solve? Issue Number: None Related PR: #69273, #70177 Problem Summary: A statement that has to run on the master FE (INSERT INTO ... SELECT, CTAS, ...) and arrives at a follower or an observer is forwarded with the session variables that `SessionVariable#getForwardVariables()` sends, and the master applies them to a session that starts from its own global variables. Only variables whose `@VarAttr` sets `needForward` are sent. #69273 flagged the planner and executor variables, `enable_local_exchange_before_agg` among them, but not four that also decide the local exchanges of a plan: `enable_local_exchange_before_streaming_agg` (a local hash exchange before a streaming pre-aggregation, BE `StreamingAggOperatorX` and the FE planner with #70177), `enable_streaming_agg_hash_join_force_passthrough` and `enable_distinct_streaming_agg_force_passthrough` (a passthrough local exchange below a streaming pre-aggregation, FE `AggregationNode` and the BE operators), and `force_to_local_shuffle` (every scan serial, `ScanNode#isSerialNode`). A session-level SET of one of them was dropped when the statement was forwarded, so the master planned it with its global value and EXPLAIN on the follower showed a plan the master did not run. Results do not change, only the plan and its speed. The four now set `needForward = true`, as #69273 does for the others. ### Release note The session variables enable_local_exchange_before_streaming_agg, enable_streaming_agg_hash_join_force_passthrough, enable_distinct_streaming_agg_force_passthrough and force_to_local_shuffle now apply to statements that a follower or observer FE forwards to the master. ### Check List (For Author) - Test: Unit Test - `SessionVariablesTest#testForwardPlanAndExecutionKnobs` lists the four; with `SessionVariable.java` reverted it fails. The test classes that call `getForwardVariables()` or `setForwardedSessionVariables()` pass, and fe-core Checkstyle reports nothing. Not run on a cluster with two FEs. - Behavior changed: Yes. A forwarded statement uses the session's value of these four variables instead of the master's global value. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 0eb5131 commit 097b15b

2 files changed

Lines changed: 9 additions & 6 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1451,16 +1451,16 @@ public void checkQuerySlotCount(String slotCnt) {
14511451
@VarAttrDef.VarAttr(name = ENABLE_DISTINCT_STREAMING_AGGREGATION, needForward = true, fuzzy = true)
14521452
public boolean enableDistinctStreamingAggregation = true;
14531453

1454-
@VarAttrDef.VarAttr(name = ENABLE_STREAMING_AGG_HASH_JOIN_FORCE_PASSTHROUGH, fuzzy = true)
1454+
@VarAttrDef.VarAttr(name = ENABLE_STREAMING_AGG_HASH_JOIN_FORCE_PASSTHROUGH, needForward = true, fuzzy = true)
14551455
public boolean enableStreamingAggHashJoinForcePassthrough = true;
14561456

14571457
@VarAttrDef.VarAttr(name = ENABLE_LOCAL_EXCHANGE_BEFORE_AGG, needForward = true, fuzzy = true)
14581458
public boolean enableLocalExchangeBeforeAgg = true;
14591459

1460-
@VarAttrDef.VarAttr(name = ENABLE_LOCAL_EXCHANGE_BEFORE_STREAMING_AGG, fuzzy = true)
1460+
@VarAttrDef.VarAttr(name = ENABLE_LOCAL_EXCHANGE_BEFORE_STREAMING_AGG, needForward = true, fuzzy = true)
14611461
public boolean enableLocalExchangeBeforeStreamingAgg = false;
14621462

1463-
@VarAttrDef.VarAttr(name = ENABLE_DISTINCT_STREAMING_AGG_FORCE_PASSTHROUGH, fuzzy = true)
1463+
@VarAttrDef.VarAttr(name = ENABLE_DISTINCT_STREAMING_AGG_FORCE_PASSTHROUGH, needForward = true, fuzzy = true)
14641464
public boolean enableDistinctStreamingAggForcePassthrough = true;
14651465

14661466
@VarAttrDef.VarAttr(name = ENABLE_BROADCAST_JOIN_FORCE_PASSTHROUGH, needForward = true, fuzzy = true)
@@ -1777,7 +1777,7 @@ public boolean isEnableDynamicPartitionDropProtection() {
17771777

17781778
@VarAttrDef.VarAttr(
17791779
name = FORCE_TO_LOCAL_SHUFFLE, fuzzy = false, varType = VariableAnnotation.EXPERIMENTAL,
1780-
description = "Whether to force to local shuffle on pipelineX engine.")
1780+
description = "Whether to force to local shuffle on pipelineX engine.", needForward = true)
17811781
private boolean forceToLocalShuffle = false;
17821782

17831783
@VarAttrDef.VarAttr(

‎fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -564,12 +564,15 @@ public void testForwardPlanAndExecutionKnobs() {
564564
SessionVariable.ENABLE_BUCKET_SHUFFLE_JOIN, SessionVariable.ENABLE_COST_BASED_JOIN_REORDER,
565565
SessionVariable.ENABLE_CTE_MATERIALIZE, SessionVariable.ENABLE_DECOUPLED_RUNTIME_FILTER,
566566
SessionVariable.ENABLE_DISTINCT_STREAMING_AGGREGATION,
567+
SessionVariable.ENABLE_DISTINCT_STREAMING_AGG_FORCE_PASSTHROUGH,
567568
SessionVariable.ENABLE_DPHYP_OPTIMIZER, SessionVariable.ENABLE_LOCAL_EXCHANGE,
568-
SessionVariable.ENABLE_LOCAL_EXCHANGE_BEFORE_AGG, SessionVariable.ENABLE_PARALLEL_OUTFILE,
569+
SessionVariable.ENABLE_LOCAL_EXCHANGE_BEFORE_AGG,
570+
SessionVariable.ENABLE_LOCAL_EXCHANGE_BEFORE_STREAMING_AGG, SessionVariable.ENABLE_PARALLEL_OUTFILE,
569571
SessionVariable.ENABLE_PARTITION_TOPN, SessionVariable.ENABLE_PUSH_DOWN_NO_GROUP_AGG,
570572
SessionVariable.ENABLE_SHARED_EXCHANGE_SINK_BUFFER, SessionVariable.ENABLE_SQL_CACHE,
573+
SessionVariable.ENABLE_STREAMING_AGG_HASH_JOIN_FORCE_PASSTHROUGH,
571574
SessionVariable.EXCHANGE_MULTI_BLOCKS_BYTE_SIZE,
572-
SessionVariable.EXPAND_RUNTIME_FILTER_BY_INNER_JION,
575+
SessionVariable.EXPAND_RUNTIME_FILTER_BY_INNER_JION, SessionVariable.FORCE_TO_LOCAL_SHUFFLE,
573576
SessionVariable.GLOBAL_PARTITION_TOPN_THRESHOLD, SessionVariable.IGNORE_RUNTIME_FILTER_IDS,
574577
SessionVariable.INLINE_CTE_REFERENCED_THRESHOLD,
575578
SessionVariable.LOCAL_EXCHANGE_FREE_BLOCKS_LIMIT,

0 commit comments

Comments
 (0)