Skip to content

Commit 0af95bc

Browse files
committed
[improvement](fe) Add a per-scan split producer limit
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Partition-batch scans derive their producer count from the shared external metadata pool. On an overloaded object store, speculative LIST requests can delay the first batch and cause gaps in subsequent Split supply. Add an opt-in session limit per scan node without shrinking global executors or changing batch sizes, queue bounds, cancellation, or wait deadlines. Zero keeps the existing behavior; positive values also respect executor and partition-batch bounds. Whole-scan streaming retains one producer. On the local million-file fixture with BE 4.1.3 and the global pool fixed at 64, three default-batch LIMIT samples per setting show a median reduction from 1443 ms to 654 ms with a producer limit of four, and median LIST calls from 26 to 12. In the partition-batch-size-one ABBA control, six LIMIT samples per setting improve from 6564 ms to 560 ms. Complete 10000-file scans succeed in 2/6 uncapped and 6/6 capped samples, but the uncapped successful subset is faster; this is not a general full-scan throughput improvement. Keep the default at zero and expose the setting in the optional regression benchmark for workload-specific tuning. ### Release note Add the FE session variable split_producer_parallelism to cap partition-batch Split producers per plugin scan node. The default zero preserves existing parallelism. Negative values are rejected. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 58 focused FE unit tests pass with zero failures, errors or skips - Standard seven-scenario Hive performance regression passes with BE 4.1.3 - 32 uninstrumented SQL controls match unique FE audit queries; complete successful scans read 10000 rows with exact COUNT and SUM results - FE build through build.sh --fe and Checkstyle pass - Behavior changed: Yes, an explicit positive session limit bounds producers; default behavior, batch sizes and deadlines remain unchanged - Does this need documentation: Yes, setting and benchmark usage documented in regression-test/script/hive_split_early_stop/README.md
1 parent d56ef95 commit 0af95bc

5 files changed

Lines changed: 94 additions & 2 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2235,6 +2235,10 @@ List<SplitProducer.BatchSource> createPartitionBatchSources(ConnectorScanPlanPro
22352235
request.getScanContext().setTotalPartitions(partitions.size());
22362236
AtomicInteger nextPartition = new AtomicInteger();
22372237
int sourceCount = Math.min(parallelism, (partitions.size() - 1) / partitionsPerBatch + 1);
2238+
int producerLimit = sessionVariable.getSplitProducerParallelism();
2239+
if (producerLimit > 0) {
2240+
sourceCount = Math.min(sourceCount, producerLimit);
2241+
}
22382242
List<SplitProducer.BatchSource> sources = new ArrayList<>(sourceCount);
22392243
// Bound active partition batches even when full queues make producers yield their workers.
22402244
// Hive streams within each batch; the SPI adapter retains one list for older implementations.

‎fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -598,6 +598,8 @@ public String toString() {
598598

599599
public static final String SPLIT_QUEUE_CAPACITY = "split_queue_capacity";
600600

601+
public static final String SPLIT_PRODUCER_PARALLELISM = "split_producer_parallelism";
602+
601603
public static final String FIRST_SPLIT_WAIT_TIMEOUT_MS = "first_split_wait_timeout_ms";
602604
public static final String FETCH_SPLITS_MAX_WAIT_TIME = "fetch_splits_max_wait_time_ms";
603605

@@ -2632,6 +2634,11 @@ public Map<String, Action> getForceEagerAggHintMap() {
26322634
description = "Maximum number of queued split batches per backend.")
26332635
public int splitQueueCapacity = 32;
26342636

2637+
@VarAttrDef.VarAttr(name = SPLIT_PRODUCER_PARALLELISM, checker = "checkSplitProducerParallelism",
2638+
needForward = true, description = "Maximum partition-batch split producers per scan node. "
2639+
+ "Zero uses the FE external metadata loader pool size. Whole-scan streams use one producer.")
2640+
public int splitProducerParallelism = 0;
2641+
26352642
@VarAttrDef.VarAttr(name = FIRST_SPLIT_WAIT_TIMEOUT_MS, checker = "checkFirstSplitWaitTimeout",
26362643
needForward = true, description = "First split wait timeout in milliseconds.")
26372644
public long firstSplitWaitTimeoutMs = 30000;
@@ -2652,6 +2659,10 @@ public void checkSplitQueueCapacity(String value) throws Exception {
26522659
checkFieldValue(SPLIT_QUEUE_CAPACITY, 1, value);
26532660
}
26542661

2662+
public void checkSplitProducerParallelism(String value) throws Exception {
2663+
checkFieldValue(SPLIT_PRODUCER_PARALLELISM, 0, value);
2664+
}
2665+
26552666
@VarAttrDef.VarAttr(
26562667
name = FETCH_SPLITS_MAX_WAIT_TIME,
26572668
description = "The max wait time of getting splits in batch mode.",
@@ -4994,6 +5005,10 @@ public int getSplitQueueCapacity() {
49945005
return splitQueueCapacity;
49955006
}
49965007

5008+
public int getSplitProducerParallelism() {
5009+
return splitProducerParallelism;
5010+
}
5011+
49975012
public long getFetchSplitsMaxWaitTime() {
49985013
return fetchSplitsMaxWaitTime;
49995014
}

‎fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodePartitionBatchTest.java‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,54 @@ void emptyPartitionBatchesAndPartialLastBatchDoNotEndTheScanEarly() throws Excep
171171
}
172172
}
173173

174+
@Test
175+
void producerLimitBoundsActiveSourcesWithoutLosingPartitions() throws Exception {
176+
for (int limit : Arrays.asList(0, 2, 100)) {
177+
Fixture fixture = new Fixture(1, Collections.emptySet());
178+
Mockito.when(fixture.variables.getSplitProducerParallelism()).thenReturn(limit);
179+
fixture.start(7, 1, 8);
180+
fixture.executor.runUntilIdle();
181+
int expectedProducers = limit == 2 ? 2 : 7;
182+
Assertions.assertEquals(expectedProducers, fixture.loaded.size());
183+
Assertions.assertEquals(0, fixture.finished.get());
184+
185+
List<Long> actual = fixture.drain();
186+
Collections.sort(actual);
187+
List<Long> expected = new ArrayList<>();
188+
for (long partition = 0; partition < 7; partition++) {
189+
for (long range = 0; range < 3; range++) {
190+
expected.add(partition * 10 + range);
191+
}
192+
}
193+
Assertions.assertEquals(expected, actual);
194+
Assertions.assertEquals(7, fixture.loaded.size());
195+
Assertions.assertEquals(7, new HashSet<>(fixture.loaded).size());
196+
Assertions.assertEquals(expectedProducers, fixture.finished.get());
197+
}
198+
}
199+
200+
@Test
201+
void producerLimitCannotExceedExecutorParallelism() throws Exception {
202+
Fixture fixture = new Fixture(1, Collections.emptySet());
203+
Mockito.when(fixture.variables.getSplitProducerParallelism()).thenReturn(100);
204+
List<SplitProducer.BatchSource> sources = fixture.sources(20, 1, 3);
205+
Assertions.assertEquals(3, sources.size());
206+
for (SplitProducer.BatchSource source : sources) {
207+
source.close();
208+
}
209+
}
210+
211+
@Test
212+
void producerLimitAcceptsZeroAndRejectsNegativeValues() throws Exception {
213+
SessionVariable variables = new SessionVariable();
214+
Assertions.assertEquals(0, variables.getSplitProducerParallelism());
215+
variables.checkSplitProducerParallelism("0");
216+
variables.checkSplitProducerParallelism("4");
217+
Exception failure = Assertions.assertThrows(Exception.class,
218+
() -> variables.checkSplitProducerParallelism("-1"));
219+
Assertions.assertTrue(failure.getMessage().contains(SessionVariable.SPLIT_PRODUCER_PARALLELISM));
220+
}
221+
174222
@Test
175223
void cancellationDuringEmptyPartitionLoadingStopsFurtherCalls() throws Exception {
176224
Fixture fixture = new Fixture(1, Collections.emptySet());
@@ -193,6 +241,7 @@ private static class Fixture {
193241
final SplitAssignment assignment;
194242
final PluginDrivenScanNode node;
195243
final ConnectorScanPlanProvider provider;
244+
final SessionVariable variables = Mockito.mock(SessionVariable.class);
196245

197246
Fixture(int batchSize, Set<String> emptyPartitions) throws Exception {
198247
this(batchSize, emptyPartitions, 3);
@@ -211,6 +260,7 @@ private static class Fixture {
211260
Collections.emptyMap(), Collections.emptyList(), true, batchSize, 1);
212261
node = Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS);
213262
Deencapsulation.setField(node, "splitAssignment", assignment);
263+
Deencapsulation.setField(node, "sessionVariable", variables);
214264
provider = Mockito.mock(ConnectorScanPlanProvider.class, Mockito.CALLS_REAL_METHODS);
215265
Mockito.when(provider.supportsBatchScan(Mockito.any(), Mockito.any())).thenReturn(true);
216266
Mockito.doAnswer(call -> {

‎regression-test/script/hive_split_early_stop/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,21 @@ hiveSplitPerfRepetitions = "3"
226226
hiveSplitPerfCacheTable = "hive_split_physical_cache.doris_split_physical.files_10000_partitions_100_files"
227227
```
228228

229+
To compare partition-batch producer limits on a version supporting
230+
`split_producer_parallelism`, set `hiveSplitPerfProducerParallelism = "4"` in a
231+
benchmark config, and use `"0"` for its control. The suite restores the session
232+
value afterward. Omit this entry when testing older versions.
233+
234+
The session variable caps active partition-batch producers **per scan node**.
235+
Zero preserves the existing FE external metadata pool limit; a positive value
236+
cannot exceed that pool's parallelism or the number of partition batches. It
237+
does not change the global executor, the configured Split batch size, queue
238+
capacity or wait deadlines. Whole-scan connector streams still use one producer.
239+
Lower limits can reduce speculative LIST calls for LIMIT queries and contention
240+
on an overloaded object store, but can reduce full-scan throughput on a fast
241+
store. Compare both workloads with the default fetch deadline before selecting a
242+
value; four is an experimental setting, not a new default.
243+
229244
The synchronous control explicitly disables `enable_external_table_batch_mode`;
230245
raising `num_partitions_in_batch_mode` alone can still select file-count streaming.
231246
If a connector's synchronous planner enumerates partitions outside the engine's

‎regression-test/suites/external_table_p0/hive/test_hive_split_early_stop_performance.groovy‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ suite("test_hive_split_early_stop_performance", "external") {
3232
String table = context.config.otherConfigs.get("hiveSplitEarlyStopTable")
3333
String cacheTable = context.config.otherConfigs.get("hiveSplitPerfCacheTable")
3434
String syncTable = context.config.otherConfigs.get("hiveSplitPerfSyncTable") ?: table
35+
String producerParallelism = context.config.otherConfigs.get("hiveSplitPerfProducerParallelism")
3536
boolean planningOnly = context.config.otherConfigs.get("hiveSplitPerfPlanningCancelOnly")
3637
?.toString()?.toBoolean() ?: false
3738
boolean steadyOnly = context.config.otherConfigs.get("hiveSplitPerfSteadyLimitOnly")
@@ -109,11 +110,18 @@ suite("test_hive_split_early_stop_performance", "external") {
109110
}
110111
return signalledAt.get()
111112
}
112-
def saved = ["query_timeout", "num_partitions_in_batch_mode", "enable_file_cache", "enable_sql_cache",
113-
"enable_external_table_batch_mode"].collectEntries {
113+
def settings = ["query_timeout", "num_partitions_in_batch_mode", "enable_file_cache", "enable_sql_cache",
114+
"enable_external_table_batch_mode"]
115+
if (producerParallelism != null) {
116+
settings.add("split_producer_parallelism")
117+
}
118+
def saved = settings.collectEntries {
114119
name -> [(name): sql("show variables like '${name}'")[0][1]]
115120
}
116121
try {
122+
if (producerParallelism != null) {
123+
sql "set split_producer_parallelism=${producerParallelism}"
124+
}
117125
sql "set query_timeout=120"
118126
sql "set enable_file_cache=false"
119127
sql "set enable_sql_cache=false"

0 commit comments

Comments
 (0)