Skip to content

Commit 5a8470c

Browse files
committed
[improvement](fe) Default external scans to bounded split batches
### What problem does this PR solve? Issue Number: None Problem Summary: External scans previously entered batch mode only above 1024 partitions or files. When enabled, each producer batch held 128 splits and each backend queue was bounded by 32 batches, so small LIMIT queries could wait for unnecessary work while the queue capacity depended on batch size. Enable batch mode from one partition or file, separate the Hive partition-processing size from that trigger, and store individual splits in a queue bounded by 1024 splits per backend. Producers still offer splits in batches; use eight splits per batch by default. In a matched one-BE Release benchmark on the integrated test branch, refreshed-metadata LIMIT medians at batch sizes 8 versus 32 were 75.8 versus 102.5 ms for Hive and 23.5 versus 30.6 ms for Iceberg. Warm full scans showed no stable winner across those sizes. Preserve request batch limits, cancellation, backpressure, and producer wakeups. The new connector SPI method advances the plugin API major version to 25. ### Release note External table Batch Mode now starts at one partition or file by default. The default producer batch size is eight splits, queue capacity is 1024 splits per backend, and split_partition_batch_size controls Hive partition-processing batches separately. ### Check List (For Author) - Test: Unit Test. Before rebasing onto current master, FE build and ten focused FE test classes passed (162 tests). On current master, ConnectorPluginSurfaceTest (2), ConnectorMetadataSurfaceTest (4), and HiveScanBatchModeTest (36) passed. FE-core Maven tests could not start locally because snapshot reactor artifacts were unresolved without -am; the full -am route requires an x86 grpc generator unavailable on this ARM host. One-BE Release benchmark results above were recorded on the integrated test branch. - Behavior changed: Yes (Batch Mode thresholds and split buffering defaults change as described above) - Does this need documentation: No (connector scheduling document updated in this commit)
1 parent ba3a660 commit 5a8470c

22 files changed

Lines changed: 387 additions & 174 deletions

File tree

‎fe/fe-connector/docs/scan-task-scheduling.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,11 +108,11 @@ shared cache loader. Normal completion retains the separate read-lease cleanup.
108108

109109
### Consumer completion notification
110110

111-
Each backend has a bounded split queue and its own data-ready wait channel. A consumer registers before checking queue contents and terminal state, then waits only when neither is available. Successful enqueue, producer completion, failure and cancellation signal that channel. The empty queue is not periodically polled to discover EOF: otherwise finishing just after a poll begins adds 100 ms despite completed production. Producers avoid the channel monitor when no consumer waits; callbacks that resume producers run after releasing the consumer channel monitor. Cancellation signals consumers before closing SDK resources.
111+
Each backend has a split queue bounded by the number of queued splits and its own data-ready wait channel. Producers offer batches, but individual splits are the queue entries. A consumer registers before checking queue contents and terminal state, then waits only when neither is available. Successful enqueue, producer completion, failure and cancellation signal that channel. The empty queue is not periodically polled to discover EOF: otherwise finishing just after a poll begins adds 100 ms despite completed production. Queue bookkeeping uses a short per-backend lock; callbacks that resume producers run after releasing it. Cancellation signals consumers before closing SDK resources.
112112

113113
### Initial delivery and producer lookahead
114114

115-
`split_initial_batch_size` (default 64) bounds the first Split fetch per backend, also capped by the BE request size. Subsequent fetches use the BE request size. The bound is applied before compatible file ranges are compacted into RPC envelopes. It is independent of `split_batch_size` (producer batch size) and `split_queue_capacity` (buffered batches per backend). Producers continue listing files and preparing splits before BE consumption, yielding only for task fairness and configured queue backpressure. A small initial RPC therefore does not suspend useful directory prefetch while BE starts.
115+
`split_initial_batch_size` (default 64) bounds the first Split fetch per backend, also capped by the BE request size. Subsequent fetches use the BE request size. The bound is applied before compatible file ranges are compacted into RPC envelopes. `split_batch_size` (default 8) limits producer batches, while `split_queue_capacity` (default 1024) limits buffered splits per backend. A configured queue capacity smaller than the batch size also reduces the effective producer batch size. Producers continue listing files and preparing splits before BE consumption, yielding only for task fairness and configured queue backpressure. A small initial RPC therefore does not suspend useful directory prefetch while BE starts.
116116

117117
Actual Manifest readers retain a lease on the exact read-table generation until the callable exits. Query cleanup releases the statement owner, while an already running shared reader keeps its FileIO alive. Future cancellation is not evidence that the callable has finished.
118118

‎fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ public class HiveScanPlanProvider implements ConnectorScanPlanProvider {
8383
/** Maximum number of partitions to list from HMS. */
8484
private static final int MAX_PARTITIONS = 100000;
8585
private static final int DEFAULT_BATCH_THRESHOLD = 1024;
86+
private static final int DEFAULT_PARTITION_BATCH_SIZE = 16;
8687

8788
/**
8889
* Session variable gating the OpenX-JSON "read the whole JSON row into one CSV column" mode (legacy
@@ -213,6 +214,12 @@ public boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle
213214
return partKeyNames != null && !partKeyNames.isEmpty();
214215
}
215216

217+
@Override
218+
public int partitionBatchSize(ConnectorSession session, int partitionCount, int parallelism) {
219+
int configured = sessionInt(session, "split_partition_batch_size", DEFAULT_PARTITION_BATCH_SIZE);
220+
return Math.min(configured, (partitionCount - 1) / parallelism + 1);
221+
}
222+
216223
@Override
217224
public long streamingSplitEstimate(ConnectorSession session, ConnectorTableHandle handle,
218225
Optional<ConnectorExpression> filter, boolean countPushdown) {
@@ -303,7 +310,7 @@ private final class HiveSplitSource implements ConnectorSplitSource {
303310
this.splittable = tableFormat.isSplittable() && !isLzo;
304311
this.targetSplitSize = getTargetSplitSize(session);
305312
this.partitionBatchSize = Math.max(1,
306-
sessionInt(session, "num_partitions_in_batch_mode", DEFAULT_BATCH_THRESHOLD));
313+
sessionInt(session, "split_partition_batch_size", DEFAULT_PARTITION_BATCH_SIZE));
307314
recordPruningProfile(handle);
308315
}
309316

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ public void execute(Runnable task) {
216216
});
217217
HiveTableHandle handle = parquetHandle(PART_KEYS).toBuilder().prunedPartitions(Arrays.asList(
218218
part("year=2024/month=01"), part("year=2024/month=02"))).build();
219-
FakeSession session = new FakeSession(Collections.singletonMap("num_partitions_in_batch_mode", "1"),
219+
FakeSession session = new FakeSession(Collections.singletonMap("split_partition_batch_size", "1"),
220220
scheduler);
221221
try (ConnectorSplitSource source = provider.streamSplits(session,
222222
ConnectorScanRequest.builder(handle, Collections.emptyList()).build())) {
@@ -262,6 +262,40 @@ public void streamingUsesEnginePartitionSelectionWithoutListingOtherPartitions()
262262
Assertions.assertEquals(Collections.singleton("year=2024/month=02"), lister.callsPerLocation.keySet());
263263
}
264264

265+
@Test
266+
public void partitionMetadataBatchSizeDoesNotFollowBatchModeThreshold() throws Exception {
267+
List<Integer> requestedBatchSizes = new ArrayList<>();
268+
FakeHmsClient hms = new FakeHmsClient() {
269+
@Override
270+
public HmsPartitionBatchResult getExistingPartitionsWithStats(
271+
String dbName, String tableName, List<String> partitionNames) {
272+
requestedBatchSizes.add(partitionNames.size());
273+
return super.getExistingPartitionsWithStats(dbName, tableName, partitionNames);
274+
}
275+
};
276+
CountingLister lister = new CountingLister();
277+
HiveScanPlanProvider provider = provider(hms, lister);
278+
List<String> names = Arrays.asList("year=2024/month=01", "year=2024/month=02",
279+
"year=2024/month=03", "year=2024/month=04", "year=2024/month=05");
280+
Map<String, String> settings = new HashMap<>();
281+
settings.put("num_partitions_in_batch_mode", "1");
282+
settings.put("split_partition_batch_size", "2");
283+
ConnectorScanRequest request = ConnectorScanRequest.builder(
284+
parquetHandle(PART_KEYS), Collections.emptyList()).requiredPartitions(names).build();
285+
286+
int ranges = 0;
287+
try (ConnectorSplitSource source = provider.streamSplits(new FakeSession(settings), request)) {
288+
while (source.hasNext()) {
289+
source.next();
290+
ranges++;
291+
}
292+
}
293+
294+
Assertions.assertEquals(5, ranges);
295+
Assertions.assertEquals(Arrays.asList(2, 2, 1), requestedBatchSizes);
296+
Assertions.assertEquals(5, lister.totalCalls);
297+
}
298+
265299
@Test
266300
public void sharedCompressionNamesMapToNativeCodecs() {
267301
for (FileCompression codec : FileCompression.values()) {
@@ -360,7 +394,7 @@ public void partitionBatchAndStreamingMatchAcrossFormatsAndListingFailures() thr
360394
.prunedPartitions(Collections.singletonList(part("year=2023/month=12"))).build();
361395
Map<String, String> settings = new HashMap<>();
362396
settings.put("file_split_size", "8");
363-
settings.put("num_partitions_in_batch_mode", "1");
397+
settings.put("split_partition_batch_size", "1");
364398
FakeSession session = new FakeSession(settings);
365399
ConnectorScanRequest request = ConnectorScanRequest.builder(handle, Collections.emptyList())
366400
.requiredPartitions(Collections.singletonList("year=2023/month=12")).build();
@@ -437,6 +471,15 @@ public void supportsBatchScanIsTrueForPartitionedNonTransactionalTable() {
437471
Assertions.assertTrue(provider.supportsBatchScan(new FakeSession(), handle));
438472
}
439473

474+
@Test
475+
public void partitionBatchSizePreservesParallelListingAndBoundsHmsRequests() {
476+
HiveScanPlanProvider provider = provider(null, new CountingLister());
477+
Assertions.assertEquals(16, provider.partitionBatchSize(new FakeSession(), 1000, 64));
478+
Assertions.assertEquals(5, provider.partitionBatchSize(new FakeSession(), 9, 2));
479+
Assertions.assertEquals(2, provider.partitionBatchSize(
480+
new FakeSession(Collections.singletonMap("split_partition_batch_size", "2")), 1000, 64));
481+
}
482+
440483
@Test
441484
public void supportsBatchScanIsFalseForUnpartitionedTable() {
442485
HiveScanPlanProvider provider = provider(null, new CountingLister());

‎fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,16 @@ default boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle
222222
return false;
223223
}
224224

225+
/**
226+
* Number of partitions assigned to one split-production batch. The batch-mode threshold and
227+
* the amount of work in a batch serve different purposes: the former selects the asynchronous
228+
* path, while this size bounds each connector request. Connectors with cheap, incremental
229+
* partition resolution may choose a smaller size than the legacy read-session batch.
230+
*/
231+
default int partitionBatchSize(ConnectorSession session, int partitionCount, int parallelism) {
232+
return 1024;
233+
}
234+
225235
/**
226236
* Whether this connector's scan ranges carry meaningful byte lengths
227237
* ({@link ConnectorScanRange#getLength()}) so the engine can apply {@code TABLESAMPLE} by

‎fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -91,8 +91,9 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException
9191
// Major 20 adds deleteUsingMetadata to the connector write surface; major 21 adds
9292
// typed filesystem credential descriptions and statement-scoped metadata.
9393
// Major 22 adds the metadata-only DDL handle path; major 23 adds the provider's
94-
// ConnectorSecuritySurface declaration; major 24 adds the scan task scheduler.
95-
Assertions.assertEquals("24.0", version.getProperty("api.version"));
94+
// ConnectorSecuritySurface declaration; major 24 adds the scan task scheduler;
95+
// major 25 adds connector-sized partition batches.
96+
Assertions.assertEquals("25.0", version.getProperty("api.version"));
9697
}
9798

9899
/** Root entry points plus provider/handle types returned to connector plugins. */

‎fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProviderBatchScanTest.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,12 @@ public void testSupportsBatchScanDefaultsFalse() {
111111
Assertions.assertFalse(provider.supportsBatchScan(null, null));
112112
}
113113

114+
@Test
115+
public void testPartitionBatchSizeDefaultsToLegacyReadSessionSize() {
116+
ConnectorScanPlanProvider provider = new RecordingProvider();
117+
Assertions.assertEquals(1024, provider.partitionBatchSize(null, 1000, 64));
118+
}
119+
114120
@Test
115121
public void testStreamingSplitEstimateDefaultsNegative() {
116122
// FIX-M3: default MUST be < 0 so no connector silently enters file-count streaming without opting in

‎fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,7 @@ org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodePropert
188188
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#ignorePartitionPruneShortCircuit():boolean
189189
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest):java.util.List
190190
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScanForPartitionBatch(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest,java.util.List):java.util.List
191+
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#partitionBatchSize(org.apache.doris.connector.spi.ConnectorSession,int,int):int
191192
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#populateScanLevelParams(org.apache.doris.thrift.TFileScanRangeParams,java.util.Map):void
192193
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#releaseReadTransaction(java.lang.String):void
193194
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#scannedPartitionCount(java.util.List):java.util.OptionalLong

‎fe/fe-connector/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ under the License.
5555
of the latter two means bumping this property as well (and fe-extension-spi means bumping
5656
all five families).
5757
-->
58-
<connector.plugin.api.version>24.0</connector.plugin.api.version>
58+
<connector.plugin.api.version>25.0</connector.plugin.api.version>
5959
</properties>
6060

6161
<modules>

‎fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2086,7 +2086,7 @@ public int numApproximateSplits() {
20862086
}
20872087

20882088
/**
2089-
* Asynchronously generates splits in batches of {@code num_partitions_in_batch_mode} partitions,
2089+
* Asynchronously generates splits in connector-sized partition batches,
20902090
* streaming each batch into {@link #splitAssignment}. Mirrors legacy
20912091
* {@code MaxComputeScanNode.startSplit}: one read session per partition batch (built by the
20922092
* connector via {@link ConnectorScanPlanProvider#planScanForPartitionBatch}) on the shared
@@ -2142,9 +2142,12 @@ public void startSplit(int numBackends) {
21422142
sources = Collections.singletonList(createStreamingBatchSource(scanProvider,
21432143
request.withRequiredPartitions(requiredPartitions), summaryProfile));
21442144
} else {
2145+
int partitionCount = selectedPartitions.selectedPartitions.size();
2146+
int batchSize = onPluginClassLoader(scanProvider, () -> scanProvider.partitionBatchSize(
2147+
connectorSession, partitionCount, Config.external_scan_split_threads));
21452148
sources = createPartitionBatchSources(scanProvider, request,
21462149
new ArrayList<>(selectedPartitions.selectedPartitions.keySet()),
2147-
sessionVariable.getNumPartitionsInBatchMode(), Config.external_scan_split_threads);
2150+
batchSize, Config.external_scan_split_threads);
21482151
}
21492152
startSplitProducers(scanProvider, sources,
21502153
summaryProfile == null ? null : summaryProfile.getExecutionSummary());
@@ -2277,7 +2280,7 @@ public void close() {
22772280
private SplitProducer.BatchSource createStreamingBatchSource(ConnectorScanPlanProvider scanProvider,
22782281
ConnectorScanRequest request, SummaryProfile summaryProfile) {
22792282
RuntimeProfile generationProfile = new RuntimeProfile("Split Generation " + getId());
2280-
generationProfile.addInfoString("BatchSize", String.valueOf(sessionVariable.getSplitBatchSize()));
2283+
generationProfile.addInfoString("BatchSize", String.valueOf(splitAssignment.getSplitBatchSize()));
22812284
generationProfile.addInfoString("QueueCapacity", String.valueOf(sessionVariable.getSplitQueueCapacity()));
22822285
Counter generatedSplits = generationProfile.addCounter("GeneratedSplits", TUnit.UNIT,
22832286
RuntimeProfile.ROOT_COUNTER);

0 commit comments

Comments
 (0)