You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5a8470c
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](fe) Default external scans to bounded split batches
### What problem does this PR solve?
Issue Number: None
Problem Summary: External scans previously entered batch mode only above 1024 partitions or files. When enabled, each producer batch held 128 splits and each backend queue was bounded by 32 batches, so small LIMIT queries could wait for unnecessary work while the queue capacity depended on batch size. Enable batch mode from one partition or file, separate the Hive partition-processing size from that trigger, and store individual splits in a queue bounded by 1024 splits per backend. Producers still offer splits in batches; use eight splits per batch by default. In a matched one-BE Release benchmark on the integrated test branch, refreshed-metadata LIMIT medians at batch sizes 8 versus 32 were 75.8 versus 102.5 ms for Hive and 23.5 versus 30.6 ms for Iceberg. Warm full scans showed no stable winner across those sizes. Preserve request batch limits, cancellation, backpressure, and producer wakeups. The new connector SPI method advances the plugin API major version to 25.
### Release note
External table Batch Mode now starts at one partition or file by default. The default producer batch size is eight splits, queue capacity is 1024 splits per backend, and split_partition_batch_size controls Hive partition-processing batches separately.
### Check List (For Author)
- Test: Unit Test. Before rebasing onto current master, FE build and ten focused FE test classes passed (162 tests). On current master, ConnectorPluginSurfaceTest (2), ConnectorMetadataSurfaceTest (4), and HiveScanBatchModeTest (36) passed. FE-core Maven tests could not start locally because snapshot reactor artifacts were unresolved without -am; the full -am route requires an x86 grpc generator unavailable on this ARM host. One-BE Release benchmark results above were recorded on the integrated test branch.
- Behavior changed: Yes (Batch Mode thresholds and split buffering defaults change as described above)
- Does this need documentation: No (connector scheduling document updated in this commit)
Copy file name to clipboardExpand all lines: fe/fe-connector/docs/scan-task-scheduling.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -108,11 +108,11 @@ shared cache loader. Normal completion retains the separate read-lease cleanup.
108
108
109
109
### Consumer completion notification
110
110
111
-
Each backend has a bounded split queue and its own data-ready wait channel. A consumer registers before checking queue contents and terminal state, then waits only when neither is available. Successful enqueue, producer completion, failure and cancellation signal that channel. The empty queue is not periodically polled to discover EOF: otherwise finishing just after a poll begins adds 100 ms despite completed production. Producers avoid the channel monitor when no consumer waits; callbacks that resume producers run after releasing the consumer channel monitor. Cancellation signals consumers before closing SDK resources.
111
+
Each backend has a split queue bounded by the number of queued splits and its own data-ready wait channel. Producers offer batches, but individual splits are the queue entries. A consumer registers before checking queue contents and terminal state, then waits only when neither is available. Successful enqueue, producer completion, failure and cancellation signal that channel. The empty queue is not periodically polled to discover EOF: otherwise finishing just after a poll begins adds 100 ms despite completed production. Queue bookkeeping uses a short per-backend lock; callbacks that resume producers run after releasing it. Cancellation signals consumers before closing SDK resources.
112
112
113
113
### Initial delivery and producer lookahead
114
114
115
-
`split_initial_batch_size` (default 64) bounds the first Split fetch per backend, also capped by the BE request size. Subsequent fetches use the BE request size. The bound is applied before compatible file ranges are compacted into RPC envelopes. It is independent of `split_batch_size` (producer batch size) and `split_queue_capacity` (buffered batches per backend). Producers continue listing files and preparing splits before BE consumption, yielding only for task fairness and configured queue backpressure. A small initial RPC therefore does not suspend useful directory prefetch while BE starts.
115
+
`split_initial_batch_size` (default 64) bounds the first Split fetch per backend, also capped by the BE request size. Subsequent fetches use the BE request size. The bound is applied before compatible file ranges are compacted into RPC envelopes. `split_batch_size` (default 8) limits producer batches, while `split_queue_capacity` (default 1024) limits buffered splits per backend. A configured queue capacity smaller than the batch size also reduces the effective producer batch size. Producers continue listing files and preparing splits before BE consumption, yielding only for task fairness and configured queue backpressure. A small initial RPC therefore does not suspend useful directory prefetch while BE starts.
116
116
117
117
Actual Manifest readers retain a lease on the exact read-table generation until the callable exits. Query cleanup releases the statement owner, while an already running shared reader keeps its FileIO alive. Future cancellation is not evidence that the callable has finished.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java
+45-2Lines changed: 45 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -216,7 +216,7 @@ public void execute(Runnable task) {
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
+3-2Lines changed: 3 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -91,8 +91,9 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException
91
91
// Major 20 adds deleteUsingMetadata to the connector write surface; major 21 adds
92
92
// typed filesystem credential descriptions and statement-scoped metadata.
93
93
// Major 22 adds the metadata-only DDL handle path; major 23 adds the provider's
94
-
// ConnectorSecuritySurface declaration; major 24 adds the scan task scheduler.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProviderBatchScanTest.java
+6Lines changed: 6 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -111,6 +111,12 @@ public void testSupportsBatchScanDefaultsFalse() {
0 commit comments