Skip to content

Commit 8b6086f

Browse files
committed
[improvement](fe) Schedule external scans fairly and bound metadata prefetch
### What problem does this PR solve? Issue Number: None Problem Summary: External scans can monopolize shared FIFO workers, wait for work submitted to their own pool, repeat scan metadata work, and load cold Iceberg manifests serially. Introduce query-owned bounded task groups, separate split and metadata workers, cooperative continuations, bounded completion-order manifest prefetch, and query-owned cancellation/error cleanup. Keep scan path reuse, bounded backend candidate lookup, scan envelope compaction and format handling with scheduling. Notify backend consumers on data, completion, failure and cancellation to avoid terminal polling delays. Limit the first split response independently of producer batches and bounded queues. Producers continue prefetching before BE consumption, preserving overlap with BE startup. On the same Release BE and 1,000-partition Hive table, 100 four-query LIMIT groups improve from 15.38 ms to 11.82 ms median versus the pre-delivery-change FE. With a controlled 20 ms per-directory wait and 300 ms initial consumer delay, full scans remain 388.52 ms versus 386.96 ms; all COUNT/SUM results match. ### Release note External scan task groups support bounded fair scheduling, cancellation and concurrent Iceberg manifest loading. The split_initial_batch_size session variable (default 64) controls the first response per backend independently of producer batches and buffered prefetch. File formats and pinned snapshot semantics are preserved. ### Check List (For Author) - Test: Clean build.sh --fe including Checkstyle; 750 scoped FE/SPI/Hive/Iceberg unit tests; Hive streaming and Iceberg scheduled-split regression suites; three-version matched Release-BE comparisons with 1,200 LIMIT SQL and 150 full scans, all result-checked; diagnostic 400-query first-response bound and terminal-wakeup verification. - Behavior changed: Yes, external scan scheduling, bounded metadata prefetch and initial split delivery. - Does this need documentation: Yes, developer scheduling documentation included.
1 parent e0bf487 commit 8b6086f

78 files changed

Lines changed: 5894 additions & 905 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2207,6 +2207,15 @@ public class Config extends ConfigBase {
22072207
@ConfField(mutable = false, masterOnly = false)
22082208
public static int max_external_cache_loader_thread_pool_size = 64;
22092209

2210+
@ConfField(description = "Workers for fair split production")
2211+
public static int external_scan_split_threads = 64;
2212+
2213+
@ConfField(description = "Workers for independent metadata tasks")
2214+
public static int external_scan_metadata_threads = 64;
2215+
2216+
@ConfField(description = "Maximum queued tasks per query in each external scan scheduler")
2217+
public static int external_scan_task_queue_capacity = 128;
2218+
22102219
/**
22112220
* Max cache num of external catalog's file
22122221
* Decrease this value if FE's memory is small

‎fe/fe-connector/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,9 @@ tests and embedded use.
120120

121121
## Reading the API
122122

123+
For the scan task scheduler SPI, admission, cancellation and configuration, see
124+
[Cooperative external scan scheduling](docs/scan-task-scheduling.md).
125+
123126
This document never lists SPI methods. The truth lives in code, behind four
124127
mechanisms:
125128

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Cooperative external scan scheduling
2+
3+
A statement obtains its stage schedulers through
4+
`ConnectorSession.getTaskScheduler`. The engine owns the task groups and closes
5+
them with the statement. Connectors submit neutral `Runnable` tasks through this
6+
SPI; they do not import FE executor classes.
7+
8+
The two stages have independent workers:
9+
10+
- `SPLIT` advances scan producers and assigns ranges to the BE split queues.
11+
- `METADATA` performs independent operations such as cold Manifest reads.
12+
13+
A producer may need metadata tasks to finish. Keeping those tasks out of the
14+
producer pool avoids a pool full of parents waiting for children queued behind
15+
them. A task must not block waiting for children submitted to its own executor.
16+
17+
## Admission and continuation
18+
19+
Each executor has a concurrent FIFO ready queue containing task groups. A
20+
worker claims a task from the next group, republishes that group if it still
21+
has pending work, and then executes the task outside queue synchronization.
22+
This allows several workers to serve one query and gives other ready queries
23+
an opportunity at subsequent task boundaries. It does not preempt a running
24+
task or guarantee equal CPU time.
25+
26+
`tryExecute` never waits for space. A full group queue registers the supplied
27+
`onSpace` callback and returns `false`; the caller retains the task until it
28+
can retry. Reuse the same callback identity and remove it when the task is
29+
cancelled or no longer needs admission. Failures are delivered through the
30+
failure callback rather than silently dropping work.
31+
32+
A serial producer can use `tryExecuteContinuation` for bounded further work.
33+
The group still returns behind other ready groups, but a worker need not wake
34+
another worker just to execute its own serial continuation. Initial tasks,
35+
external admission and recovery from backpressure retain the normal wakeup
36+
protocol. Split production yields after at most 16 source batches or after
37+
observing 1 ms elapsed between batches. One blocking batch can exceed that
38+
interval; the bound is cooperative.
39+
40+
## Iceberg Manifest reads
41+
42+
Cold Manifest prefetch uses a bounded window and a statement-wide read budget.
43+
Within that window, completed reads can be consumed before a slower earlier
44+
Manifest. First-split production does not wait for speculative prefetch when
45+
one Manifest already supplies output; subsequent BE demand enables further
46+
prefetch. Filtered or rewrite-scoped scans must still advance while no output
47+
has been found.
48+
49+
Closing a scan stops new submissions and wakes completion waiters. Load failures
50+
propagate through the corresponding future. A running shared cache load is not
51+
interrupted just because one waiting scan stops. Authentication and plugin
52+
thread-context class loading remain scoped around each actual metadata read.
53+
Cached partition values stay immutable: adapt a file task to the pinned scan
54+
schema at the scan boundary. Select the format from each file range rather
55+
than planning files merely to infer a table-wide format.
56+
57+
The captured scheduler exposes its closed state even after query cleanup removes
58+
the statement registry. A Manifest reader waiting for another scan's read permit
59+
checks this state at most 100 ms apart and rechecks after admission. Cancellation
60+
therefore stops that waiter without releasing another scan's permit or starting
61+
another remote read. This does not interrupt an already running shared cache load.
62+
Hive checks the same captured state between directory requests, including when
63+
empty partitions are skipped within one iterator call.
64+
65+
## Configuration and limits
66+
67+
| Setting | Default | Meaning |
68+
|---|---:|---|
69+
| FE `external_scan_split_threads` | 64 | Worker threads in the producer scheduler |
70+
| FE `external_scan_metadata_threads` | 64 | Worker threads in the metadata scheduler |
71+
| FE `external_scan_task_queue_capacity` | 128 | Pending tasks per query in each stage |
72+
| Session `iceberg_manifest_load_concurrency` | 8 | Manifest read budget per query and window bound per scan; 1 uses sequential loading |
73+
74+
These task queues are distinct from `split_queue_capacity`, which bounds the
75+
produced Split buffer. Queue capacity does not reserve that many running
76+
threads. Worker counts limit the respective pools, and the query read budget
77+
also covers caller-assisted Manifest loading.
78+
79+
Workers are ordinary threads. Synchronous remote calls occupy workers until
80+
they return; FIFO admission cannot recover a worker blocked inside an SDK.
81+
Effective remote timeouts and dependency fault isolation are separate concerns.
82+
83+
Planning-only attempts also run their query-finish callbacks when statement
84+
execution returns. This releases batch Split producers and metadata leases for
85+
EXPLAIN and planning failures even when no coordinator was registered. Normal
86+
coordinator unregistration remains idempotent. Deferred Arrow Flight queries keep
87+
those callbacks until their retained executor finalizes the query after consumption.
88+
89+
Backend assignment has its own policy-state lock. First-split readiness and stop
90+
notifications use a separate short-held lock, so an already converted range can
91+
wake the planner while another lane is still assigning its batch. The policy's
92+
mutable load-balancing state remains serialized.
93+
94+
The engine captures task-group cancellation on the statement executor before
95+
asynchronous work starts. KILL and timeout close those groups even before a
96+
coordinator exists; a group captured after cancellation is immediately closed.
97+
Cancellation listeners run outside their registry operation and are claimed once.
98+
This stops queued work and wakes future waiters without interrupting a running
99+
shared cache loader. Normal completion retains the separate read-lease cleanup.
100+
101+
### Consumer completion notification
102+
103+
Each backend has a bounded split queue and its own data-ready wait channel. A consumer registers before checking queue contents and terminal state, then waits only when neither is available. Successful enqueue, producer completion, failure and cancellation signal that channel. The empty queue is not periodically polled to discover EOF: otherwise finishing just after a poll begins adds 100 ms despite completed production. Producers avoid the channel monitor when no consumer waits; callbacks that resume producers run after releasing the consumer channel monitor. Cancellation signals consumers before closing SDK resources.
104+
105+
### Initial delivery and producer lookahead
106+
107+
`split_initial_batch_size` (default 64) bounds the first Split fetch per backend, also capped by the BE request size. Subsequent fetches use the BE request size. The bound is applied before compatible file ranges are compacted into RPC envelopes. It is independent of `split_batch_size` (producer batch size) and `split_queue_capacity` (buffered batches per backend). Producers continue listing files and preparing splits before BE consumption, yielding only for task fairness and configured queue backpressure. A small initial RPC therefore does not suspend useful directory prefetch while BE starts.

‎fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.doris.connector.spi.ConnectorContext;
2626
import org.apache.doris.connector.spi.ConnectorSession;
2727
import org.apache.doris.connector.spi.ConnectorStorageContext;
28+
import org.apache.doris.connector.spi.ConnectorTaskScheduler;
2829
import org.apache.doris.connector.spi.DorisConnectorException;
2930
import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
3031
import org.apache.doris.connector.spi.handle.ConnectorTableHandle;
@@ -304,6 +305,7 @@ private static int sessionInt(ConnectorSession session, String name, int default
304305

305306
private final class HiveSplitSource implements ConnectorSplitSource {
306307
private final HiveTableHandle handle;
308+
private final ConnectorTaskScheduler scheduler;
307309
private final FileSystem fs;
308310
private final HiveFileFormat tableFormat;
309311
private final boolean isLzo;
@@ -321,6 +323,7 @@ private final class HiveSplitSource implements ConnectorSplitSource {
321323

322324
HiveSplitSource(ConnectorSession session, HiveTableHandle handle, List<String> requiredPartitions) {
323325
this.handle = handle;
326+
this.scheduler = session.getTaskScheduler(ConnectorTaskScheduler.Stage.SPLIT);
324327
this.partitionNames = requiredPartitions.isEmpty() ? null : requiredPartitions;
325328
this.fs = storage().getFileSystem(session);
326329
this.tableFormat = HiveFileFormat.detect(handle.getInputFormat(), handle.getSerializationLib(),
@@ -335,7 +338,10 @@ private final class HiveSplitSource implements ConnectorSplitSource {
335338

336339
@Override
337340
public boolean hasNext() {
338-
while (!closed) {
341+
// Empty partitions can keep hasNext() inside the source for many directory requests.
342+
// Observe query cleanup here, before advancing to another listing, rather than waiting
343+
// for the producer to regain control and close this source after its active read.
344+
while (!closed && !scheduler.isClosed()) {
339345
if (ranges.hasNext()) {
340346
return true;
341347
}

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import org.apache.doris.connector.hms.HmsTableInfo;
2727
import org.apache.doris.connector.spi.ConnectorSession;
2828
import org.apache.doris.connector.spi.ConnectorStatementScope;
29+
import org.apache.doris.connector.spi.ConnectorTaskScheduler;
2930
import org.apache.doris.connector.spi.DorisConnectorException;
3031
import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
3132
import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider;
@@ -53,7 +54,9 @@
5354
import java.util.Map;
5455
import java.util.Optional;
5556
import java.util.Set;
57+
import java.util.concurrent.atomic.AtomicBoolean;
5658
import java.util.concurrent.atomic.AtomicInteger;
59+
import java.util.function.Consumer;
5760
import java.util.function.Supplier;
5861

5962
/**
@@ -184,6 +187,46 @@ public void streamingAndSynchronousSplitsMatchAcrossPartitionsAndFileBoundaries(
184187
}
185188
}
186189

190+
@Test
191+
public void cancellationBetweenEmptyPartitionsDoesNotListRemainingPartitions() throws Exception {
192+
AtomicBoolean cancelled = new AtomicBoolean();
193+
AtomicInteger listings = new AtomicInteger();
194+
ConnectorTaskScheduler scheduler = new ConnectorTaskScheduler() {
195+
@Override
196+
public boolean isClosed() {
197+
return cancelled.get();
198+
}
199+
200+
@Override
201+
public boolean tryExecute(Runnable task, Consumer<Throwable> failure, Runnable onSpace) {
202+
throw new UnsupportedOperationException("This source only observes the query lifetime");
203+
}
204+
205+
@Override
206+
public void removeSpaceListener(Runnable onSpace) {
207+
}
208+
209+
@Override
210+
public void execute(Runnable task) {
211+
throw new UnsupportedOperationException("This source only observes the query lifetime");
212+
}
213+
};
214+
HiveScanPlanProvider provider = provider(new FakeHmsClient(), (location, fs) -> {
215+
listings.incrementAndGet();
216+
cancelled.set(true);
217+
return Collections.emptyList();
218+
});
219+
HiveTableHandle handle = parquetHandle(PART_KEYS).toBuilder().prunedPartitions(Arrays.asList(
220+
part("year=2024/month=01"), part("year=2024/month=02"))).build();
221+
FakeSession session = new FakeSession(Collections.singletonMap("num_partitions_in_batch_mode", "1"),
222+
scheduler);
223+
try (ConnectorSplitSource source = provider.streamSplits(session,
224+
ConnectorScanRequest.builder(handle, Collections.emptyList()).build())) {
225+
Assertions.assertFalse(source.hasNext());
226+
Assertions.assertEquals(1, listings.get(), "cancellation must stop listing the next empty partition");
227+
}
228+
}
229+
187230
@Test
188231
public void closingFirstPartitionDoesNotListRemainingPartitions() throws Exception {
189232
CountingLister lister = new CountingLister();
@@ -1263,13 +1306,24 @@ public void close() {
12631306
/** Minimal {@link ConnectorSession} (no split-size override, empty session properties). */
12641307
private static final class FakeSession implements ConnectorSession {
12651308
private final Map<String, String> properties;
1309+
private final ConnectorTaskScheduler scheduler;
12661310

12671311
FakeSession() {
12681312
this(Collections.emptyMap());
12691313
}
12701314

12711315
FakeSession(Map<String, String> properties) {
1316+
this(properties, ConnectorTaskScheduler.DIRECT);
1317+
}
1318+
1319+
FakeSession(Map<String, String> properties, ConnectorTaskScheduler scheduler) {
12721320
this.properties = properties;
1321+
this.scheduler = scheduler;
1322+
}
1323+
1324+
@Override
1325+
public ConnectorTaskScheduler getTaskScheduler(ConnectorTaskScheduler.Stage stage) {
1326+
return scheduler;
12731327
}
12741328

12751329
private final Map<String, Object> statementMemos = new HashMap<>();

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCacheSizeEstimator.java‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -246,10 +246,7 @@ private static long estimateWrappedReferenceList(List<?> values) {
246246
}
247247

248248
static long estimateManifestKey(IcebergManifestEntryKey key) {
249-
long bytes = add(MANIFEST_KEY_SHALLOW_BYTES, JvmSizeUtils.stringSize(key.getManifestPath()));
250-
// The key retains its bound partition type. A non-empty value's files share it, but an empty value (a delete
251-
// manifest whose entries are all DELETED) holds nothing that would account for it.
252-
return add(bytes, estimateIcebergType(key.getPartitionType()));
249+
return add(MANIFEST_KEY_SHALLOW_BYTES, JvmSizeUtils.stringSize(key.getManifestPath()));
253250
}
254251

255252
static long estimateManifestValue(ManifestCacheValue value) {

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@
4343
* disclosure). Vended-credentials uses a single static catalog identity, so every user loads the same table and
4444
* sharing a comment is safe.
4545
*
46-
* <p><b>No credential gate</b> (unlike {@link IcebergTableCache}, like {@link IcebergPartitionCache} /
47-
* {@link IcebergFormatCache}): the cached value is a bare comment {@link String} with no {@code FileIO} /
46+
* <p><b>No credential gate</b> (unlike {@link IcebergTableCache}, like {@link IcebergPartitionCache}): the cached value
47+
* is a bare comment {@link String} with no {@code FileIO} /
4848
* credential. A comment changes only via external DDL, picked up through the REFRESH invalidate hooks. The
4949
* connector builds it with the table entry's {@code meta.cache.iceberg.table.(enable|ttl-second|capacity)};
5050
* {@code enable=false}, {@code ttl-second <= 0} or {@code capacity=0} disables it (read live), so a no-cache

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -221,9 +221,6 @@ public class IcebergConnector implements Connector {
221221
// The value is pure metadata (no FileIO/credential), but under session=user it is an authorization-sensitive
222222
// projection (a shared hit would disclose one user's partitions), so it is disabled there (see constructor).
223223
private final IcebergPartitionCache partitionCache; // null under session=user
224-
// PERF-03: cross-query inferred-file-format cache (the whole-table planFiles() fallback result, keyed by
225-
// (table, snapshotId)). Same authorization-sensitive treatment as partitionCache: disabled under session=user.
226-
private final IcebergFormatCache formatCache; // null under session=user
227224
// PERF-05: cross-query table-comment cache (value = the 'comment' property string). Built ONLY for a REST
228225
// vended-credentials catalog that is NOT session=user -- plain catalogs already reuse tableCache (PERF-01) for
229226
// the comment path, and session=user must stay live because the loadTable itself carries per-user
@@ -236,7 +233,7 @@ public class IcebergConnector implements Connector {
236233
// rebuild, not just the remote scan. Two typed fields because the two SPI hooks return structurally different
237234
// derived types and neither derives from the other; the shared raw PARTITIONS scan underneath is already
238235
// deduped by partitionCache, so two derived caches never double-scan remotely. Authorization-sensitive
239-
// projection like partitionCache/formatCache: disabled (null) under iceberg.rest.session=user so a shared
236+
// projection like partitionCache: disabled (null) under iceberg.rest.session=user so a shared
240237
// (no user dimension) hit cannot disclose one user's partition view; kept for every other flavor.
241238
private final ConnectorMetadataCache<ConnectorMvccPartitionView> // null under session=user
242239
mvccPartitionViewCache;
@@ -313,11 +310,6 @@ public IcebergConnector(Map<String, String> properties, ConnectorContext context
313310
? null
314311
: new IcebergPartitionCache(
315312
metaCache, cacheSpec("partition"));
316-
// PERF-03: inferred-file-format cache. Same authorization-sensitive treatment as partitionCache (disabled
317-
// under session=user, kept otherwise); readers already tolerate a null cache (resolveFileFormatName).
318-
this.formatCache = isUserSessionEnabled()
319-
? null
320-
: new IcebergFormatCache(metaCache, tableFactsCacheSpec);
321313
// PERF-05: table-comment cache, built ONLY for a REST vended-credentials catalog that is NOT session=user.
322314
// Plain catalogs (tableCache on) already serve the comment path from tableCache; session=user is excluded
323315
// because a shared comment cache would bypass the per-user loadTable authorization (a metadata disclosure).
@@ -811,11 +803,6 @@ IcebergPartitionCache partitionCacheForTest() {
811803
return partitionCache;
812804
}
813805

814-
/** Test-only: the cross-query inferred-file-format cache (PERF-03), or {@code null} for a session=user catalog. */
815-
IcebergFormatCache formatCacheForTest() {
816-
return formatCache;
817-
}
818-
819806
/** Test-only: the table-comment cache (PERF-05), or {@code null} unless vended-credentials and non-session. */
820807
IcebergCommentCache commentCacheForTest() {
821808
return commentCache;
@@ -840,7 +827,7 @@ public ConnectorScanPlanProvider getScanPlanProvider() {
840827
// threaded for parity with the legacy single per-catalog IcebergMetadataOps.
841828
return new IcebergScanPlanProvider(catalogProps,
842829
this::newCatalogBackedOps, context, manifestCache,
843-
tableCache, formatCache, catalogResourceTracker);
830+
tableCache, catalogResourceTracker);
844831
}
845832

846833
@Override

0 commit comments

Comments
 (0)