[fix](fe) Stop split enumeration when queries finish or terminate by CalvinKirs · Pull Request #69835 · apache/doris

CalvinKirs · 2026-10-29T09:19:27Z

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

External scans can continue listing files and retain split queues after a LIMIT query finishes, a query is cancelled, or its deadline expires. This change makes split enumeration follow the query's lifetime and adds a configurable first-split deadline with progress in the error message.

Area	What changes	How it works
Query lifetime	Stop split work on LIMIT completion, cancellation, timeout, and failed/replaced planning attempts.	Query- and attempt-owned scan scopes propagate cancellation; statement resources remain owned until the associated tasks actually exit.
Producer scheduling	A stopped producer cannot be restarted by a late wakeup or left in the executor queue.	Explicit closing/closed states prevent resubmission; cancellation removes queued tasks and handles submission races. Idle/queued sources close on a separate cleanup pool; a running worker closes its own source before completing its termination future.
Hive/HMS and file listing	Stop requesting more partition metadata, files, and listing pages after cancellation.	Check the scan context at metadata RPC/retry, partition, file, and S3 pagination boundaries. Share completed cache entries while isolating cancellable in-flight loads from healthy callers.
Iceberg SDK planning	Stop owned manifest/data/delete planning jobs and wait for their actual exit.	A source-owned executor view and cancellable FileIO reuse the existing SDK pool. Queued jobs settle independently; SDK bodies exit before iterators and statement resources close.
First split wait	Make the initial wait configurable and report enumeration progress on timeout.	Add `first_split_wait_timeout_ms` (default `30000`, minimum `1`, supports `SET_VAR`). The error includes processed/total partitions, enumerated files, generated splits, and active producers. This is separate from the existing subsequent split-fetch timeout.
Bounded work	Keep backpressure and allow an explicit producer cap per scan.	Preserve upstream query-owned fair schedulers and queues bounded by individual splits. Schedule partition-batch producers as capacity becomes available. Add `split_producer_parallelism`: `0` retains pool-derived parallelism; whole-scan streams use one producer.

Cancellation is cooperative. An already-running SDK operation that cannot be interrupted may still need to return before its owner can finish cleanup.

Release note

External scan split enumeration now stops when its query finishes or terminates. Add the first_split_wait_timeout_ms and split_producer_parallelism session variables. Connector/filesystem plugin API versions become 26.0/7.0; deploy plugins built with the matching FE. Native streaming Iceberg planning uses request-local FileIO without the SDK content cache; catalog FileIO and the Doris manifest cache are retained.

Check List (For Author)

Test:
- Unit Test
- Manual test
- Regression test (deterministic session-variable suite retained; no claim of a complete regression run on the latest master)
Behavior changed:
- No.
- Yes: query termination stops split scheduling/enumeration and releases owned resources; first-split waits have an explicit deadline.
Does this need documentation?
- No.
- Yes: connector cancellation contract and variables are documented in fe/fe-connector/README.md; user-facing variable reference follow-up is pending. Manual acceptance/performance tools and evidence remain local and are excluded from this PR.

Validation	Result
FE compilation and packaging (`00dfe97a`)	Complete `build.sh --fe` exited 0 with Java style validation. Maven build cache was disabled. Source/class/package provenance passed for all 36 changed production sources, 143 classes including nested classes, 24 plugin assemblies, matching API resources, and the actual Version marker. A separate frozen package was reverified before benchmark use.
Related FE tests (`bfe19cde`)	All 761 selected tests across 61 classes completed with zero failures, errors, or skips through `run-fe-ut.sh`. The runner subsequently exited 137 while shading the unrelated BE Java `paimon-s3-shade` module; the complete runner did not pass. Coverage includes queue/admission races, owner-thread source closure and actual termination on read/close/completion errors, query/attempt ownership, cache isolation, and native Iceberg SDK work.
Functional observer (`bfe19cde`)	Observer compiled with JDK 17; all 16 report-analysis tests passed. It reads the current split-count queue structure and rejects the obsolete CSV schema.
HMS cancellation fix (`00dfe97a`)	All 246 related HMS/Hive tests across 19 classes passed with zero failures, errors, or skips; targeted `run-fe-ut.sh` invocation exited 0 using a per-invocation Maven module-selection wrapper. Negative control reproduced both pool-wait cancellation failures before the fix. Coverage includes exhausted-pool waiters, healthy connection reuse, original total wait budget, creation failure propagation, authenticated dispatch, names-cache isolation, and the whole-stream entry path.
Independent review (`00dfe97a`)	APPROVE: Blocker/Major/Minor/Nit all zero. The prior HMS issue is fixed. Main and independent full reviewer covered all 81 files and 17 checkpoints; all 53 anchors in each language passed validation. Source review does not substitute for current package, end-to-end, or performance validation.
Baseline package (`2a4191c1`)	Complete `build.sh --fe` exited 0 with Java style validation and Maven build cache disabled. Independently verified source/class/package provenance and the actual Version marker; all five PR-added production classes are absent from all 2,260 deployed baseline jars. Both comparison packages were frozen with materialized dependencies before either timed run.
Real OSS query validation (`00dfe97a` vs `2a4191c1`)	All 1,440 scheduled queries succeeded: 800 warmups and 640 measured queries, 40 measured samples per version/scenario. No failed or outlier samples excluded. Aggregations returned the expected COUNT/SUM and each read exactly 10,000 physical files according to native BE reader counters.
Real OSS LIMIT/cancel/timeout cleanup (`00dfe97a`)	9/9 passed on 10,000 partitions / 1,000,000 files, three repetitions per trigger. Maximum observed actual exit: LIMIT 90ms, cancellation 135ms, query timeout 475ms. Queues, producers, source resources, listeners and callback registrations were released. A separate observer at the OSS SDK HTTP dispatch boundary saw no new dispatch after 10s during each 12s observation window. Already-entered transport calls/internal retries are not separate dispatches in this observer. No instrumentation was loaded in the performance runs.
Real OSS first-split deadline (`00dfe97a`)	Both 1000ms and 2500ms budgets produced the expected deadline and partition/file/split/producer progress in the error. End-to-end SQL errors took 1167ms / 2761ms, including planning before the wait. A deliberately delayed already-entered SDK operation returned before source closure; cleanup finished 4133ms / 2608ms after the respective error. These are separate acceptance cases, not performance measurements.
PR scope cleanup (`0e882462`)	Remove all 15 local acceptance-tool and dependent-suite files (2,003 added lines). Both fixture-dependent suites and their generated outputs are removed together; no tracked references remain. All FE implementation and unit-test files are byte-identical to the reviewed and runtime-tested `00dfe97a`. Retain only the deterministic session-variable regression (SET, SET_VAR, restoration and invalid values), which has no storage access or elapsed-time assertions.
Unit test scope cleanup (`c3b37ae5`)	Reduce another 144 lines: remove the entire JDK CompletableFuture callback test class, a test that manually constructed two attempt scopes, duplicate failure/queue cases and constant fake-executor assertions. Combine batching cases with both partition-batch parameters and all 300 split identities checked. Keep the actual query-retry path, HMS/SDK cancellation races, resource lifetime, late-fetch EOF and timeout lower-bound oracles. Independent test-quality review completed. Focused `run-fe-ut.sh`: all 47 tests across four classes passed, zero failures/errors/skips, runner exit 0, Checkstyle explicitly enabled and passed, build cache disabled. The Iceberg class with only two constant assertions deleted compiled and passed style; its five tests were not rerun in this focused core selection. Current PR scope: 64 files, +4,075/-276; production Java is +1,743/-245 across 36 files.
Latest independent skill review (`c3b37ae5`)	APPROVE: Blocker/Major/Minor/Nit all zero, one round converged. Main and new independent helper verified 64/64 paths and all 17 checkpoints. Reuse the completed full review for 60 byte-identical paths; freshly review four retained test changes and both paths leaving the PR. Preserve 45 prior dismissals and two superseded historical dispositions; retire three removed-tool dispositions. Both language reports passed 53 anchors each. Author test evidence was verified; reviewers ran no builds, tests or cluster operations. The disclosed OSS performance slowdown remains unresolved.
HDFS query validation (historical `60351f1`)	All 488 scans succeeded: 240 measured queries, 240 warmups, four million-file SUM validations, and four separate profile queries. Full scans were checked against native reader counts and expected SUM results.
HDFS LIMIT/cancel/timeout cleanup (historical `60351f1`)	9/9 cases passed, three repetitions per trigger. Maximum observed exit was 871ms, measured from LIMIT return, KILL send, or the configured query deadline. Producers, queues, sources, listeners, and callback registrations were released; a direct HDFS request observer saw no new listing requests after 10s. Instrumentation was used for this separate resource test, not performance timing.
Configurable first-split timeout (historical `60351f1`)	Manual waits of 1000ms and 2500ms failed near their configured deadlines with progress in the error; invalid nonpositive values are rejected.

The latest built and runtime-tested revision is 00dfe97a935d12c268044171c98103be28372edd, merged with master 2a4191c1dc94e789e1725d6cb345f46c31a17272. Current head c3b37ae5084ebcbeb16a9387551f7612eeb9ffa6 removes local tools and redundant tests; all production sources and API-version POMs are unchanged. Both complete packages, the zero-finding independent source review at 00dfe97a, fresh OSS acceptance (11/11), and the four-round OSS comparison are complete. The PR remains a draft: the OSS cold-catalog 10,000-partition LIMIT is 3.77% slower by median and 4.05% by mean in the formal run; attribution remains open. Shared-host load differs materially, and a separate profiled replay reverses that result without establishing causality. Historical MinIO slowdowns are also not resolved. The HDFS figures below refer to 60351f1 on baseline d30058a, before this merge; they are not evidence for the merged head or object-storage performance.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

HDFS performance measurements

Setting	Value
Data	10,000 actual HMS partitions × 100 physical Parquet files = 1,000,000 files; 481 bytes and one row (`id = 1`) per file.
Storage	Dedicated HDFS 3.2.1 NameNode/DataNode, with the same data and server settings throughout.
FE comparison	Before: `d30058a80337e56f0f4930d7a75c0df50aa39ffc`; after: `60351f116ece1427a0a7e04a58869a52dc22a57e`.
Common BE	`4.1.3-rc02-7126cf65d96` for both versions.
Execution	One online cluster, one query and one producer at a time. Fresh FE/BE runtimes for each round; order: before → after → after → before.
Repetitions	Fixed 30 warmup cycles, then 20 measured queries per scenario per round: 40 measured samples per version/scenario.
Configuration	Native query/catalog cache and timeout defaults. Single-producer setting on both sides; `experimental_enable_local_shuffle_planner=false` and `be_exec_version=10` on both for BE 4.1.3 compatibility.
Cold definition	Fresh FE catalog; HDFS/OS caches were not forcibly flushed. Timing used no proxy or Java agent.

In the following tables, t is the million-file HDFS table and p is its integer partition key. The 10,000-file aggregation uses WHERE p < 100; it is not a separate 100-partition table.

These are observed shared-host measurements, not a guaranteed speedup. Baseline median latency for the 10,000-file aggregation fell 13.5% between its two rounds, and CPU affinity was not exclusive. All successful samples are retained; both medians and means are shown. Negative changes mean lower elapsed time.

Scenario / SQL	Samples per version	Before median (ms)	After median (ms)	Median change	Before mean (ms)	After mean (ms)	Mean change
Fresh-catalog `SELECT * FROM t LIMIT 10`	40	863.56	774.17	-10.35%	905.38	813.43	-10.16%
Warm-catalog `SELECT * FROM t LIMIT 10`	40	114.20	109.90	-3.77%	124.91	115.60	-7.45%
`SELECT COUNT(*), SUM(id) FROM t WHERE p < 100` (10,000 files)	40	1167.90	1052.15	-9.91%	1404.32	1145.69	-18.42%

Per-round results expose drift. Million-file SUMs are separate correctness/workload validations, one per runtime, rather than repeated steady-state performance samples.

Round	Version	Fresh-catalog LIMIT median (ms)	Warm-catalog LIMIT median (ms)	10,000-file aggregation median (ms)	Million-file SUM validation (s; result = 1,000,000)
1	Before	989.08	116.32	1273.19	282.615
2	After	753.10	99.75	1052.15	232.395
3	After	785.57	115.46	1061.70	232.503
4	Before	781.45	102.26	1101.28	238.465

Real OSS performance measurements (current merged head)

Setting	Value
FE comparison	Before: master `2a4191c1dc94e789e1725d6cb345f46c31a17272`; after: `00dfe97a935d12c268044171c98103be28372edd`. Both complete FE builds and independently verified frozen packages.
Common BE	`4.1.3-rc02-7126cf65d96`, identical binary on both sides.
Data/storage	Direct real OSS endpoint, 1,000,000 verified physical Parquet objects, 481 bytes and one row (`id=1`) each. Three HMS tables expose 100 / 1,000 / 10,000 partitions × 100 files. Smaller tables reuse the corresponding subset of the same object paths. Local Docker HMS.
Execution	One online FE + two BEs, one SQL query and one split producer at a time. Fresh runtime each round; after → before → before → after. Each process pinned to its own four CPUs; FE heap 4GiB, each BE memory 3GiB. CPU affinity is not exclusive on this shared host.
Repetitions	Fixed before formal measurement: 30 LIMIT warmup cycles plus 10 full-scan warmups per aggregation, then 20 measured queries per scenario per round. 800 warmups + 640 measured, all successful; 40 measured samples per version/scenario.
Split/cache configuration	Current defaults on both sides: batch 8, queue 1024 splits, initial batch 64, partition batch 16, batch thresholds 1, fetch wait 1000ms. One split/metadata worker; PR producer cap 1. Native BE file cache is disabled (default); FE session file-cache variable is enabled (default). SQL cache enabled, Hive SQL cache disabled.
Compatibility settings	`experimental_enable_local_shuffle_planner=false`, `be_exec_version=10` on both sides for the common BE 4.1.3.
Timing boundary	SQL client wall time includes query execution. Fresh-catalog cold cases recreate the FE catalog; HMS/OS caches are not flushed. Required native scheduler/reclaimer queues and BE readers drain before the next query, outside the SQL timer. No Java agent, JFR, profiler, replay or proxy in these timed runs.
Correctness	Full and filtered-full queries each returned COUNT/SUM 10,000 and each incremented the native S3 physical file-reader counter by exactly 10,000.
Qualification	All attempts retained. Shared host average load was approximately 132 for PR samples and 162 for baseline samples. Consequently this is an observed comparison, not proof of a causal improvement or absence of regression.

The 10,000-partition cold LIMIT slowdown appears in both adjacent formal round pairs (+5.45% and +3.04% median). Its FE CPU usage is nearly unchanged (237.5ms before / 238.0ms after, mean process CPU per query). A separate post-timing JFR replay (20 samples/version/scene) instead observed cold LIMIT 481.51ms before / 430.32ms after, with average host load about 398 / 131. That replay is instrumented, unbalanced, and excluded from the table below; it does not explain away the formal slowdown. Sampled common costs include partition-view construction, range construction, URI parsing, HMS reads and HTTP reads; the available evidence does not identify a causal extra 13ms in this PR. Further attribution requires controlled host contention. The 100-partition warm LIMIT median is also reported as slower, while its mean is lower; neither statistic is hidden.

Native FE audit logs reconcile all 1,440 query markers exactly once, including phase timings for all 640 formal queries. For the 10,000-partition cold LIMIT, mean client time increases 14.11ms; mean native planning time increases 12.53ms (216.60 → 229.12ms), including an 8.23ms increase in analysis (96.30 → 104.53ms). Mean finalize_scan_node time decreases 2.17ms (58.50 → 56.33ms). These nested stage times localize the observed difference mostly to planning; they must not be added together or treated as a causal explanation of the shared-host difference.

Negative changes mean lower elapsed time. t100, t1000, and t10000 denote the corresponding OSS table scales.

Partitions	Scenario / SQL	Samples per version	Before median (ms)	After median (ms)	Median change	Before mean (ms)	After mean (ms)	Mean change
100	Fresh-catalog `SELECT * FROM t100 LIMIT 10`	40	172.70	156.51	-9.37%	179.45	167.35	-6.74%
100	`SELECT COUNT(*), SUM(id) FROM t100` (10,000 files)	40	1537.61	1477.29	-3.92%	1553.59	1535.61	-1.16%
100	Warm-catalog `SELECT * FROM t100 LIMIT 10`	40	40.58	42.50	+4.71%	46.99	45.80	-2.54%
1000	Fresh-catalog `SELECT * FROM t1000 LIMIT 10`	40	266.82	266.11	-0.26%	268.92	273.72	+1.78%
1000	Warm-catalog `SELECT * FROM t1000 LIMIT 10`	40	125.00	122.20	-2.24%	125.96	124.64	-1.04%
10000	Fresh-catalog `SELECT * FROM t10000 LIMIT 10`	40	347.26	360.37	+3.77%	347.94	362.05	+4.05%
10000	`SELECT COUNT(*), SUM(id) FROM t10000 WHERE p < 100` (10,000 files)	40	1563.96	1541.16	-1.46%	1599.73	1569.72	-1.88%
10000	Warm-catalog `SELECT * FROM t10000 LIMIT 10`	40	133.35	123.73	-7.21%	132.67	130.09	-1.95%

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Large external-table scans could keep enumerating files after LIMIT, cancellation or timeout, including while planning waited for its first Split. Queued ranges and callbacks survived termination, and closing a source could race its reader. Own Split tasks from statement planning onward, fence late publication, close each source serially, and discard stopped queues. Retain statement metadata until all producers close and read transactions release. Isolate cancellable Hive listing loads while continuing to share completed cache entries. Bound first-Split waiting with a monotonic deadline that includes executor queueing and reports progress. ### Release note External streaming scans stop background Split work on LIMIT, cancellation and timeout. Add positive session variable first_split_wait_timeout_ms (default 30000), with SET_VAR and FE forwarding. Uninterruptible SDK calls may finish later. Connector SPI major 20 and filesystem SPI major 2 require separately packaged plugins to be rebuilt against the matching APIs. ### Check List (For Author) - Test: 468 targeted FE unit tests; 14 filesystem unit tests; two regression suites. - FE build and Checkstyle for all changed Java files passed. - Manual acceptance: real HMS with 10,000 partitions and an S3 fixture exposing 1,000,000 logical Parquet paths, using BE 4.1.3. Seven stop scenarios passed twice on the final FE build; all producers exited within 1.21 seconds, queues cleared, and no file-list requests arrived after 10 seconds. - First-Split deadlines of 1000/2500 ms errored in 1068/2572 ms in the final run, with progress details. SDK calls that cannot be interrupted remain excluded. - Behavior changed: Yes; stopped scans discard queued ranges and first-Split waiting is configurable. - Does this need documentation: Yes; updated fe/fe-connector/README.md in this commit.

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Early-stop acceptance depended on local, untracked fixtures and resource observers. A successful regression runner could also skip the optional case when launched with the wrong configuration. Add parameterized S3 fixture and physical-proxy modes, owned HMS metadata setup, a buildable test observer, and reports that require complete stop scenarios, fresh resource samples and request cessation. Replace raw SQL threads with framework futures. Add a shared file-cache regression that cancels one in-flight reader while another completes and then reuses the completed cache entry. ### Release note None ### Check List (For Author) - Test: 18 fixture/analyzer tests; probe compilation; Python and shell syntax checks. - Seven stop scenarios passed on two BE 4.1.3 nodes, with all producers exiting within 329 ms, empty queues/listeners/resources, and no LIST requests after 10 s. - Shared-cache cancellation regression passed generation and comparison runs; the survivor and cached follow-up both returned count=100,sum=100. - Physical S3 proxy verified 10,000 existing objects with pagination, HEAD and Range GET. The requested physical million-file benchmark still needs its FE/HMS connection; it is not claimed as validated. - Newer-FE/BE-4.1.3 test settings: be_exec_version=10 and local shuffle disabled. - Behavior changed: No; production FE/BE logic is unchanged. - Does this need documentation: Yes; added acceptance tool README in this commit.

… data ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Existing reproducible acceptance covered one million logical file paths but did not create the physical dataset. Add a resumable uploader and full key/count/length/MD5 verification for 10000 partitions with 100 real Parquet objects each. Require matching verified manifests during setup and reporting. Delay real upstream LIST responses to test first-Split cancellation and deadlines, and reject synthetic listings in physical acceptance reports. Document physical setup, HMS storage access and the test controls. ### Release note None ### Check List (For Author) - Test: 27 fixture/analyzer/physical-validation unit tests passed; Python and shell syntax and whitespace checks passed. Real MinIO process-pool create, resume and verify-only smoke checks passed. - Fully verified 1000000 physical Parquet objects across 10000 HMS partitions. - Seven early-stop scenarios passed on two BE 4.1.3 nodes. All producers exited within 5511 ms; queues/listeners/resources cleared and no LISTs arrived after 10 seconds. First-Split 1000/2500 ms deadlines errored in 1146/2930 ms with progress details. - Physical shared-cache cancellation passed: the cancelled reader exited in 135 ms, the survivor completed, and the follow-up reused the listing. Both complete queries returned count=100,sum=100; the callback registry was empty after both suites. - This is a self-created physical dataset of the requested shape; the original benchmark table has not been connected. Production code is unchanged. - Behavior changed: No; optional acceptance tools and controls only. - Does this need documentation: Yes; updated the acceptance README in this commit.

…ests ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Early-stop acceptance established resource-disposal deadlines but did not measure benefits against the implementation's parent revision. Add an optional physical-file benchmark for LIMIT, execution cancellation, timeout, complete scans and cache hits, plus focused pre-read cancellation and consecutive warm LIMIT checks. Sample process CPU/RSS externally, reject Java agents and synthetic listings, validate request/sample coverage, and publish medians and ranges bound to the verified physical dataset. Keep query latency separate from cancellation-response latency and post-return background work. ### Release note None ### Check List (For Author) - Test: 34 Python tool tests; Python syntax and changed-source whitespace checks. - Baseline FE built with build.sh; no production Java source changes. Baseline Checkstyle was skipped. Complete before/after plugin packages were used. - Nine scenarios, three before/after repetitions: 54 primary observations against a verified 10000-partition, 1000000-physical-file dataset and two BE 4.1.3 nodes, without artificial LIST delay or Java agents. An additional 21-case changed-version pass assessed drift. - Expected SQL results were generated with the standard runner's -forceGenOut and compared by changed-version and focused benchmark passes. - Pre-read cancellation response median: 7000 to 6 ms; LIST 129 to 32. Configured 3-second timeout: SQL elapsed median 7307 to 3184 ms. - Complete 10000-file scans differed by approximately 3-4 percent. No stable LIMIT/throughput speedup is claimed. Main warm-LIMIT median regression was retained; supplementary prewarmed repeats did not reproduce a stable regression (7837 versus 7549 ms). Shared-host variability is documented. - Behavior changed: No; optional benchmark tooling only. - Does this need documentation: Yes; updated the acceptance README in this commit.

…ion files ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: The comparative Split benchmark completed only 10000-file scans. Add an optional unfiltered million-file SUM and require its observation in both comparison passes. Index and stream request logs so millions of reads do not dominate analyzer memory. Reject unrelated early timeouts and changed-version early-stop observations with late or still-active file requests. On the physical 10000-partition by 100-file fixture with BE 4.1.3, three-repeat pre-read cancellation median total SQL time decreased from 6302 ms to 189 ms, cancellation response from 6181 ms to 7 ms, and LIST requests from 129 to 58. Three-second query timeout median decreased from 6884 ms to 3511 ms. No stable general LIMIT or full-scan throughput improvement is claimed: single complete million-file scans took 1028.394 s and 1154.918 s while host load spiked during the latter. The original supplementary LIMIT trial was rejected because proxy requests remained in flight during a 10.164 s sampler stall. No requests began after SQL returned; all eventually drained. The diagnostic is retained separately from the passing retry. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 37 Python tool tests passed; seven standard regression suite runs completed; 56 selected comparison observations and 59 matching FE audit records verified. Both complete scans returned count=1000000,sum=1000000 without SQL result-cache hits. One original supplementary observation was rejected and retained; retry passed. - Behavior changed: No (benchmark tools and tests only) - Does this need documentation: Yes (updated the benchmark README)

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Benchmarking smaller initial Split batches exposed complete-scan failures at the existing 1000 ms subsequent-fetch wait. Preserve the current production batching policy and cover configured initial sizes, complete enumeration across partition boundaries, large partition groups, and whole-scan source closure. Correct the synchronous benchmark control to explicitly disable external batch mode, since increasing the partition threshold alone can still select file-count streaming. Allow a separate synchronous-control table over the same physical files and document its use. ### Release note None ### Check List (For Author) - Test: Targeted FE unit tests and Checkstyle; manual prototype evaluation with BE 4.1.3. - Unit Test: 55 tests across the partition-batch, Hive streaming, producer, source, assignment and concurrency suites using run-fe-ut.sh. - Manual test: 72 unique main observations and 24 default-wait controls across two rejected prototypes; existing production code restored after default-fetch failures. - Style: Maven Checkstyle reports 0 violations. - Behavior changed: No production behavior change. The optional benchmark now explicitly disables batch mode for its synchronous control and restores the session setting afterward. - Does this need documentation: No user-facing feature. Benchmark setup is documented in the in-tree README.

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Partition-batch scans derive their producer count from the shared external metadata pool. On an overloaded object store, speculative LIST requests can delay the first batch and cause gaps in subsequent Split supply. Add an opt-in session limit per scan node without shrinking global executors or changing batch sizes, queue bounds, cancellation, or wait deadlines. Zero keeps the existing behavior; positive values also respect executor and partition-batch bounds. Whole-scan streaming retains one producer. On the local million-file fixture with BE 4.1.3 and the global pool fixed at 64, three default-batch LIMIT samples per setting show a median reduction from 1443 ms to 654 ms with a producer limit of four, and median LIST calls from 26 to 12. In the partition-batch-size-one ABBA control, six LIMIT samples per setting improve from 6564 ms to 560 ms. Complete 10000-file scans succeed in 2/6 uncapped and 6/6 capped samples, but the uncapped successful subset is faster; this is not a general full-scan throughput improvement. Keep the default at zero and expose the setting in the optional regression benchmark for workload-specific tuning. ### Release note Add the FE session variable split_producer_parallelism to cap partition-batch Split producers per plugin scan node. The default zero preserves existing parallelism. Negative values are rejected. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 58 focused FE unit tests pass with zero failures, errors or skips - Standard seven-scenario Hive performance regression passes with BE 4.1.3 - 32 uninstrumented SQL controls match unique FE audit queries; complete successful scans read 10000 rows with exact COUNT and SUM results - FE build through build.sh --fe and Checkstyle pass - Behavior changed: Yes, an explicit positive session limit bounds producers; default behavior, batch sizes and deadlines remain unchanged - Does this need documentation: Yes, setting and benchmark usage documented in regression-test/script/hive_split_early_stop/README.md

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: The early-stop resource suite waited for ten LIST calls before cancelling a first-batch query. A scan capped at four producers can reach that threshold only after the first batch is available, so the test could exercise the execution phase instead. Add an optional producer-limit control to the main and shared-cache suites, bound the initial LIST trigger by that limit, and reject first-Split cancellation or timeout observations containing earlier BE reads. Add negative and positive analyzer coverage and document the acceptance option. The exact final FE package from 0af95bc passes all seven resource scenarios at both limits zero and four on one million physical Hive/S3 files with two BE 4.1.3 nodes. The slowest observed background termination is 5762 ms and 5263 ms respectively, with empty queues/resources, completed termination, no LISTs after ten seconds, and empty query-finish callback registries. Both shared-cache cancellation/isolation checks also pass. Production code is unchanged in this follow-up. ### Release note None ### Check List (For Author) - Test: Regression test / Unit Test - 38 acceptance-tool unit tests pass, including wrong-phase rejection - Standard main and cache regression suites pass at producer limits 0 and 4 - Resource probe confirms all counters clear and termination completes in under ten seconds for all 14 stop observations - First-Split deadlines 1000/2500 ms and progress messages verified twice - Prior physical acceptance still passes the stricter phase validator - Python compilation and git diff whitespace checks pass - Existing 58 FE unit tests cover the unchanged production package; not rerun - Behavior changed: No, production behavior is unchanged - Does this need documentation: Yes, acceptance configuration is documented in regression-test/script/hive_split_early_stop/README.md

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: A cloud replan can reuse the connector scope closed by the failed query attempt and fail its next scan registration. Hive split enumeration also checks cancellation only before a logical partition lookup, allowing subsequent physical HMS chunks or size fallback requests to continue after cancellation. Reset the connector scope at the outer replan boundary while retaining old producer resources until termination. Propagate the scan context to physical HMS dispatch, including after client acquisition, and isolate cancellable metadata loads while sharing completed cache values. ### Release note Fix cloud query replanning with external scans and stop subsequent Hive partition metadata requests after scan cancellation or timeout. ### Check List (For Author) - Test: Unit Test - run-fe-ut.sh: 190 focused passing executions; final lifecycle/cancellation suite: 13 passing executions - Negative control: all 13 new cases fail when the relevant fixes are removed; fixes restored before the final passing run - Checkstyle checked through the standard build.sh --fe entry point - Behavior changed: Yes (isolate connector scopes between outer retries and stop further scan-owned HMS requests) - Does this need documentation: No (existing cancellation contract; internal overload documented)

### What problem does this PR solve? Issue Number: None Problem Summary: With the Iceberg manifest cache enabled, a streaming scan whose files are all rejected by metrics or rewrite filters can advance through many manifests inside one hasNext call. Cancelling the scan therefore leaves its producer reading more metadata and delays source cleanup. Eager delete-manifest loading can also continue or fall back to SDK planning after cancellation. Check the scan token at file and manifest boundaries and before SDK fallback, while allowing an already-entered shared manifest load to complete without poisoning other queries. Update lifecycle comments to describe producer joins, asynchronous cleanup, and deferred disposal of retained statement scopes. ### Release note Iceberg streaming split enumeration observes cancellation between files and manifests, including delete-manifest preparation, and avoids starting SDK fallback after cancellation. ### Check List (For Author) - Test: Unit Test / Manual test - 319 targeted FE unit tests passed through run-fe-ut.sh, including five new cancellation cases. - All five new cases failed for the expected missing-cancellation behavior against the previous implementation. - build.sh --fe and integrated Checkstyle passed; packaged provider and iterator classes match the compiled classes. - Behavior changed: Yes (cancelled Iceberg streaming scans stop at scan-owned enumeration boundaries) - Does this need documentation: No (existing cancellation contract; lifecycle comments updated)

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: A streaming Iceberg scan can remain inside SDK manifest iteration after query cancellation. Prefetched data-manifest jobs and eager delete-index jobs can continue issuing file requests, and cancelling their SDK futures does not prove the suppliers have exited. Add a source-owned executor view and cancellable FileIO over native local scans. Settle queued SDK commands on independent cleanup workers, retain the original shared worker pool and scan context, and join actual job exits before closing SDK iterators and releasing statement resources. Preserve custom scan semantics and the SDK parallel-planning setting. ### Release note Streaming Iceberg split planning stops new manifest reads after LIMIT completion or cancellation. The connector plugin API is now 21.0. Native streaming SDK planning uses a request-local FileIO without the SDK content cache; the catalog FileIO and Doris manifest cache remain unchanged. ### Check List (For Author) - Test: Unit Test: 495 related tests via run-fe-ut.sh, plus 2 tests rerun after relocating their package; verified the old-provider cancellation failure; build.sh --fe and Checkstyle passed; packaged production classes match the tested classes. - Behavior changed: Yes (stop owned Iceberg SDK work and wait for actual exit; bump connector plugin API) - Does this need documentation: No (internal SDK task ownership; cancellation notification contract documented in SPI)

hello-stephen · 2026-10-29T09:19:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69835 Problem Summary: Merge fixed master 2a4191c and adapt split cancellation to its query-owned fair scheduler, per-split bounded queues, asynchronous metadata cache, and Iceberg planning lifecycle. Remove queued admissions and capacity callbacks without waiting for query-group closure, preserve source closure on the owning worker, and release statement resources only after actual producer termination. Retain upstream split defaults, retryable fetch timeouts, and planning behavior. Preserve original failure causes and finish producer cleanup even when source work, source closure, or completion throws an Error. Update connector/filesystem API contracts and the functional observer, and remove the requested performance drivers from the PR. ### Release note Split generation stops after LIMIT completion, query cancellation, or timeout. First-split waiting is configurable and timeout errors include progress. ### Check List (For Author) - Test: Unit Test: 61 selected FE classes, 761 tests, zero failures/errors/skips. The standard run-fe-ut.sh runner was subsequently killed (exit 137) while shading the unrelated BE Java paimon-s3-shade module; the complete runner did not pass. Functional observer compiled with JDK17 and its 16 report tests passed. A negative-control source Error test reproduced the termination leak before the fix. Final build.sh --fe packaging is pending. - Behavior changed: Yes; preserve early-stop behavior with the current master scheduler and retain upstream defaults and fetch retry semantics. - Does this need documentation: No; existing PR documentation updated.

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69835 Problem Summary: A split producer could wait up to 60 seconds for a local HMS connection after cancellation. Whole-scan streaming also listed unpruned partition names without its scan context and could join another query's cold cache load or dispatch a names RPC after cancellation. Poll only scan-owned pool borrows at 100ms intervals while preserving the original total wait budget and non-scan behavior. Carry the scan context through partition-name cache loads and authenticated dispatch, isolate in-flight loads by owner, share completed values, and preserve cancellation without tainting healthy connections. ### Release note Cancelled Hive split generation exits local HMS pool waits promptly and stops partition-name requests before dispatch. ### Check List (For Author) - Test: Unit Test: 19 relevant HMS/Hive classes, 246 tests, zero failures/errors/skips, run-fe-ut.sh exit0 with a per-invocation CUSTOM_MVN wrapper selecting only the HMS/Hive dependency closure. The negative control had 12 tests with the 2 newly added pool-wait cancellation cases failing before the fix. Full build.sh --fe packaging is pending. - Behavior changed: Yes; scan-owned HMS waits and whole-stream partition-name enumeration observe cancellation without affecting healthy queries. - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69835 Problem Summary: Local split acceptance fixtures, data generators, observers, reports and their dedicated suites unnecessarily enlarge the split early-stop change. Remove these local-only tools and the two suites that require their custom fixture. Retain FE unit coverage and the deterministic session-variable regression, which checks configuration semantics without timing external I/O. Production code is unchanged from the reviewed and benchmarked revision. ### Release note None ### Check List (For Author) - Test: Manual test - Verified no remaining references to the removed fixture, no performance scripts in the PR, identical production sources, and git diff --check. - No rebuild or runtime tests: only local test tools and dependent suites are removed; all FE implementation and unit tests are unchanged. - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69835 Problem Summary: Split cancellation tests included a whole class repeating JDK CompletableFuture guarantees, a retry test that only constructed two objects, coverage already present in stronger cleanup tests, and constant fake-executor assertions. Remove these cases, combine equivalent batching tests while checking all split identities, and retain the actual query-retry, cancellation, queue and resource-lifetime failure paths. Drop a host-sensitive timeout upper bound while retaining the configured deadline lower bound and progress/termination checks. Production code is unchanged. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - Checkstyle passed through the standard FE test entry point with style explicitly enabled; independent read-only test quality review completed. - Focused run-fe-ut.sh run is compiling/running at commit time; final execution results will be recorded separately in the PR description. - Verified unchanged production/POM content, retained behavioral oracles, no removed-tool references, and git diff --check. - Behavior changed: No - Does this need documentation: No

CalvinKirs · 2026-10-30T01:36:24Z

run buildall

CalvinKirs · 2026-10-30T01:40:41Z

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69835
commit: c3b37ae5084ebcbeb16a9387551f7612eeb9ffa6
base: 2a4191c1dc94e789e1725d6cb345f46c31a17272
reviewed_at: 2026-10-30T09:40+08:00
reviewer: CalvinKirs
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

Scope: 64 paths and all 17 checkpoints covered, with one converged round and no findings. This reuses the completed full review at 00dfe97 and scope review at 0e88246: 60 unchanged paths were verified by exact content, four retained test changes were freshly reviewed, and two paths leaving the PR were checked independently.
Validation: the author's focused run-fe-ut.sh completed with 47 tests across four classes, zero failures/errors/skips, and Checkstyle enabled and passed. Reviewers verified the archived XML evidence; they did not run builds, tests, or a cluster. The five Iceberg cases affected only by removing two constant assertions were compiled/style checked but were not rerun in that focused selection.
Known performance limitation: on real OSS, the 10,000-partition fresh-catalog SELECT * ... LIMIT 10 remains 3.77% slower by median and 4.05% by mean in the disclosed comparison. Attribution is unresolved; this source-review PASS does not certify absence of a performance regression or a newly benchmarked package.

_{Reviewed locally with the doris-repo-review pipeline (a local port of .github/workflows/code-review-runner.yml). This is not a CI status check.}

924060929

External scans enumerate files in the FE and stream splits to BEs. This PR ties enumeration to query/attempt termination, delays resource disposal until producers actually exit, and adds first-split timeout and producer-parallelism controls.

APPROVE for the completed FE static review of all 64 changed paths at c3b37ae. No actionable P0/P1/P2 defects were found. Reviewed cancellation and retry flows, scheduling races, resource ownership, cache isolation, connector/filesystem boundaries, API versions, and test changes. No builds or tests were run.

The supplied context contains a current-head local PASS receipt but no formal approval or inline comments. Compilation, FE UT, and performance checks remain pending. The disclosed OSS latency difference remains unattributed; this review does not certify performance or merge readiness.

Automated FE-focused static review (GPT-6.1 Sol, medium). No build or tests were run.

hello-stephen · 2026-10-30T02:22:03Z

TPC-H: Total hot run time: 28874 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3b37ae5084ebcbeb16a9387551f7612eeb9ffa6, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17658	5241	5149	5149
q2	2143	313	263	263
q3	10920	1377	857	857
q4	4680	460	332	332
q5	7427	791	521	521
q6	186	187	153	153
q7	753	782	593	593
q8	9249	1330	1079	1079
q9	5489	4195	4206	4195
q10	6825	1292	1015	1015
q11	430	240	228	228
q12	637	393	289	289
q13	18083	3556	3028	3028
q14	274	267	244	244
q15	q16	726	732	661	661
q17	1198	748	566	566
q18	7021	6164	6091	6091
q19	1093	949	642	642
q20	383	336	225	225
q21	4882	2550	2456	2456
q22	355	321	287	287
Total cold run time: 100412 ms
Total hot run time: 28874 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5939	5914	5896	5896
q2	733	610	562	562
q3	5605	5712	5415	5415
q4	2020	2115	1594	1594
q5	5754	5857	5651	5651
q6	238	189	142	142
q7	2294	1989	1744	1744
q8	2861	2509	2526	2509
q9	7517	7477	7482	7477
q10	3617	3565	3148	3148
q11	591	421	382	382
q12	661	671	509	509
q13	3303	3585	3033	3033
q14	288	283	263	263
q15	q16	662	666	603	603
q17	7694	7023	6896	6896
q18	13055	12185	13014	12185
q19	934	802	796	796
q20	2142	2142	1901	1901
q21	5654	4739	4792	4739
q22	512	465	428	428
Total cold run time: 72074 ms
Total hot run time: 65873 ms

hello-stephen · 2026-10-30T02:31:53Z

TPC-DS: Total hot run time: 151936 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3b37ae5084ebcbeb16a9387551f7612eeb9ffa6, data reload: false

query5	4322	590	454	454
query6	425	220	191	191
query7	4802	443	235	235
query8	327	189	171	171
query9	8792	3264	3227	3227
query10	482	313	266	266
query11	6039	3849	3588	3588
query12	148	99	89	89
query13	1235	458	346	346
query14	6137	3944	3930	3930
query14_1	3451	3396	3454	3396
query15	212	200	187	187
query16	944	434	406	406
query17	846	647	549	549
query18	2430	424	314	314
query19	195	165	143	143
query20	88	85	82	82
query21	212	131	109	109
query22	12895	12952	12883	12883
query23	13355	12781	12468	12468
query23_1	12609	12597	12531	12531
query24	7370	824	494	494
query24_1	486	489	493	489
query25	535	420	359	359
query26	1264	276	147	147
query27	2763	448	278	278
query28	4613	2061	2066	2061
query29	1566	615	469	469
query30	300	220	185	185
query31	1022	964	828	828
query32	143	101	109	101
query33	544	317	254	254
query34	902	856	513	513
query35	745	753	649	649
query36	801	802	713	713
query37	135	103	92	92
query38	1801	1731	1680	1680
query39	791	797	743	743
query39_1	741	742	745	742
query40	234	124	99	99
query41	73	69	68	68
query42	93	87	85	85
query43	361	370	323	323
query44	1408	738	731	731
query45	196	194	174	174
query46	846	991	621	621
query47	3076	3054	2807	2807
query48	319	324	223	223
query49	591	412	298	298
query50	701	277	213	213
query51	10386	10502	10658	10502
query52	78	81	67	67
query53	183	206	149	149
query54	251	217	200	200
query55	73	66	66	66
query56	236	214	201	201
query57	1671	1640	1557	1557
query58	287	267	253	253
query59	2162	2142	1942	1942
query60	296	240	229	229
query61	147	146	146	146
query62	401	333	306	306
query63	194	155	162	155
query64	2733	963	767	767
query65	3321	3289	3273	3273
query66	1807	429	303	303
query67	20030	20161	19820	19820
query68	3042	1009	642	642
query69	385	293	253	253
query70	858	753	771	753
query71	286	248	216	216
query72	2749	2770	2270	2270
query73	523	534	298	298
query74	4664	4481	4321	4321
query75	2348	2270	1902	1902
query76	2309	1024	595	595
query77	340	372	275	275
query78	9720	9760	9179	9179
query79	1007	916	551	551
query80	800	449	335	335
query81	593	319	267	267
query82	582	134	106	106
query83	288	195	180	180
query84	320	125	107	107
query85	981	442	368	368
query86	351	238	231	231
query87	2008	1944	1804	1804
query88	3826	2826	2796	2796
query89	328	283	241	241
query90	1702	169	167	167
query91	163	156	125	125
query92	98	91	92	91
query93	1037	1007	586	586
query94	491	299	288	288
query95	584	338	305	305
query96	692	495	242	242
query97	2477	2416	2333	2333
query98	165	151	147	147
query99	774	753	637	637
Total cold run time: 234887 ms
Total hot run time: 151936 ms

hello-stephen · 2026-10-30T02:36:44Z

ClickBench: Total hot run time: 24.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3b37ae5084ebcbeb16a9387551f7612eeb9ffa6, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.10	0.10
query5	0.23	0.23	0.24
query6	1.16	0.68	0.68
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.40	0.35	0.36
query10	0.57	0.59	0.60
query11	0.21	0.15	0.13
query12	0.19	0.16	0.17
query13	0.50	0.49	0.49
query14	1.04	1.01	0.99
query15	0.64	0.61	0.61
query16	0.36	0.35	0.35
query17	1.18	1.15	1.20
query18	0.21	0.19	0.20
query19	2.14	2.06	1.93
query20	0.02	0.01	0.02
query21	15.49	0.18	0.13
query22	4.97	0.06	0.05
query23	16.30	0.33	0.12
query24	3.18	0.51	0.32
query25	0.11	0.09	0.05
query26	1.03	0.22	0.17
query27	0.04	0.03	0.03
query28	3.10	0.82	0.37
query29	12.49	4.02	3.23
query30	0.28	0.14	0.10
query31	2.77	0.57	0.33
query32	3.23	0.59	0.51
query33	3.14	3.28	3.26
query34	15.30	3.98	3.37
query35	3.34	3.34	3.34
query36	0.57	0.45	0.42
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.78 s
Total hot run time: 24.31 s

hello-stephen · 2026-10-30T03:26:34Z

FE UT Coverage Report

Increment line coverage 77.52% (300/387) 🎉
Increment coverage report
Complete coverage report

…hema (#69488) ### What problem does this PR solve? Issue Number: None Related PR: #66538, #57531, #65851 Problem Summary: **In short.** On an Iceberg table, `DEFAULT(column)` anywhere except `UPDATE ... SET` and `MERGE ... VALUES` reads the default from the read schema, which carries no Iceberg write-default. This PR resolves it against the write schema that the statement pinned for its target, as branch-4.1 and branch-4.2 do. **Background.** An Iceberg `write-default` applies to rows written after it is set. So it lives in the writer schema that a write statement pins (`IcebergWriteSchemaContext`), not in the columns of the read schema. #65851 resolves `DEFAULT(column)` against that pinned schema in `UPDATE ... SET`, `MERGE ... UPDATE SET` and `MERGE ... INSERT VALUES` (`ConnectorWriteSchemaUtils`). Every other `DEFAULT(column)` either fails at binding or reaches `RewriteDefaultExpression` (#57531), which reads the default from the bound column. branch-4.1 #66538, also in branch-4.2, resolves these against the pinned write target too. It rejects `DEFAULT(column)` on an Iceberg column in a statement that does not write the table. **The problem, and what it cost.** Reproduced on a local cluster with a `file://` format-version 3 table: `CREATE TABLE t (id INT, c INT)`, `ALTER TABLE t ADD COLUMN d INT DEFAULT 42`, one row `(0, 0)`, so `d = 42`. | Statement | Master today | After this PR (same as branch-4.2) | |---|---|---| | `INSERT INTO t (id, c, d) VALUES (1, 1, DEFAULT(d))` | `Unknown column 'd' in 'table list'` | writes 42 | | `INSERT INTO t (id, c, d) SELECT 2, 2, DEFAULT(d) FROM t WHERE id = 0` | writes NULL | writes 42 | | `INSERT INTO t (id, c, d) SELECT 3, 3, DEFAULT(d)` | `Unknown column 'd'` | writes 42 | | `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates nothing: `d = NULL` makes the plan an empty set | updates every row, `d = 42` is pushed down to the scan | | `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d = DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | matches nothing | updates row 0 | | `SELECT DEFAULT(d) FROM t` | NULL | `DEFAULT(d) must reference a column of the table that this statement writes` | The `UPDATE` and `MERGE` cases fail silently: the statement succeeds and changes nothing. **How this PR fixes it.** - fe-core cannot tell from a column whether its connector keeps defaults in the read schema. Hive does: an HMS default shows up in both schemas. Iceberg does not. The new connector capability `SUPPORTS_WRITE_ONLY_DEFAULTS` says which kind a table is. The Iceberg connector declares it. The Hive connector passes it on to iceberg-on-HMS tables, like the other per-table capabilities, but only through the two-argument `getTableSchema`. Since #69299, a statement that pins the table at its latest snapshot reads the schema from the snapshot overload, which reflects none of them, so on an iceberg-on-HMS table such a statement keeps master's results (for example, `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` still changes no row). Reflecting the sibling's capabilities there changes planning for all of them and is left to a follow-up PR. - For a target that declares it, `InsertUtils.normalizePlan` resolves `DEFAULT(column)` in `INSERT ... VALUES` and in the select lists of `INSERT ... SELECT`, including the replacements of `SELECT * REPLACE(...)`, against the pinned write schema. It does this by name, before the source relations are bound, so the name always refers to a target column, as in branch-4.2. In MySQL, `DEFAULT(column)` in a `SELECT` refers to a source column instead. A derived table in `FROM` is a query of its own and is left alone: `DEFAULT(column)` in it still names a column of its own `FROM` clause. - `RewriteDefaultExpression` takes the default for a `DEFAULT(column)` bound to such a table from the pinned write column with the same field id. That covers the `WHERE` of `UPDATE` and the `ON` of `MERGE`. A MERGE resolves it whatever its WHEN branches, also when they all delete, as branch-4.2 does. If the statement pinned no write schema for that table, it fails. A `DELETE` pins the writer schema too, for its delete sink, but it writes no row values, so `RowLevelDmlCommand` marks its target and `DEFAULT(column)` on it fails with `DEFAULT(d) cannot be used in a DELETE: the defaults of this table apply only to the rows that a statement writes`, as in branch-4.2, whose DELETE installs no write schema context. Master reads NULL there, so `DELETE FROM t WHERE d = DEFAULT(d)` deletes nothing; resolved to the write-default it would delete every row with `d = 42`. - A resolved default is cast to the type of its writer column before the expression around it is bound, as `RewriteDefaultExpression` already does for a bound `DEFAULT(column)`. The parsed literal `-1` is a TINYINT, so without the cast `bit_count(DEFAULT(c))` on an INT column would count 8 bits instead of 32. The external `UPDATE`, `MERGE` and changelog paths use the same helper. `RewriteDefaultExpression` rejects a bound `DEFAULT(column)` whose writer column has another type than the read column the statement was bound with, which happens only when the table changed outside Doris after Doris cached it, and asks to refresh the table: the expressions around it were bound against the read type. - Hive, Paimon and internal tables do not declare the capability, so they behave as before. - The new enum constant changes the connector plugin API surface, so `connector.plugin.api.version` goes from 26.0 to 27.0, as for the other recent changes to that surface (#69299, #69588, #69298, #69683 and #69835 took 22.0 to 26.0). **Where `DEFAULT(column)` resolves on an Iceberg target (direct Iceberg catalogs).** | Statement | Position of `DEFAULT(column)` | Result | branch-4.2 | |---|---|---|---| | `INSERT` / `INSERT OVERWRITE` | `VALUES` rows | the target column's write-default, typed as the column | same | | | the INSERT's own select lists (with or without `FROM`, set operation branches, `SELECT * REPLACE`) | the target column's write-default, by name, before binding | same, except `* REPLACE`, which branch-4.2 binds to the source | | | `GROUP BY` / `ROLLUP` outputs, a derived table in `FROM`, `WITH` definitions, subqueries, `WHERE` / `ON` | bound like any `DEFAULT(column)` (last row) | same, except derived tables, which branch-4.2 resolves against the target | | | a `VALUES` branch of a set operation | fails with an unknown column, as on master | same | | | a qualifier other than the target (`DEFAULT(src.c)`, `DEFAULT(ctl.db.t.c)`) in the positions above | fails: `column in assignment list is invalid` | same | | `UPDATE` | `SET` / `WHERE` | write-default | same | | `MERGE` | `ON` and `WHEN ... AND` conditions, whatever the WHEN branches, also when they all delete | write-default | same | | | `UPDATE SET` / `INSERT VALUES` | write-default | same | | `DELETE` (also `DELETE ... USING`) | anywhere | fails: `cannot be used in a DELETE` | fails: its DELETE has no write schema context | | any | a bound `DEFAULT(x.c)` | `x` is the written table: its write-default by field id; an OLAP or Hive table: that column's default; another Iceberg table: fails with `must reference a column of the table that this statement writes` | same | A bound `DEFAULT(column)` whose writer column has another type than the read column fails and asks to refresh the table. Where `DEFAULT(column)` is bound rather than resolved by name (the third row), it keeps its meaning from before this PR and names a column of its own `FROM` clause, as on master and branch-4.2: for example in `INSERT INTO t SELECT id, DEFAULT(d) FROM src GROUP BY id, d` with an OLAP or Hive `src`, it takes the default of `src.d`. **Results.** - Local cluster, with the table above: the table shows what each statement did before and after. `EXPLAIN UPDATE t SET c = 5 WHERE d = DEFAULT(d)` goes from an empty set to a scan with `d = 42` pushed down to Iceberg. - Unit tests: 6 new tests in `InsertUtilsTest`, 6 in the new `RewriteDefaultExpressionTest` and 1 in `RowLevelDmlCommandTest`. They cover VALUES, and INSERT ... SELECT both from a staging relation whose own column is also `name` and without FROM: each resolves to the pinned default, and an unknown column fails. `DEFAULT(column)` inside a derived table, qualified or not and also under a UNION branch, is left to binding. A `SELECT * REPLACE(DEFAULT(name) AS name)` replacement resolves like the rest of the select list and keeps the star's qualifier and `EXCEPT` list. The pinned write column wins over the read column. A statement that writes nothing fails. A table without the capability is unchanged. With master's `InsertUtils` and `RewriteDefaultExpression`, the 5 tests of the fix fail and the 2 tests of the unchanged path pass. The `SELECT * REPLACE` test fails with this PR's previous `InsertUtils`, where the replacement stays `DEFAULT(name)`. `IcebergConnectorTest`, `HiveConnectorMetadataSiblingDelegationTest` and `ConnectorPluginSurfaceTest` pass. So do the fe-core tests around the write schema: `ConnectorWriteSchemaUtilsTest`, `ExternalRowLevelUpdatePlanBuilderTest`, `ExternalRowLevelMergePlanBuilderTest`, `ChangelogRowLevelDmlTransformTest` and `PluginDrivenExternalTableTest`. Two of the tests pin the typing: `bit_count(DEFAULT(bits))` gets the default as an INT in VALUES and in SELECT, and fails without the cast; a BIGINT writer column over an INT read column is rejected, directly and under an IF bound to INT. `DEFAULT(column)` on the target of a DELETE fails even with a pinned write-default, and only a DELETE marks its target (an UPDATE does not); skipping the check, or marking UPDATE instead of DELETE, makes exactly these two tests fail. `PositionDeleteRowLevelDmlTransformTest`, `PhysicalPlanTranslatorIcebergRowLevelDmlTest`, `DeleteFromCommandTest` and `UpdateCommandTest` pass. - Regression: `test_iceberg_initial_defaults` now writes rows 20 to 22 with `DEFAULT(default_int)` in VALUES and in INSERT ... SELECT with and without FROM. It then updates them with `WHERE default_int = DEFAULT(default_int)` and `MERGE ... ON ... = DEFAULT(...)`, for Parquet and ORC, and checks the rows in the `.out`. A plain `SELECT DEFAULT(default_int)` must fail, and so must a `DELETE ... WHERE default_int = DEFAULT(default_int)` on rows 20 to 22, which hold the write-default. The suite passed locally before the DELETE block was added, and the whole suite, DELETE block included, has since passed in two External regression runs on CI. With master's FE it fails at the first of these statements: `Unknown column 'default_int' in 'table list'`.

### What problem does this PR solve? Issue Number: N/A Related PR: apache#69835 Problem Summary: On branch-4.1, cancellable Hive global file-cache misses must retain the catalog-specific filesystem lister, and TABLESAMPLE must pass cancellation through selection and per-file splitting. Iceberg weighted caches return a query-scoped table that must keep snapshot isolation when installing guarded FileIO and tracking manifest planning tasks. Preserve producer failures during endpoint cleanup and close registered scan tasks when internal-query planning fails before a coordinator exists. ### Release note Complete split early-stop behavior for sampled Hive scans, weighted Iceberg metadata caches, and internal-query planning failures. ### Check List (For Author) - Test: Unit Test - 476 FE tests passed across 20 relevant classes using run-fe-ut.sh; Checkstyle passed with 0 violations - Behavior changed: Yes (stop omitted scan paths and preserve native cache isolation and error details) - Does this need documentation: No (covered by the original early-stop change)

…erminate (#70004) #69835

CalvinKirs added 11 commits October 22, 2026 18:13

CalvinKirs added 4 commits October 29, 2026 19:05

CalvinKirs marked this pull request as ready for review October 30, 2026 01:36

CalvinKirs requested review from 924060929, gavinchou and morningman as code owners October 30, 2026 01:36

CalvinKirs requested review from englefly, morrySnow and starocean999 as code owners October 30, 2026 01:36

924060929 approved these changes Oct 30, 2026

View reviewed changes

924060929 merged commit ea3558d into apache:master Oct 30, 2026
61 of 64 checks passed

morningman mentioned this pull request Oct 31, 2026

[fix](fe) Resolve DEFAULT(column) on Iceberg tables from the write schema #69488

Merged

5 of 16 tasks

CalvinKirs added dev/4.1.x dev/4.2.x labels Nov 2, 2026

github-actions Bot added the dev/4.2.x-conflict label Nov 2, 2026

CalvinKirs added the usercase Important user case type label label Nov 2, 2026

github-actions Bot added the dev/4.1.x-conflict label Nov 2, 2026

CalvinKirs mentioned this pull request Nov 3, 2026

branch-4.1: [fix](fe) Stop split enumeration when queries finish or terminate #70004

Merged

yiguolei pushed a commit that referenced this pull request Nov 4, 2026

branch-4.1: [fix](fe) Stop split enumeration when queries finish or t…

a58b5dd

…erminate (#70004) #69835

yiguolei added dev/4.1.6-merged and removed dev/4.1.x dev/4.1.x-conflict labels Nov 4, 2026

Navigation Menu

Repository navigation

[fix](fe) Stop split enumeration when queries finish or terminate - #69835

[fix](fe) Stop split enumeration when queries finish or terminate#69835
924060929 merged 15 commits into
apache:masterfrom
CalvinKirs:stream_split

CalvinKirs commented Oct 29, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Oct 29, 2026

Uh oh!

CalvinKirs commented Oct 30, 2026

Uh oh!

CalvinKirs commented Oct 30, 2026

Uh oh!

924060929 left a comment

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

CalvinKirs commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

HDFS performance measurements

Real OSS performance measurements (current merged head)

Uh oh!

hello-stephen commented Oct 29, 2026

Uh oh!

CalvinKirs commented Oct 30, 2026

Uh oh!

CalvinKirs commented Oct 30, 2026

Local pipeline review — ✅ PASS

Uh oh!

924060929 left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

hello-stephen commented Oct 30, 2026

Uh oh!

hello-stephen commented Oct 30, 2026

FE UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CalvinKirs commented Oct 29, 2026 •

edited

Loading