You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0af95bc
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](fe) Add a per-scan split producer limit
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: Partition-batch scans derive their producer count from the
shared external metadata pool. On an overloaded object store, speculative LIST
requests can delay the first batch and cause gaps in subsequent Split supply.
Add an opt-in session limit per scan node without shrinking global executors or
changing batch sizes, queue bounds, cancellation, or wait deadlines. Zero keeps
the existing behavior; positive values also respect executor and partition-batch
bounds. Whole-scan streaming retains one producer.
On the local million-file fixture with BE 4.1.3 and the global pool fixed at 64,
three default-batch LIMIT samples per setting show a median reduction from
1443 ms to 654 ms with a producer limit of four, and median LIST calls from 26
to 12. In the partition-batch-size-one ABBA control, six LIMIT samples per setting
improve from 6564 ms to 560 ms. Complete 10000-file scans succeed in 2/6 uncapped
and 6/6 capped samples, but the uncapped successful subset is faster; this is not
a general full-scan throughput improvement. Keep the default at zero and expose
the setting in the optional regression benchmark for workload-specific tuning.
### Release note
Add the FE session variable split_producer_parallelism to cap partition-batch
Split producers per plugin scan node. The default zero preserves existing
parallelism. Negative values are rejected.
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- 58 focused FE unit tests pass with zero failures, errors or skips
- Standard seven-scenario Hive performance regression passes with BE 4.1.3
- 32 uninstrumented SQL controls match unique FE audit queries; complete
successful scans read 10000 rows with exact COUNT and SUM results
- FE build through build.sh --fe and Checkstyle pass
- Behavior changed: Yes, an explicit positive session limit bounds producers;
default behavior, batch sizes and deadlines remain unchanged
- Does this need documentation: Yes, setting and benchmark usage documented in
regression-test/script/hive_split_early_stop/README.md
0 commit comments