Repository navigation
[improvement](be) Reuse Parquet profile counters across external scan files - #69300
Conversation
|
run buildall |
|
/review |
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Static review of all 19 changed files at e3948fd. One P2 finding: the custom Hudi and Iceberg delete-file Parquet factories omit the cache, retaining per-file counter registration on those paths. No additional verified defect was found. There were no existing inline threads to carry forward and no additional user-provided focus.
Required checkpoint conclusions:
- Goal and proof: cached generic readers avoid repeated registration and reduce Variant ownership copies; the inline finding identifies uncovered native paths. Added unit tests inspect identity, concurrency, lock avoidance, aggregation, isolation, disabled profiles and retained lifetime.
- Scope and clarity: changes are focused on profile handles, ownership and propagation across the scan stack; decoding and table semantics are unchanged.
- Concurrency: scanner workers share one cache per local scan state. call_once publishes its immutable handles; existing atomic counter updates remain additive. No new reverse lock order was found; registration is confined to initialization.
- Lifecycle and static initialization: worker context ownership retains RuntimeState/LocalState during cache use. Close does not dereference the cache. Deferred Variant state copies retain the shared counter bundle, including selected/subtree states. Disabled defaults are function-local statics with no cross-translation-unit dependency.
- Configuration: no new configuration or dynamic-update behavior.
- Compatibility: constructor changes are internal and default the cache for standalone readers; no wire or stored-format change.
- Thrift/protobuf: no changed definitions or field IDs.
- Parallel paths: Hive, native Iceberg data, Paimon and Fluss forwarding inspected; Hudi base files and Iceberg delete files are the omissions in the inline finding. Other concrete FileReader initialization paths retain their explicit profile setup.
- Conditional checks: cached/standalone and enabled/disabled profile branches have concrete ownership purposes and explanatory comments; no silent error continuation introduced.
- Test coverage: new tests exercise the cache directly and direct ParquetReader use, but do not exercise cache consumption through the custom table factories. Actual wrapper-path coverage should accompany the fix.
- Test-result changes: no expected-result artifacts were changed.
- Observability: counter names, units, hierarchy and per-reader delta snapshots are preserved; cached and fallback readers safely reuse counter identities.
- Transactions and persistence: no transaction, EditLog, failover or persistence behavior changed.
- Data writes and atomicity: no write or commit path changed.
- FE/BE variables: no new transmitted variable or sending path.
- Performance: lazy registration and one retained Variant bundle remove repeated work where the cache is consumed; the finding covers missed factories. Benchmark claims were not independently reproduced.
- Other invariants: no additional data-correctness, version/delete visibility, memory ownership or error-propagation regression was established in the changed paths.
Validation limits: static review only, as required; no builds, tests, sanitizer runs or performance measurements were executed. Added tests and the author's reported results were inspected, not reproduced. No unresolved code-path candidate remains beyond the verified finding.
Review converged after 2 round(s).
TPC-H: Total hot run time: 28664 ms |
TPC-DS: Total hot run time: 150252 ms |
ClickBench: Total hot run time: 24.27 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
e3948fd to
83312d8
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed PR #69300 at 83312d80085fa2aff2bfd494dd06f6357b3ee626 using the supplied aggregate diff and changed-file list. One new P2 finding: Iceberg $position_deletes scans still register Parquet counters for each split because their nested reader omits the new cache. No new P0/P1 or correctness defect was substantiated.
Existing review context: thread 4317030285 is fixed in this head. Both the Hudi custom factory and the ordinary Iceberg delete-file factory now pass the scan cache; the new system-table finding follows a different class and child profile. No existing P0/P1 remains to carry forward. There were no additional user-provided focus points.
Critical checkpoint conclusions:
| Checkpoint | Conclusion |
|---|---|
| Goal and evidence | Scan-owned lazy caching removes repeated registration from ordinary native Parquet scans; focused identity, concurrency, reuse and lifetime tests cover the core contract. The nested system-table path remains incomplete as described inline. |
| Scope and reuse | Changes are focused on profile ownership/plumbing and redundant initialization removal. Existing factories and counter registration APIs are reused. |
| Concurrency and locking | Parallel scanner workers can enter the first cache lookup. std::call_once publishes a complete immutable handle set; counter updates remain atomic. Existing profile mutexes protect registration. No reverse lock acquisition or new deadlock path was found. |
| Lifecycle and memory | Running scanner work retains TaskExecutionContext, which owns its RuntimeState/local-state cache. Close uses copied handles. Deferred Variant state retains one shared bundle of independently owned counters; all deferred accesses use that bundle. No ownership cycle, cross-TU initializer dependency or new unbounded allocation was found. |
| Configuration | No configuration item is added or changed; dynamic propagation is inapplicable. |
| Compatibility | No persisted format or FE/BE wire contract changes. Constructor additions are internal and retain defaults for standalone callers; rolling-upgrade behavior is unaffected. |
| Thrift and Protobuf | No definitions changed; field-ID compatibility is inapplicable. |
| Parallel paths | Reviewed generic, Hudi, ordinary Iceberg delete, Paimon, Fluss and standalone schema readers, plus every concrete format initializer affected by the base-call removal. The additional position-delete system-table path is the inline finding. |
| Conditions and errors | Cached/standalone and disabled-profile cases have concrete purposes and comments. Concrete readers initialize before timers; cancellation, EOF, error propagation and close behavior remain unchanged. |
| Test coverage | Added tests check lazy and concurrent initialization, lock avoidance after initialization, instance isolation, disabled profiles, retained/released Variant owners, one init per file and real custom-factory reuse. Full scanner teardown and the nested system-table path lack direct new coverage. All tests were inspected statically and were not executed here. |
| Test results | No expected-output files changed. Author-reported build/test/benchmark results were not independently reproduced. |
| Observability | Counter names, units, hierarchy, atomic aggregation and publication remain unchanged in the modified reader paths. The missing nested-profile cache must preserve child-profile attribution when fixed. |
| Transactions and persistence | No transaction, visibility-version, delete-bitmap, EditLog or failover behavior is changed. |
| Writes and atomicity | No write path or crash-recovery protocol is changed. |
| FE/BE variables | No new transmitted variable or sending path is introduced. |
| Performance | One immutable counter set per scan and one Variant bundle reference per copy remove repeated work without adding per-row work. The inline P2 identifies the remaining per-file registration path; no speedup was measured during this review. |
| Other architecture/header checks | Forward declarations and out-of-line destruction avoid heavy header inclusion. No PCH, template-instantiation or hygiene-table edits occur. The opaque Parquet cache observer in generic TableReadOptions is a concrete-format instrumentation dependency that departs from the guide's neutral-contract preference; it does not expose physical metadata or alter table semantics, and no separate actionable defect was established. |
Static review covered all 22 changed paths and related control flow. No builds, tests, source changes or runtime validation were performed, as required by the review contract. No unresolved candidate or static coverage gap remains. Review convergence is reported separately by the delivery metadata.
Review converged after 2 round(s).
83312d8 to
eb06287
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
No new actionable findings in the 25 changed paths at eb06287. The two existing P2 concerns are fixed: the Hudi and ordinary Iceberg delete-file factories forward the cache, and the position-delete system table resolves a distinct child-profile cache across splits and scanners. No existing P0/P1 finding remains to carry forward. No additional user focus was supplied.
The code-review skill checkpoints have these conclusions:
- Goal and proof: the code removes repeated Parquet counter registration and the duplicate common init hook. Added tests check actual reader initialization, shared counter identity and accumulation, independent scans, nested factories, concurrent publication and retained Variant ownership.
- Scope and abstraction: changes stay focused on profiling ownership, initialization and forwarding. Headers use forward declarations where ownership permits; physical decoding and table-schema mapping remain in their existing layers.
- Concurrency and critical state: scan workers share immutable counter handles published by std::call_once. The child-cache map has its own mutex; RuntimeProfile serializes registration and child creation. Counter values retain their atomic update behavior.
- Lock cost and order: the child-cache lock covers lookup/allocation only, and counter registration occurs after it is released. Profile child creation also releases its lock before cache lookup. No new reverse lock order was found, and warm cache reads avoid the counter-registration lock.
- Lifecycle and static initialization: workers retain the PipelineFragmentContext, which owns RuntimeStates and their local states/cache, even during asynchronous scanner stop. Queued tasks with expired execution context return before scanner access. Deferred Variant states retain the shared counter bundle without a profile/cache ownership cycle. No cross-translation-unit initializer dependency was introduced.
- Configuration: no configuration item or dynamic-update behavior changes.
- Compatibility: constructor additions preserve existing callers through a default null cache. No stored-file, on-disk metadata or wire-format change requires rolling-upgrade adaptation.
- Thrift/Protobuf IDs: no gensrc definitions changed; branch field-ID comparison is not applicable.
- Parallel paths: generic, Hudi, ordinary Iceberg delete, nested position-delete, Paimon and Fluss routes were traced. Standalone schema/row-id readers keep the uncached fallback. ORC and other concrete readers retain their own initialization; removing the base hook does not omit a concrete profile setup.
- Conditional checks and failures: null-cache fallback and child-cache creation have reachable standalone/nested uses and explanatory comments. Status/error propagation and null-profile behavior remain intact; no silent-success failure path was found.
- Test coverage: focused unit tests exercise the modified boundaries, including negative null-profile behavior and contention/lifetime cases. Hudi and nested tests inspect real factories without reading data; Parquet and Iceberg delete-file tests open actual local files. Full runtime, sanitizer, cancellation and performance validation was not reproduced in this static-only review.
- Test-result changes: no regression .out files changed; the new unit-test assertions match the counter identity/aggregation contracts. Their execution results remain unverified here.
- Observability: existing names, hierarchy and accumulation remain; nested position-delete counters stay on the child profile, and non-Parquet scans do not eagerly acquire Parquet counters. No new per-row logging or timers were added.
- Transactions and persistence: no transaction, EditLog, recovery or persistent state changes; visibility/version and MoW delete-bitmap invariants are unaffected by the reviewed changes.
- Data writes and crash atomicity: no data-write or commit path changes; resource lifetime changes were reviewed through scanner shutdown and deferred materialization.
- FE/BE propagation: no new transmitted variable or sending path; the cache is an in-process observer propagated through the relevant reader factories.
- Performance: initialization work moves from each file to one counter set per scan profile, with one retained Variant bundle reference per copy. Child caches are retained rather than recreated per split. The author's benchmark gains were not independently verified.
- Other risks and user concerns: cache identity, teardown, missing parallel initialization and deferred ownership risks were dismissed with caller evidence. Both existing review threads were revalidated without duplicate comments; no unresolved code candidate remains.
Coverage is complete for the supplied aggregate diff and affected call chains. Validation was static only: no builds, tests, source modifications or benchmark runs were performed under this review contract.
All three round-1 reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty finding set; main verification dismissed every initial risk and recorded no unresolved candidate. Review converged in one round.
Review converged after 1 round(s).
… files
### What problem does this PR solve?
Issue Number: None
Problem Summary: External small-file scans repeatedly initialize the same Parquet profile counter hierarchy for each file, contending on the shared RuntimeProfile registration lock. FileReader::init also repeats the concrete reader's initialization. Initialize counters once per scan instance and pass the immutable handles through table-reader wrappers to Parquet readers. Keep distinct scan instances separate. Group the shared Variant counters into one retained bundle so deferred materialization can safely outlive the scanner without copying a shared_ptr for every counter.
This change applies to native Parquet reads across Hive, Iceberg, Hudi, Paimon and Fluss table-reader wrappers. It does not add speculative I/O or change the file reader's selection and decoding behavior.
Review follow-up: forward the scanner-owned profile cache through the actual Hudi base-file and Iceberg delete-file Parquet factories. New tests exercise repeated reader construction and shared counter identity, including opening a real delete Parquet file. Validation: full ASAN BE build, 442 affected reader/profile/Variant tests, clang-tidy on changed lines, clang-format 16 and build hygiene passed. The scoped UT harness restores its CMake selection after execution and is not part of this commit.
Review follow-up: retain scan-owned caches for nested Parquet profiles, including the Iceberg position-deletes system table, without attributing counters to the outer profile. Added repeated split/scanner factory coverage and concurrent child-cache isolation tests. ASAN build, 446 scoped unit tests, clang-format 16, header hygiene and clang-tidy 22 passed.
CI follow-up: initialize the Hudi factory test through TableReader::init instead of assigning an incomplete set of internal fields; Linux ASAN exposed the uninitialized scan-parameter pointer. Revalidated 446 scoped ASAN tests, clang-format16 and clang-tidy22. Production behavior is unchanged.
### Release note
Reduce profile initialization and counter ownership overhead when scanning many small Parquet files.
### Check List (For Author)
- Test: Unit Test and Manual Test
- New profile-cache tests cover concurrent initialization, counter aggregation, independent scan instances, no-profile readers and retained Variant counter lifetime
- Reader tests cover one initialization per file and sharing across files
- run-be-ut.sh: 585 tests across 13 Parquet, ORC, Variant and Hive/Iceberg/Hudi/Paimon/table-reader suites passed with ASAN
- Fresh Linux AArch64 build.sh --be -j3 baseline and optimized RELEASE builds passed
- clang-format16 and header hygiene passed; preset clang-tidy passed all 19 changed files against the matching ASAN compilation database
- Isolated Release SQL results and a comparison with identical FE source are reported below
- Behavior changed: No; preserve results and profile counter semantics
- Does this need documentation: No
### Performance validation
Complete SQL medians in milliseconds, matching Linux AArch64 Release BE, single backend, 6 CPUs, warm footer/data cache. Both settings are fixed: `split_batch_size=1`, `split_queue_capacity=4096`. Each single-query result has 40 measured samples in forward/reverse order after identical warmup. Cold denotes catalog/manifest cache invalidation, not a cold JVM or BE data cache. COUNT and SUM results are checked, and post-timing profiles verify footer/page hits.
To isolate the BE change from a pre-existing FE EOF polling race, both columns below use exactly the same FE source containing its notification fix; only the Release BE package changes.
| Warm full-scan scenario | Original BE, same fixed FE | This BE, same fixed FE | Reduction |
|---|---:|---:|---:|
| Hive 1,000 partitions | 60.77 | 16.39 | 73.0% |
| Hive 10,000 partitions | 738.04 | 120.15 | 83.7% |
| Hive one partition / 10,000 files | 495.15 | 74.42 | 85.0% |
| Iceberg 1,000 manifests / 10,000 files | 503.25 | 81.46 | 83.8% |
On unchanged baseline FE, independent BE-only single-query medians were 22.89 ms for Hive 1,000 partitions, 230.37 ms for Hive 10,000 partitions, 117.95 ms for Hive one partition with 10,000 files and 138.38 ms for Iceberg 1,000 manifests. The faster BE exposed an existing FE terminal-queue poll: 78 delayed empty requests in a 400-SQL diagnostic; one finished production at 0.148 ms but returned at 100.911 ms. The companion FE scan/scheduling PR fixes terminal notification, verified with zero such delays on replay. Merge the FE notification fix first when assessing concurrent latency; the BE patch has no code dependency on it.
eb06287 to
024ea81
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed all 25 changed files (19 production files and six test files) at 024ea81a94780949e4a26d6ebe4f78f0a1495964. No new actionable findings. The review converged in round 1: both complete-scope reviewers and the dedicated risk reviewer returned NO_NEW_VALUABLE_FINDINGS, and all risk questions and candidate dispositions were independently checked.
Both existing P2 issues are fixed at this head: custom Hudi/Iceberg factories now forward the cache, and nested position-delete system-table readers retain a separate cache for their child RuntimeProfile across splits/scanners. Neither is reposted. No existing P0/P1 blocker remains in the reviewed thread context. No additional user-provided review focus was supplied.
- Goal and evidence: the cache registers Parquet handles once per scan/profile; the Variant bundle reduces ownership copies to one shared reference. Added tests inspect identity, aggregation, registration-lock avoidance, repeated files, custom factories and retained counter lifetime.
- Scope and design: the changes are focused on counter ownership, forwarding and removal of duplicate initialization. Schema mapping, decoding, predicate/delete semantics and physical file selection are unaffected by the changed logic.
- Concurrency and locking: scanner workers share immutable handles published by
call_once; child-cache insertion uses its own mutex and counter values remain atomic. Child-map locks are released before profile initialization. No new lock inversion or production thread entry was found. - Lifecycle and static initialization: worker execution pins the fragment context and its RuntimeState/local cache through scanner use and cleanup. Close preserves cache/profile ownership; failed opens finalize, and recursive rebuild creates a new fragment context. Deferred Variant states retain their shared counters after scanner/profile destruction without a cycle. Disabled bundles use function-local initialization; no cross-TU static dependency is introduced.
- Configuration: no configuration items are added; dynamic updates are not applicable.
- Compatibility: changes are internal source interfaces with standalone defaults. No storage/wire format or dependency/toolchain change requires rolling-upgrade compatibility handling; binary artifacts were not built in this review.
- Thrift/protobuf: no definitions are changed, so branch field-ID comparison is not applicable.
- Parallel paths: generic, Hudi and Iceberg factories plus Hudi/Paimon/Fluss hybrid, union-lake and tail-reader paths forward the cache with the matching profile. Nested position deletes use child counters; standalone/no-profile paths retain their defaults. Every current concrete reader that needs profile initialization still calls it explicitly.
- Conditional checks: the cache-present, standalone and child-ownership branches correspond to concrete lifecycle cases and have explanatory comments; no changed decision suppresses a data error.
- Test coverage: six changed BE test files cover the central cache, ownership and factory contracts. Fluss/Paimon forwarding, cancellation and recursive teardown were traced statically. The patch adds no SQL integration tests; no missing coverage was tied to a verified defect.
- Test results: no expected-result artifacts change. The new assertions match the inspected identity, accumulation and ownership contracts. Author-reported test and benchmark results were not independently executed.
- Observability: counter names, types and hierarchy remain intact. Per-file statistics and published-delta snapshots remain reader-local, while nested counters stay on the child profile; no additional metric family is needed for handle reuse.
- Transactions and persistence: no visibility/version, delete-bitmap, EditLog, replay or failover logic changes.
- Data writes and crash behavior: no data-writing path changes. New query-scoped owners use RAII and introduce no persistent recovery state.
- FE/BE variables: no transmitted field is added; the new option is internal BE plumbing.
- Performance: repeated Parquet counter registration and per-counter shared ownership copies are removed. Cache creation is bounded by scan/profile count, with no added per-row work or changed data I/O. Speedup magnitude was not independently measured.
- Other invariants: no substantiated change to row results, nullability, error propagation or resource ownership was found. The incomplete-type constructor concern was dismissed by the actual header include chain.
Validation was static only, as required: no builds, tests, sanitizer runs or performance measurements were performed. No changed paths remain uncovered and no candidate or risk question remains unresolved within that scope.
Review converged after 1 round(s).
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28334 ms |
TPC-DS: Total hot run time: 150676 ms |
ClickBench: Total hot run time: 24.23 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Merging master 7acfd9b left two Delta gaps. First, apache#69717 replaced table_reader.h's full ShardedKVCache definition with a forward declaration, but deltalake_reader_test.cpp constructs a ShardedKVCache and relied on the transitive include. Include format/format_common.h directly, as hudi_reader_test.cpp does. Second, apache#69300 made Parquet readers in one scan share their profile counters through TableReader's parquet profile cache. Hudi, Iceberg and the base reader pass that cache, but DeltaReader still built its ParquetReader without it, so every Delta file created its own set of counters. Pass the cache, and keep binary-UUID preservation disabled as before. ### Release note None ### Check List (For Author) - Test: No local build. The merge was conflict-free. Both changes follow the matching master code. COMPILE and BE UT are left to CI. - Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do. - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Merging master 7acfd9b left two Delta gaps. First, apache#69717 replaced table_reader.h's full ShardedKVCache definition with a forward declaration, but deltalake_reader_test.cpp constructs a ShardedKVCache and relied on the transitive include. Include format/format_common.h directly, as hudi_reader_test.cpp does. Second, apache#69300 made Parquet readers in one scan share their profile counters through TableReader's parquet profile cache. Hudi, Iceberg and the base reader pass that cache, but DeltaReader still built its ParquetReader without it, so every Delta file created its own set of counters. Pass the cache, and keep binary-UUID preservation disabled as before. ### Release note None ### Check List (For Author) - Test: No local build. The merge was conflict-free. Both changes follow the matching master code. COMPILE and BE UT are left to CI. - Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do. - Does this need documentation: No
What problem does this PR solve?
Issue Number: None
Problem Summary: External small-file scans repeatedly initialize the same Parquet profile counter hierarchy for each file, contending on the shared RuntimeProfile registration lock. FileReader::init also repeats the concrete reader's initialization. Initialize counters once per scan instance and pass the immutable handles through table-reader wrappers to Parquet readers. Keep distinct scan instances separate. Group the shared Variant counters into one retained bundle so deferred materialization can safely outlive the scanner without copying a shared_ptr for every counter.
This change applies to native Parquet reads across Hive, Iceberg, Hudi, Paimon and Fluss table-reader wrappers. It does not add speculative I/O or change the file reader's selection and decoding behavior.
Main benefit
Avoid rebuilding the same Parquet counters for every small file. In matched Release-BE experiments with the FE held fixed, warm complete-SQL full scans improve by 73–85%: Hive with 10,000 partitions falls from 738.04 to 120.15 ms, and Iceberg with 10,000 files from 503.25 to 81.46 ms. The full table and independent-BE-only results are below.
The mechanism is scan-owned immutable counter handles, reused across files and forwarded through the real table-reader factories. Nested readers retain their own child-profile caches, preserving attribution. Deferred Variant work retains one shared counter bundle rather than copying each counter owner separately. File decoding, predicates and delete application are unchanged.
The headline A/B table uses exactly the same FE source and settings in both variants; only the Release BE package changes. The elapsed-time reduction therefore measures this BE optimization. No standalone latency claim is made for Hudi, Paimon or Fluss merely because their factory paths forward the cache.
Release note
Reduce profile initialization and counter ownership overhead when scanning many small Parquet files.
Check List (For Author)
Latest validation at
024ea81a947: all 446 scoped reader/profile/Variant ASAN tests passed, as did clang-format 16 and clang-tidy 22. The Linux CI crash in the added Hudi factory test was traced to bypassingTableReader::init; the fixture now follows normal initialization instead of leaving scan parameters uninitialized. This follow-up changes tests only. The Release performance measurements below predate the later factory/child-profile review repairs and are not a fresh A/B of this head. CI status remains tracked by the checks on this PR.Performance validation
Complete SQL medians in milliseconds, matching Linux AArch64 Release BE, single backend, 6 CPUs, warm footer/data cache. Both settings are fixed:
split_batch_size=1,split_queue_capacity=4096. Each single-query result has 40 measured samples in forward/reverse order after identical warmup. Cold denotes catalog/manifest cache invalidation, not a cold JVM or BE data cache. COUNT and SUM results are checked, and post-timing profiles verify footer/page hits.To isolate the BE change from a pre-existing FE EOF polling race, both columns below use exactly the same FE source containing its notification fix; only the Release BE package changes.
A separate BE-only experiment on an older, unchanged FE measured single-query medians of 22.89 ms for Hive 1,000 partitions, 230.37 ms for Hive 10,000 partitions, 117.95 ms for Hive one partition with 10,000 files, and 138.38 ms for Iceberg 1,000 manifests. That FE had an existing terminal-queue polling delay: a 400-SQL diagnostic found 78 delayed empty requests, including one whose production ended at 0.148 ms but whose response returned at 100.911 ms. The headline table above uses a fixed FE without that delay in both columns, so the BE comparison is not dominated by FE polling. The two experiments use different fixed FE versions and their absolute timings should not be mixed.