Skip to content

[improvement](be) Reuse Parquet profile counters across external scan files - #69300

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
924060929:codex/lake-pr-parquet-profile
Oct 20, 2026
Merged

Gabriel39 merged 1 commit into
apache:masterfrom
924060929:codex/lake-pr-parquet-profile

Conversation

@924060929

@924060929 924060929 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Problem Summary: External small-file scans repeatedly initialize the same Parquet profile counter hierarchy for each file, contending on the shared RuntimeProfile registration lock. FileReader::init also repeats the concrete reader's initialization. Initialize counters once per scan instance and pass the immutable handles through table-reader wrappers to Parquet readers. Keep distinct scan instances separate. Group the shared Variant counters into one retained bundle so deferred materialization can safely outlive the scanner without copying a shared_ptr for every counter.

This change applies to native Parquet reads across Hive, Iceberg, Hudi, Paimon and Fluss table-reader wrappers. It does not add speculative I/O or change the file reader's selection and decoding behavior.

Main benefit

Avoid rebuilding the same Parquet counters for every small file. In matched Release-BE experiments with the FE held fixed, warm complete-SQL full scans improve by 73–85%: Hive with 10,000 partitions falls from 738.04 to 120.15 ms, and Iceberg with 10,000 files from 503.25 to 81.46 ms. The full table and independent-BE-only results are below.

The mechanism is scan-owned immutable counter handles, reused across files and forwarded through the real table-reader factories. Nested readers retain their own child-profile caches, preserving attribution. Deferred Variant work retains one shared counter bundle rather than copying each counter owner separately. File decoding, predicates and delete application are unchanged.

The headline A/B table uses exactly the same FE source and settings in both variants; only the Release BE package changes. The elapsed-time reduction therefore measures this BE optimization. No standalone latency claim is made for Hudi, Paimon or Fluss merely because their factory paths forward the cache.

Release note

Reduce profile initialization and counter ownership overhead when scanning many small Parquet files.

Check List (For Author)

Latest validation at 024ea81a947: all 446 scoped reader/profile/Variant ASAN tests passed, as did clang-format 16 and clang-tidy 22. The Linux CI crash in the added Hudi factory test was traced to bypassing TableReader::init; the fixture now follows normal initialization instead of leaving scan parameters uninitialized. This follow-up changes tests only. The Release performance measurements below predate the later factory/child-profile review repairs and are not a fresh A/B of this head. CI status remains tracked by the checks on this PR.

  • Test: Unit Test and Manual Test
    • New profile-cache tests cover concurrent initialization, counter aggregation, independent scan instances, no-profile readers and retained Variant counter lifetime
    • Reader tests cover one initialization per file and sharing across files
    • run-be-ut.sh: 585 tests across 13 Parquet, ORC, Variant and Hive/Iceberg/Hudi/Paimon/table-reader suites passed with ASAN
    • Fresh Linux AArch64 build.sh --be -j3 baseline and optimized RELEASE builds passed
    • clang-format16 and header hygiene passed; preset clang-tidy passed all 19 changed files against the matching ASAN compilation database
    • Isolated Release SQL results and a comparison with identical FE source are reported below
  • Behavior changed: No; preserve results and profile counter semantics
  • Does this need documentation: No

Performance validation

Complete SQL medians in milliseconds, matching Linux AArch64 Release BE, single backend, 6 CPUs, warm footer/data cache. Both settings are fixed: split_batch_size=1, split_queue_capacity=4096. Each single-query result has 40 measured samples in forward/reverse order after identical warmup. Cold denotes catalog/manifest cache invalidation, not a cold JVM or BE data cache. COUNT and SUM results are checked, and post-timing profiles verify footer/page hits.

To isolate the BE change from a pre-existing FE EOF polling race, both columns below use exactly the same FE source containing its notification fix; only the Release BE package changes.

Warm full-scan scenario Original BE, same fixed FE This BE, same fixed FE Reduction
Hive 1,000 partitions 60.77 16.39 73.0%
Hive 10,000 partitions 738.04 120.15 83.7%
Hive one partition / 10,000 files 495.15 74.42 85.0%
Iceberg 1,000 manifests / 10,000 files 503.25 81.46 83.8%

A separate BE-only experiment on an older, unchanged FE measured single-query medians of 22.89 ms for Hive 1,000 partitions, 230.37 ms for Hive 10,000 partitions, 117.95 ms for Hive one partition with 10,000 files, and 138.38 ms for Iceberg 1,000 manifests. That FE had an existing terminal-queue polling delay: a 400-SQL diagnostic found 78 delayed empty requests, including one whose production ended at 0.148 ms but whose response returned at 100.911 ms. The headline table above uses a fixed FE without that delay in both columns, so the BE comparison is not dominated by FE polling. The two experiments use different fixed FE versions and their absolute timings should not be mixed.

@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of all 19 changed files at e3948fd. One P2 finding: the custom Hudi and Iceberg delete-file Parquet factories omit the cache, retaining per-file counter registration on those paths. No additional verified defect was found. There were no existing inline threads to carry forward and no additional user-provided focus.

Required checkpoint conclusions:

  • Goal and proof: cached generic readers avoid repeated registration and reduce Variant ownership copies; the inline finding identifies uncovered native paths. Added unit tests inspect identity, concurrency, lock avoidance, aggregation, isolation, disabled profiles and retained lifetime.
  • Scope and clarity: changes are focused on profile handles, ownership and propagation across the scan stack; decoding and table semantics are unchanged.
  • Concurrency: scanner workers share one cache per local scan state. call_once publishes its immutable handles; existing atomic counter updates remain additive. No new reverse lock order was found; registration is confined to initialization.
  • Lifecycle and static initialization: worker context ownership retains RuntimeState/LocalState during cache use. Close does not dereference the cache. Deferred Variant state copies retain the shared counter bundle, including selected/subtree states. Disabled defaults are function-local statics with no cross-translation-unit dependency.
  • Configuration: no new configuration or dynamic-update behavior.
  • Compatibility: constructor changes are internal and default the cache for standalone readers; no wire or stored-format change.
  • Thrift/protobuf: no changed definitions or field IDs.
  • Parallel paths: Hive, native Iceberg data, Paimon and Fluss forwarding inspected; Hudi base files and Iceberg delete files are the omissions in the inline finding. Other concrete FileReader initialization paths retain their explicit profile setup.
  • Conditional checks: cached/standalone and enabled/disabled profile branches have concrete ownership purposes and explanatory comments; no silent error continuation introduced.
  • Test coverage: new tests exercise the cache directly and direct ParquetReader use, but do not exercise cache consumption through the custom table factories. Actual wrapper-path coverage should accompany the fix.
  • Test-result changes: no expected-result artifacts were changed.
  • Observability: counter names, units, hierarchy and per-reader delta snapshots are preserved; cached and fallback readers safely reuse counter identities.
  • Transactions and persistence: no transaction, EditLog, failover or persistence behavior changed.
  • Data writes and atomicity: no write or commit path changed.
  • FE/BE variables: no new transmitted variable or sending path.
  • Performance: lazy registration and one retained Variant bundle remove repeated work where the cache is consumed; the finding covers missed factories. Benchmark claims were not independently reproduced.
  • Other invariants: no additional data-correctness, version/delete visibility, memory ownership or error-propagation regression was established in the changed paths.

Validation limits: static review only, as required; no builds, tests, sanitizer runs or performance measurements were executed. Added tests and the author's reported results were inspected, not reproduced. No unresolved code-path candidate remains beyond the verified finding.

Review converged after 2 round(s).

Comment thread be/src/format_v2/table/hudi_reader.cpp Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28664 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3948fd2d21839559025e9ffc3449b5908073854, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17558	5158	5042	5042
q2	2094	312	261	261
q3	10244	1401	807	807
q4	4679	459	332	332
q5	7429	799	531	531
q6	183	188	154	154
q7	764	806	601	601
q8	9260	1342	1042	1042
q9	5466	4237	4201	4201
q10	6819	1282	1048	1048
q11	434	245	242	242
q12	632	405	279	279
q13	18096	3485	3021	3021
q14	267	271	247	247
q15	q16	736	706	653	653
q17	1206	751	580	580
q18	7055	6153	6103	6103
q19	1094	962	617	617
q20	385	343	230	230
q21	4889	2397	2414	2397
q22	396	331	276	276
Total cold run time: 99686 ms
Total hot run time: 28664 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5939	5908	5873	5873
q2	714	557	526	526
q3	5555	5749	5242	5242
q4	2016	2162	1549	1549
q5	5819	5757	5794	5757
q6	263	198	150	150
q7	2280	1988	1732	1732
q8	2984	2622	2627	2622
q9	7588	7500	7466	7466
q10	3602	3552	3169	3169
q11	577	410	377	377
q12	679	690	509	509
q13	3259	3543	3046	3046
q14	291	279	264	264
q15	q16	660	685	614	614
q17	7703	7014	6840	6840
q18	12963	12132	12862	12132
q19	931	797	789	789
q20	2133	2173	1944	1944
q21	5603	4620	4816	4620
q22	509	457	418	418
Total cold run time: 72068 ms
Total hot run time: 65639 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150252 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3948fd2d21839559025e9ffc3449b5908073854, data reload: false

query5	4303	573	448	448
query6	430	206	207	206
query7	4809	442	233	233
query8	316	161	153	153
query9	8769	3048	3092	3048
query10	449	329	241	241
query11	5966	3818	3573	3573
query12	147	95	84	84
query13	1249	432	328	328
query14	6098	3946	3658	3658
query14_1	3363	3399	3395	3395
query15	205	190	178	178
query16	938	443	412	412
query17	855	674	553	553
query18	2397	417	293	293
query19	189	155	128	128
query20	82	81	80	80
query21	201	136	109	109
query22	13053	12971	12780	12780
query23	13400	12942	12554	12554
query23_1	12567	12603	12498	12498
query24	7391	808	452	452
query24_1	474	459	460	459
query25	502	390	337	337
query26	1265	249	133	133
query27	2788	430	263	263
query28	4664	1903	1902	1902
query29	1548	585	426	426
query30	303	222	187	187
query31	983	925	806	806
query32	140	91	85	85
query33	532	293	230	230
query34	945	912	510	510
query35	712	754	633	633
query36	858	809	736	736
query37	118	90	77	77
query38	1782	1706	1654	1654
query39	788	805	737	737
query39_1	724	724	737	724
query40	221	117	91	91
query41	65	61	60	60
query42	85	85	81	81
query43	347	368	307	307
query44	1263	674	668	668
query45	180	180	165	165
query46	832	966	559	559
query47	2790	2981	2897	2897
query48	297	311	220	220
query49	593	408	302	302
query50	683	289	213	213
query51	10398	10470	10109	10109
query52	86	82	72	72
query53	188	210	157	157
query54	251	227	220	220
query55	70	68	67	67
query56	232	225	228	225
query57	1648	1593	1553	1553
query58	298	260	239	239
query59	2094	2154	1934	1934
query60	292	246	221	221
query61	147	147	142	142
query62	382	328	281	281
query63	193	149	155	149
query64	2716	918	739	739
query65	3308	3259	3250	3250
query66	1763	416	298	298
query67	20339	20261	19978	19978
query68	2966	996	621	621
query69	385	287	251	251
query70	840	813	793	793
query71	293	228	190	190
query72	2682	2562	2386	2386
query73	514	525	288	288
query74	4588	4516	4341	4341
query75	2310	2278	1887	1887
query76	2304	1011	594	594
query77	329	365	267	267
query78	9630	9698	9147	9147
query79	864	852	520	520
query80	465	405	327	327
query81	523	314	274	274
query82	199	135	110	110
query83	194	195	184	184
query84	289	127	103	103
query85	791	411	358	358
query86	256	233	224	224
query87	1968	1955	1822	1822
query88	3564	2660	2631	2631
query89	319	277	243	243
query90	2105	166	168	166
query91	157	150	124	124
query92	98	88	96	88
query93	978	977	569	569
query94	441	303	260	260
query95	581	321	309	309
query96	661	483	237	237
query97	2433	2441	2281	2281
query98	158	152	148	148
query99	752	769	650	650
Total cold run time: 233221 ms
Total hot run time: 150252 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3948fd2d21839559025e9ffc3449b5908073854, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.04	0.05
query3	0.26	0.09	0.08
query4	1.60	0.10	0.10
query5	0.24	0.24	0.24
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.35	0.35
query10	0.56	0.58	0.58
query11	0.20	0.14	0.15
query12	0.19	0.14	0.15
query13	0.48	0.50	0.49
query14	1.01	1.02	1.01
query15	0.63	0.61	0.60
query16	0.34	0.34	0.35
query17	1.15	1.13	1.15
query18	0.21	0.19	0.19
query19	2.10	2.03	1.95
query20	0.01	0.01	0.01
query21	15.48	0.19	0.13
query22	4.90	0.05	0.05
query23	16.27	0.33	0.12
query24	2.96	0.47	0.45
query25	0.10	0.08	0.05
query26	1.29	0.20	0.14
query27	0.05	0.05	0.05
query28	3.40	0.77	0.36
query29	12.47	4.12	3.27
query30	0.28	0.14	0.10
query31	2.77	0.55	0.32
query32	3.22	0.59	0.50
query33	3.14	3.11	3.22
query34	15.32	4.00	3.42
query35	3.35	3.32	3.34
query36	0.53	0.42	0.40
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.79 s
Total hot run time: 24.27 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.26% (35814/46356)
Line Coverage 62.51% (404318/646849)
Region Coverage 58.99% (341637/579147)
Branch Coverage 59.92% (157828/263407)

@924060929
924060929 force-pushed the codex/lake-pr-parquet-profile branch from e3948fd to 83312d8 Compare October 20, 2026 03:54
@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR #69300 at 83312d80085fa2aff2bfd494dd06f6357b3ee626 using the supplied aggregate diff and changed-file list. One new P2 finding: Iceberg $position_deletes scans still register Parquet counters for each split because their nested reader omits the new cache. No new P0/P1 or correctness defect was substantiated.

Existing review context: thread 4317030285 is fixed in this head. Both the Hudi custom factory and the ordinary Iceberg delete-file factory now pass the scan cache; the new system-table finding follows a different class and child profile. No existing P0/P1 remains to carry forward. There were no additional user-provided focus points.

Critical checkpoint conclusions:

Checkpoint Conclusion
Goal and evidence Scan-owned lazy caching removes repeated registration from ordinary native Parquet scans; focused identity, concurrency, reuse and lifetime tests cover the core contract. The nested system-table path remains incomplete as described inline.
Scope and reuse Changes are focused on profile ownership/plumbing and redundant initialization removal. Existing factories and counter registration APIs are reused.
Concurrency and locking Parallel scanner workers can enter the first cache lookup. std::call_once publishes a complete immutable handle set; counter updates remain atomic. Existing profile mutexes protect registration. No reverse lock acquisition or new deadlock path was found.
Lifecycle and memory Running scanner work retains TaskExecutionContext, which owns its RuntimeState/local-state cache. Close uses copied handles. Deferred Variant state retains one shared bundle of independently owned counters; all deferred accesses use that bundle. No ownership cycle, cross-TU initializer dependency or new unbounded allocation was found.
Configuration No configuration item is added or changed; dynamic propagation is inapplicable.
Compatibility No persisted format or FE/BE wire contract changes. Constructor additions are internal and retain defaults for standalone callers; rolling-upgrade behavior is unaffected.
Thrift and Protobuf No definitions changed; field-ID compatibility is inapplicable.
Parallel paths Reviewed generic, Hudi, ordinary Iceberg delete, Paimon, Fluss and standalone schema readers, plus every concrete format initializer affected by the base-call removal. The additional position-delete system-table path is the inline finding.
Conditions and errors Cached/standalone and disabled-profile cases have concrete purposes and comments. Concrete readers initialize before timers; cancellation, EOF, error propagation and close behavior remain unchanged.
Test coverage Added tests check lazy and concurrent initialization, lock avoidance after initialization, instance isolation, disabled profiles, retained/released Variant owners, one init per file and real custom-factory reuse. Full scanner teardown and the nested system-table path lack direct new coverage. All tests were inspected statically and were not executed here.
Test results No expected-output files changed. Author-reported build/test/benchmark results were not independently reproduced.
Observability Counter names, units, hierarchy, atomic aggregation and publication remain unchanged in the modified reader paths. The missing nested-profile cache must preserve child-profile attribution when fixed.
Transactions and persistence No transaction, visibility-version, delete-bitmap, EditLog or failover behavior is changed.
Writes and atomicity No write path or crash-recovery protocol is changed.
FE/BE variables No new transmitted variable or sending path is introduced.
Performance One immutable counter set per scan and one Variant bundle reference per copy remove repeated work without adding per-row work. The inline P2 identifies the remaining per-file registration path; no speedup was measured during this review.
Other architecture/header checks Forward declarations and out-of-line destruction avoid heavy header inclusion. No PCH, template-instantiation or hygiene-table edits occur. The opaque Parquet cache observer in generic TableReadOptions is a concrete-format instrumentation dependency that departs from the guide's neutral-contract preference; it does not expose physical metadata or alter table semantics, and no separate actionable defect was established.

Static review covered all 22 changed paths and related control flow. No builds, tests, source changes or runtime validation were performed, as required by the review contract. No unresolved candidate or static coverage gap remains. Review convergence is reported separately by the delivery metadata.

Review converged after 2 round(s).

Comment thread be/src/format_v2/table_reader.h Resolved
@924060929
924060929 force-pushed the codex/lake-pr-parquet-profile branch from 83312d8 to eb06287 Compare October 20, 2026 04:54
@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new actionable findings in the 25 changed paths at eb06287. The two existing P2 concerns are fixed: the Hudi and ordinary Iceberg delete-file factories forward the cache, and the position-delete system table resolves a distinct child-profile cache across splits and scanners. No existing P0/P1 finding remains to carry forward. No additional user focus was supplied.

The code-review skill checkpoints have these conclusions:

  • Goal and proof: the code removes repeated Parquet counter registration and the duplicate common init hook. Added tests check actual reader initialization, shared counter identity and accumulation, independent scans, nested factories, concurrent publication and retained Variant ownership.
  • Scope and abstraction: changes stay focused on profiling ownership, initialization and forwarding. Headers use forward declarations where ownership permits; physical decoding and table-schema mapping remain in their existing layers.
  • Concurrency and critical state: scan workers share immutable counter handles published by std::call_once. The child-cache map has its own mutex; RuntimeProfile serializes registration and child creation. Counter values retain their atomic update behavior.
  • Lock cost and order: the child-cache lock covers lookup/allocation only, and counter registration occurs after it is released. Profile child creation also releases its lock before cache lookup. No new reverse lock order was found, and warm cache reads avoid the counter-registration lock.
  • Lifecycle and static initialization: workers retain the PipelineFragmentContext, which owns RuntimeStates and their local states/cache, even during asynchronous scanner stop. Queued tasks with expired execution context return before scanner access. Deferred Variant states retain the shared counter bundle without a profile/cache ownership cycle. No cross-translation-unit initializer dependency was introduced.
  • Configuration: no configuration item or dynamic-update behavior changes.
  • Compatibility: constructor additions preserve existing callers through a default null cache. No stored-file, on-disk metadata or wire-format change requires rolling-upgrade adaptation.
  • Thrift/Protobuf IDs: no gensrc definitions changed; branch field-ID comparison is not applicable.
  • Parallel paths: generic, Hudi, ordinary Iceberg delete, nested position-delete, Paimon and Fluss routes were traced. Standalone schema/row-id readers keep the uncached fallback. ORC and other concrete readers retain their own initialization; removing the base hook does not omit a concrete profile setup.
  • Conditional checks and failures: null-cache fallback and child-cache creation have reachable standalone/nested uses and explanatory comments. Status/error propagation and null-profile behavior remain intact; no silent-success failure path was found.
  • Test coverage: focused unit tests exercise the modified boundaries, including negative null-profile behavior and contention/lifetime cases. Hudi and nested tests inspect real factories without reading data; Parquet and Iceberg delete-file tests open actual local files. Full runtime, sanitizer, cancellation and performance validation was not reproduced in this static-only review.
  • Test-result changes: no regression .out files changed; the new unit-test assertions match the counter identity/aggregation contracts. Their execution results remain unverified here.
  • Observability: existing names, hierarchy and accumulation remain; nested position-delete counters stay on the child profile, and non-Parquet scans do not eagerly acquire Parquet counters. No new per-row logging or timers were added.
  • Transactions and persistence: no transaction, EditLog, recovery or persistent state changes; visibility/version and MoW delete-bitmap invariants are unaffected by the reviewed changes.
  • Data writes and crash atomicity: no data-write or commit path changes; resource lifetime changes were reviewed through scanner shutdown and deferred materialization.
  • FE/BE propagation: no new transmitted variable or sending path; the cache is an in-process observer propagated through the relevant reader factories.
  • Performance: initialization work moves from each file to one counter set per scan profile, with one retained Variant bundle reference per copy. Child caches are retained rather than recreated per split. The author's benchmark gains were not independently verified.
  • Other risks and user concerns: cache identity, teardown, missing parallel initialization and deferred ownership risks were dismissed with caller evidence. Both existing review threads were revalidated without duplicate comments; no unresolved code candidate remains.

Coverage is complete for the supplied aggregate diff and affected call chains. Validation was static only: no builds, tests, source modifications or benchmark runs were performed under this review contract.

All three round-1 reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty finding set; main verification dismissed every initial risk and recorded no unresolved candidate. Review converged in one round.

Review converged after 1 round(s).

… files

### What problem does this PR solve?

Issue Number: None

Problem Summary: External small-file scans repeatedly initialize the same Parquet profile counter hierarchy for each file, contending on the shared RuntimeProfile registration lock. FileReader::init also repeats the concrete reader's initialization. Initialize counters once per scan instance and pass the immutable handles through table-reader wrappers to Parquet readers. Keep distinct scan instances separate. Group the shared Variant counters into one retained bundle so deferred materialization can safely outlive the scanner without copying a shared_ptr for every counter.

This change applies to native Parquet reads across Hive, Iceberg, Hudi, Paimon and Fluss table-reader wrappers. It does not add speculative I/O or change the file reader's selection and decoding behavior.

Review follow-up: forward the scanner-owned profile cache through the actual Hudi base-file and Iceberg delete-file Parquet factories. New tests exercise repeated reader construction and shared counter identity, including opening a real delete Parquet file. Validation: full ASAN BE build, 442 affected reader/profile/Variant tests, clang-tidy on changed lines, clang-format 16 and build hygiene passed. The scoped UT harness restores its CMake selection after execution and is not part of this commit.

Review follow-up: retain scan-owned caches for nested Parquet profiles, including the Iceberg position-deletes system table, without attributing counters to the outer profile. Added repeated split/scanner factory coverage and concurrent child-cache isolation tests. ASAN build, 446 scoped unit tests, clang-format 16, header hygiene and clang-tidy 22 passed.

CI follow-up: initialize the Hudi factory test through TableReader::init instead of assigning an incomplete set of internal fields; Linux ASAN exposed the uninitialized scan-parameter pointer. Revalidated 446 scoped ASAN tests, clang-format16 and clang-tidy22. Production behavior is unchanged.

### Release note

Reduce profile initialization and counter ownership overhead when scanning many small Parquet files.

### Check List (For Author)

- Test: Unit Test and Manual Test
    - New profile-cache tests cover concurrent initialization, counter aggregation, independent scan instances, no-profile readers and retained Variant counter lifetime
    - Reader tests cover one initialization per file and sharing across files
    - run-be-ut.sh: 585 tests across 13 Parquet, ORC, Variant and Hive/Iceberg/Hudi/Paimon/table-reader suites passed with ASAN
    - Fresh Linux AArch64 build.sh --be -j3 baseline and optimized RELEASE builds passed
    - clang-format16 and header hygiene passed; preset clang-tidy passed all 19 changed files against the matching ASAN compilation database
    - Isolated Release SQL results and a comparison with identical FE source are reported below
- Behavior changed: No; preserve results and profile counter semantics
- Does this need documentation: No

### Performance validation

Complete SQL medians in milliseconds, matching Linux AArch64 Release BE, single backend, 6 CPUs, warm footer/data cache. Both settings are fixed: `split_batch_size=1`, `split_queue_capacity=4096`. Each single-query result has 40 measured samples in forward/reverse order after identical warmup. Cold denotes catalog/manifest cache invalidation, not a cold JVM or BE data cache. COUNT and SUM results are checked, and post-timing profiles verify footer/page hits.

To isolate the BE change from a pre-existing FE EOF polling race, both columns below use exactly the same FE source containing its notification fix; only the Release BE package changes.

| Warm full-scan scenario | Original BE, same fixed FE | This BE, same fixed FE | Reduction |
|---|---:|---:|---:|
| Hive 1,000 partitions | 60.77 | 16.39 | 73.0% |
| Hive 10,000 partitions | 738.04 | 120.15 | 83.7% |
| Hive one partition / 10,000 files | 495.15 | 74.42 | 85.0% |
| Iceberg 1,000 manifests / 10,000 files | 503.25 | 81.46 | 83.8% |

On unchanged baseline FE, independent BE-only single-query medians were 22.89 ms for Hive 1,000 partitions, 230.37 ms for Hive 10,000 partitions, 117.95 ms for Hive one partition with 10,000 files and 138.38 ms for Iceberg 1,000 manifests. The faster BE exposed an existing FE terminal-queue poll: 78 delayed empty requests in a 400-SQL diagnostic; one finished production at 0.148 ms but returned at 100.911 ms. The companion FE scan/scheduling PR fixes terminal notification, verified with zero such delays on replay. Merge the FE notification fix first when assessing concurrent latency; the BE patch has no code dependency on it.
@924060929
924060929 force-pushed the codex/lake-pr-parquet-profile branch from eb06287 to 024ea81 Compare October 20, 2026 05:56
@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 25 changed files (19 production files and six test files) at 024ea81a94780949e4a26d6ebe4f78f0a1495964. No new actionable findings. The review converged in round 1: both complete-scope reviewers and the dedicated risk reviewer returned NO_NEW_VALUABLE_FINDINGS, and all risk questions and candidate dispositions were independently checked.

Both existing P2 issues are fixed at this head: custom Hudi/Iceberg factories now forward the cache, and nested position-delete system-table readers retain a separate cache for their child RuntimeProfile across splits/scanners. Neither is reposted. No existing P0/P1 blocker remains in the reviewed thread context. No additional user-provided review focus was supplied.

  1. Goal and evidence: the cache registers Parquet handles once per scan/profile; the Variant bundle reduces ownership copies to one shared reference. Added tests inspect identity, aggregation, registration-lock avoidance, repeated files, custom factories and retained counter lifetime.
  2. Scope and design: the changes are focused on counter ownership, forwarding and removal of duplicate initialization. Schema mapping, decoding, predicate/delete semantics and physical file selection are unaffected by the changed logic.
  3. Concurrency and locking: scanner workers share immutable handles published by call_once; child-cache insertion uses its own mutex and counter values remain atomic. Child-map locks are released before profile initialization. No new lock inversion or production thread entry was found.
  4. Lifecycle and static initialization: worker execution pins the fragment context and its RuntimeState/local cache through scanner use and cleanup. Close preserves cache/profile ownership; failed opens finalize, and recursive rebuild creates a new fragment context. Deferred Variant states retain their shared counters after scanner/profile destruction without a cycle. Disabled bundles use function-local initialization; no cross-TU static dependency is introduced.
  5. Configuration: no configuration items are added; dynamic updates are not applicable.
  6. Compatibility: changes are internal source interfaces with standalone defaults. No storage/wire format or dependency/toolchain change requires rolling-upgrade compatibility handling; binary artifacts were not built in this review.
  7. Thrift/protobuf: no definitions are changed, so branch field-ID comparison is not applicable.
  8. Parallel paths: generic, Hudi and Iceberg factories plus Hudi/Paimon/Fluss hybrid, union-lake and tail-reader paths forward the cache with the matching profile. Nested position deletes use child counters; standalone/no-profile paths retain their defaults. Every current concrete reader that needs profile initialization still calls it explicitly.
  9. Conditional checks: the cache-present, standalone and child-ownership branches correspond to concrete lifecycle cases and have explanatory comments; no changed decision suppresses a data error.
  10. Test coverage: six changed BE test files cover the central cache, ownership and factory contracts. Fluss/Paimon forwarding, cancellation and recursive teardown were traced statically. The patch adds no SQL integration tests; no missing coverage was tied to a verified defect.
  11. Test results: no expected-result artifacts change. The new assertions match the inspected identity, accumulation and ownership contracts. Author-reported test and benchmark results were not independently executed.
  12. Observability: counter names, types and hierarchy remain intact. Per-file statistics and published-delta snapshots remain reader-local, while nested counters stay on the child profile; no additional metric family is needed for handle reuse.
  13. Transactions and persistence: no visibility/version, delete-bitmap, EditLog, replay or failover logic changes.
  14. Data writes and crash behavior: no data-writing path changes. New query-scoped owners use RAII and introduce no persistent recovery state.
  15. FE/BE variables: no transmitted field is added; the new option is internal BE plumbing.
  16. Performance: repeated Parquet counter registration and per-counter shared ownership copies are removed. Cache creation is bounded by scan/profile count, with no added per-row work or changed data I/O. Speedup magnitude was not independently measured.
  17. Other invariants: no substantiated change to row results, nullability, error propagation or resource ownership was found. The incomplete-type constructor concern was dismissed by the actual header include chain.

Validation was static only, as required: no builds, tests, sanitizer runs or performance measurements were performed. No changed paths remain uncovered and no candidate or risk question remains unresolved within that scope.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (9/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 66.41% (31772/47841)
Line Coverage 51.19% (334687/653863)
Region Coverage 46.53% (270479/581259)
Branch Coverage 48.26% (127436/264061)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28334 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 024ea81a94780949e4a26d6ebe4f78f0a1495964, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17595	5189	5050	5050
q2	2095	318	289	289
q3	10224	1372	812	812
q4	4673	461	336	336
q5	7431	808	552	552
q6	183	183	151	151
q7	740	803	602	602
q8	9266	1336	996	996
q9	5521	4202	4167	4167
q10	6831	1292	1036	1036
q11	439	247	238	238
q12	638	398	278	278
q13	18114	3623	2997	2997
q14	267	278	240	240
q15	q16	722	704	667	667
q17	1179	765	580	580
q18	6960	6170	6120	6120
q19	1131	956	583	583
q20	371	332	223	223
q21	5212	2160	2227	2160
q22	343	294	257	257
Total cold run time: 99935 ms
Total hot run time: 28334 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5636	5591	5573	5573
q2	667	592	558	558
q3	5183	5426	4972	4972
q4	1983	2142	1565	1565
q5	5263	5165	5513	5165
q6	268	201	152	152
q7	2253	1983	1860	1860
q8	2992	2645	2676	2645
q9	8083	7923	8083	7923
q10	3724	3699	3226	3226
q11	600	582	430	430
q12	700	680	534	534
q13	3525	3840	3239	3239
q14	327	316	305	305
q15	q16	711	743	666	666
q17	8243	7409	6894	6894
q18	12964	12133	12932	12133
q19	902	770	777	770
q20	2151	2168	1916	1916
q21	5591	4568	4706	4568
q22	500	435	421	421
Total cold run time: 72266 ms
Total hot run time: 65515 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150676 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 024ea81a94780949e4a26d6ebe4f78f0a1495964, data reload: false

query5	4304	587	452	452
query6	434	210	194	194
query7	4808	446	221	221
query8	322	163	155	155
query9	8770	3135	3103	3103
query10	469	312	259	259
query11	5957	3858	3568	3568
query12	152	95	86	86
query13	1251	446	317	317
query14	6100	3930	3659	3659
query14_1	3431	3430	3376	3376
query15	211	201	178	178
query16	946	404	402	402
query17	859	676	535	535
query18	2424	432	311	311
query19	190	163	166	163
query20	86	86	86	86
query21	216	125	109	109
query22	12964	12969	12768	12768
query23	13403	12834	12510	12510
query23_1	12573	12430	12629	12430
query24	7346	841	461	461
query24_1	465	458	467	458
query25	505	389	334	334
query26	1271	258	138	138
query27	2769	428	264	264
query28	4652	1924	1943	1924
query29	1563	585	441	441
query30	305	222	191	191
query31	997	934	810	810
query32	149	85	85	85
query33	513	301	238	238
query34	878	844	487	487
query35	717	757	635	635
query36	819	831	735	735
query37	116	105	93	93
query38	1768	1730	1682	1682
query39	796	797	726	726
query39_1	735	722	733	722
query40	213	114	92	92
query41	65	63	63	63
query42	89	84	82	82
query43	342	361	316	316
query44	1282	688	689	688
query45	180	174	161	161
query46	842	953	569	569
query47	2958	2930	2780	2780
query48	298	305	217	217
query49	577	405	294	294
query50	711	271	207	207
query51	10278	10642	10224	10224
query52	78	77	72	72
query53	185	199	149	149
query54	236	234	182	182
query55	72	66	61	61
query56	235	224	207	207
query57	1705	1613	1554	1554
query58	279	261	254	254
query59	2063	2144	1986	1986
query60	282	243	223	223
query61	151	147	146	146
query62	381	336	286	286
query63	195	157	172	157
query64	2736	973	764	764
query65	3310	3262	3282	3262
query66	1779	415	308	308
query67	20150	20175	19895	19895
query68	2913	1013	602	602
query69	381	290	256	256
query70	840	779	752	752
query71	303	251	218	218
query72	2885	2718	2670	2670
query73	519	529	291	291
query74	4596	4501	4347	4347
query75	2336	2298	1935	1935
query76	2313	1040	592	592
query77	357	384	285	285
query78	9831	9692	9132	9132
query79	997	839	503	503
query80	1228	441	350	350
query81	577	318	264	264
query82	579	134	107	107
query83	303	202	177	177
query84	311	124	105	105
query85	871	418	382	382
query86	396	229	240	229
query87	2037	1916	1817	1817
query88	3668	2665	2637	2637
query89	328	284	238	238
query90	1910	164	167	164
query91	161	150	126	126
query92	95	86	93	86
query93	1059	1000	579	579
query94	631	307	276	276
query95	576	349	310	310
query96	646	489	227	227
query97	2422	2408	2301	2301
query98	163	150	157	150
query99	771	763	661	661
Total cold run time: 234831 ms
Total hot run time: 150676 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 024ea81a94780949e4a26d6ebe4f78f0a1495964, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.06
query3	0.26	0.09	0.08
query4	1.61	0.10	0.10
query5	0.24	0.24	0.24
query6	1.17	0.68	0.67
query7	0.04	0.01	0.00
query8	0.06	0.03	0.04
query9	0.40	0.34	0.35
query10	0.62	0.61	0.59
query11	0.19	0.14	0.15
query12	0.18	0.15	0.14
query13	0.48	0.48	0.48
query14	1.03	1.02	1.02
query15	0.63	0.60	0.60
query16	0.34	0.36	0.35
query17	1.16	1.14	1.14
query18	0.20	0.20	0.19
query19	2.03	1.99	1.98
query20	0.02	0.01	0.01
query21	15.48	0.18	0.13
query22	4.91	0.06	0.05
query23	16.28	0.32	0.12
query24	2.65	0.66	0.33
query25	0.10	0.06	0.06
query26	1.23	0.22	0.16
query27	0.05	0.05	0.03
query28	3.25	0.78	0.35
query29	12.45	4.15	3.25
query30	0.28	0.12	0.10
query31	2.77	0.55	0.32
query32	3.22	0.59	0.51
query33	3.17	3.18	3.26
query34	15.25	3.97	3.38
query35	3.35	3.35	3.34
query36	0.57	0.43	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.28 s
Total hot run time: 24.23 s

@Gabriel39
Gabriel39 merged commit 7073c51 into apache:master Oct 20, 2026
50 of 51 checks passed
@924060929
924060929 deleted the codex/lake-pr-parquet-profile branch October 20, 2026 08:50
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.23% (35800/46357)
Line Coverage 62.52% (404392/646867)
Region Coverage 59.00% (341701/579161)
Branch Coverage 59.92% (157843/263418)

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

yiguolei pushed a commit that referenced this pull request Oct 29, 2026
xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Merging master 7acfd9b left two Delta gaps. First,
apache#69717 replaced table_reader.h's full ShardedKVCache definition with a
forward declaration, but deltalake_reader_test.cpp constructs a
ShardedKVCache and relied on the transitive include. Include
format/format_common.h directly, as hudi_reader_test.cpp does. Second,
apache#69300 made Parquet readers in one scan share their profile counters
through TableReader's parquet profile cache. Hudi, Iceberg and the base
reader pass that cache, but DeltaReader still built its ParquetReader
without it, so every Delta file created its own set of counters. Pass
the cache, and keep binary-UUID preservation disabled as before.

### Release note

None

### Check List (For Author)

- Test: No local build. The merge was conflict-free. Both changes follow
  the matching master code. COMPILE and BE UT are left to CI.
- Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do.
- Does this need documentation: No
xylaaaaa added a commit to xylaaaaa/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: Merging master 7acfd9b left two Delta gaps. First,
apache#69717 replaced table_reader.h's full ShardedKVCache definition with a
forward declaration, but deltalake_reader_test.cpp constructs a
ShardedKVCache and relied on the transitive include. Include
format/format_common.h directly, as hudi_reader_test.cpp does. Second,
apache#69300 made Parquet readers in one scan share their profile counters
through TableReader's parquet profile cache. Hudi, Iceberg and the base
reader pass that cache, but DeltaReader still built its ParquetReader
without it, so every Delta file created its own set of counters. Pass
the cache, and keep binary-UUID preservation disabled as before.

### Release note

None

### Check List (For Author)

- Test: No local build. The merge was conflict-free. Both changes follow
  the matching master code. COMPILE and BE UT are left to CI.
- Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do.
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.2.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants