Skip to content

[feature](be) Add session controls for Parquet page indexes - #69971

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/parquet-write-page-index
Nov 4, 2026
Merged

Gabriel39 merged 1 commit into
apache:masterfrom
Gabriel39:dev/parquet-write-page-index

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Parquet output relies on implicit page-index and data-page-size defaults, so users cannot explicitly omit indexes or tune the target page size.

Add two validated session variables shared by OUTFILE, EXPORT, Hive and Iceberg Parquet writers:

  • enable_parquet_write_page_index: write ColumnIndex/OffsetIndex metadata; defaults to true, preserving the current master behavior.
  • parquet_write_data_page_size: target uncompressed data-page size in bytes; defaults to 1048576, accepts positive 32-bit integers, and applies with indexes enabled or disabled.

Keep the existing 128 KiB target cap for Hadoop-framed LZ4. Other codecs use the requested target. Row-group size, dictionary-page size, and data-page/format versions are unchanged. Optional query fields preserve existing defaults when omitted by older FEs.

Release note

Add session controls for Parquet page-index output and target data-page size.

Validation

  • 16 BE ASAN tests passed, including real file metadata/readback, index on/off, dictionary on/off, null/all-null columns, Parquet versions, Data Page V1/V2, page offset bounds, Hadoop-framed LZ4 limits, and Hive/Iceberg partition writers.
  • 9 FE tests passed for defaults, variable forwarding, Thrift serialization, valid bounds, and invalid input.
  • The new tests failed for the missing settings before implementation.
  • FE Checkstyle, clang-format 16, build hygiene, and clang-tidy passed.
  • No cluster SQL run or performance benchmark was performed locally; CI validation is pending.

Check List (For Author)

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Problem Summary: Parquet output currently relies on implicit page-index and data-page-size defaults. Add validated session variables for controlling ColumnIndex/OffsetIndex output and the target data-page size across OUTFILE, EXPORT, Hive and Iceberg writes. Preserve the existing enabled index default, 1 MiB page target and Hadoop-framed LZ4 compatibility limit.

### Release note

Add enable_parquet_write_page_index (default true) and parquet_write_data_page_size (default 1048576 bytes) for new Parquet output.

### Check List (For Author)

- Test: 16 BE ASAN tests and 9 FE tests passed; clang-format 16, build hygiene, FE Checkstyle and clang-tidy passed.
- Behavior changed: Yes; users can explicitly disable page indexes and adjust the target data-page size. Existing defaults are preserved.
- Does this need documentation: Yes; paired English and Chinese Parquet documentation.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41404136298

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28790 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b209062081937f8f0b5303341b754539d0be957, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17584	5191	5084	5084
q2	2142	314	253	253
q3	10244	1406	853	853
q4	4684	458	339	339
q5	7431	803	546	546
q6	179	185	156	156
q7	763	794	591	591
q8	9261	1375	1051	1051
q9	5455	4212	4201	4201
q10	6844	1312	1039	1039
q11	437	253	240	240
q12	641	407	273	273
q13	18094	3936	3012	3012
q14	272	272	253	253
q15	q16	745	727	666	666
q17	1220	741	580	580
q18	7018	6166	6147	6147
q19	1117	957	594	594
q20	389	339	223	223
q21	5792	2467	2413	2413
q22	376	319	276	276
Total cold run time: 100688 ms
Total hot run time: 28790 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6078	5945	6102	5945
q2	756	589	604	589
q3	5615	5800	5342	5342
q4	2030	2175	1565	1565
q5	5803	5891	5617	5617
q6	242	194	144	144
q7	2245	2039	1817	1817
q8	3039	2668	2710	2668
q9	7999	7642	7469	7469
q10	3620	3594	3161	3161
q11	590	418	382	382
q12	665	671	502	502
q13	3315	3647	3055	3055
q14	285	285	265	265
q15	q16	665	721	612	612
q17	7792	7143	6896	6896
q18	13028	12189	12919	12189
q19	899	779	783	779
q20	2152	2157	1939	1939
q21	6160	4688	4712	4688
q22	511	457	393	393
Total cold run time: 73489 ms
Total hot run time: 66017 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151775 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b209062081937f8f0b5303341b754539d0be957, data reload: false

query5	4304	585	438	438
query6	429	204	194	194
query7	4816	438	237	237
query8	323	168	154	154
query9	8770	3197	3228	3197
query10	496	317	257	257
query11	6024	3870	3591	3591
query12	140	93	86	86
query13	1254	456	339	339
query14	6076	3922	3646	3646
query14_1	3502	3674	3406	3406
query15	214	199	181	181
query16	939	442	420	420
query17	875	642	531	531
query18	2415	408	296	296
query19	193	158	133	133
query20	81	82	80	80
query21	210	127	106	106
query22	12994	13117	12905	12905
query23	13444	12820	12540	12540
query23_1	12585	12598	12427	12427
query24	7344	850	474	474
query24_1	475	474	482	474
query25	507	393	334	334
query26	1256	258	135	135
query27	2787	437	276	276
query28	4624	2028	2008	2008
query29	1521	582	448	448
query30	295	215	185	185
query31	998	926	796	796
query32	142	86	88	86
query33	520	292	233	233
query34	894	849	495	495
query35	742	746	681	681
query36	817	819	702	702
query37	132	98	82	82
query38	1765	1727	1666	1666
query39	788	791	742	742
query39_1	735	740	737	737
query40	218	118	101	101
query41	74	68	77	68
query42	89	85	84	84
query43	348	370	314	314
query44	1343	707	697	697
query45	180	176	173	173
query46	836	966	569	569
query47	2974	2869	2688	2688
query48	304	320	229	229
query49	583	403	300	300
query50	686	286	221	221
query51	10490	10447	10398	10398
query52	84	82	71	71
query53	191	215	157	157
query54	262	204	195	195
query55	73	72	67	67
query56	234	222	219	219
query57	1667	1623	1564	1564
query58	287	276	278	276
query59	2156	2186	2017	2017
query60	299	253	237	237
query61	170	164	168	164
query62	396	335	283	283
query63	199	166	169	166
query64	2838	1071	851	851
query65	3351	3305	3305	3305
query66	1752	414	294	294
query67	20227	20435	20210	20210
query68	3333	1022	627	627
query69	392	292	251	251
query70	844	787	812	787
query71	308	234	204	204
query72	2824	2664	2341	2341
query73	534	525	310	310
query74	4679	4495	4381	4381
query75	2359	2289	1899	1899
query76	2404	1008	597	597
query77	339	351	273	273
query78	9820	9908	9201	9201
query79	1006	867	540	540
query80	1340	445	343	343
query81	594	321	272	272
query82	592	135	111	111
query83	291	207	197	197
query84	311	119	103	103
query85	1042	442	362	362
query86	401	236	215	215
query87	2002	1940	1812	1812
query88	3845	2752	2768	2752
query89	339	284	245	245
query90	1883	174	177	174
query91	171	153	135	135
query92	102	89	98	89
query93	1050	1027	584	584
query94	635	310	266	266
query95	569	342	327	327
query96	690	512	239	239
query97	2455	2460	2335	2335
query98	166	155	145	145
query99	788	762	641	641
Total cold run time: 236599 ms
Total hot run time: 151775 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.2 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b209062081937f8f0b5303341b754539d0be957, data reload: false

query1	0.01	0.00	0.00
query2	0.14	0.09	0.08
query3	0.35	0.13	0.14
query4	1.61	0.12	0.13
query5	0.26	0.26	0.24
query6	1.15	0.65	0.64
query7	0.04	0.01	0.00
query8	0.08	0.07	0.08
query9	0.48	0.35	0.35
query10	0.59	0.60	0.58
query11	0.31	0.17	0.17
query12	0.33	0.19	0.18
query13	0.52	0.51	0.52
query14	0.95	0.93	0.93
query15	0.66	0.58	0.58
query16	0.36	0.38	0.35
query17	1.04	1.05	1.04
query18	0.28	0.28	0.27
query19	1.96	1.91	1.87
query20	0.02	0.01	0.01
query21	15.41	0.35	0.30
query22	4.92	0.12	0.13
query23	15.86	0.50	0.29
query24	2.32	0.50	0.38
query25	0.15	0.10	0.10
query26	0.75	0.26	0.21
query27	0.10	0.10	0.10
query28	3.63	0.83	0.38
query29	12.66	4.33	3.42
query30	0.34	0.17	0.16
query31	2.76	0.59	0.33
query32	3.23	0.62	0.48
query33	3.05	3.00	2.95
query34	15.90	4.05	3.45
query35	3.36	3.40	3.37
query36	0.59	0.53	0.51
query37	0.12	0.09	0.08
query38	0.07	0.06	0.07
query39	0.06	0.06	0.06
query40	0.19	0.18	0.16
query41	0.10	0.05	0.06
query42	0.06	0.06	0.06
query43	0.07	0.06	0.06
Total cold run time: 96.84 s
Total hot run time: 25.2 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.70% (35889/46191)
Line Coverage 63.05% (403722/640312)
Region Coverage 59.64% (341634/572808)
Branch Coverage 60.75% (158109/260265)

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No actionable defect was found in the five changed files at head 3b20906. No existing inline comments require retention, and no additional user focus was supplied.

The reviewed flow is:

SET -> SessionVariable -> forwarding/session snapshot -> toThrift/TQueryOptions
    -> coordinator -> RuntimeState -> common VParquetWriter properties -> output file
EXPORT captures the session and runs generated OUTFILE statements through this flow.
Hive, Iceberg and native Parquet TVF sinks share the common writer.

Applicable critical-checkpoint conclusions:

Checkpoint Conclusion
Goal and proof The two controls reach the common writer. Added FE tests inspect defaults, bounds and forwarding; BE tests inspect real metadata and readback. These tests were inspected, not executed.
Scope and reuse The implementation is focused: existing session reflection, query-options transport, compression helper and Arrow property builder are reused.
Concurrency and locking Values are captured into query options and read during per-file open. No new threads, shared mutable state, lock order or heavy locked operation is introduced.
Lifecycle and static initialization Existing writer, converter and output-stream ownership/close paths remain. The new constant is constexpr; there is no cross-translation-unit initialization dependency or new resource cycle.
Dynamic configuration Checked SET values affect subsequent statements without restart. Existing writers use their captured settings. Page size rejects zero, negative, nonnumeric and values above INT_MAX before assignment.
Format and rolling compatibility Data-page/format versions, row-group policy and dictionary limit remain unchanged. Optional absent fields retain true/1048576 on new BEs; old BEs ignore the new fields and cannot implement the new controls. Defaults match the tagged Arrow 24 properties.
Thrift IDs Read TQueryOptions on HEAD and live-verified branch-3.1, 4.0, 4.1 and 4.2. Added IDs 238/239 are unused on every release branch, optional, and have explicit defaults. The existing field237 mismatch between master and branch-4.1 predates this PR and is excluded from new findings.
Parallel paths OUTFILE, EXPORT, Hive, Iceberg and native Parquet TVF writers receive the shared policy. Point-query and expression-only constant-folding paths were checked for relevance; no missing Parquet-writer sender was found.
Conditions and error handling The index toggle is explicit; the documented Hadoop LZ4 branch preserves min(requested,131072). The positive-size assertion matches checked FE inputs and the nonzero Thrift default. No ignored Status or added exception-boundary issue was found.
Test coverage Added tests cover index on/off, dictionary on/off, format versions, V1/V2 data pages, nullable/all-null values, metadata offset bounds, LZ4 target properties and Hive/Iceberg partition writers. FE tests cover valid/invalid bounds and map forwarding. SQL end-to-end and binary Thrift round-trip coverage are absent from the additions.
Expected test results New counts, null counts and metadata expectations match the generated input by inspection. The FE test named ThriftSerialization constructs options; it does not execute a binary round trip. No golden output files changed.
Observability Settings and output-file metadata expose the policy; existing error reporting remains. No new asynchronous lifecycle requires additional tracing or metrics.
Transactions and persistence Generic annotated session persistence includes both primitive fields; missing saved keys retain defaults. EXPORT captures a deep clone and is cancelled on replay/failover. Transaction visibility, version and delete-bitmap paths are unchanged.
Writes, atomicity and crashes File properties change within existing sink open/write/close and commit/abort protocols. No new ownership, crash-cleanup or atomicity defect was found.
FE-to-BE propagation Both fields are set in common toThrift, marked needForward, copied through coordinators and RuntimeState, and consumed by common writer setup. Export worker contexts retain the captured session.
CPU and memory Added setup work is constant per file and preserves the Arrow memory pool. Small and large page targets deliberately trade page overhead against buffering; size remains a target checked at batch boundaries, with Arrow's row limit retained. No concrete unintended performance regression was found.
Further recheck Checked absent-index sequential fallback, page/header statistics, Iceberg split offsets, dictionary offsets, subclass overrides, export snapshots and extreme target handling. No unresolved static correctness candidate remains.

BE header-hygiene review found no hub-header, PCH, explicit-instantiation, gate-table or source-.cpp test inclusion change. No new libc API or linker change is introduced.

Coverage is complete for the assigned static review. Builds, FE/BE tests, cluster SQL, mixed-version execution, external JVM-reader interoperability, stress tests and benchmarks were not run; the author's reported validation was not reproduced. These are validation limits, with no unresolved candidate withheld from the inline set.

Review converged after 2 round(s).

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parquet output currently uses shared writer defaults. This PR carries two session controls through TQueryOptions to the common writer, preserving default behavior and the Hadoop LZ4 page-size cap.

Independent static review of head 3b20906 found no actionable P0/P1/P2 defects within the FE scope and necessary FE/BE boundaries. Checked validation, SET_VAR restoration, FE forwarding, EXPORT session snapshots, coordinator transport, shared writer consumers, optional-field defaults, release-branch field IDs, and added tests. The implementation uses the appropriate shared abstractions.

APPROVE for the reviewed scope. No builds or tests were run. The supplied current-head snapshot reports successful P0/cloud regressions and FE/BE unit tests, but a failed FE coverage check; required-check rules are unavailable, so this is not a merge-readiness assertion. The existing automated review reports no actionable defects and agrees with this independent assessment.

Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.

@Gabriel39
Gabriel39 merged commit 035a843 into apache:master Nov 4, 2026
43 of 44 checks passed
Gabriel39 added a commit to apache/doris-website that referenced this pull request Nov 4, 2026
Describe the new Parquet output session controls in the English and
Chinese current documentation: `enable_parquet_write_page_index`
(default `true`) and `parquet_write_data_page_size` (default 1 MiB).
Include an OUTFILE example, disabling/restoring the settings,
Hive/Iceberg/EXPORT scope, the Hadoop-framed LZ4 page target cap, and
tuning tradeoffs.

Implementation: apache/doris#69971

Version scope: current/Dev only. These controls are newly introduced on
master and are not available in released 4.x or 3.x, so their versioned
pages are intentionally unchanged.

Validation: both changed pages compile as Docusaurus-compatible MDX;
`git diff --check` passes. Changed-link lint reports two existing
`parquet_metadata_size_limit` anchor findings already present before
this change; the new OUTFILE links resolve. I18n lint flags the
intentionally current-only scope described above. A full website build
was not run.
morningman added a commit that referenced this pull request Nov 4, 2026
…berg WriteInfo (#70209)

### What problem does this PR solve?

Issue Number: None

Related PR: #69392, #69971

Problem Summary:

**In short.** `doris_be_test` does not compile on master since #69971
was merged, so the BE UT of every PR fails with `BE BUILD FAIL`. This
fixes the one test case that breaks it.

**The problem.** #69392 replaced the `write_path`, `original_write_path`
and `target_path` fields of `IPartitionWriterBase::WriteInfo` with one
`IcebergOutputLocation location`, and updated the callers it knew about.
#69971, merged after it, added
`VParquetWriterTest.HiveAndIcebergParquetPageIndexOptions`, which still
sets the old fields:

```
be/test/format/transformer/vparquet_writer_test.cpp:380:41: error: no matching constructor for initialization of 'VIcebergPartitionWriter'
```

Each PR built alone, so both were green. TeamCity's BE UT has failed
this way for PRs merged with master `0eb5131db62` (for example build
1092024 of #69605).

**How this PR fixes it.** The case builds its `WriteInfo` as the other
Iceberg case in the same file does, `.location =
IcebergOutputLocation(".", ".")`. Nothing else changes.
yiguolei pushed a commit that referenced this pull request Nov 5, 2026
…/JSON/WAL fixes (#70185)

### What problem does this PR solve?

Backport the following master PRs to `branch-4.2`, in master merge
order:

1. #69672 — Preserve source values and alias semantics in ES scans.
2. #69971 — Add session controls for Parquet page indexes and data-page
size.
3. #70155 — Repair ES source predicate regression cases.
4. #70079 — Fix ORC condition-cache seeks past filtered stripe suffixes.
5. #70154 — Preserve JSON and WAL rows with empty file projections.

The ES changes are adapted to the existing 4.2 FE planner and BE source
locations. Mapping safety, scan-wide read mode, local residuals, native
`esquery` handling and alias routing retain the upstream behavior
without introducing the master connector framework. Added adapter tests
cover both scan protocols, aliases, missing-value negations and native
residual rejection.

Preserve existing branch-specific ES regression coverage. Bring in the
previously absent predicate suite, adapting its plan assertions to 4.2's
`LOCAL_PREDICATES` output and retaining LIMIT result checks. Parquet
variables use the 4.2 annotation interface and the original optional
Thrift field IDs. The ORC dependency is pinned to the same fixed commit
as #70079.

The ES implementation also preserves source numeric and JSONB predicate
semantics after scan-wide fallback. Numeric-token rounding must not make
a remote predicate discard a row matching the value Doris materializes;
empty JSONB objects remain nonnull even when ES indexes no leaf.
Docvalue eligibility and pushdown are retained.

Mapped date formats and their order are carried per concrete backing
index through the existing property map to both scan protocols.
Calendar/epoch alternatives now distinguish basic_date from
epoch_millis, for strings and source numeric tokens. Numeric docvalues
and older FE numeric requests retain their epoch decoder. Typed alias
selection consistently uses the requested mapping type in schema
inference and planning. The pinned ORC decoder's libdeflate dependency
is included on ARM, matching master.

### Release note

Fix ES source-value filtering, mapped dates and typed alias query
correctness, ORC condition-cache seek crashes/missing rows and its ARM
link dependency, and JSON/WAL row counts for empty file projections. Add
Parquet output page-index and data-page-size session controls.

### Validation

- FE compilation and Checkstyle passed. ES validation: 81 tests (48
datasource/planner tests and 33 legacy ES tests), 0 failures/errors, 1
external-environment skip. The 9 Parquet session-variable tests passed.
- Rebuilt the pinned ORC dependency under AddressSanitizer: 56 focused
tests passed, including 32 filtered-stripe-suffix seek cases.
- Compiled all 10 affected Doris C++ implementation/test translation
units and the ORC fixture generator. The regenerated fixture is
byte-for-byte identical to the committed fixture.
- clang-format 16 checks passed for all affected Doris C++ sources and
headers. Modified Groovy suites compiled successfully.
- The ES reader/parser implementations and test translation units
compiled. The shared DATETIMEV2 date-format algorithm passed 21
standalone ASAN checks. CMake dependency checks passed for aarch64,
arm64 and x86_64; an actual ARM link was not run locally.
- Added `test_es_source_review_cases.groovy` with SQL-to-SQL checks
against an internal reference table for numeric/JSONB source predicates,
LIMIT, epoch-only formats, and explicit legacy DATETIMEV2 basic_date
columns. It covers both scanners and parallel-scroll settings, waits for
legacy metadata readiness, and restores session settings. The suite
compiled; no golden files were handwritten.
- The applicable ES fixes are synchronized to master in #70205. Master
already links libdeflate on every architecture.
- Full Doris BE unit-test execution and live external SQL regression
were not performed locally; CI validation is requested.

### Check List (For Author)

- Test
  - [x] Unit Test
  - [x] Regression tests included
- Behavior changed:
  - [x] Yes; the upstream behavior is backported to 4.2.
- Does this need documentation?
- [x] Parquet controls are documented by the upstream documentation PR:
apache/doris-website#4581

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants