[fix](iceberg) Support GZIP for iceberg/hive Parquet writers. by hubgeter · Pull Request #68854 · apache/doris

hubgeter · 2026-10-10T03:27:12Z

What problem does this PR solve?

Issue Number: N/A

Related PR: #64723

Problem Summary:

Iceberg tables with write.parquet.compression-codec = gzip (the default before Iceberg 1.4) and Hive
tables with parquet.compression = GZIP could not be written by Doris: the BE Parquet writers only
accepted PLAIN/SNAPPY/ZSTD/LZ4 and failed with [INTERNAL_ERROR]Unsupported compress type GZ with parquet, while Spark/Trino wrote the same table fine.

BE: map TFileCompressType.GZ to TParquetCompressionType.GZIP, and move the whole
TFileCompressType -> TParquetCompressionType mapping into ParquetBuildHelper so the Iceberg
data-file writer, the Iceberg delete-file writer and the Hive Parquet writer share one copy.
FE: accept gzip for Parquet in the Hive CREATE TABLE compression allow-list.

Tests: the Iceberg Parquet-compression suite now writes and reads gzip data and merge-on-read delete
files and checks the codec with parquet_meta(); the Hive DDL suite adds gzip to the Parquet
compression matrix; FE unit tests cover the gzip allow-list and the codec mapping.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes. Iceberg/Hive Parquet writes with GZIP no longer fail; the file is written with the standard Parquet GZIP codec, the same codec Spark/Trino write for that table property.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-10-10T03:27:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

The BE Parquet writers only accepted PLAIN/SNAPPY/ZSTD/LZ4 and rejected the GZ codec that Iceberg's write.parquet.compression-codec=gzip (and Hive's parquet.compression=GZIP) map to, so INSERT/DELETE failed with 'Unsupported compress type GZ with parquet' while Spark/Trino wrote the same table fine. Factor the whole TFileCompressType -> TParquetCompressionType mapping into ParquetBuildHelper (GZ -> GZIP included) so the Iceberg data-file writer, the Iceberg delete-file writer and the Hive Parquet writer share one definition instead of three copies of the switch, and accept gzip in the Hive CREATE TABLE parquet compression allow-list.

hubgeter · 2026-10-10T06:56:32Z

run buildall

hubgeter · 2026-10-10T07:07:40Z

/review

github-actions

Complete static review of PR #68854 at head 500cad6 after two converged review rounds. One low-priority test-standard finding is inline; no production correctness defect or existing P0/P1 thread was confirmed. There were no additional user focus points.

Critical checkpoints:

Goal and proof: Hive CREATE accepts Parquet gzip; FE sends the existing GZ enum, and BE selects Arrow Parquet GZIP for Hive, Iceberg data, and format-v2 position-delete files. Added FE unit and external regression cases cover this intent, but this review did not execute them.
Scope and parallel paths: The shared helper preserves PLAIN, SNAPPY, ZSTD, and LZ4 mappings. Iceberg INSERT/OVERWRITE/REWRITE, DELETE, and UPDATE/MERGE use the appropriate table, delete, and merge sink fields; no omitted parallel codec path was found.
Concurrency and lifecycle: The change adds no shared mutable state, thread, lock, or static initializer. Existing writer open/close, created-file cleanup, commit-data reporting, and failure propagation were traced; no newly reachable leak, deadlock, or invalid commit was substantiated.
Configuration, compatibility, and persistence: No new config, thrift/protobuf field, FE-to-BE variable, edit-log state, or transaction protocol is introduced. The existing GZ wire value and standard Parquet GZIP codec are used; no new mixed-version data-corruption path was found.
Conditions, errors, and data writes: Unsupported codecs still return non-OK status; valid gzip reaches the existing atomic write/commit flow. The ORC gzip regression case is intentionally excluded because ORC uses zlib.
Tests and results: The new SQL checks read rows and inspect Parquet footer codec. Their deterministic expected results use assertions instead of generated .out cases; the inline P3 asks for the repository-required oracle. No changed expected-output file was presented as executed evidence.
Observability, memory, and performance: The mapping runs once per file, adds no hot-row work or new significant allocation, and existing status/log paths are sufficient. No new external request or dynamic artifact loading was introduced.

Review was static by the runner contract: no builds, tests, or source modifications were performed. The final sweep covered all 10 changed paths and found no unresolved candidate.

github-actions · 2026-10-10T07:19:54Z

+                        "mode" = "parquet_metadata"
+                    )
+                """
+                assertEquals("GZIP", gzipCodec[0][0].toString().toUpperCase())


[P3] Record deterministic GZIP query results in generated .out files

These new row and footer checks use assertEquals for fixed SQL results, but the repository's regression-test standard requires qt/order_qt cases with .out generated by the test runner. The neighboring LZ4 codec checks already use that pattern. Please convert this Hive GZIP check and the new Iceberg GZIP row/footer checks to recorded queries, with ORDER BY where needed, and generate their expected outputs through the regression script.

hello-stephen · 2026-10-10T07:49:58Z

TPC-H: Total hot run time: 29256 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 500cad6dc5b3cd4fbe448b7aa63aa2bc365427bc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17602	5679	5547	5547
q2	2094	309	276	276
q3	10238	1402	856	856
q4	4678	477	347	347
q5	7433	798	543	543
q6	188	193	160	160
q7	767	804	613	613
q8	9249	1346	1073	1073
q9	5777	4482	4476	4476
q10	6822	1314	1031	1031
q11	446	264	239	239
q12	637	405	282	282
q13	18043	3053	2405	2405
q14	284	263	246	246
q15	q16	739	744	694	694
q17	1188	738	573	573
q18	7151	6239	6183	6183
q19	1102	951	588	588
q20	391	338	234	234
q21	5443	2570	2599	2570
q22	406	361	320	320
Total cold run time: 100678 ms
Total hot run time: 29256 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6524	6476	6513	6476
q2	722	614	575	575
q3	5538	5769	5195	5195
q4	2302	2178	1578	1578
q5	5794	5702	5726	5702
q6	248	194	153	153
q7	2250	1984	1815	1815
q8	3064	2777	2814	2777
q9	8288	7774	7750	7750
q10	3659	3605	3216	3216
q11	607	402	376	376
q12	662	694	492	492
q13	2768	3086	2369	2369
q14	281	285	275	275
q15	q16	669	680	619	619
q17	7790	6965	6860	6860
q18	13116	12209	13099	12209
q19	905	789	803	789
q20	2174	2185	1959	1959
q21	5839	4715	4894	4715
q22	515	487	407	407
Total cold run time: 73715 ms
Total hot run time: 66307 ms

hello-stephen · 2026-10-10T07:59:46Z

TPC-DS: Total hot run time: 151855 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 500cad6dc5b3cd4fbe448b7aa63aa2bc365427bc, data reload: false

query5	4312	585	451	451
query6	432	214	198	198
query7	4799	450	236	236
query8	323	187	172	172
query9	8770	3974	3960	3960
query10	480	333	276	276
query11	5903	3430	3192	3192
query12	148	90	86	86
query13	1254	437	316	316
query14	6548	4783	4445	4445
query14_1	4217	4250	4256	4250
query15	208	204	189	189
query16	975	480	460	460
query17	862	697	535	535
query18	2419	425	317	317
query19	207	162	138	138
query20	83	81	77	77
query21	206	135	115	115
query22	12923	13025	12786	12786
query23	13272	12668	12280	12280
query23_1	12344	12287	12246	12246
query24	7098	829	466	466
query24_1	473	469	476	469
query25	529	427	367	367
query26	1259	264	143	143
query27	2751	444	269	269
query28	4619	1942	1928	1928
query29	1572	597	468	468
query30	301	227	186	186
query31	909	767	657	657
query32	148	88	90	88
query33	520	304	248	248
query34	947	839	504	504
query35	757	760	651	651
query36	822	804	780	780
query37	127	100	90	90
query38	1791	1766	1737	1737
query39	725	709	673	673
query39_1	667	692	665	665
query40	230	120	140	120
query41	67	63	62	62
query42	85	82	84	82
query43	363	384	334	334
query44	1273	684	669	669
query45	185	181	159	159
query46	840	960	561	561
query47	3028	3022	2898	2898
query48	304	303	213	213
query49	554	391	289	289
query50	680	286	216	216
query51	10376	10499	10295	10295
query52	79	80	67	67
query53	185	209	158	158
query54	243	185	170	170
query55	75	68	61	61
query56	225	206	206	206
query57	1591	1567	1526	1526
query58	278	256	248	248
query59	2353	2370	2114	2114
query60	278	234	229	229
query61	148	137	138	137
query62	397	341	282	282
query63	196	161	168	161
query64	2750	928	768	768
query65	3409	3362	3395	3362
query66	1770	405	298	298
query67	20099	19956	19918	19918
query68	2912	973	595	595
query69	375	276	241	241
query70	925	831	830	830
query71	293	226	210	210
query72	2761	2553	2293	2293
query73	516	518	284	284
query74	4615	4491	4300	4300
query75	2332	2295	1951	1951
query76	2290	1016	600	600
query77	362	388	293	293
query78	9039	8992	8486	8486
query79	1008	844	501	501
query80	1082	456	330	330
query81	572	323	270	270
query82	578	140	110	110
query83	300	190	171	171
query84	300	119	100	100
query85	835	409	400	400
query86	352	238	232	232
query87	2012	1962	1813	1813
query88	3584	2693	2657	2657
query89	338	293	253	253
query90	1820	181	177	177
query91	161	147	124	124
query92	95	87	87	87
query93	972	986	550	550
query94	530	312	274	274
query95	589	336	382	336
query96	661	523	228	228
query97	2451	2415	2323	2323
query98	160	152	145	145
query99	783	768	651	651
Total cold run time: 234133 ms
Total hot run time: 151855 ms

hello-stephen · 2026-10-10T08:04:36Z

ClickBench: Total hot run time: 24.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 500cad6dc5b3cd4fbe448b7aa63aa2bc365427bc, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.04
query3	0.27	0.12	0.12
query4	1.61	0.13	0.13
query5	0.28	0.25	0.24
query6	1.17	0.68	0.69
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.43	0.36	0.35
query10	0.59	0.58	0.63
query11	0.21	0.14	0.14
query12	0.19	0.14	0.14
query13	0.48	0.50	0.49
query14	0.97	0.97	0.97
query15	0.62	0.60	0.63
query16	0.34	0.34	0.36
query17	1.13	1.15	1.12
query18	0.23	0.21	0.20
query19	2.11	2.01	2.06
query20	0.02	0.01	0.02
query21	15.48	0.28	0.14
query22	4.50	0.06	0.05
query23	16.28	0.32	0.12
query24	2.69	0.64	0.86
query25	0.10	0.07	0.05
query26	1.85	0.25	0.15
query27	0.05	0.04	0.04
query28	2.60	0.61	0.39
query29	12.56	4.13	3.27
query30	0.28	0.13	0.14
query31	2.77	0.58	0.34
query32	3.22	0.61	0.51
query33	3.19	3.18	3.19
query34	15.40	4.01	3.41
query35	3.33	3.33	3.37
query36	0.55	0.44	0.45
query37	0.10	0.06	0.06
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.23 s
Total hot run time: 24.75 s

hello-stephen · 2026-10-10T10:12:45Z

BE UT Coverage Report

Increment line coverage 35.29% (6/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	65.24% (30944/47434)
Line Coverage	50.04% (325451/650336)
Region Coverage	45.48% (263093/578462)
Branch Coverage	47.14% (123421/261804)

hello-stephen · 2026-10-10T10:19:08Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-10T10:20:31Z

BE Regression && UT Coverage Report

Increment line coverage 35.29% (6/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.77% (35247/45913)
Line Coverage	61.98% (398544/642989)
Region Coverage	58.42% (336535/576078)
Branch Coverage	59.28% (154712/261003)

hello-stephen · 2026-10-10T10:28:58Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/646) 🎉
Increment coverage report
Complete coverage report

…#68854) Issue Number: N/A Related PR: apache#64723 Problem Summary: Iceberg tables with `write.parquet.compression-codec = gzip` (the default before Iceberg 1.4) and Hive tables with `parquet.compression = GZIP` could not be written by Doris: the BE Parquet writers only accepted PLAIN/SNAPPY/ZSTD/LZ4 and failed with `[INTERNAL_ERROR]Unsupported compress type GZ with parquet`, while Spark/Trino wrote the same table fine. - BE: map `TFileCompressType.GZ` to `TParquetCompressionType.GZIP`, and move the whole `TFileCompressType` -> `TParquetCompressionType` mapping into `ParquetBuildHelper` so the Iceberg data-file writer, the Iceberg delete-file writer and the Hive Parquet writer share one copy. - FE: accept `gzip` for Parquet in the Hive `CREATE TABLE` compression allow-list. Tests: the Iceberg Parquet-compression suite now writes and reads gzip data and merge-on-read delete files and checks the codec with `parquet_meta()`; the Hive DDL suite adds gzip to the Parquet compression matrix; FE unit tests cover the gzip allow-list and the codec mapping. None - Test  - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes.  Iceberg/Hive Parquet writes with GZIP no longer fail; the file is written with the standard Parquet GZIP codec, the same codec Spark/Trino write for that table property. - Does this need documentation? - [x] No. - [ ] Yes.  - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? Issue Number: N/A Related PR: apache#66348, apache#68854, apache#69376 Problem Summary: Partition writers call open before registering a new writer in the partition map. If the output file is created but the Parquet or ORC transformer then fails to open, the table writer cannot reach that writer through its close path. Clean up the created file at the open failure boundary so the error cannot leave an orphan, matching the lifecycle hardening introduced on master by apache#66348. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [x] Manual test: repository clang-format check and direct compilation of the modified BE writer translation unit passed. - Behavior changed: - [x] No. - Does this need documentation? - [x] No.

#69338) pick #68854

Issue Number: N/A Related PR: apache#68854, apache#69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path. None - Test - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it. - [ ] Regression test - [ ] Manual test - [ ] No need to test or manual test. Explain why - Behavior changed: - [x] No. - Does this need documentation? - [x] No.

) ### What problem does this PR solve? Issue Number: N/A Related PR: #68854, #69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling `create_file` so invalid codecs fail without creating any output file, and add a regression test for the failure path. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test: added `UnsupportedParquetCodecDoesNotCreateFile`; CI will run it. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

hubgeter requested review from 924060929 and CalvinKirs as code owners October 10, 2026 03:27

hubgeter force-pushed the fix_iceberg_hive_parquet_gzip branch from 749af5f to 500cad6 Compare October 10, 2026 06:48

github-actions Bot reviewed Oct 10, 2026

View reviewed changes

924060929 approved these changes Oct 12, 2026

View reviewed changes

Gabriel39 merged commit dda6861 into apache:master Oct 12, 2026
42 of 43 checks passed

Gabriel39 added the dev/4.1.x label Oct 12, 2026

github-actions Bot added the dev/4.1.x-conflict label Oct 12, 2026

Gabriel39 added dev/4.2.x and removed dev/4.1.x dev/4.1.x-conflict dev/4.2.x labels Oct 15, 2026

github-actions Bot added the dev/4.2.x-conflict label Oct 15, 2026

hubgeter mentioned this pull request Oct 20, 2026

[fix](iceberg) Support GZIP for iceberg/hive Parquet writers. (#68854) #69338

Merged

hubgeter mentioned this pull request Oct 21, 2026

[fix](iceberg) Validate Parquet compression before file creation #69376

Merged

3 of 16 tasks

yiguolei pushed a commit that referenced this pull request Oct 22, 2026

[fix](iceberg) Support GZIP for iceberg/hive Parquet writers. (#68854) (

3eebe29

#69338) pick #68854

yiguolei removed the dev/4.2.x-conflict label Oct 22, 2026

yiguolei added the dev/4.2.0-merged label Oct 22, 2026

Navigation Menu

Repository navigation

[fix](iceberg) Support GZIP for iceberg/hive Parquet writers. - #68854

[fix](iceberg) Support GZIP for iceberg/hive Parquet writers.#68854
Gabriel39 merged 1 commit into
apache:masterfrom
hubgeter:fix_iceberg_hive_parquet_gzip

hubgeter commented Oct 10, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hubgeter commented Oct 10, 2026

Uh oh!

hubgeter commented Oct 10, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

hubgeter commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hubgeter commented Oct 10, 2026

Uh oh!

hubgeter commented Oct 10, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

Uh oh!

hello-stephen commented Oct 10, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 10, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 10, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Oct 10, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hubgeter commented Oct 10, 2026 •

edited

Loading