You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fa30df3
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](iceberg) Support GZIP for iceberg/hive Parquet writers. (#68854)
Issue Number: N/A
Related PR: #64723
Problem Summary:
Iceberg tables with `write.parquet.compression-codec = gzip` (the
default before Iceberg 1.4) and Hive
tables with `parquet.compression = GZIP` could not be written by Doris:
the BE Parquet writers only
accepted PLAIN/SNAPPY/ZSTD/LZ4 and failed with
`[INTERNAL_ERROR]Unsupported compress type GZ with
parquet`, while Spark/Trino wrote the same table fine.
- BE: map `TFileCompressType.GZ` to `TParquetCompressionType.GZIP`, and
move the whole
`TFileCompressType` -> `TParquetCompressionType` mapping into
`ParquetBuildHelper` so the Iceberg
data-file writer, the Iceberg delete-file writer and the Hive Parquet
writer share one copy.
- FE: accept `gzip` for Parquet in the Hive `CREATE TABLE` compression
allow-list.
Tests: the Iceberg Parquet-compression suite now writes and reads gzip
data and merge-on-read delete
files and checks the codec with `parquet_meta()`; the Hive DDL suite
adds gzip to the Parquet
compression matrix; FE unit tests cover the gzip allow-list and the
codec mapping.
None
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change --> Iceberg/Hive Parquet
writes with GZIP no longer fail; the file is written with the standard
Parquet GZIP codec, the same codec Spark/Trino write for that table
property.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
0 commit comments