Skip to content

[fix](iceberg) Support GZIP for iceberg/hive Parquet writers. (#68854) - #69338

Merged
yiguolei merged 3 commits into
apache:branch-4.2from
hubgeter:pick/doris-68854-parquet-gzip-branch-4.2
Oct 22, 2026
Merged

yiguolei merged 3 commits into
apache:branch-4.2from
hubgeter:pick/doris-68854-parquet-gzip-branch-4.2

Conversation

@hubgeter

Copy link
Copy Markdown
Contributor

pick #68854

@hubgeter
hubgeter requested a review from yiguolei as a code owner October 20, 2026 09:03
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter hubgeter changed the title [fix](iceberg) Support GZIP for iceberg/hive Parquet writers. (pick #68854) [fix](iceberg) Support GZIP for iceberg/hive Parquet writers. (#68854) Oct 20, 2026
@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed pinned head fa30df3 across all nine changed files. One verified P1 requires changes: the Iceberg writer's new codec-error branch references undeclared open_status and has no final cleanup/status return. Static inspection establishes the build blocker; the inline comment covers this single incomplete refactor.

The intended codec flow is consistent: Hive/Iceberg table property -> existing FE GZ enum -> data/delete/MERGE writer -> shared Parquet GZIP mapping -> Arrow GZIP pages/footer -> existing footer-selected decoder. No additional user focus was supplied. No existing inline findings required carry-forward.

Critical checkpoint conclusions:

  1. Goal and tests: GZIP support reaches Hive/Iceberg Parquet data and position-delete writers, but the P1 prevents the BE implementation from compiling. Positive FE property and Hive/Iceberg read/footer tests were added; none were run.
  2. Scope: the shared mapping replaces three duplicate switches, with a focused FE whitelist change and regression additions. The error-handling branch is unfinished.
  3. Concurrency: no new threads, shared mutable state, locks, heavy operations under locks, or lock-order changes. The helper is stateless.
  4. Lifecycle and statics: open/write/close, sorted rollover, ownership and rollback paths were traced. No new global initializer or ownership cycle. The introduced invalid error path is covered inline; older dynamic-partition open-failure cleanup weaknesses are pre-existing.
  5. Configuration: no new process/session configuration. Existing table codec properties are selected when binding each sink.
  6. Compatibility: existing compression enums and standard Parquet GZIP pages/footer are reused; prior codec mappings and LZ4 framing remain consistent. Older binaries retain their prior unsupported-codec behavior. No storage-schema migration is introduced.
  7. Thrift/protobuf: no definitions, IDs, enum values, required fields or wire variables changed; release-branch ID comparison is not applicable.
  8. Parallel paths: Hive data, Iceberg regular/static/dynamic/sorted data, DELETE and MERGE data/deletes were checked. Existing legacy/native-v2 readers support footer GZIP, including non-x86 decoding. ORC/text and v3 Puffin deletion vectors retain separate paths.
  9. Special conditions: the new Result failure branch is defective as reported. Other mappings preserve unsupported-codec rejection; the Hive test matrix correctly excludes ORC gzip and retains zlib there.
  10. Test coverage: new positive coverage includes property storage, writes, ordered reads and actual footer codecs, including position deletes. No direct helper negative tests, multi-file stress or comprehensive merge/update cases were added.
  11. Expected results: no output snapshots changed; added checks use explicit assertions and existing qt labels/results remain. Fixture warehouse and committed Iceberg metadata-file paths match the assertions by static inspection.
  12. Observability: existing codec errors and Arrow writer status/logging paths are reused; no additional metric or distributed lifecycle was introduced. The invalid error return is the inline finding.
  13. Transactions, persistence and failover: no EditLog, commit descriptor, snapshot/version or failover algorithm changes. Hive uses its normal table parameter; Iceberg preserves codec choice in the existing pinned write context.
  14. Writes, atomicity and crashes: successful codec selection uses existing close/commit/abort paths, with no new commit point or visibility mechanism. Static inspection does not establish crash/runtime guarantees; the introduced error branch is reported.
  15. FE/BE passing: the existing compression field reaches all relevant data/delete/MERGE descriptors and all three changed BE callers.
  16. Performance and memory: a constant-size mapping replaces equivalent switches, without new significant owning allocations, copies, scans or repeated IO. Existing Arrow memory accounting is reused; GZIP CPU cost follows the requested codec.
  17. Additional sweep: checked filename suffixes versus internal compression, pinned/default properties, Arrow zlib build capability, standalone unit-test setup, reader variants and available parallel rejection expectations. No other substantiated introduced defect was found.

Core data visibility, internal visible-version and MoW bitmap machinery are unchanged. The affected error-propagation invariant is covered by the P1.

Review converged after two rounds: both complete reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the exact unchanged inline set, with all changed paths and risk/candidate dispositions complete. Remaining validation limits: static analysis only; no builds, tests, submodule initialization or Spark/Trino execution. The separate external compression product suite referenced by the old comment is unavailable in this checkout; no stale executable GZIP rejection was found in available sources. No unresolved static candidate remains.

Review converged after 2 round(s).

if (!parquet_compression_type.has_value()) {
// The target file is already created, so let the shared cleanup below remove it
// instead of returning early.
open_status = parquet_compression_type.error();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Complete the codec error-handling block before using open_status

open_status is not declared in open(), VIcebergPartitionWriter, or its base class, so this translation unit fails to compile in every BE build, regardless of the selected codec. The new break also exits the switch directly into the end of this Status-returning function: there is no shared cleanup/return block below it. Complete the intended cleanup and status return, or propagate the helper error through a valid return path.

…#68854)

Issue Number: N/A

Related PR: apache#64723

Problem Summary:

Iceberg tables with `write.parquet.compression-codec = gzip` (the
default before Iceberg 1.4) and Hive
tables with `parquet.compression = GZIP` could not be written by Doris:
the BE Parquet writers only
accepted PLAIN/SNAPPY/ZSTD/LZ4 and failed with
`[INTERNAL_ERROR]Unsupported compress type GZ with
parquet`, while Spark/Trino wrote the same table fine.

- BE: map `TFileCompressType.GZ` to `TParquetCompressionType.GZIP`, and
move the whole
`TFileCompressType` -> `TParquetCompressionType` mapping into
`ParquetBuildHelper` so the Iceberg
data-file writer, the Iceberg delete-file writer and the Hive Parquet
writer share one copy.
- FE: accept `gzip` for Parquet in the Hive `CREATE TABLE` compression
allow-list.

Tests: the Iceberg Parquet-compression suite now writes and reads gzip
data and merge-on-read delete
files and checks the codec with `parquet_meta()`; the Hive DDL suite
adds gzip to the Parquet
compression matrix; FE unit tests cover the gzip allow-list and the
codec mapping.

None

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
- [x] Yes. <!-- Explain the behavior change --> Iceberg/Hive Parquet
writes with GZIP no longer fail; the file is written with the standard
Parquet GZIP codec, the same codec Spark/Trino write for that table
property.

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@hubgeter
hubgeter force-pushed the pick/doris-68854-parquet-gzip-branch-4.2 branch from fa30df3 to 413ec38 Compare October 20, 2026 09:51
@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all nine changed paths at 413ec38, including upstream property handling, downstream writer/commit behavior, and both regression suites. No new actionable defect found. Review converged in round 1: all three independent reviewers returned NO_NEW_VALUABLE_FINDINGS against the same empty finding set, and all risk candidates were dismissed with source evidence.

Existing P1 comment 4323545185 is fixed at this head: VIcebergPartitionWriter::open now returns the codec error explicitly; the undeclared open_status and fallthrough break are gone. No existing P0/P1 finding remains applicable, and no duplicate comment is posted.

Critical checkpoint conclusions:

  1. Goal and proof: Hive DDL now accepts gzip, and all three Parquet sink writers map GZ to the existing Arrow GZIP codec. Added tests inspect metadata, written rows and actual data/delete footer codecs.
  2. Scope and clarity: the focused helper replaces three duplicated switches while preserving the four existing mappings.
  3. Concurrency and locks: the helper is stateless; no threads, shared mutable state, atomics, lock scope or lock ordering are introduced.
  4. Lifecycle: rejected codec selection occurs before transformer creation or appends. HDFS writer ownership still closes its handle; S3 has no upload in flight at that point. No new ownership cycle or static initialization dependency was found.
  5. Configuration: existing table properties are reused; no new dynamic configuration or restart behavior is introduced.
  6. Compatibility: existing codec enums and Parquet encodings are reused, including Hadoop-framed LZ4. No wire or storage-schema change; GZIP writes still require a supporting BE during rolling upgrades.
  7. Generated definitions: no Thrift/protobuf definitions or IDs change, so release-branch ID comparison is inapplicable.
  8. Parallel paths: Hive data, Iceberg data and position deletes use the helper; sorted/partitioned writes and MERGE data/delete fan-out transport the same enum. ORC/text paths retain their existing handling.
  9. Conditional/error paths: unsupported codecs return the original error; cleanup failure is logged. The Parquet-only gzip regression and ORC skip match their respective codec support.
  10. Test coverage: new FE metadata and Hive/Iceberg write/read/footer assertions cover the feature, including position deletes. Existing codec and ORC/LZ4 checks remain. Direct helper negative tests and cleanup failure injection are absent; error propagation was checked statically.
  11. Expected results: no golden output changes are required because new cases use direct assertions; existing qt labels remain unchanged. Ordered reads and expected GZIP footer values match the code path.
  12. Observability: codec errors retain their codec context, and the new cleanup warning includes file path and status. No new distributed stage requires added metrics.
  13. Transactions and persistence: no EditLog, replay, transaction protocol, visibility-version or delete-bitmap changes; existing external metadata/commit paths are reused.
  14. Writes and failures: successful close/commit ordering is unchanged. Rejected-open errors propagate through async writer status and prevent commit-data creation; no new crash-cleanup defect was substantiated.
  15. FE/BE transport: existing GZ reaches Hive and Iceberg INSERT/DELETE/MERGE sinks without a missing send path; no new transmitted field is needed.
  16. Performance and memory: mapping remains constant time with no new large allocation, repeated scan or hot-path copy. GZIP compression cost follows the selected table codec.
  17. Other concerns and user focus: no additional user focus was supplied. Rechecks of Result/header availability, defaults, property precedence, filename handling, test setup and unchanged codec behavior found no further issue.

Limits: static review only. No builds, tests, runtime failure injection or Spark/Trino interoperability checks were executed, and repository sources/submodules were untouched. No unresolved candidate or uncovered changed path remains within that scope.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 38.71% (12/31) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.37% (34407/45052)
Line Coverage 61.20% (388542/634857)
Region Coverage 57.95% (326978/564220)
Branch Coverage 58.97% (150851/255825)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 38.71% (12/31) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.37% (34407/45052)
Line Coverage 61.20% (388542/634857)
Region Coverage 57.95% (326978/564220)
Branch Coverage 58.97% (150851/255825)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 38.71% (12/31) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.38% (34412/45052)
Line Coverage 61.19% (388484/634857)
Region Coverage 57.94% (326918/564220)
Branch Coverage 58.97% (150856/255825)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 38.71% (12/31) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.39% (34413/45052)
Line Coverage 61.20% (388505/634857)
Region Coverage 57.95% (326946/564220)
Branch Coverage 58.97% (150869/255825)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 38.71% (12/31) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.38% (34410/45052)
Line Coverage 61.18% (388410/634857)
Region Coverage 57.94% (326894/564220)
Branch Coverage 58.96% (150842/255825)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@@ -97,34 +98,19 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil

switch (_file_format_type) {
case TFileFormatType::FORMAT_PARQUET: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你把这个拿compression type的逻辑,放到_fs->create_file(file_description.path, &_file_writer, &file_writer_options)); 这个之前,这样就不会错误的创建完文件,然后再删除了。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hubgeter 你把对应的master 的代码也改改吧

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 10 changed files at d25ac89. No new actionable findings were identified. Review converged in 1 round: both complete-scope reviewers and the separate lifecycle-risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty inline set; all candidates and risk focuses were independently verified and dismissed with evidence.

Existing review context: P1 comment 4323545185 is fixed: open() now propagates codec errors with DORIS_TRY and every format branch returns a Status; the undeclared open_status and erroneous break are gone. The creation-order request in thread 4332582986 is also implemented: all three Parquet writers validate the codec before creating the output file. No existing P0/P1 still applies, so existing_blocking_comment_ids is empty. No additional user-provided focus was supplied.

Critical checkpoint conclusions:

  • Goal and evidence: Hive/Iceberg Parquet GZIP now follows the existing FE property -> GZ -> shared GZIP mapping -> Arrow GZIP route, including Iceberg position-delete files. New unit/regression tests cover property translation, invalid-codec file absence, row reads, and GZIP footers; these tests were inspected statically.
  • Scope and reuse: one shared conversion replaces three duplicate switches; the FE allowlist and test changes are focused on GZIP and early rejection.
  • Concurrency and locks: no new thread, shared mutable state, lock or lock order. Conversion is stateless; local create_file lambdas do not escape. Existing filesystem dispatch and sorted-writer synchronization remain in use.
  • Lifecycle and static initialization: no new global initialization dependency or ownership cycle. Checked static/unpartitioned retained writers, dynamic writers, sorted/spill paths, cancellation, rollover, delete cleanup and MERGE sibling cleanup. Error paths guard absent transformers/file writers and avoid publishing commits; asynchronous filesystem dispatch completes within the caller lifetime.
  • Configuration: no configuration/session variable is added; existing per-table codec properties and sink binding are reused.
  • Compatibility: existing GZ/GZIP wire enums and standard Parquet codec are used; previous mappings and timestamp/version choices are preserved. Older BEs retain their prior explicit unsupported-codec error until upgraded.
  • Thrift/protobuf: no definitions, IDs, enum values or required fields changed; release-branch schema comparison is not triggered.
  • Parallel paths: Hive and Iceberg data/delete Parquet writers all use the helper; ORC and Hive CSV still create files before transformer setup with the same options. Generic Parquet already supports GZIP; Iceberg v3 deletion vectors retain their separate path.
  • Special conditions and errors: unsupported codecs return non-OK Status before file creation. LZ4 retains LZ4_HADOOP and its existing page limits. The regression's ORC/GZIP exclusion documents the zlib spelling; no new unchecked Status or speculative null fallback was identified.
  • Test coverage: FE metadata unit test, BE unsupported-codec test, Hive GZIP write/read/footer and Iceberg data/format-v2 position-delete write/read/footer checks were reviewed. Existing codec loops remain. The BE target's -fno-access-control permits the test's private-field access.
  • Expected results: no golden files changed; new explicit row/codec assertions match the inputs and metadata schema. Ordered row assertions and existing result labels are preserved. No surviving GZIP expected-failure product test was found in this checkout.
  • Observability: existing unsupported-codec Status, file-close/delete warnings and sink counters cover these unchanged operations; no new distributed phase calls for separate instrumentation.
  • Transactions and persistence: no EditLog, visible-version, delete-bitmap or commit-protocol change; existing success-only commit metadata and rollback paths remain in use. No new failover-persisted state is introduced.
  • Writes, atomicity and crash boundaries: successful creation/close/publication follows the existing transaction path; invalid codecs avoid creating output resources. Failure/cancellation propagation and cleanup were traced; no new write-boundary defect was identified.
  • FE-BE variables: no variable is added. Existing compression fields were traced through Hive, Iceberg data/delete and MERGE routes.
  • Performance and memory: constant-time shared conversion adds no row-path scan, large allocation or ownership container; invalid-codec file creation is removed. Existing memory pools and tracking context are unchanged.
  • Further issues: rechecked Result/include visibility, test fixture conventions, filename versus internal Parquet compression, metadata URLs/columns, property defaults/overrides, and cleanup after failed open. No unresolved correctness candidate remains.

Validation limits: this was a static review only, as required; no builds, tests or submodule initialization were performed. Runtime cross-engine, mixed-version and crash/remote-store behavior were not exercised. The new BE test covers no-file rejection/destruction; retained-writer failure-close behavior was verified by source tracing rather than a direct integration test. No external product-test suite is present in this checkout. All assigned in-repository scope is covered; there are no remaining unresolved review questions.

Review converged after 1 round(s).

@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#66348, apache#68854, apache#69376

Problem Summary:
Partition writers call open before registering a new writer in the partition map. If the output file is created but the Parquet or ORC transformer then fails to open, the table writer cannot reach that writer through its close path. Clean up the created file at the open failure boundary so the error cannot leave an orphan, matching the lifecycle hardening introduced on master by apache#66348.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [ ] Unit Test
    - [x] Manual test: repository clang-format check and direct compilation of the modified BE writer translation unit passed.
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.
@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all ten changed paths at f7456aa9b98e87244750713f9f3c8c40deb594f3. No new actionable findings. Hive and Iceberg data/delete/merge paths carry the existing GZ value into the shared helper, which selects Parquet GZIP and the existing Arrow GZIP builder. Existing codecs, including Hadoop-framed LZ4, remain intact.

Existing P1 comment 4323545185 is resolved: open_status is declared, assigned in both supported cases, and returned. The validation-before-file-creation request in thread 4332582986 is also implemented in all three Parquet writers. Neither is duplicated; no existing P0/P1 remains applicable. No additional user focus was supplied. The requested master-branch follow-up is outside this backport's changed scope.

Critical checkpoint conclusions:

  1. Goal and tests: GZIP admission and routing are complete statically. Added tests cover Hive metadata translation, Hive INSERT/readback/footer, Iceberg data and position-delete readback/footer, and unsupported-codec rejection without file creation.
  2. Scope: the shared mapping replaces three switches; changes and tests are focused on compression and open-time file handling.
  3. Concurrency and locks: no new threads, shared mutable codec state, locks, or lock-order changes. The file-creation lambdas run synchronously within open().
  4. Lifecycle and initialization: transformer/FileWriter ownership, static/nonpartitioned versus dynamic registration, and S3/HDFS/local close/destruction were traced. No introduced regression or static-initialization issue was established. The new best-effort delete is not a universal S3 orphan-prevention guarantee: later stream destruction can publish after it, but that same failed-open publication behavior predates this PR.
  5. Configuration: no process configuration added; existing table properties select the codec.
  6. Compatibility: existing GZ/GZIP values and standard Parquet page compression are reused; no changed wire layout, stored schema, or removed symbol.
  7. Generated definitions: no thrift/protobuf changes; cross-release field-ID comparison is inapplicable.
  8. Parallel paths: Hive new/existing/overwrite partitions, Iceberg data/delete/merge and sorted delegation were checked; ORC/CSV retain file creation and their existing transformers.
  9. Conditions and errors: unsupported Parquet codecs fail before file creation; open failures propagate their Status. ORC/GZIP is correctly excluded from the regression loop.
  10. Coverage: positive codec/footer assertions and a negative BE no-file test are present. Multi-file, external-engine and backend-fault-injection execution are not established.
  11. Expected results: no output snapshots changed; new explicit assertions agree with source behavior, with deterministic row checks. The BE test target allows its direct private-member access through the existing -fno-access-control flag.
  12. Observability: codec/open errors remain propagated and cleanup failures are logged; no additional metric-critical path identified.
  13. Persistence and failover: no EditLog, replay or transaction-protocol changes; existing sink commit/rollback paths remain in use.
  14. Writes and crashes: row conversion and visibility/commit semantics are unchanged; no introduced atomicity or crash-safety regression found within static review.
  15. FE-BE transport: no new variable; existing compression fields were traced through data, delete and merge sinks.
  16. Performance and memory: constant-time mapping adds no success-path allocation or per-row work; no new accounting or reservation requirement identified.
  17. Further recheck: reviewed codec aliases/property precedence, pinned write context, file suffixes, position-delete triggering, metadata TVF prerequisites and parallel/error paths. No further actionable issue found.

Review converged in one round: both complete-scope reviewers and the separate lifecycle-risk reviewer returned NO_NEW_VALUABLE_FINDINGS; all changed paths, candidates and initial risks have evidence-backed dispositions. Static inspection only: no builds or tests were run, no source files changed, and no submodules initialized. Compilation, runtime behavior and cross-engine interoperability are not claimed as validated.

Review converged after 1 round(s).

@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 54.17% (26/48) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.07% (29721/46390)
Line Coverage 48.92% (313267/640316)
Region Coverage 44.74% (252025/563294)
Branch Coverage 46.45% (118977/256162)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 54.17% (26/48) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.45% (34449/45059)
Line Coverage 61.27% (389043/634989)
Region Coverage 58.10% (327847/564277)
Branch Coverage 59.13% (151303/255902)

@yiguolei
yiguolei merged commit 3eebe29 into apache:branch-4.2 Oct 22, 2026
29 of 31 checks passed
hubgeter added a commit to hubgeter/doris that referenced this pull request Oct 22, 2026
Issue Number: N/A

Related PR: apache#68854, apache#69338

Problem Summary:
The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path.

None

- Test
    - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it.
    - [ ] Regression test
    - [ ] Manual test
    - [ ] No need to test or manual test. Explain why
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added approved Indicates a PR has been approved by one committer. labels Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

hubgeter added a commit to hubgeter/doris that referenced this pull request Oct 27, 2026
Issue Number: N/A

Related PR: apache#68854, apache#69338

Problem Summary:
The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path.

None

- Test
    - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it.
    - [ ] Regression test
    - [ ] Manual test
    - [ ] No need to test or manual test. Explain why
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 3, 2026
Issue Number: N/A

Related PR: apache#68854, apache#69338

Problem Summary:
The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path.

None

- Test
    - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it.
    - [ ] Regression test
    - [ ] Manual test
    - [ ] No need to test or manual test. Explain why
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 5, 2026
Issue Number: N/A

Related PR: apache#68854, apache#69338

Problem Summary:
The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path.

None

- Test
    - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it.
    - [ ] Regression test
    - [ ] Manual test
    - [ ] No need to test or manual test. Explain why
- Behavior changed:
    - [x] No.
- Does this need documentation?
    - [x] No.
Gabriel39 pushed a commit that referenced this pull request Nov 5, 2026
)

### What problem does this PR solve?

Issue Number: N/A

Related PR: #68854, #69338

Problem Summary:
The Iceberg data, position-delete, and Hive Parquet writers created
their output file before validating the requested Parquet compression
codec. For an unsupported codec, the Iceberg data writer then attempted
to delete an open file, while the other writers returned after creating
the file. This made error cleanup depend on filesystem behavior and
could leave an orphan file. Resolve the codec before calling
`create_file` so invalid codecs fail without creating any output file,
and add a regression test for the failure path.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
- [x] Unit Test: added `UnsupportedParquetCodecDoesNotCreateFile`; CI
will run it.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason

- Behavior changed:
    - [x] No.
    - [ ] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants