Repository navigation
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed pinned head fa30df3 across all nine changed files. One verified P1 requires changes: the Iceberg writer's new codec-error branch references undeclared open_status and has no final cleanup/status return. Static inspection establishes the build blocker; the inline comment covers this single incomplete refactor.
The intended codec flow is consistent: Hive/Iceberg table property -> existing FE GZ enum -> data/delete/MERGE writer -> shared Parquet GZIP mapping -> Arrow GZIP pages/footer -> existing footer-selected decoder. No additional user focus was supplied. No existing inline findings required carry-forward.
Critical checkpoint conclusions:
- Goal and tests: GZIP support reaches Hive/Iceberg Parquet data and position-delete writers, but the P1 prevents the BE implementation from compiling. Positive FE property and Hive/Iceberg read/footer tests were added; none were run.
- Scope: the shared mapping replaces three duplicate switches, with a focused FE whitelist change and regression additions. The error-handling branch is unfinished.
- Concurrency: no new threads, shared mutable state, locks, heavy operations under locks, or lock-order changes. The helper is stateless.
- Lifecycle and statics: open/write/close, sorted rollover, ownership and rollback paths were traced. No new global initializer or ownership cycle. The introduced invalid error path is covered inline; older dynamic-partition open-failure cleanup weaknesses are pre-existing.
- Configuration: no new process/session configuration. Existing table codec properties are selected when binding each sink.
- Compatibility: existing compression enums and standard Parquet GZIP pages/footer are reused; prior codec mappings and LZ4 framing remain consistent. Older binaries retain their prior unsupported-codec behavior. No storage-schema migration is introduced.
- Thrift/protobuf: no definitions, IDs, enum values, required fields or wire variables changed; release-branch ID comparison is not applicable.
- Parallel paths: Hive data, Iceberg regular/static/dynamic/sorted data, DELETE and MERGE data/deletes were checked. Existing legacy/native-v2 readers support footer GZIP, including non-x86 decoding. ORC/text and v3 Puffin deletion vectors retain separate paths.
- Special conditions: the new Result failure branch is defective as reported. Other mappings preserve unsupported-codec rejection; the Hive test matrix correctly excludes ORC gzip and retains zlib there.
- Test coverage: new positive coverage includes property storage, writes, ordered reads and actual footer codecs, including position deletes. No direct helper negative tests, multi-file stress or comprehensive merge/update cases were added.
- Expected results: no output snapshots changed; added checks use explicit assertions and existing qt labels/results remain. Fixture warehouse and committed Iceberg metadata-file paths match the assertions by static inspection.
- Observability: existing codec errors and Arrow writer status/logging paths are reused; no additional metric or distributed lifecycle was introduced. The invalid error return is the inline finding.
- Transactions, persistence and failover: no EditLog, commit descriptor, snapshot/version or failover algorithm changes. Hive uses its normal table parameter; Iceberg preserves codec choice in the existing pinned write context.
- Writes, atomicity and crashes: successful codec selection uses existing close/commit/abort paths, with no new commit point or visibility mechanism. Static inspection does not establish crash/runtime guarantees; the introduced error branch is reported.
- FE/BE passing: the existing compression field reaches all relevant data/delete/MERGE descriptors and all three changed BE callers.
- Performance and memory: a constant-size mapping replaces equivalent switches, without new significant owning allocations, copies, scans or repeated IO. Existing Arrow memory accounting is reused; GZIP CPU cost follows the requested codec.
- Additional sweep: checked filename suffixes versus internal compression, pinned/default properties, Arrow zlib build capability, standalone unit-test setup, reader variants and available parallel rejection expectations. No other substantiated introduced defect was found.
Core data visibility, internal visible-version and MoW bitmap machinery are unchanged. The affected error-propagation invariant is covered by the P1.
Review converged after two rounds: both complete reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the exact unchanged inline set, with all changed paths and risk/candidate dispositions complete. Remaining validation limits: static analysis only; no builds, tests, submodule initialization or Spark/Trino execution. The separate external compression product suite referenced by the old comment is unavailable in this checkout; no stale executable GZIP rejection was found in available sources. No unresolved static candidate remains.
Review converged after 2 round(s).
| if (!parquet_compression_type.has_value()) { | ||
| // The target file is already created, so let the shared cleanup below remove it | ||
| // instead of returning early. | ||
| open_status = parquet_compression_type.error(); |
There was a problem hiding this comment.
[P1] Complete the codec error-handling block before using open_status
open_status is not declared in open(), VIcebergPartitionWriter, or its base class, so this translation unit fails to compile in every BE build, regardless of the selected codec. The new break also exits the switch directly into the end of this Status-returning function: there is no shared cleanup/return block below it. Complete the intended cleanup and status return, or propagate the helper error through a valid return path.
…#68854) Issue Number: N/A Related PR: apache#64723 Problem Summary: Iceberg tables with `write.parquet.compression-codec = gzip` (the default before Iceberg 1.4) and Hive tables with `parquet.compression = GZIP` could not be written by Doris: the BE Parquet writers only accepted PLAIN/SNAPPY/ZSTD/LZ4 and failed with `[INTERNAL_ERROR]Unsupported compress type GZ with parquet`, while Spark/Trino wrote the same table fine. - BE: map `TFileCompressType.GZ` to `TParquetCompressionType.GZIP`, and move the whole `TFileCompressType` -> `TParquetCompressionType` mapping into `ParquetBuildHelper` so the Iceberg data-file writer, the Iceberg delete-file writer and the Hive Parquet writer share one copy. - FE: accept `gzip` for Parquet in the Hive `CREATE TABLE` compression allow-list. Tests: the Iceberg Parquet-compression suite now writes and reads gzip data and merge-on-read delete files and checks the codec with `parquet_meta()`; the Hive DDL suite adds gzip to the Parquet compression matrix; FE unit tests cover the gzip allow-list and the codec mapping. None - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> Iceberg/Hive Parquet writes with GZIP no longer fail; the file is written with the standard Parquet GZIP codec, the same codec Spark/Trino write for that table property. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
fa30df3 to
413ec38
Compare
|
/review |
|
run buildall |
There was a problem hiding this comment.
Reviewed all nine changed paths at 413ec38, including upstream property handling, downstream writer/commit behavior, and both regression suites. No new actionable defect found. Review converged in round 1: all three independent reviewers returned NO_NEW_VALUABLE_FINDINGS against the same empty finding set, and all risk candidates were dismissed with source evidence.
Existing P1 comment 4323545185 is fixed at this head: VIcebergPartitionWriter::open now returns the codec error explicitly; the undeclared open_status and fallthrough break are gone. No existing P0/P1 finding remains applicable, and no duplicate comment is posted.
Critical checkpoint conclusions:
- Goal and proof: Hive DDL now accepts gzip, and all three Parquet sink writers map GZ to the existing Arrow GZIP codec. Added tests inspect metadata, written rows and actual data/delete footer codecs.
- Scope and clarity: the focused helper replaces three duplicated switches while preserving the four existing mappings.
- Concurrency and locks: the helper is stateless; no threads, shared mutable state, atomics, lock scope or lock ordering are introduced.
- Lifecycle: rejected codec selection occurs before transformer creation or appends. HDFS writer ownership still closes its handle; S3 has no upload in flight at that point. No new ownership cycle or static initialization dependency was found.
- Configuration: existing table properties are reused; no new dynamic configuration or restart behavior is introduced.
- Compatibility: existing codec enums and Parquet encodings are reused, including Hadoop-framed LZ4. No wire or storage-schema change; GZIP writes still require a supporting BE during rolling upgrades.
- Generated definitions: no Thrift/protobuf definitions or IDs change, so release-branch ID comparison is inapplicable.
- Parallel paths: Hive data, Iceberg data and position deletes use the helper; sorted/partitioned writes and MERGE data/delete fan-out transport the same enum. ORC/text paths retain their existing handling.
- Conditional/error paths: unsupported codecs return the original error; cleanup failure is logged. The Parquet-only gzip regression and ORC skip match their respective codec support.
- Test coverage: new FE metadata and Hive/Iceberg write/read/footer assertions cover the feature, including position deletes. Existing codec and ORC/LZ4 checks remain. Direct helper negative tests and cleanup failure injection are absent; error propagation was checked statically.
- Expected results: no golden output changes are required because new cases use direct assertions; existing qt labels remain unchanged. Ordered reads and expected GZIP footer values match the code path.
- Observability: codec errors retain their codec context, and the new cleanup warning includes file path and status. No new distributed stage requires added metrics.
- Transactions and persistence: no EditLog, replay, transaction protocol, visibility-version or delete-bitmap changes; existing external metadata/commit paths are reused.
- Writes and failures: successful close/commit ordering is unchanged. Rejected-open errors propagate through async writer status and prevent commit-data creation; no new crash-cleanup defect was substantiated.
- FE/BE transport: existing GZ reaches Hive and Iceberg INSERT/DELETE/MERGE sinks without a missing send path; no new transmitted field is needed.
- Performance and memory: mapping remains constant time with no new large allocation, repeated scan or hot-path copy. GZIP compression cost follows the selected table codec.
- Other concerns and user focus: no additional user focus was supplied. Rechecks of Result/header availability, defaults, property precedence, filename handling, test setup and unchanged codec behavior found no further issue.
Limits: static review only. No builds, tests, runtime failure injection or Spark/Trino interoperability checks were executed, and repository sources/submodules were untouched. No unresolved candidate or uncovered changed path remains within that scope.
Review converged after 1 round(s).
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
| @@ -97,34 +98,19 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil | |||
|
|
|||
| switch (_file_format_type) { | |||
| case TFileFormatType::FORMAT_PARQUET: { | |||
There was a problem hiding this comment.
你把这个拿compression type的逻辑,放到_fs->create_file(file_description.path, &_file_writer, &file_writer_options)); 这个之前,这样就不会错误的创建完文件,然后再删除了。
|
/review |
There was a problem hiding this comment.
Reviewed all 10 changed files at d25ac89. No new actionable findings were identified. Review converged in 1 round: both complete-scope reviewers and the separate lifecycle-risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty inline set; all candidates and risk focuses were independently verified and dismissed with evidence.
Existing review context: P1 comment 4323545185 is fixed: open() now propagates codec errors with DORIS_TRY and every format branch returns a Status; the undeclared open_status and erroneous break are gone. The creation-order request in thread 4332582986 is also implemented: all three Parquet writers validate the codec before creating the output file. No existing P0/P1 still applies, so existing_blocking_comment_ids is empty. No additional user-provided focus was supplied.
Critical checkpoint conclusions:
- Goal and evidence: Hive/Iceberg Parquet GZIP now follows the existing FE property -> GZ -> shared GZIP mapping -> Arrow GZIP route, including Iceberg position-delete files. New unit/regression tests cover property translation, invalid-codec file absence, row reads, and GZIP footers; these tests were inspected statically.
- Scope and reuse: one shared conversion replaces three duplicate switches; the FE allowlist and test changes are focused on GZIP and early rejection.
- Concurrency and locks: no new thread, shared mutable state, lock or lock order. Conversion is stateless; local create_file lambdas do not escape. Existing filesystem dispatch and sorted-writer synchronization remain in use.
- Lifecycle and static initialization: no new global initialization dependency or ownership cycle. Checked static/unpartitioned retained writers, dynamic writers, sorted/spill paths, cancellation, rollover, delete cleanup and MERGE sibling cleanup. Error paths guard absent transformers/file writers and avoid publishing commits; asynchronous filesystem dispatch completes within the caller lifetime.
- Configuration: no configuration/session variable is added; existing per-table codec properties and sink binding are reused.
- Compatibility: existing GZ/GZIP wire enums and standard Parquet codec are used; previous mappings and timestamp/version choices are preserved. Older BEs retain their prior explicit unsupported-codec error until upgraded.
- Thrift/protobuf: no definitions, IDs, enum values or required fields changed; release-branch schema comparison is not triggered.
- Parallel paths: Hive and Iceberg data/delete Parquet writers all use the helper; ORC and Hive CSV still create files before transformer setup with the same options. Generic Parquet already supports GZIP; Iceberg v3 deletion vectors retain their separate path.
- Special conditions and errors: unsupported codecs return non-OK Status before file creation. LZ4 retains LZ4_HADOOP and its existing page limits. The regression's ORC/GZIP exclusion documents the zlib spelling; no new unchecked Status or speculative null fallback was identified.
- Test coverage: FE metadata unit test, BE unsupported-codec test, Hive GZIP write/read/footer and Iceberg data/format-v2 position-delete write/read/footer checks were reviewed. Existing codec loops remain. The BE target's -fno-access-control permits the test's private-field access.
- Expected results: no golden files changed; new explicit row/codec assertions match the inputs and metadata schema. Ordered row assertions and existing result labels are preserved. No surviving GZIP expected-failure product test was found in this checkout.
- Observability: existing unsupported-codec Status, file-close/delete warnings and sink counters cover these unchanged operations; no new distributed phase calls for separate instrumentation.
- Transactions and persistence: no EditLog, visible-version, delete-bitmap or commit-protocol change; existing success-only commit metadata and rollback paths remain in use. No new failover-persisted state is introduced.
- Writes, atomicity and crash boundaries: successful creation/close/publication follows the existing transaction path; invalid codecs avoid creating output resources. Failure/cancellation propagation and cleanup were traced; no new write-boundary defect was identified.
- FE-BE variables: no variable is added. Existing compression fields were traced through Hive, Iceberg data/delete and MERGE routes.
- Performance and memory: constant-time shared conversion adds no row-path scan, large allocation or ownership container; invalid-codec file creation is removed. Existing memory pools and tracking context are unchanged.
- Further issues: rechecked Result/include visibility, test fixture conventions, filename versus internal Parquet compression, metadata URLs/columns, property defaults/overrides, and cleanup after failed open. No unresolved correctness candidate remains.
Validation limits: this was a static review only, as required; no builds, tests or submodule initialization were performed. Runtime cross-engine, mixed-version and crash/remote-store behavior were not exercised. The new BE test covers no-file rejection/destruction; retained-writer failure-close behavior was verified by source tracing rather than a direct integration test. No external product-test suite is present in this checkout. All assigned in-repository scope is covered; there are no remaining unresolved review questions.
Review converged after 1 round(s).
|
run buildall |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#66348, apache#68854, apache#69376 Problem Summary: Partition writers call open before registering a new writer in the partition map. If the output file is created but the Parquet or ORC transformer then fails to open, the table writer cannot reach that writer through its close path. Clean up the created file at the open failure boundary so the error cannot leave an orphan, matching the lifecycle hardening introduced on master by apache#66348. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [x] Manual test: repository clang-format check and direct compilation of the modified BE writer translation unit passed. - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
|
/review |
There was a problem hiding this comment.
Reviewed all ten changed paths at f7456aa9b98e87244750713f9f3c8c40deb594f3. No new actionable findings. Hive and Iceberg data/delete/merge paths carry the existing GZ value into the shared helper, which selects Parquet GZIP and the existing Arrow GZIP builder. Existing codecs, including Hadoop-framed LZ4, remain intact.
Existing P1 comment 4323545185 is resolved: open_status is declared, assigned in both supported cases, and returned. The validation-before-file-creation request in thread 4332582986 is also implemented in all three Parquet writers. Neither is duplicated; no existing P0/P1 remains applicable. No additional user focus was supplied. The requested master-branch follow-up is outside this backport's changed scope.
Critical checkpoint conclusions:
- Goal and tests: GZIP admission and routing are complete statically. Added tests cover Hive metadata translation, Hive INSERT/readback/footer, Iceberg data and position-delete readback/footer, and unsupported-codec rejection without file creation.
- Scope: the shared mapping replaces three switches; changes and tests are focused on compression and open-time file handling.
- Concurrency and locks: no new threads, shared mutable codec state, locks, or lock-order changes. The file-creation lambdas run synchronously within
open(). - Lifecycle and initialization: transformer/FileWriter ownership, static/nonpartitioned versus dynamic registration, and S3/HDFS/local close/destruction were traced. No introduced regression or static-initialization issue was established. The new best-effort delete is not a universal S3 orphan-prevention guarantee: later stream destruction can publish after it, but that same failed-open publication behavior predates this PR.
- Configuration: no process configuration added; existing table properties select the codec.
- Compatibility: existing GZ/GZIP values and standard Parquet page compression are reused; no changed wire layout, stored schema, or removed symbol.
- Generated definitions: no thrift/protobuf changes; cross-release field-ID comparison is inapplicable.
- Parallel paths: Hive new/existing/overwrite partitions, Iceberg data/delete/merge and sorted delegation were checked; ORC/CSV retain file creation and their existing transformers.
- Conditions and errors: unsupported Parquet codecs fail before file creation; open failures propagate their Status. ORC/GZIP is correctly excluded from the regression loop.
- Coverage: positive codec/footer assertions and a negative BE no-file test are present. Multi-file, external-engine and backend-fault-injection execution are not established.
- Expected results: no output snapshots changed; new explicit assertions agree with source behavior, with deterministic row checks. The BE test target allows its direct private-member access through the existing
-fno-access-controlflag. - Observability: codec/open errors remain propagated and cleanup failures are logged; no additional metric-critical path identified.
- Persistence and failover: no EditLog, replay or transaction-protocol changes; existing sink commit/rollback paths remain in use.
- Writes and crashes: row conversion and visibility/commit semantics are unchanged; no introduced atomicity or crash-safety regression found within static review.
- FE-BE transport: no new variable; existing compression fields were traced through data, delete and merge sinks.
- Performance and memory: constant-time mapping adds no success-path allocation or per-row work; no new accounting or reservation requirement identified.
- Further recheck: reviewed codec aliases/property precedence, pinned write context, file suffixes, position-delete triggering, metadata TVF prerequisites and parallel/error paths. No further actionable issue found.
Review converged in one round: both complete-scope reviewers and the separate lifecycle-risk reviewer returned NO_NEW_VALUABLE_FINDINGS; all changed paths, candidates and initial risks have evidence-backed dispositions. Static inspection only: no builds or tests were run, no source files changed, and no submodules initialized. Compilation, runtime behavior and cross-engine interoperability are not claimed as validated.
Review converged after 1 round(s).
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Issue Number: N/A Related PR: apache#68854, apache#69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path. None - Test - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it. - [ ] Regression test - [ ] Manual test - [ ] No need to test or manual test. Explain why - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
|
PR approved by anyone and no changes requested. |
1 similar comment
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
1 similar comment
|
PR approved by at least one committer and no changes requested. |
Issue Number: N/A Related PR: apache#68854, apache#69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path. None - Test - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it. - [ ] Regression test - [ ] Manual test - [ ] No need to test or manual test. Explain why - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
Issue Number: N/A Related PR: apache#68854, apache#69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path. None - Test - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it. - [ ] Regression test - [ ] Manual test - [ ] No need to test or manual test. Explain why - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
Issue Number: N/A Related PR: apache#68854, apache#69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling create_file so invalid codecs fail without creating any output file, and add a regression test for the failure path. None - Test - [x] Unit Test: added UnsupportedParquetCodecDoesNotCreateFile; not executed locally per request, CI will run it. - [ ] Regression test - [ ] Manual test - [ ] No need to test or manual test. Explain why - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
) ### What problem does this PR solve? Issue Number: N/A Related PR: #68854, #69338 Problem Summary: The Iceberg data, position-delete, and Hive Parquet writers created their output file before validating the requested Parquet compression codec. For an unsupported codec, the Iceberg data writer then attempted to delete an open file, while the other writers returned after creating the file. This made error cleanup depend on filesystem behavior and could leave an orphan file. Resolve the codec before calling `create_file` so invalid codecs fail without creating any output file, and add a regression test for the failure path. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test: added `UnsupportedParquetCodecDoesNotCreateFile`; CI will run it. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
pick #68854