Repository navigation
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes: two verified regressions remain: [P1] cleanup can delete files from an already published REST snapshot after response loss; [P2] a thrown allocation failure during sorted EOS can bypass file cleanup. Both are reported inline. There are no existing inline findings to carry forward, and no additional user-provided focus.
Reviewed the complete 15-file authoritative diff at head 8b8880d and traced the INSERT/overwrite, DELETE, MERGE, rewrite, sorter/spill, serializer, report, and transaction-owner paths. Static review only: no builds, tests, benchmarks, source edits, or submodule initialization.
Critical checkpoint conclusions:
- Goal and proof: required NULL/map-key checks run before partition writer creation; unit and ORC/Parquet regression cases cover the intended rejection and optional-parent behavior. The two failure paths below remain uncovered by those tests.
- Scope: changes stay within Iceberg validation, close/failure cleanup, and associated tests.
- Concurrency: report append and rollback snapshot/clear share the transaction monitor; remote deletion occurs outside it. Sort write/close/revoke share the sorter mutex. No new lock inversion identified.
- Lifecycle: reset destroys streams before waiting for S3/HDFS writers and invoking cleanup callbacks; retained FE authentication/FileIO survives cleanup. Sorted exception unwinding has the inline P2 defect. No new global initialization dependency.
- Configuration: no new configuration item.
- Compatibility: no changed wire/storage layout or function signature; existing mixed-version behavior remains.
- Thrift/protobuf: no gensrc definitions changed, so cross-release ID comparisons are inapplicable.
- Parallel paths: ordinary, sorted/spilled, and MERGE data writers share validation; append/overwrite/DELETE/MERGE/rewrite transaction cleanup was traced.
- Conditions: optional container parents mask descendants; constant materialization and padded collection offsets agree with this branch's column invariants and serializers.
- Test coverage: scalar/constant/nested/list/map rejection, optional parents, pre-creation rejection, close Status failure, repeated failed close, lease ordering, and explicit unknown commit state are covered in inspected tests. REST response loss and thrown EOS allocation failure need targeted cases.
- Expected results: no .out changes; Groovy baselines check published snapshot/file/row counts. Nested invalid input can be rejected by FE planning.
- Observability: validation reports field path and row/ordinal; cleanup logs per-file deletion failures and retains the original query error. No separate missing metric identified.
- Transactions/persistence: publication remains Iceberg metadata CAS with no new Doris EditLog state. The inline P1 violates committed-file visibility after an indeterminate REST commit; MoW/version machinery is untouched.
- Writes and crashes: rollback deletes newly reported paths, not referenced source files. Per-file deletion errors continue cleanup. No new crash recovery is implemented; sorted exceptions leave the inline P2 orphan.
- FE/BE propagation: no new transmitted variable; pinned schema/projection and row-lineage shapes were checked.
- Performance/memory: primitive required fields use a null-map fast path; rollback deduplicates paths and scales linearly. No substantiated additional performance or memory-accounting defect found.
- Other issues and limits: all changed-file scope and initial risks have dispositions. No additional distinct issue found. Physical orphan absence, async upload failure behavior, and REST reconciliation were not runtime-validated; metadata-only regression assertions do not prove those properties.
Review converged after 2 round(s).
| } | ||
| throw e; | ||
| } catch (RuntimeException e) { | ||
| cleanupReportedFiles(); |
There was a problem hiding this comment.
[P1] Preserve files for indeterminate REST commit failures
A REST server can publish the snapshot and then lose or truncate the response. In the pinned Iceberg 1.11.0, HTTPClient.execute converts response I/O/parsing failures to RESTException, so these bypass the CommitStateUnknownException catch. BaseTransaction deliberately preserves files for such non-CleanableFailure exceptions under strict cleanup. This new catch instead deletes every reported data/delete file, leaving the published snapshot referencing missing objects. Gate deletion on a definitive commit failure and preserve the files through the subsequent rollback() too; the explicit unknown-state test does not cover this response-loss case.
| return internal_status; | ||
| } | ||
| // Form the return value only after the underlying close runs; a deferred assignment is too late. | ||
| Status close_status = |
There was a problem hiding this comment.
[P2] Keep partition finalization safe when sorting throws
For a spill-enabled sorted INSERT, EOS reaches this code from SpillIcebergTableSinkLocalState::_close_writer(). do_sort()/merge/write can throw, for example when get_permutation() or permute() allocation raises MEM_ALLOC_FAILED. Removing the Defer lets that bypass partition close, while close() still sets _closed = true during unwinding. The scheduler converts the exception and retries outer close with the failure, but the sorted writer now returns immediately: the current path never enters _closed_files or BE commit data. Its stream is eventually closed during destruction, leaving an orphan that neither BE nor FE rollback knows about (HDFS already creates it at open). Convert intermediate exceptions into internal_status before unconditional partition finalization, or retain exception-safe cleanup ownership. Add an EOS allocation-failure case that checks physical file removal.
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
8b8880d to
261176c
Compare
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69327, apache#69842 Problem Summary: A REST catalog can publish an Iceberg snapshot before a lost or malformed response raises an exception. Preserve reported files after SDK commit entry unless a CleanableFailure proves deletion safe, including on a later rollback. Convert sorted EOS allocation exceptions to Status before unconditional partition finalization so outer cleanup can remove the failed file. Add published-snapshot and persisted-file tests, cover missing container validation branches, and update nullable regression error expectations. ### Release note Prevent deletion of potentially committed Iceberg files after an indeterminate commit response, and remove failed sorted-write files after EOS allocation errors. ### Check List (For Author) - Test: Unit Test - IcebergTransactionTest, 39 passed; local FE changed executable line coverage 34/38 (89.47%). Related BE production sources compiled under ASAN; the focused BE unit-test build is still running. Java Checkstyle and BE clang-format/check-format passed. External regression requires new-head CI. - Behavior changed: Yes - preserve files when commit publication is uncertain and finalize failed sorted output before outer file cleanup. - Does this need documentation: No
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69327, apache#69842 Problem Summary: MAP constant materialization reconstructs the column and checks that its final offset matches both child lengths. The out-of-range offset fixture invalidated that final offset and aborted before reaching the Iceberg validator. Keep the final offset consistent and use an invalid intermediate offset so the validator can reject the first-row range. ### Release note None ### Check List (For Author) - Test: Unit Test - run-be-ut.sh focused Iceberg validator, partition writer and table writer suites: 37 ASAN tests passed, zero failures/errors/disabled tests. clang-format, check-format and git diff --check passed. Changed-file clang-tidy has no diagnostics; full analysis retains the existing unmatched NOLINTEND in core/types.h:578. - Behavior changed: No - test fixture only. - Does this need documentation: No
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Issue Number: N/A Related PR: apache#69327 Problem Summary: Iceberg ORC and Parquet writes could accept NULL required values, including nested STRUCT/LIST/MAP fields and MAP keys. Failed writes could also leave physical files produced by writers or completed BE fragments behind. Validate every prepared block before creating or dispatching partition writers, while preserving optional-parent NULL masking. Destroy failed writer streams before deferring file deletion to the existing outer cleanup lifecycle. Adapt FE cleanup to branch-4.2's legacy IcebergTransaction and retained catalog lease. Clean failed-commit files before releasing the lease, and retain files when Iceberg reports an unknown commit state. Carry over the necessary sorted writer close error propagation that the original PR's baseline already has. Split container validation into focused helpers for the target lint thresholds. Iceberg ORC/Parquet writes reject required NULL values and clean up failed files without publishing a snapshot. - Test: Modified Java files pass Checkstyle; repository clang-format 16 and whitespace checks pass; regression framework compilation and 47 framework unit tests pass. FE build and focused unit-test attempts were interrupted by confirmed host OOM kills. BE ASAN product/UT configuration succeeds, but full compilation and BE unit execution remain incomplete. Both focused regression suites were attempted and failed connecting to the isolated FE. Validator clang-tidy diagnostics were fixed; the check remains blocked by an existing unmatched NOLINTEND in core/types.h. - Behavior changed: Yes, invalid required NULL writes are rejected and failed external files are cleaned up. - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69327, apache#69842 Problem Summary: A REST catalog can publish an Iceberg snapshot before a lost or malformed response raises an exception. Preserve reported files after SDK commit entry unless a CleanableFailure proves deletion safe, including on a later rollback. Convert sorted EOS allocation exceptions to Status before unconditional partition finalization so outer cleanup can remove the failed file. Add published-snapshot and persisted-file tests, cover missing container validation branches, and update nullable regression error expectations. ### Release note Prevent deletion of potentially committed Iceberg files after an indeterminate commit response, and remove failed sorted-write files after EOS allocation errors. ### Check List (For Author) - Test: Unit Test - IcebergTransactionTest, 39 passed; local FE changed executable line coverage 34/38 (89.47%). Related BE production sources compiled under ASAN; the focused BE unit-test build is still running. Java Checkstyle and BE clang-format/check-format passed. External regression requires new-head CI. - Behavior changed: Yes - preserve files when commit publication is uncertain and finalize failed sorted output before outer file cleanup. - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69327, apache#69842 Problem Summary: MAP constant materialization reconstructs the column and checks that its final offset matches both child lengths. The out-of-range offset fixture invalidated that final offset and aborted before reaching the Iceberg validator. Keep the final offset consistent and use an invalid intermediate offset so the validator can reject the first-row range. ### Release note None ### Check List (For Author) - Test: Unit Test - run-be-ut.sh focused Iceberg validator, partition writer and table writer suites: 37 ASAN tests passed, zero failures/errors/disabled tests. clang-format, check-format and git diff --check passed. Changed-file clang-tidy has no diagnostics; full analysis retains the existing unmatched NOLINTEND in core/types.h:578. - Behavior changed: No - test fixture only. - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69327, apache#69842 Problem Summary: Keep sorted Iceberg writer finalization consistent with the master baseline used by this backport. Close the underlying partition writer explicitly before choosing the returned status, and retain the unit test that checks propagation of file-close failures. ### Release note None ### Check List (For Author) - Test: BE clang-format 16, check-format, and git diff --check passed. FE and BE builds are in progress; final-head CI will validate the rebased branch. - Behavior changed: Yes - use the master sorted-writer finalization flow. - Does this need documentation: No
19001f1 to
8a12e58
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69327, apache#69842 Problem Summary: EOS sorting, spill, merge, and write operations can throw a Doris allocation exception before the partition writer is finalized. Convert these exceptions to a failure Status before closing the partition writer and cleaning spill streams, so outer cleanup can remove the failed physical file. Include the allocation-failure unit test that checks file removal. ### Release note Fix orphan files after Iceberg sorted-write allocation failures during finalization. ### Check List (For Author) - Test: clang-format 16, repository check-format, and git diff --check passed. Local compilation, unit tests, and regression tests were not rerun as requested. - Behavior changed: Yes - finalize failed sorted output after caught Doris allocation exceptions and preserve the failure status for outer cleanup. - Does this need documentation: No
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
pick #69327