Skip to content

branch-4.2: [fix](iceberg) Validate required NULLs before file creation (#69327) - #69842

Open
hubgeter wants to merge 5 commits into
apache:branch-4.2from
hubgeter:backport/69327-branch-4.2
Open

hubgeter wants to merge 5 commits into
apache:branch-4.2from
hubgeter:backport/69327-branch-4.2

Conversation

@hubgeter

@hubgeter hubgeter commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

pick #69327

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two verified regressions remain: [P1] cleanup can delete files from an already published REST snapshot after response loss; [P2] a thrown allocation failure during sorted EOS can bypass file cleanup. Both are reported inline. There are no existing inline findings to carry forward, and no additional user-provided focus.

Reviewed the complete 15-file authoritative diff at head 8b8880d and traced the INSERT/overwrite, DELETE, MERGE, rewrite, sorter/spill, serializer, report, and transaction-owner paths. Static review only: no builds, tests, benchmarks, source edits, or submodule initialization.

Critical checkpoint conclusions:

  • Goal and proof: required NULL/map-key checks run before partition writer creation; unit and ORC/Parquet regression cases cover the intended rejection and optional-parent behavior. The two failure paths below remain uncovered by those tests.
  • Scope: changes stay within Iceberg validation, close/failure cleanup, and associated tests.
  • Concurrency: report append and rollback snapshot/clear share the transaction monitor; remote deletion occurs outside it. Sort write/close/revoke share the sorter mutex. No new lock inversion identified.
  • Lifecycle: reset destroys streams before waiting for S3/HDFS writers and invoking cleanup callbacks; retained FE authentication/FileIO survives cleanup. Sorted exception unwinding has the inline P2 defect. No new global initialization dependency.
  • Configuration: no new configuration item.
  • Compatibility: no changed wire/storage layout or function signature; existing mixed-version behavior remains.
  • Thrift/protobuf: no gensrc definitions changed, so cross-release ID comparisons are inapplicable.
  • Parallel paths: ordinary, sorted/spilled, and MERGE data writers share validation; append/overwrite/DELETE/MERGE/rewrite transaction cleanup was traced.
  • Conditions: optional container parents mask descendants; constant materialization and padded collection offsets agree with this branch's column invariants and serializers.
  • Test coverage: scalar/constant/nested/list/map rejection, optional parents, pre-creation rejection, close Status failure, repeated failed close, lease ordering, and explicit unknown commit state are covered in inspected tests. REST response loss and thrown EOS allocation failure need targeted cases.
  • Expected results: no .out changes; Groovy baselines check published snapshot/file/row counts. Nested invalid input can be rejected by FE planning.
  • Observability: validation reports field path and row/ordinal; cleanup logs per-file deletion failures and retains the original query error. No separate missing metric identified.
  • Transactions/persistence: publication remains Iceberg metadata CAS with no new Doris EditLog state. The inline P1 violates committed-file visibility after an indeterminate REST commit; MoW/version machinery is untouched.
  • Writes and crashes: rollback deletes newly reported paths, not referenced source files. Per-file deletion errors continue cleanup. No new crash recovery is implemented; sorted exceptions leave the inline P2 orphan.
  • FE/BE propagation: no new transmitted variable; pinned schema/projection and row-lineage shapes were checked.
  • Performance/memory: primitive required fields use a null-map fast path; rollback deduplicates paths and scales linearly. No substantiated additional performance or memory-accounting defect found.
  • Other issues and limits: all changed-file scope and initial risks have dispositions. No additional distinct issue found. Physical orphan absence, async upload failure behavior, and REST reconciliation were not runtime-validated; metadata-only regression assertions do not prove those properties.

Review converged after 2 round(s).

}
throw e;
} catch (RuntimeException e) {
cleanupReportedFiles();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve files for indeterminate REST commit failures

A REST server can publish the snapshot and then lose or truncate the response. In the pinned Iceberg 1.11.0, HTTPClient.execute converts response I/O/parsing failures to RESTException, so these bypass the CommitStateUnknownException catch. BaseTransaction deliberately preserves files for such non-CleanableFailure exceptions under strict cleanup. This new catch instead deletes every reported data/delete file, leaving the published snapshot referencing missing objects. Gate deletion on a definitive commit failure and preserve the files through the subsequent rollback() too; the explicit unknown-state test does not cover this response-loss case.

return internal_status;
}
// Form the return value only after the underlying close runs; a deferred assignment is too late.
Status close_status =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep partition finalization safe when sorting throws

For a spill-enabled sorted INSERT, EOS reaches this code from SpillIcebergTableSinkLocalState::_close_writer(). do_sort()/merge/write can throw, for example when get_permutation() or permute() allocation raises MEM_ALLOC_FAILED. Removing the Defer lets that bypass partition close, while close() still sets _closed = true during unwinding. The scheduler converts the exception and retries outer close with the failure, but the sorted writer now returns immediately: the current path never enters _closed_files or BE commit data. Its stream is eventually closed during destruction, leaving an orphan that neither BE nor FE rollback knows about (HDFS already creates it at open). Convert intermediate exceptions into internal_status before unconditional partition finalization, or retain exception-safe cleanup ownership. Add an EOS allocation-failure case that checks physical file removal.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 73.72% (244/331) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.59% (30167/46702)
Line Coverage 49.44% (318835/644894)
Region Coverage 45.20% (256713/567917)
Branch Coverage 46.88% (121234/258621)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 75.68% (28/37) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 73.72% (244/331) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.33% (34619/45356)
Line Coverage 61.18% (391241/639511)
Region Coverage 57.98% (329796/568806)
Branch Coverage 58.95% (152275/258327)

@hubgeter
hubgeter force-pushed the backport/69327-branch-4.2 branch from 8b8880d to 261176c Compare November 3, 2026 11:22
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: A REST catalog can publish an Iceberg snapshot before a lost
or malformed response raises an exception. Preserve reported files after SDK
commit entry unless a CleanableFailure proves deletion safe, including on a
later rollback. Convert sorted EOS allocation exceptions to Status before
unconditional partition finalization so outer cleanup can remove the failed
file. Add published-snapshot and persisted-file tests, cover missing container
validation branches, and update nullable regression error expectations.

### Release note

Prevent deletion of potentially committed Iceberg files after an indeterminate
commit response, and remove failed sorted-write files after EOS allocation
errors.

### Check List (For Author)

- Test: Unit Test - IcebergTransactionTest, 39 passed; local FE changed executable
  line coverage 34/38 (89.47%). Related BE production sources compiled under ASAN;
  the focused BE unit-test build is still running. Java Checkstyle and BE
  clang-format/check-format passed. External regression requires new-head CI.
- Behavior changed: Yes - preserve files when commit publication is uncertain
  and finalize failed sorted output before outer file cleanup.
- Does this need documentation: No
@hubgeter

hubgeter commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

hubgeter commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41627042124

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: MAP constant materialization reconstructs the column and
checks that its final offset matches both child lengths. The out-of-range
offset fixture invalidated that final offset and aborted before reaching the
Iceberg validator. Keep the final offset consistent and use an invalid
intermediate offset so the validator can reject the first-row range.

### Release note

None

### Check List (For Author)

- Test: Unit Test - run-be-ut.sh focused Iceberg validator, partition writer and
  table writer suites: 37 ASAN tests passed, zero failures/errors/disabled tests.
  clang-format, check-format and git diff --check passed. Changed-file
  clang-tidy has no diagnostics; full analysis retains the existing unmatched
  NOLINTEND in core/types.h:578.
- Behavior changed: No - test fixture only.
- Does this need documentation: No
@hubgeter

hubgeter commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

hubgeter commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41639116676

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Issue Number: N/A

Related PR: apache#69327

Problem Summary:
Iceberg ORC and Parquet writes could accept NULL required values, including
nested STRUCT/LIST/MAP fields and MAP keys. Failed writes could also leave
physical files produced by writers or completed BE fragments behind.

Validate every prepared block before creating or dispatching partition writers,
while preserving optional-parent NULL masking. Destroy failed writer streams
before deferring file deletion to the existing outer cleanup lifecycle.
Adapt FE cleanup to branch-4.2's legacy IcebergTransaction and retained catalog
lease. Clean failed-commit files before releasing the lease, and retain files
when Iceberg reports an unknown commit state. Carry over the necessary sorted
writer close error propagation that the original PR's baseline already has.
Split container validation into focused helpers for the target lint thresholds.

Iceberg ORC/Parquet writes reject required NULL values and clean up failed
files without publishing a snapshot.

- Test: Modified Java files pass Checkstyle; repository clang-format 16 and
  whitespace checks pass; regression framework compilation and 47 framework
  unit tests pass. FE build and focused unit-test attempts were interrupted by
  confirmed host OOM kills. BE ASAN product/UT configuration succeeds, but
  full compilation and BE unit execution remain incomplete. Both focused
  regression suites were attempted and failed connecting to the isolated FE.
  Validator clang-tidy diagnostics were fixed; the check remains blocked by
  an existing unmatched NOLINTEND in core/types.h.
- Behavior changed: Yes, invalid required NULL writes are rejected and failed
  external files are cleaned up.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: A REST catalog can publish an Iceberg snapshot before a lost
or malformed response raises an exception. Preserve reported files after SDK
commit entry unless a CleanableFailure proves deletion safe, including on a
later rollback. Convert sorted EOS allocation exceptions to Status before
unconditional partition finalization so outer cleanup can remove the failed
file. Add published-snapshot and persisted-file tests, cover missing container
validation branches, and update nullable regression error expectations.

### Release note

Prevent deletion of potentially committed Iceberg files after an indeterminate
commit response, and remove failed sorted-write files after EOS allocation
errors.

### Check List (For Author)

- Test: Unit Test - IcebergTransactionTest, 39 passed; local FE changed executable
  line coverage 34/38 (89.47%). Related BE production sources compiled under ASAN;
  the focused BE unit-test build is still running. Java Checkstyle and BE
  clang-format/check-format passed. External regression requires new-head CI.
- Behavior changed: Yes - preserve files when commit publication is uncertain
  and finalize failed sorted output before outer file cleanup.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: MAP constant materialization reconstructs the column and
checks that its final offset matches both child lengths. The out-of-range
offset fixture invalidated that final offset and aborted before reaching the
Iceberg validator. Keep the final offset consistent and use an invalid
intermediate offset so the validator can reject the first-row range.

### Release note

None

### Check List (For Author)

- Test: Unit Test - run-be-ut.sh focused Iceberg validator, partition writer and
  table writer suites: 37 ASAN tests passed, zero failures/errors/disabled tests.
  clang-format, check-format and git diff --check passed. Changed-file
  clang-tidy has no diagnostics; full analysis retains the existing unmatched
  NOLINTEND in core/types.h:578.
- Behavior changed: No - test fixture only.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: Keep sorted Iceberg writer finalization consistent with the
master baseline used by this backport. Close the underlying partition writer
explicitly before choosing the returned status, and retain the unit test that
checks propagation of file-close failures.

### Release note

None

### Check List (For Author)

- Test: BE clang-format 16, check-format, and git diff --check passed. FE and BE
  builds are in progress; final-head CI will validate the rebased branch.
- Behavior changed: Yes - use the master sorted-writer finalization flow.
- Does this need documentation: No
@hubgeter
hubgeter force-pushed the backport/69327-branch-4.2 branch from 19001f1 to 8a12e58 Compare November 10, 2026 07:33
@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/42776256656

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: EOS sorting, spill, merge, and write operations can throw a
Doris allocation exception before the partition writer is finalized. Convert
these exceptions to a failure Status before closing the partition writer and
cleaning spill streams, so outer cleanup can remove the failed physical file.
Include the allocation-failure unit test that checks file removal.

### Release note

Fix orphan files after Iceberg sorted-write allocation failures during finalization.

### Check List (For Author)

- Test: clang-format 16, repository check-format, and git diff --check passed.
  Local compilation, unit tests, and regression tests were not rerun as requested.
- Behavior changed: Yes - finalize failed sorted output after caught Doris
  allocation exceptions and preserve the failure status for outer cleanup.
- Does this need documentation: No
@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/42785416782

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hubgeter
hubgeter marked this pull request as ready for review November 10, 2026 09:24
@hubgeter
hubgeter requested a review from yiguolei as a code owner November 10, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants