### What problem does this PR solve?

Issue Number: None

Related PR: apache#69032, apache#69327

Problem Summary: Merge official upstream/master at 67f31fa. Its unified failed-file cleanup overlaps with the spatial writer open-error cleanup. Keep upstream rollback callbacks and repeated-close behavior while retaining blocking close and writer destruction before callback or final deletion. Preserve both sets of tests and cover asynchronous close failure before outer cleanup.

### Release note

None

### Check List (For Author)

- Test: Unit Test - VIcebergPartitionWriterTest.*, VIcebergTableWriterTest.*, IcebergWriteValidatorTest.* (25 cases, clang21/ASAN); clang-format 16, build hygiene and clang-tidy (no warnings).
- Behavior changed: No; preserve spatial failed-write cleanup while integrating upstream lifecycle handling.
- Does this need documentation: No; merge resolution only.

### What problem does this PR solve?

Related PR: apache#69327

The Iceberg validator now rejects required NULLs before format-specific
writers run. Two regression suites still expect generic file-writer errors.
Update all six assertions, preserving the offending field and NULL reason,
including the full arr.element and m.value paths. Keep metadata, write,
post-ALTER nullability, and Spark comparison coverage unchanged.

### Release note

None

### Check List (For Author)

- Test: Regression test
    - Same CI binary package: both original suites fail locally.
    - Both corrected suites pass: 2 passed, 0 failed, 0 skipped.
    - Full diff self-review and git diff --check passed.
- Behavior changed: No product behavior change.
- Does this need documentation: No.

…9416)

### What problem does this PR solve?

This PR corrects three external-regression test issues:

- `test_iceberg_modify_binary_column`: the initial golden treats the
required `digest` field as nullable. Regenerate that expectation as
non-nullable; retain the nullable expectation after `ALTER ... NULL`.
- `test_iceberg_show_nullable` and `test_iceberg_modify_binary_column`:
their six negative-write assertions still expect generic file-writer
errors. After #69327, Iceberg validates required NULLs before creating
writers/files and reports `Iceberg required field '<path>' contains
NULL`. Update the expected diagnostics for `id`, `value`, `digest`,
`arr.element`, and `m.value`. The first two errors stop their respective
suites; the remaining four assertions have the same stale expectation.
The writes correctly fail, so this fixes test expectations without
changing product behavior or weakening the required-field checks.
- `test_paimon_hms_catalog`: use the recognized `fs.gcs.support`
property instead of `fs.gs.support` for explicit GCS-provider selection,
retaining both credential-property variants.

The Iceberg coverage still checks metadata nullability, rejected NULL
writes, successful writes after constraints are relaxed, nested fields,
and Spark read-back. Both TVF suites retain their original fixture
handling.

Related PRs: #68640, #69285, #69327.

### Validation

Current failure fix, using the same CI binary package (FE `436941a8bb`,
BE `0a77e2dd98`; package master revision `e779124b5f60`):
- Ran both original suites locally with `run-regression-test.sh`: 2
failed, reproducing the exact stale exception assertions for `digest`
and `id`.
- Updated all six assertions and reran both complete suites in normal
comparison mode: **2 passed, 0 failed, 0 fatal, 0 skipped**. All six
expected NULL rejections, post-ALTER writes, metadata/snapshot checks,
and Spark read-back passed. No golden output was regenerated for this
assertion-only update.
- One intermediate run encountered `SQLITE_BUSY` while the local REST
fixture cleaned up an existing table. Restarted that fixture, then reran
both complete suites successfully; the environment failure is not
counted as a passing validation.
- Reviewed the full diff and passed `git diff --check`. Only regression
scripts change in this update; no FE/BE source changes.

Earlier validation:
- Reproduced the initial Iceberg `digest` nullability mismatch with
runtime `6ca5ef36d8`. Generated the golden with `run-regression-test.sh
-forceGenOut`, then passed the complete binary-column suite in normal
comparison mode, including Spark read-back.
- Loaded the actual GCS directory plugin and checked both `gs.*` and
`s3.*` credential variants: the old flag fails `gs` filesystem
resolution; the corrected flag selects GCS and resolves `S3AFileSystem`.
This checks plugin binding and filesystem resolution; the full remote
Paimon HMS/GCS suite was not run locally.
- Regression framework build and its 50 tests passed.

### Release note

None

### Check List (For Author)

- Test:
    - [x] Regression test
    - [x] Manual test
- Behavior changed:
    - [x] No product behavior change.
- Does this need documentation?
    - [x] No.

…ash (#69487)

### What problem does this PR solve?

Issue Number: None

Related PR: #69223, #69174, #69327, #69183

Problem Summary:

BE UT of every PR based on master fails since 2026-10-22. Two test-only
breakages were merged by PRs whose own CI ran on an older master.

1. doris_be_test does not compile (TeamCity: "BE BUILD FAIL"):

be/test/exprs/function/function_string_test.cpp:2971: error: no matching
conversion for functional-style cast from 'std::string' to 'Field'
be/test/exprs/function/function_string_test.cpp:2982: error: no member
       named 'FRAGMENT_LOCAL' in 'doris::FunctionContext'

function_translate_mixed_const_map_test, added by #69223, builds a Field
   directly from std::string and uses FunctionContext::FRAGMENT_LOCAL.
Field must be created with Field::create_field<TYPE_STRING>() now, and
#69174 made FunctionStateScope an enum class. Use both current
spellings,
   as the rest of be/test does.

2. Once it compiles, doris_be_test aborts (TeamCity: "CORE DUMPED") in
DefaultAndConfiguredLimits/ExternalTableWriterPartitionLimitTest.
IcebergRejectsNextPartitionBeforeOpeningAFile/0:

       shared_ptr_base.h:1344: [_Tp = doris::iceberg::Schema]:
       Assertion '__p != nullptr' failed.

#69327 made VIcebergTableWriter validate every block against the schema
pinned by open(): `iceberg::validate_write_block(*_schema, ...)`. The
test added by #69183 skips open() and calls write_prepared_block()
directly, so _schema is null. open() always sets _schema in production,
so the writer stays unchanged. The test now pins a one-field schema that
matches its block (optional INT "p", the source field of its identity
partition), as viceberg_table_writer_test.cpp does.

Co-authored-by: csun5285 <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>

…rom master

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69327, apache#69183, apache#69487

Problem Summary: The first commit of this PR takes the async-writer
dependencies out of VIcebergTableWriter, so its constructor takes only the
sink and the output expressions. Master has since added three tests that
still pass two null dependencies: RejectsRequiredNullBeforeWriterCreation
and RejectsNestedRequiredNullBeforeWriterCreation in
viceberg_table_writer_test.cpp (apache#69327), and
IcebergRejectsNextPartitionBeforeOpeningAFile in
external_table_writer_partition_limit_test.cpp (apache#69183, which apache#69487 made
pin the writer schema). On that master doris_be_test does not compile.
Construct the writer in these tests with the two arguments; nothing else in
them changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - doris_be_test builds on master bf278a5 with this PR; 63 tests in
      VIcebergTableWriterTest, VIcebergTableWriterLifecycleTest,
      ExternalTableWriterPartitionLimitTest, IcebergTableSinkOperatorTest,
      SpillIcebergTableSinkOperatorTest, VIcebergPartitionWriterTest,
      IcebergWriteValidatorTest, FullSorterTest and SorterReserveMemoryTest
      pass (ASAN)
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…rom master

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69327, apache#69183, apache#69487, apache#69358

Problem Summary: The first commit of this PR takes the async-writer
dependencies out of VIcebergTableWriter, so its constructor takes only the
sink and the output expressions. Master has since added four tests that
still pass two null dependencies: RejectsRequiredNullBeforeWriterCreation
and RejectsNestedRequiredNullBeforeWriterCreation (apache#69327) and
VoidPartitionFieldNeedsNoSourceColumn (apache#69358) in
viceberg_table_writer_test.cpp, and
IcebergRejectsNextPartitionBeforeOpeningAFile in
external_table_writer_partition_limit_test.cpp (apache#69183, which apache#69487 made
pin the writer schema). On that master doris_be_test does not compile.
Construct the writer in these tests with the two arguments; nothing else in
them changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - doris_be_test builds on master 2e7f79e with this PR; 64 tests in
      VIcebergTableWriterTest, VIcebergTableWriterLifecycleTest,
      ExternalTableWriterPartitionLimitTest, IcebergTableSinkOperatorTest,
      SpillIcebergTableSinkOperatorTest, VIcebergPartitionWriterTest,
      IcebergWriteValidatorTest, FullSorterTest and SorterReserveMemoryTest
      pass (ASAN)
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Issue Number: N/A

Related PR: apache#69327

Problem Summary:
Iceberg ORC and Parquet writes could accept NULL required values, including
nested STRUCT/LIST/MAP fields and MAP keys. Failed writes could also leave
physical files produced by writers or completed BE fragments behind.

Validate every prepared block before creating or dispatching partition writers,
while preserving optional-parent NULL masking. Destroy failed writer streams
before deferring file deletion to the existing outer cleanup lifecycle.
Adapt FE cleanup to branch-4.2's legacy IcebergTransaction and retained catalog
lease. Clean failed-commit files before releasing the lease, and retain files
when Iceberg reports an unknown commit state. Carry over the necessary sorted
writer close error propagation that the original PR's baseline already has.
Split container validation into focused helpers for the target lint thresholds.

Iceberg ORC/Parquet writes reject required NULL values and clean up failed
files without publishing a snapshot.

- Test: Modified Java files pass Checkstyle; repository clang-format 16 and
  whitespace checks pass; regression framework compilation and 47 framework
  unit tests pass. FE build and focused unit-test attempts were interrupted by
  confirmed host OOM kills. BE ASAN product/UT configuration succeeds, but
  full compilation and BE unit execution remain incomplete. Both focused
  regression suites were attempted and failed connecting to the isolated FE.
  Validator clang-tidy diagnostics were fixed; the check remains blocked by
  an existing unmatched NOLINTEND in core/types.h.
- Behavior changed: Yes, invalid required NULL writes are rejected and failed
  external files are cleaned up.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: A REST catalog can publish an Iceberg snapshot before a lost
or malformed response raises an exception. Preserve reported files after SDK
commit entry unless a CleanableFailure proves deletion safe, including on a
later rollback. Convert sorted EOS allocation exceptions to Status before
unconditional partition finalization so outer cleanup can remove the failed
file. Add published-snapshot and persisted-file tests, cover missing container
validation branches, and update nullable regression error expectations.

### Release note

Prevent deletion of potentially committed Iceberg files after an indeterminate
commit response, and remove failed sorted-write files after EOS allocation
errors.

### Check List (For Author)

- Test: Unit Test - IcebergTransactionTest, 39 passed; local FE changed executable
  line coverage 34/38 (89.47%). Related BE production sources compiled under ASAN;
  the focused BE unit-test build is still running. Java Checkstyle and BE
  clang-format/check-format passed. External regression requires new-head CI.
- Behavior changed: Yes - preserve files when commit publication is uncertain
  and finalize failed sorted output before outer file cleanup.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: MAP constant materialization reconstructs the column and
checks that its final offset matches both child lengths. The out-of-range
offset fixture invalidated that final offset and aborted before reaching the
Iceberg validator. Keep the final offset consistent and use an invalid
intermediate offset so the validator can reject the first-row range.

### Release note

None

### Check List (For Author)

- Test: Unit Test - run-be-ut.sh focused Iceberg validator, partition writer and
  table writer suites: 37 ASAN tests passed, zero failures/errors/disabled tests.
  clang-format, check-format and git diff --check passed. Changed-file
  clang-tidy has no diagnostics; full analysis retains the existing unmatched
  NOLINTEND in core/types.h:578.
- Behavior changed: No - test fixture only.
- Does this need documentation: No

Issue Number: N/A

Related PR: apache#69327

Problem Summary:
Iceberg ORC and Parquet writes could accept NULL required values, including
nested STRUCT/LIST/MAP fields and MAP keys. Failed writes could also leave
physical files produced by writers or completed BE fragments behind.

Validate every prepared block before creating or dispatching partition writers,
while preserving optional-parent NULL masking. Destroy failed writer streams
before deferring file deletion to the existing outer cleanup lifecycle.
Adapt FE cleanup to branch-4.2's legacy IcebergTransaction and retained catalog
lease. Clean failed-commit files before releasing the lease, and retain files
when Iceberg reports an unknown commit state. Carry over the necessary sorted
writer close error propagation that the original PR's baseline already has.
Split container validation into focused helpers for the target lint thresholds.

Iceberg ORC/Parquet writes reject required NULL values and clean up failed
files without publishing a snapshot.

- Test: Modified Java files pass Checkstyle; repository clang-format 16 and
  whitespace checks pass; regression framework compilation and 47 framework
  unit tests pass. FE build and focused unit-test attempts were interrupted by
  confirmed host OOM kills. BE ASAN product/UT configuration succeeds, but
  full compilation and BE unit execution remain incomplete. Both focused
  regression suites were attempted and failed connecting to the isolated FE.
  Validator clang-tidy diagnostics were fixed; the check remains blocked by
  an existing unmatched NOLINTEND in core/types.h.
- Behavior changed: Yes, invalid required NULL writes are rejected and failed
  external files are cleaned up.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: A REST catalog can publish an Iceberg snapshot before a lost
or malformed response raises an exception. Preserve reported files after SDK
commit entry unless a CleanableFailure proves deletion safe, including on a
later rollback. Convert sorted EOS allocation exceptions to Status before
unconditional partition finalization so outer cleanup can remove the failed
file. Add published-snapshot and persisted-file tests, cover missing container
validation branches, and update nullable regression error expectations.

### Release note

Prevent deletion of potentially committed Iceberg files after an indeterminate
commit response, and remove failed sorted-write files after EOS allocation
errors.

### Check List (For Author)

- Test: Unit Test - IcebergTransactionTest, 39 passed; local FE changed executable
  line coverage 34/38 (89.47%). Related BE production sources compiled under ASAN;
  the focused BE unit-test build is still running. Java Checkstyle and BE
  clang-format/check-format passed. External regression requires new-head CI.
- Behavior changed: Yes - preserve files when commit publication is uncertain
  and finalize failed sorted output before outer file cleanup.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: MAP constant materialization reconstructs the column and
checks that its final offset matches both child lengths. The out-of-range
offset fixture invalidated that final offset and aborted before reaching the
Iceberg validator. Keep the final offset consistent and use an invalid
intermediate offset so the validator can reject the first-row range.

### Release note

None

### Check List (For Author)

- Test: Unit Test - run-be-ut.sh focused Iceberg validator, partition writer and
  table writer suites: 37 ASAN tests passed, zero failures/errors/disabled tests.
  clang-format, check-format and git diff --check passed. Changed-file
  clang-tidy has no diagnostics; full analysis retains the existing unmatched
  NOLINTEND in core/types.h:578.
- Behavior changed: No - test fixture only.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: Keep sorted Iceberg writer finalization consistent with the
master baseline used by this backport. Close the underlying partition writer
explicitly before choosing the returned status, and retain the unit test that
checks propagation of file-close failures.

### Release note

None

### Check List (For Author)

- Test: BE clang-format 16, check-format, and git diff --check passed. FE and BE
  builds are in progress; final-head CI will validate the rebased branch.
- Behavior changed: Yes - use the master sorted-writer finalization flow.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69327, apache#69842

Problem Summary: EOS sorting, spill, merge, and write operations can throw a
Doris allocation exception before the partition writer is finalized. Convert
these exceptions to a failure Status before closing the partition writer and
cleaning spill streams, so outer cleanup can remove the failed physical file.
Include the allocation-failure unit test that checks file removal.

### Release note

Fix orphan files after Iceberg sorted-write allocation failures during finalization.

### Check List (For Author)

- Test: clang-format 16, repository check-format, and git diff --check passed.
  Local compilation, unit tests, and regression tests were not rerun as requested.
- Behavior changed: Yes - finalize failed sorted output after caught Doris
  allocation exceptions and preserve the failure status for outer cleanup.
- Does this need documentation: No