Repository navigation
[feature](paimon) Support writes using paimon-rust - #69886
suxiaogang223 wants to merge 6 commits into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69886 Problem Summary: When one BE fails, FE can roll back and unregister the query before a successful BE reports its prepared Paimon files. The late report is then discarded, leaving uncommitted files behind. This is reproducible with two BEs by delaying the successful writer's final report. Make rollback terminal and atomically hand off reported files under the transaction lock. Retain bounded contexts for recently aborted inserts so reports received after query removal, or racing with transaction removal, can still abort their files. Release payload references and perform storage I/O outside the lock. Never register committing, committed, or unknown-outcome transactions for late file deletion. Add fault injection and distributed regression coverage for BE prepare failure, delayed reports, failure before FE commit, errors after snapshot publication, and unknown commit outcomes. Process crashes before reporting files still require orphan-file maintenance. ### Release note Clean up uncommitted Paimon files when a successful BE reports after another BE has failed and FE has rolled back the INSERT. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - 10 Rust BE tests pass under ASAN; 31 focused FE tests pass. - Distributed failure suite passes on both FE distribution planners; expected output generated by the standard runner and verified normally. - Existing Paimon Rust append regression passes. - Manual two-BE SIGKILL test confirms failed INSERT publishes no snapshot, cleans the healthy writer's files, and succeeds after BE restart. - Full ASAN BE/FE build, updated FE build with Checkstyle, and clang-format 16 preset check pass. Clang-tidy is blocked by existing NOLINTEND and local system-header parsing errors. - Behavior changed: Yes, late reports for aborted Paimon inserts clean files - Does this need documentation: No, internal failure cleanup
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
9ae32af to
c489854
Compare
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: A pipeline can finish early or be cancelled before its sink opens. PaimonTableWriter.close must return the incoming status in that case, because neither its backend nor its profile has been created. Guard the whole close path instead of only its final profile update. Add coverage for successful early completion alongside cancellation. ### Release note Avoid a Paimon writer crash when a pipeline finishes before opening its sink. ### Check List (For Author) - Test: Added early-completion unit coverage; clang-format 16 and BE build hygiene passed. Native compilation and tests have not been run on master. - Behavior changed: Yes. Pre-open completion preserves its original status. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: The master plugin transaction manager removes a transaction before invoking the connector commit. A failure before snapshot publication therefore cannot rely on the engine's later rollback to clean up acknowledged BE files. Roll back within the Paimon commit failure path while still PREPARED; leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real prepared data files and verify that subsequent rollback does not abort twice. Add an exact license-check exclusion for the Rust-generated base64 protocol fixture, following existing golden-file exclusions; text headers would change its decoded bytes. Remove an obsolete comment describing FFI as future work. ### Release note Clean up unpublished Paimon data files when FE fails before attempting commit. ### Check List (For Author) - Test: FE and Java plugin build with Checkstyle passed. Paimon connector 904 tests, no failures/errors, one skipped, including the real-file cleanup and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax and license YAML validation passed. BE compilation and remaining FE/JNI tests are running; native and SQL validation are not claimed as passed. - Behavior changed: Yes. Pre-publication commit failures clean up immediately; uncertain publication outcomes still preserve files. - Does this need documentation: No.
|
run buildall |
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: Use the existing JNI Thrift serialization helper for the shared Paimon table descriptor, including its length and exception checks. Separate Java open arguments and cached Rust table initialization from their caller functions, and shorten repetitive comments so modified functions meet clang-tidy complexity and size limits. Apply pointer qualification and explicit aggregate member initialization reported by clang-tidy. ### Release note None ### Check List (For Author) - Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed for the affected writer/descriptor/reader paths; JNI source also passed full-file clang-tidy. FE/Java build and targeted tests passed before this C++ refactor. Final master ASAN BE build is running; native/SQL tests are not claimed as passed. - Behavior changed: No. Reuse existing checked JNI serialization. - Does this need documentation: No.
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
d2f5e2b to
496da98
Compare
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: A pipeline can finish early or be cancelled before its sink opens. PaimonTableWriter.close must return the incoming status in that case, because neither its backend nor its profile has been created. Guard the whole close path instead of only its final profile update. Add coverage for successful early completion alongside cancellation. ### Release note Avoid a Paimon writer crash when a pipeline finishes before opening its sink. ### Check List (For Author) - Test: Added early-completion unit coverage; clang-format 16 and BE build hygiene passed. Native compilation and tests have not been run on master. - Behavior changed: Yes. Pre-open completion preserves its original status. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: The master plugin transaction manager removes a transaction before invoking the connector commit. A failure before snapshot publication therefore cannot rely on the engine's later rollback to clean up acknowledged BE files. Roll back within the Paimon commit failure path while still PREPARED; leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real prepared data files and verify that subsequent rollback does not abort twice. Add an exact license-check exclusion for the Rust-generated base64 protocol fixture, following existing golden-file exclusions; text headers would change its decoded bytes. Remove an obsolete comment describing FFI as future work. ### Release note Clean up unpublished Paimon data files when FE fails before attempting commit. ### Check List (For Author) - Test: FE and Java plugin build with Checkstyle passed. Paimon connector 904 tests, no failures/errors, one skipped, including the real-file cleanup and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax and license YAML validation passed. BE compilation and remaining FE/JNI tests are running; native and SQL validation are not claimed as passed. - Behavior changed: Yes. Pre-publication commit failures clean up immediately; uncertain publication outcomes still preserve files. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: Use the existing JNI Thrift serialization helper for the shared Paimon table descriptor, including its length and exception checks. Separate Java open arguments and cached Rust table initialization from their caller functions, and shorten repetitive comments so modified functions meet clang-tidy complexity and size limits. Apply pointer qualification and explicit aggregate member initialization reported by clang-tidy. ### Release note None ### Check List (For Author) - Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed for the affected writer/descriptor/reader paths; JNI source also passed full-file clang-tidy. FE/Java build and targeted tests passed before this C++ refactor. Final master ASAN BE build is running; native/SQL tests are not claimed as passed. - Behavior changed: No. Reuse existing checked JNI serialization. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886, apache#70075, apache#69343 Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon 2.1.0. Rebase the writer on master instead of carrying separate dependency upgrades. Reuse the upstream Java v14 serializer with bounded accumulated output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate the real two-writer interop fixture. Reject old protocol requests before writer admission. Preserve master's vector-search field 42 and use optional field 43 for the shared descriptor; retain master's SPI additions and bump the write diagnostics surface to API 30. The previous BE UT crash was caused by a validation test bypassing init() and reading an uninitialized scan-params pointer. Explicitly initialize its missing-descriptor input. ### Release note Rust Paimon writes use the Java v14 commit protocol matching master's Paimon 2.1.0 SDK and inherit master's paimon-rust dependency version. ### Check List (For Author) - Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds, version rejection, message ranges and resource release; FE/Java build with Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests; six Paimon JNI static layout checks; Thrift generation/field audit; clang-format 16, header hygiene, shell syntax and whitespace checks. JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests, clang-tidy against the new build and SQL regression are deferred at the user's request to leave resources for another PR. - Behavior changed: Yes; Rust commit messages now use v14 instead of v11. - Does this need documentation: Yes; existing feature documentation remains pending with the draft PR.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: The upgraded paimon-rust writer needs native coverage of credit release, cleanup and the v14 transport. Use the same Parquet buffer option as the production descriptor instead of the generic write-buffer-size option, which does not control the Parquet buffer. Cover empty writes, prepare failure after rolling a file, and sustained 32 MiB dictionary input under a 16 MiB SDK input budget without rolling the file. Extend the memory-failure case to retry the same batch with a larger runtime limit. Check complete big-endian frame fields and the metadata credit retained until report acknowledgement. The exercised paths required no production logic changes. ### Release note None ### Check List (For Author) - Test: ASAN BE compilation and packaging with build.sh; 164 related BE tests passed through run-be-ut.sh (20 FFI writer tests, no failures or skips); clang-tidy for five modified production translation units and the full updated FFI test file; clang-format 16, header hygiene and whitespace checks. SQL/distributed regression was not rerun. - Behavior changed: No; test-only changes. - Does this need documentation: No; test-only changes.
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
|
run beut |
Related PR: apache#67395 Problem Summary: Port the branch-4.2 Rust append writer to master's connector framework and Paimon Java SDK 2.1.0. Prefer Rust for supported writes, share a pinned Thrift table descriptor across Rust reads and Rust/JNI writes, and retain online writer memory/concurrency settings. Use the existing acknowledged file-report protocol for ownership and cleanup, with a separate estimated credit for native commit metadata until reporting completes. Expose transaction diagnostics through connector-neutral SPI hooks (API 29.0). Keep Java v11 commit encoding in the released paimon-rust 0.4.0 patch; verify that SDK 2.1.0 decodes it. Preserve Rust 1.94.0. Fix closing a writer before open and reserve Fluss field 41 by assigning Paimon scan field 42. Support Paimon append writes using paimon-rust, enabled by default for supported targets with JNI selection for unsupported writes. Runtime failures fail the statement. Writer limits can be changed online for new writers; users can tune limits and retry confirmed uncommitted failures. - Test: FE/Java compilation and packaging with Checkstyle; Paimon connector 902 tests (one skipped); SPI 152; JNI Paimon 186; FE integration 36; S3/MinIO properties 37; plugin layout; clang-format 16; BE header hygiene. Master BE compilation/native tests and SQL regression remain pending. - Behavior changed: Yes. Supported append writes prefer Rust; table descriptions and transaction diagnostics use master's connector model. - Does this need documentation: Yes. User documentation is pending.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: A pipeline can finish early or be cancelled before its sink opens. PaimonTableWriter.close must return the incoming status in that case, because neither its backend nor its profile has been created. Guard the whole close path instead of only its final profile update. Add coverage for successful early completion alongside cancellation. ### Release note Avoid a Paimon writer crash when a pipeline finishes before opening its sink. ### Check List (For Author) - Test: Added early-completion unit coverage; clang-format 16 and BE build hygiene passed. Native compilation and tests have not been run on master. - Behavior changed: Yes. Pre-open completion preserves its original status. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: The master plugin transaction manager removes a transaction before invoking the connector commit. A failure before snapshot publication therefore cannot rely on the engine's later rollback to clean up acknowledged BE files. Roll back within the Paimon commit failure path while still PREPARED; leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real prepared data files and verify that subsequent rollback does not abort twice. Add an exact license-check exclusion for the Rust-generated base64 protocol fixture, following existing golden-file exclusions; text headers would change its decoded bytes. Remove an obsolete comment describing FFI as future work. ### Release note Clean up unpublished Paimon data files when FE fails before attempting commit. ### Check List (For Author) - Test: FE and Java plugin build with Checkstyle passed. Paimon connector 904 tests, no failures/errors, one skipped, including the real-file cleanup and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax and license YAML validation passed. BE compilation and remaining FE/JNI tests are running; native and SQL validation are not claimed as passed. - Behavior changed: Yes. Pre-publication commit failures clean up immediately; uncertain publication outcomes still preserve files. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: Use the existing JNI Thrift serialization helper for the shared Paimon table descriptor, including its length and exception checks. Separate Java open arguments and cached Rust table initialization from their caller functions, and shorten repetitive comments so modified functions meet clang-tidy complexity and size limits. Apply pointer qualification and explicit aggregate member initialization reported by clang-tidy. ### Release note None ### Check List (For Author) - Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed for the affected writer/descriptor/reader paths; JNI source also passed full-file clang-tidy. FE/Java build and targeted tests passed before this C++ refactor. Final master ASAN BE build is running; native/SQL tests are not claimed as passed. - Behavior changed: No. Reuse existing checked JNI serialization. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#69886, apache#70075, apache#69343 Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon 2.1.0. Rebase the writer on master instead of carrying separate dependency upgrades. Reuse the upstream Java v14 serializer with bounded accumulated output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate the real two-writer interop fixture. Reject old protocol requests before writer admission. Preserve master's vector-search field 42 and use optional field 43 for the shared descriptor; retain master's SPI additions and bump the write diagnostics surface to API 30. The previous BE UT crash was caused by a validation test bypassing init() and reading an uninitialized scan-params pointer. Explicitly initialize its missing-descriptor input. ### Release note Rust Paimon writes use the Java v14 commit protocol matching master's Paimon 2.1.0 SDK and inherit master's paimon-rust dependency version. ### Check List (For Author) - Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds, version rejection, message ranges and resource release; FE/Java build with Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests; six Paimon JNI static layout checks; Thrift generation/field audit; clang-format 16, header hygiene, shell syntax and whitespace checks. JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests, clang-tidy against the new build and SQL regression are deferred at the user's request to leave resources for another PR. - Behavior changed: Yes; Rust commit messages now use v14 instead of v11. - Does this need documentation: Yes; existing feature documentation remains pending with the draft PR.
### What problem does this PR solve? Related PR: apache#69886 Problem Summary: The upgraded paimon-rust writer needs native coverage of credit release, cleanup and the v14 transport. Use the same Parquet buffer option as the production descriptor instead of the generic write-buffer-size option, which does not control the Parquet buffer. Cover empty writes, prepare failure after rolling a file, and sustained 32 MiB dictionary input under a 16 MiB SDK input budget without rolling the file. Extend the memory-failure case to retry the same batch with a larger runtime limit. Check complete big-endian frame fields and the metadata credit retained until report acknowledgement. The exercised paths required no production logic changes. ### Release note None ### Check List (For Author) - Test: ASAN BE compilation and packaging with build.sh; 164 related BE tests passed through run-be-ut.sh (20 FFI writer tests, no failures or skips); clang-tidy for five modified production translation units and the full updated FFI test file; clang-format 16, header hygiene and whitespace checks. SQL/distributed regression was not rerun. - Behavior changed: No; test-only changes. - Does this need documentation: No; test-only changes.
8cf2388 to
4078a16
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
Local review with the Doris Reviewed the Rust/JNI selection, shared pinned descriptor, v14 commit transport, memory admission/release, report ownership and FE commit/rollback flow. The six feature patches are unchanged by the rebase. Current-head CI compilation, FE UT and Cloud UT have passed. BE UT and regression jobs are still running. This review does not mark unfinished tests as passed. Earlier local validation covered 164 related BE tests (20 FFI writer tests) and 1,541 FE/Java tests (one skip); those were not rerun during this review. Local formatting with clang-format 16, build hygiene, shell syntax and cross-release Thrift checks passed. Doris skill checkpoints
schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69886
commit: 4078a16c9b67a4467275f2dfc31e125eba4a42d1
base: 913532ce1673595cd552bd6f582e5b2640f45dd9
reviewed_at: 2026-11-05T07:37:01+00:00
reviewer: suxiaogang223
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
What problem does this PR solve?
Problem Summary:
Support Paimon append writes through the paimon-rust C FFI.
bucket=-1. Unsupported writes select JNI before consuming input; runtime failures fail the statement.enable_paimon_rust_writer=falseforces JNI.Detected memory or admission failures fail the INSERT. Users can adjust
paimon_rust_writer_memory_limit_bytes(default 64 MiB) andpaimon_rust_writer_max_concurrency(default 8 per BE) online through/api/update_config, then retry confirmed uncommitted failures without restarting Doris. Updates apply to new writers. Verify unknown commit outcomes before retrying to avoid duplicate appends.Memory accounting is estimated, not a complete Rust allocator or RSS limit. Spill and memory revocation are outside this implementation.
BE accesses the existing Paimon catalog storage endpoint. Configuration requires existing catalog privileges; writes require INSERT privileges.
Release note
Support Paimon append writes using paimon-rust, preferred by default for supported tables, with JNI for unsupported writes. Writer memory and concurrency limits can be adjusted online for subsequent writers without restarting Doris.
Check List (For Author)
Test:
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)