Skip to content

[feature](paimon) Support writes using paimon-rust - #69886

Open
suxiaogang223 wants to merge 6 commits into
apache:masterfrom
suxiaogang223:codex/branch-4.2-paimon-rust-write
Open

suxiaogang223 wants to merge 6 commits into
apache:masterfrom
suxiaogang223:codex/branch-4.2-paimon-rust-write

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Problem Summary:

Support Paimon append writes through the paimon-rust C FFI.

  • Prefer Rust by default for supported main-branch, unpartitioned append-only Parquet tables with bucket=-1. Unsupported writes select JNI before consuming input; runtime failures fail the statement. enable_paimon_rust_writer=false forces JNI.
  • Share a lightweight, pinned table descriptor between Rust reads and Rust/JNI writes. BE produces Java-compatible commit messages; FE publishes the snapshot through the existing transaction framework.
  • Bound input batches, reserve estimated writer memory, limit concurrent writers per BE, and expose writer/resource/transaction diagnostics.
  • Release memory and admission slots on close or failure. Clean up files on confirmed aborts; preserve files when report ownership or commit outcome is uncertain.

Detected memory or admission failures fail the INSERT. Users can adjust paimon_rust_writer_memory_limit_bytes (default 64 MiB) and paimon_rust_writer_max_concurrency (default 8 per BE) online through /api/update_config, then retry confirmed uncommitted failures without restarting Doris. Updates apply to new writers. Verify unknown commit outcomes before retrying to avoid duplicate appends.

Memory accounting is estimated, not a complete Rust allocator or RSS limit. Spill and memory revocation are outside this implementation.

BE accesses the existing Paimon catalog storage endpoint. Configuration requires existing catalog privileges; writes require INSERT privileges.

Release note

Support Paimon append writes using paimon-rust, preferred by default for supported tables, with JNI for unsupported writes. Writer memory and concurrency limits can be adjusted online for subsequent writers without restarting Doris.

Check List (For Author)

  • Test:

    • Unit Test: 164 related BE tests passed, including 20 FFI writer tests covering memory release, online limit adjustment and retry, failure cleanup, report ownership and commit serialization.
    • FE/Java tests: 1,541 tests, one skipped, no failures or errors; includes decoding and re-encoding real Rust commit messages in Java.
    • ASAN BE and FE builds; Checkstyle, clang-format 16, clang-tidy and BE header hygiene checks passed.
    • SQL regression and distributed failure verification on the current revision.
  • Behavior changed:

    • No.
    • Yes. Supported append writes prefer Rust; unsupported writes select JNI before consuming input. Runtime failures fail the statement.
  • Does this need documentation?

    • No.
    • Yes. User documentation for the switch, supported tables and online limits is pending.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Oct 30, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69886

Problem Summary: When one BE fails, FE can roll back and unregister the query
before a successful BE reports its prepared Paimon files. The late report is
then discarded, leaving uncommitted files behind. This is reproducible with
two BEs by delaying the successful writer's final report.

Make rollback terminal and atomically hand off reported files under the
transaction lock. Retain bounded contexts for recently aborted inserts so
reports received after query removal, or racing with transaction removal,
can still abort their files. Release payload references and perform storage
I/O outside the lock. Never register committing, committed, or unknown-outcome
transactions for late file deletion.

Add fault injection and distributed regression coverage for BE prepare
failure, delayed reports, failure before FE commit, errors after snapshot
publication, and unknown commit outcomes. Process crashes before reporting
files still require orphan-file maintenance.

### Release note

Clean up uncommitted Paimon files when a successful BE reports after another
BE has failed and FE has rolled back the INSERT.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - 10 Rust BE tests pass under ASAN; 31 focused FE tests pass.
    - Distributed failure suite passes on both FE distribution planners;
      expected output generated by the standard runner and verified normally.
    - Existing Paimon Rust append regression passes.
    - Manual two-BE SIGKILL test confirms failed INSERT publishes no snapshot,
      cleans the healthy writer's files, and succeeds after BE restart.
    - Full ASAN BE/FE build, updated FE build with Checkstyle, and clang-format
      16 preset check pass. Clang-tidy is blocked by existing NOLINTEND and
      local system-header parsing errors.
- Behavior changed: Yes, late reports for aborted Paimon inserts clean files
- Does this need documentation: No, internal failure cleanup
@suxiaogang223 suxiaogang223 changed the title [feature](be) Add opt-in Paimon Rust append writes [feature](be) Add Paimon Rust append writes with shared table metadata Oct 30, 2026
@suxiaogang223 suxiaogang223 changed the title [feature](be) Add Paimon Rust append writes with shared table metadata [feature](paimon) Support writes using paimon-rust Nov 2, 2026
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 82.92% (403/486) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.81% (30368/46860)
Line Coverage 49.66% (321327/646990)
Region Coverage 45.46% (259080/569878)
Branch Coverage 47.10% (122358/259795)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 82.89% (402/485) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.41% (34775/45511)
Line Coverage 61.25% (392983/641599)
Region Coverage 58.07% (331451/570735)
Branch Coverage 58.99% (153067/259483)

@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from 9ae32af to c489854 Compare November 3, 2026 08:21
@suxiaogang223
suxiaogang223 changed the base branch from branch-4.2 to master November 3, 2026 08:22
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 23.81% (5/21) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from d2f5e2b to 496da98 Compare November 4, 2026 03:03
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886, apache#70075, apache#69343

Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon
2.1.0. Rebase the writer on master instead of carrying separate dependency
upgrades. Reuse the upstream Java v14 serializer with bounded accumulated
output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate
the real two-writer interop fixture. Reject old protocol requests before
writer admission. Preserve master's vector-search field 42 and use optional
field 43 for the shared descriptor; retain master's SPI additions and bump
the write diagnostics surface to API 30.

The previous BE UT crash was caused by a validation test bypassing init()
and reading an uninitialized scan-params pointer. Explicitly initialize its
missing-descriptor input.

### Release note

Rust Paimon writes use the Java v14 commit protocol matching master's Paimon
2.1.0 SDK and inherit master's paimon-rust dependency version.

### Check List (For Author)

- Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds,
  version rejection, message ranges and resource release; FE/Java build with
  Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests;
  six Paimon JNI static layout checks; Thrift generation/field audit;
  clang-format 16, header hygiene, shell syntax and whitespace checks.
  JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests,
  clang-tidy against the new build and SQL regression are deferred at the
  user's request to leave resources for another PR.
- Behavior changed: Yes; Rust commit messages now use v14 instead of v11.
- Does this need documentation: Yes; existing feature documentation remains
  pending with the draft PR.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The upgraded paimon-rust writer needs native coverage of
credit release, cleanup and the v14 transport. Use the same Parquet buffer
option as the production descriptor instead of the generic write-buffer-size
option, which does not control the Parquet buffer.

Cover empty writes, prepare failure after rolling a file, and sustained
32 MiB dictionary input under a 16 MiB SDK input budget without rolling the
file. Extend the memory-failure case to retry the same batch with a larger
runtime limit. Check complete big-endian frame fields and the metadata
credit retained until report acknowledgement. The exercised paths required
no production logic changes.

### Release note

None

### Check List (For Author)

- Test: ASAN BE compilation and packaging with build.sh; 164 related BE
  tests passed through run-be-ut.sh (20 FFI writer tests, no failures or
  skips); clang-tidy for five modified production translation units and
  the full updated FFI test file; clang-format 16, header hygiene and
  whitespace checks. SQL/distributed regression was not rerun.
- Behavior changed: No; test-only changes.
- Does this need documentation: No; test-only changes.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 65.99% (38786/58772)
Region Coverage 53.76% (36417/67743)
Branch Coverage 57.27% (11868/20722)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 23.81% (5/21) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223

Copy link
Copy Markdown
Member Author

run beut

Related PR: apache#67395

Problem Summary:

Port the branch-4.2 Rust append writer to master's connector framework and
Paimon Java SDK 2.1.0. Prefer Rust for supported writes, share a pinned
Thrift table descriptor across Rust reads and Rust/JNI writes, and retain
online writer memory/concurrency settings. Use the existing acknowledged
file-report protocol for ownership and cleanup, with a separate estimated
credit for native commit metadata until reporting completes. Expose
transaction diagnostics through connector-neutral SPI hooks (API 29.0).

Keep Java v11 commit encoding in the released paimon-rust 0.4.0 patch;
verify that SDK 2.1.0 decodes it. Preserve Rust 1.94.0. Fix closing a writer
before open and reserve Fluss field 41 by assigning Paimon scan field 42.

Support Paimon append writes using paimon-rust, enabled by default for
supported targets with JNI selection for unsupported writes. Runtime
failures fail the statement. Writer limits can be changed online for new
writers; users can tune limits and retry confirmed uncommitted failures.

- Test: FE/Java compilation and packaging with Checkstyle; Paimon connector
  902 tests (one skipped); SPI 152; JNI Paimon 186; FE integration 36;
  S3/MinIO properties 37; plugin layout; clang-format 16; BE header hygiene.
  Master BE compilation/native tests and SQL regression remain pending.
- Behavior changed: Yes. Supported append writes prefer Rust; table
  descriptions and transaction diagnostics use master's connector model.
- Does this need documentation: Yes. User documentation is pending.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
### What problem does this PR solve?

Related PR: apache#69886, apache#70075, apache#69343

Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon
2.1.0. Rebase the writer on master instead of carrying separate dependency
upgrades. Reuse the upstream Java v14 serializer with bounded accumulated
output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate
the real two-writer interop fixture. Reject old protocol requests before
writer admission. Preserve master's vector-search field 42 and use optional
field 43 for the shared descriptor; retain master's SPI additions and bump
the write diagnostics surface to API 30.

The previous BE UT crash was caused by a validation test bypassing init()
and reading an uninitialized scan-params pointer. Explicitly initialize its
missing-descriptor input.

### Release note

Rust Paimon writes use the Java v14 commit protocol matching master's Paimon
2.1.0 SDK and inherit master's paimon-rust dependency version.

### Check List (For Author)

- Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds,
  version rejection, message ranges and resource release; FE/Java build with
  Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests;
  six Paimon JNI static layout checks; Thrift generation/field audit;
  clang-format 16, header hygiene, shell syntax and whitespace checks.
  JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests,
  clang-tidy against the new build and SQL regression are deferred at the
  user's request to leave resources for another PR.
- Behavior changed: Yes; Rust commit messages now use v14 instead of v11.
- Does this need documentation: Yes; existing feature documentation remains
  pending with the draft PR.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The upgraded paimon-rust writer needs native coverage of
credit release, cleanup and the v14 transport. Use the same Parquet buffer
option as the production descriptor instead of the generic write-buffer-size
option, which does not control the Parquet buffer.

Cover empty writes, prepare failure after rolling a file, and sustained
32 MiB dictionary input under a 16 MiB SDK input budget without rolling the
file. Extend the memory-failure case to retry the same batch with a larger
runtime limit. Check complete big-endian frame fields and the metadata
credit retained until report acknowledgement. The exercised paths required
no production logic changes.

### Release note

None

### Check List (For Author)

- Test: ASAN BE compilation and packaging with build.sh; 164 related BE
  tests passed through run-be-ut.sh (20 FFI writer tests, no failures or
  skips); clang-tidy for five modified production translation units and
  the full updated FFI test file; clang-format 16, header hygiene and
  whitespace checks. SQL/distributed regression was not rerun.
- Behavior changed: No; test-only changes.
- Does this need documentation: No; test-only changes.
@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from 8cf2388 to 4078a16 Compare November 5, 2026 05:06
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 66.02% (38800/58772)
Region Coverage 53.77% (36425/67743)
Branch Coverage 57.19% (11851/20722)

@suxiaogang223

Copy link
Copy Markdown
Member Author

Local review with the Doris code-review skill: PASS for 4078a16c9b67a4467275f2dfc31e125eba4a42d1 against 913532ce1673595cd552bd6f582e5b2640f45dd9. No unresolved findings.

Reviewed the Rust/JNI selection, shared pinned descriptor, v14 commit transport, memory admission/release, report ownership and FE commit/rollback flow. The six feature patches are unchanged by the rebase.

Current-head CI compilation, FE UT and Cloud UT have passed. BE UT and regression jobs are still running. This review does not mark unfinished tests as passed. Earlier local validation covered 164 related BE tests (20 FFI writer tests) and 1,541 FE/Java tests (one skip); those were not rerun during this review. Local formatting with clang-format 16, build hygiene, shell syntax and cross-release Thrift checks passed.

Doris skill checkpoints
Checkpoint Conclusion
Goal Supported append writes prefer Rust; unsupported writes select JNI before consuming input.
Scope and reuse Changes reuse the existing transaction/report framework and pinned table schema.
Concurrency and locks Per-task writer calls are serialized; admission locks exclude SDK work and group refresh; abort I/O runs outside the connector state lock.
Lifetime and static initialization Native work stops before writer credit/slots are returned; report callbacks own their metadata and tracker; metrics initialize lazily.
Online settings Mutable limits apply to new admissions; active budgets retain their original value.
Compatibility Missing descriptors, unsupported commit versions and JNI ABI mismatches return errors, consistent with the agreed mixed-version scope.
Thrift/Proto Checked master and branches 3.1/4.0/4.1/4.2: optional additions, stable meanings and no ID collisions. No protobuf changes.
Parallel paths Descriptor sharing covers Rust reads and Rust/JNI writes; SPI forwarding and S3/MinIO aliases are complete.
Conditions Backend eligibility and cleanup states correspond to concrete supported/failed paths; runtime errors fail the statement.
Coverage Tests cover budgets, release, cancellation, prepare/report failures, protocol bytes and FE outcomes; current CI remains in progress.
Test results Real writer fixtures verify Java interoperability; generated regression outputs were not edited.
Observability Profiles/metrics cover selection, resources, input, timing, failures and transaction states.
Transactions and persistence FE alone publishes snapshots; pre-publication failures clean up, uncertain outcomes retry/reconcile and preserve files. No new EditLog.
Data writes and crashes BE only prepares; ACK/rejection/ambiguous ownership is handled explicitly. Crash/cleanup-I/O orphan maintenance remains an accepted limitation.
FE/BE propagation Session forwarding, sink/scan descriptors and commit version agree; JNI rebuilds the pinned table.
Performance and memory Input/commit payloads are bounded; query/native credits are estimates. Full RSS limits, spill and revocation remain outside scope.
Other integration Checked current Coordinator/profile lifetime, transaction-manager behavior, SPI version and plugin dependencies; master includes the common BE UT build fix #70209.
schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69886
commit: 4078a16c9b67a4467275f2dfc31e125eba4a42d1
base: 913532ce1673595cd552bd6f582e5b2640f45dd9
reviewed_at: 2026-11-05T07:37:01+00:00
reviewer: suxiaogang223
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 93.94% (31/33) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.44% (32207/47758)
Line Coverage 52.28% (339050/648569)
Region Coverage 47.76% (275491/576810)
Branch Coverage 49.60% (129904/261886)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 93.75% (30/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.65% (35928/46271)
Line Coverage 62.98% (404048/641511)
Region Coverage 59.48% (341489/574123)
Branch Coverage 60.67% (158318/260946)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants