[Enhancement] Support Paimon 2.0 table writes on master #66784

Search before asking

I searched existing issues and PRs. #65086 tracks general Paimon write support, and #65868 implemented the Java/JNI write path for branch-4.1. This issue focuses on bringing that capability to the current master architecture and upgrading the shared runtime to Apache Paimon 2.0.

Description

Current master can read Paimon tables through the new Connector SPI, but it still uses Paimon 1.3.1 and has no complete external-table write path. The implementation merged into branch-4.1 cannot be applied directly because master has moved catalog, transaction, planning, and scan/write contracts into the new connector modules.

The desired behavior is:

• use one Paimon 2.0 runtime version across FE serialization, BE JNI, scanner, and writer;
• support CREATE TABLE, DROP TABLE, INSERT INTO, INSERT OVERWRITE, and CTAS against Paimon external catalogs;
• support filesystem, HMS, REST, HDFS, and S3-compatible object storage catalogs;
• support append-only and primary-key tables, partitioned writes, and Paimon bucket assignment;
• preserve Doris statement atomicity with FE-coordinated commit, multi-BE commit aggregation, idempotent retry/reconciliation, and abort cleanup;
• map Doris primitive and complex types, including Paimon 2.0 BLOB plus Doris VARIANT and VECTOR encodings;
• keep read compatibility for existing Paimon tables after the runtime upgrade.

Proposed solution

Port the proven Java SDK/JNI design from #65868 to the current Connector SPI instead of restoring the old datasource implementation:

ConnectorWritePlanProvider
  -> PaimonTableSinkOperator
  -> PaimonTableWriter
  -> JNI Paimon writer
  -> Paimon CommitMessage
  -> FE ConnectorTransaction

The writer should delegate partition routing, bucket/index assignment, schema defaults, file creation, and commit semantics to the Paimon 2.0 SDK. Doris should own query lifecycle, memory limits, commit-message transport, acknowledgement, retry reconciliation, and cleanup.

BLOB and VECTOR can be represented using Paimon 2.0 physical types plus stable column directives. VARIANT should use a lossless Arrow struct carrying value and metadata binaries so nested and typed values survive round trips.

Validation

The change should include:

• Java unit tests for Arrow conversion, schema mapping, commit codec, JNI writer, connector write planning, and transaction handling;
• C++ unit tests for VARIANT transport, JNI schema metadata, report budgets, acknowledgement, and cleanup;
• regression coverage for create/drop, insert, overwrite, CTAS atomicity, and BLOB/VARIANT/VECTOR;
• live filesystem/HMS/REST/S3-compatible catalog tests against Paimon 2.0 data.

Are you willing to submit PR?

Yes. A draft implementation for master is being validated and will reference this issue.

Code of Conduct

I agree to follow the Apache Software Foundation Code of Conduct.