Tracking: Paimon table write support #65086

Background

Apache Doris already supports reading Apache Paimon tables through External Catalog. To complete the read/write experience for lakehouse workloads, we are adding support for writing data from Doris into Paimon tables, so users can write query results and computed data into Paimon tables using standard Doris SQL.

This issue tracks the feature requirements and implementation plan for Paimon write support.

Goals

• Support INSERT INTO for Paimon tables.
• Support INSERT OVERWRITE for Paimon tables.
• Support append-only tables and primary-key tables.
• Support partitioned, fixed bucket, and unaware bucket tables.
• Support primitive types and complex types (ARRAY, MAP, STRUCT).
• Align Doris transaction semantics with Paimon commit and abort.
• Preserve correctness for failure, retry, and rollback scenarios.

Current Status

Core Paimon JNI write support and the main merged follow-up work are implemented on branch-4.1:

• branch-4.1: [feature](paimon) Support table writes via JNI and schema evolution #65868: core JNI write path and schema evolution.
• [test](paimon) Add P0 write correctness and snapshot reference coverage #66325: P0 write correctness and snapshot-reference coverage.
• branch-4.1: [feature](paimon) Support row-level UPDATE, DELETE, and MERGE #66498: row-level UPDATE, DELETE, and MERGE.
• branch-4.1: [feature](external) support partition- and bucket-aware concurrent writes #66685: partition- and bucket-aware concurrent writes for supported routing modes.
• branch-4.1: [fix](paimon) Keep write cache coherent and extend coverage #67023: write-cache coherence fixes and extended regression coverage.

Status legend:

• ✅ Implemented
• 🚧 Partially implemented; remaining modes are follow-up work
• ➡️ Follow-up work

Feature Checklist

Write Path & DML

Table Types & Write Semantics

Partitioning & Bucketing

Fixed-bucket tables now use Paimon-compatible (partition, bucket) routing so each bucket has one stable writer within a write job. Bucket-unaware tables retain concurrent random distribution. Hash-dynamic, key-dynamic, custom bucket functions, and unsupported partition or bucket key types continue to use the correctness-first single-writer fallback.

Data Types

Commit & Transaction

Write Backend

Changelog & Compaction

Catalog & Storage

The HMS and REST catalog write suites have been added under external_table_p2.

Schema & DDL

Remaining Follow-up Work

1. Bucket-aware parallel routing for hash-dynamic, key-dynamic, custom bucket functions, and currently unsupported key types.
2. Rust FFI write backend.
3. Paimon Branch/Tag lifecycle and branch writes: [Feature](paimon) Support Branch/Tag lifecycle and branch writes #67336
4. Paimon maintenance procedures and actions: [Feature](catalog) Support Paimon maintenance procedures and actions #67337