Skip to content

Commit 139cd2c

Browse files
branch-4.1: [feature](paimon) Support table writes via JNI and schema evolution (#65868)
## What problem does this PR solve? Related to #65086. Doris supports reading Paimon external tables, but branch-4.1 lacks a complete write path for inserting query results into Paimon tables. This PR adds Paimon table writes through the Paimon Java SDK via JNI. It also adds schema evolution support and correctness coverage for merge engines, bucket modes, changelog producers, compaction, schema defaults, partition overwrite, and transaction handling. ## What is changed? ### Write path and DML - Support `INSERT INTO ... SELECT ...`. - Support `INSERT INTO ... VALUES ...`. - Support `INSERT OVERWRITE` for: - non-partitioned tables; - static partitions; - partial static partition specifications; - dynamic partitions; - empty input. - Support append-only and primary-key tables. - Support reordered and partial target column lists. - Apply Paimon schema defaults to omitted columns through the Paimon SDK. ### Primary-key merge engines Support writes to the following Paimon merge engines: - `deduplicate`; - `partial-update`; - `first-row`; - `aggregation`. Full-row and partial-column writes share the same schema mapping path. Partial-column writes are accepted only when the Paimon merge engine defines safe semantics for omitted fields. ### Bucket modes Support the following Paimon bucket modes: - fixed hash bucket; - unaware bucket; - postpone bucket; - hash dynamic bucket; - key dynamic bucket. Dynamic bucket modes currently use a correctness-first single-owner execution model. Paimon SDK assigners determine the bucket; Doris does not duplicate Paimon hashing or bucket assignment logic. ### Changelog and compaction - Support changelog producers: - `none`; - `input`; - `lookup`; - `full-compaction`. - Cover automatic write-time compaction correctness. - Preserve Paimon audit-log and snapshot changelog semantics. ### Transaction and commit - Add FE-coordinated Paimon transactions. - Aggregate commit messages from multiple BE writers. - Support idempotent commit retry and commit outcome reconciliation. - Deduplicate retried commit messages. - Abort uncommitted files on write failure. - Commit empty `INSERT OVERWRITE` statements correctly. ### Schema and metadata - Support Paimon external table schema evolution: - add columns; - drop columns; - rename columns; - modify column types, nullability, comments, defaults, and positions; - reorder columns. - Refresh external-table metadata after schema changes. - Load the latest remote Paimon schema when building scan projections. - Preserve typed partition identity instead of treating physical partition paths as unique logical partition identifiers. - Honor Paimon `partition.default-name` and `partition.legacy-name`. - Fall back to Paimon scan planning when Doris cannot safely represent all partitions for pruning. ### Catalog coverage - Filesystem catalog writes. - Object storage and HDFS-backed writes. - HMS and REST catalog write regression suites. ### Architecture ```text Nereids Paimon Table Sink -> PaimonTableSinkOperator -> PaimonTableWriter -> JniPaimonWriteBackend -> PaimonJniWriter -> Paimon Java SDK -> CommitMessage -> FE PaimonTransaction ``` Arrow IPC is used to transfer Doris blocks to the Java writer. Partition routing, bucket assignment, merge-engine behavior, schema validation, default conversion, file writing, and compaction are delegated to the Paimon SDK where possible. ## Scope The following features are intentionally not included in this PR: - row-level `UPDATE` / `DELETE` SQL; - bucket-aware parallel data distribution and exchange optimization; - the Rust FFI write backend. Primary-key upsert through `INSERT` is supported. Row-level SQL DML requires a separate scan/rewrite and RowKind propagation path. Dynamic bucket writes are functionally supported, but currently use a correctness-first single-owner execution model. Bucket-aware parallel routing will be implemented separately. ## Test coverage Regression coverage includes: - append-only and primary-key writes; - primitive and complex types; - merge engines; - fixed, unaware, postpone, and dynamic bucket modes; - static and dynamic partition overwrite; - transaction retry, abort, and empty overwrite; - changelog producers; - automatic compaction; - schema defaults and omitted columns; - schema changes before and after writes; - Paimon CREATE TABLE properties; - filesystem, HMS, and REST catalog suites; - failure and edge cases. Doris query results are validated using regression output files and compared with Spark/Paimon results where applicable.
1 parent 64ba650 commit 139cd2c

136 files changed

Lines changed: 13348 additions & 228 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/common/config.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1552,6 +1552,11 @@ DEFINE_mInt64(hive_sink_max_file_size, "1073741824"); // 1GB
15521552
/** Iceberg sink configurations **/
15531553
DEFINE_mInt64(iceberg_sink_max_file_size, "1073741824"); // 1GB
15541554

1555+
/** Paimon sink configurations **/
1556+
DEFINE_mInt64(paimon_jni_writer_memory_pool_limit_bytes, "536870912"); // 512MB
1557+
DEFINE_Validator(paimon_jni_writer_memory_pool_limit_bytes,
1558+
[](int64_t bytes) -> bool { return bytes > 0; });
1559+
15551560
// URI scheme to Doris file type mappings used by paimon-cpp DorisFileSystem.
15561561
// Each entry uses the format "<scheme>=<file_type>", and file_type must be one of:
15571562
// local, hdfs, s3, http, broker.

‎be/src/common/config.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1641,6 +1641,10 @@ DECLARE_mInt64(hive_sink_max_file_size);
16411641
/** Iceberg sink configurations **/
16421642
DECLARE_mInt64(iceberg_sink_max_file_size);
16431643

1644+
/** Paimon sink configurations **/
1645+
// Hard upper bound for Doris-managed Paimon write-buffer memory per JNI writer.
1646+
DECLARE_mInt64(paimon_jni_writer_memory_pool_limit_bytes);
1647+
16441648
/** Paimon file system configurations **/
16451649
DECLARE_Strings(paimon_file_system_scheme_mappings);
16461650

‎be/src/exec/operator/operator.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@
6363
#include "exec/operator/olap_scan_operator.h"
6464
#include "exec/operator/olap_table_sink_operator.h"
6565
#include "exec/operator/olap_table_sink_v2_operator.h"
66+
#include "exec/operator/paimon_table_sink_operator.h"
6667
#include "exec/operator/partition_sort_sink_operator.h"
6768
#include "exec/operator/partition_sort_source_operator.h"
6869
#include "exec/operator/partitioned_aggregation_sink_operator.h"
@@ -802,6 +803,7 @@ DECLARE_OPERATOR(OlapTableSinkV2LocalState)
802803
DECLARE_OPERATOR(HiveTableSinkLocalState)
803804
DECLARE_OPERATOR(TVFTableSinkLocalState)
804805
DECLARE_OPERATOR(IcebergTableSinkLocalState)
806+
DECLARE_OPERATOR(PaimonTableSinkLocalState)
805807
DECLARE_OPERATOR(SpillIcebergTableSinkLocalState)
806808
DECLARE_OPERATOR(IcebergDeleteSinkLocalState)
807809
DECLARE_OPERATOR(IcebergMergeSinkLocalState)
@@ -929,6 +931,7 @@ template class AsyncWriterSink<doris::VIcebergDeleteSink, IcebergDeleteSinkOpera
929931
template class AsyncWriterSink<doris::VIcebergMergeSink, IcebergMergeSinkOperatorX>;
930932
template class AsyncWriterSink<doris::VMCTableWriter, MCTableSinkOperatorX>;
931933
template class AsyncWriterSink<doris::VTVFTableWriter, TVFTableSinkOperatorX>;
934+
template class AsyncWriterSink<doris::PaimonTableWriter, PaimonTableSinkOperatorX>;
932935

933936
#ifdef BE_TEST
934937
template class OperatorX<DummyOperatorLocalState>;
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "exec/operator/paimon_table_sink_operator.h"
19+
20+
#include "common/logging.h"
21+
22+
namespace doris {
23+
24+
Status PaimonTableSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
25+
return Base::init(state, info);
26+
}
27+
28+
Status PaimonTableSinkOperatorX::sink_impl(RuntimeState* state, Block* in_block, bool eos) {
29+
auto& local_state = get_local_state(state);
30+
SCOPED_TIMER(local_state.exec_time_counter());
31+
COUNTER_UPDATE(local_state.rows_input_counter(), static_cast<int64_t>(in_block->rows()));
32+
33+
// Delegate to AsyncWriterSink → PaimonTableWriter for this pipeline instance.
34+
// Each pipeline instance has its own writer session; partition and bucket
35+
// routing is handled internally by the Paimon SDK inside IPaimonWriter::write().
36+
return local_state.sink(state, in_block, eos);
37+
}
38+
39+
} // namespace doris
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <gen_cpp/DataSinks_types.h>
21+
22+
#include <memory>
23+
#include <string>
24+
25+
#include "common/status.h"
26+
#include "core/block/block.h"
27+
#include "exec/operator/operator.h"
28+
#include "exec/sink/writer/paimon/paimon_table_writer.h"
29+
#include "runtime/runtime_state.h"
30+
31+
namespace doris {
32+
33+
/// Paimon table sink operator — simple pass-through to AsyncWriterSink.
34+
///
35+
/// Each pipeline instance (LocalState) owns one PaimonTableWriter, which in
36+
/// turn owns one IPaimonWriteBackend + IPaimonWriter. Pipeline parallelism
37+
/// determines the number of concurrent Paimon writer sessions per table.
38+
///
39+
/// Partition and bucket routing is performed internally by the Paimon SDK
40+
/// (Java via JNI, or Rust via FFI). Doris does not compute partition values
41+
/// or bucket ids; it passes complete Blocks through the backend to the SDK,
42+
/// where each row is routed via getPartition(row) + getBucket(row).
43+
///
44+
/// This mirrors Iceberg's approach: IcebergTableSinkOperatorX delegates to
45+
/// AsyncWriterSink<VIcebergTableWriter>, with partition routing inside
46+
/// VIcebergTableWriter::write().
47+
class PaimonTableSinkOperatorX;
48+
49+
class PaimonTableSinkLocalState final
50+
: public AsyncWriterSink<PaimonTableWriter, PaimonTableSinkOperatorX> {
51+
public:
52+
using Base = AsyncWriterSink<PaimonTableWriter, PaimonTableSinkOperatorX>;
53+
using Parent = PaimonTableSinkOperatorX;
54+
ENABLE_FACTORY_CREATOR(PaimonTableSinkLocalState);
55+
PaimonTableSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
56+
: Base(parent, state) {}
57+
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
58+
Status open(RuntimeState* state) override {
59+
SCOPED_TIMER(exec_time_counter());
60+
SCOPED_TIMER(_open_timer);
61+
return Base::open(state);
62+
}
63+
64+
friend class PaimonTableSinkOperatorX;
65+
};
66+
67+
class PaimonTableSinkOperatorX final : public DataSinkOperatorX<PaimonTableSinkLocalState> {
68+
public:
69+
using Base = DataSinkOperatorX<PaimonTableSinkLocalState>;
70+
PaimonTableSinkOperatorX(ObjectPool* pool, int operator_id, const RowDescriptor& row_desc,
71+
const std::vector<TExpr>& t_output_expr)
72+
: Base(operator_id, 0, 0),
73+
_row_desc(row_desc),
74+
_t_output_expr(t_output_expr),
75+
_pool(pool) {}
76+
77+
Status init(const TDataSink& thrift_sink) override {
78+
RETURN_IF_ERROR(Base::init(thrift_sink));
79+
DCHECK(thrift_sink.__isset.paimon_table_sink);
80+
RETURN_IF_ERROR(VExpr::create_expr_trees(_t_output_expr, _output_vexpr_ctxs));
81+
return Status::OK();
82+
}
83+
84+
Status prepare(RuntimeState* state) override {
85+
RETURN_IF_ERROR(Base::prepare(state));
86+
RETURN_IF_ERROR(VExpr::prepare(_output_vexpr_ctxs, state, _row_desc));
87+
return VExpr::open(_output_vexpr_ctxs, state);
88+
}
89+
90+
Status sink_impl(RuntimeState* state, Block* in_block, bool eos) override;
91+
92+
private:
93+
friend class PaimonTableSinkLocalState;
94+
template <typename Writer, typename Parent>
95+
requires(std::is_base_of_v<AsyncResultWriter, Writer>)
96+
friend class AsyncWriterSink;
97+
98+
const RowDescriptor& _row_desc;
99+
VExprContextSPtrs _output_vexpr_ctxs;
100+
const std::vector<TExpr>& _t_output_expr;
101+
ObjectPool* _pool = nullptr;
102+
};
103+
104+
} // namespace doris

‎be/src/exec/pipeline/pipeline_fragment_context.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@
8686
#include "exec/operator/olap_scan_operator.h"
8787
#include "exec/operator/olap_table_sink_operator.h"
8888
#include "exec/operator/olap_table_sink_v2_operator.h"
89+
#include "exec/operator/paimon_table_sink_operator.h"
8990
#include "exec/operator/partition_sort_sink_operator.h"
9091
#include "exec/operator/partition_sort_source_operator.h"
9192
#include "exec/operator/partitioned_aggregation_sink_operator.h"
@@ -1157,6 +1158,14 @@ Status PipelineFragmentContext::_create_data_sink(ObjectPool* pool, const TDataS
11571158
output_exprs);
11581159
break;
11591160
}
1161+
case TDataSinkType::PAIMON_TABLE_SINK: {
1162+
if (!thrift_sink.__isset.paimon_table_sink) {
1163+
return Status::InternalError("Missing paimon table sink.");
1164+
}
1165+
_sink = std::make_shared<PaimonTableSinkOperatorX>(pool, next_sink_operator_id(), row_desc,
1166+
output_exprs);
1167+
break;
1168+
}
11601169
case TDataSinkType::JDBC_TABLE_SINK: {
11611170
if (!thrift_sink.__isset.jdbc_table_sink) {
11621171
return Status::InternalError("Missing data jdbc sink.");
@@ -2175,6 +2184,20 @@ void PipelineFragmentContext::_coordinator_callback(const ReportStatusRequest& r
21752184
}
21762185
}
21772186

2187+
if (auto pcm = req.runtime_state->paimon_commit_messages(); !pcm.empty()) {
2188+
params.__isset.paimon_commit_messages = true;
2189+
params.paimon_commit_messages.insert(params.paimon_commit_messages.end(), pcm.begin(),
2190+
pcm.end());
2191+
} else if (!req.runtime_states.empty()) {
2192+
for (auto* rs : req.runtime_states) {
2193+
if (auto rs_pcm = rs->paimon_commit_messages(); !rs_pcm.empty()) {
2194+
params.__isset.paimon_commit_messages = true;
2195+
params.paimon_commit_messages.insert(params.paimon_commit_messages.end(),
2196+
rs_pcm.begin(), rs_pcm.end());
2197+
}
2198+
}
2199+
}
2200+
21782201
req.runtime_state->get_unreported_errors(&(params.error_log));
21792202
params.__isset.error_log = (!params.error_log.empty());
21802203

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "exec/sink/writer/paimon/ffi_paimon_write_backend.h"
19+
20+
namespace doris {
21+
22+
Status FfiPaimonWriteBackend::open(const TPaimonTableSink&, RuntimeState*, RuntimeProfile*) {
23+
return Status::NotSupported("Paimon Rust FFI writer is not implemented");
24+
}
25+
26+
Status FfiPaimonWriteBackend::create_writer(std::unique_ptr<IPaimonWriter>*) {
27+
return Status::NotSupported("Paimon Rust FFI writer is not implemented");
28+
}
29+
30+
Status FfiPaimonWriteBackend::close() {
31+
return Status::OK();
32+
}
33+
34+
} // namespace doris
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include "exec/sink/writer/paimon/paimon_write_backend.h"
21+
22+
namespace doris {
23+
24+
/// Placeholder for the future paimon-rust writer implementation. Keeping this
25+
/// backend in the factory makes the integration boundary explicit without
26+
/// introducing a BE commit contract that the Rust writer will not own.
27+
class FfiPaimonWriteBackend final : public IPaimonWriteBackend {
28+
public:
29+
Status open(const TPaimonTableSink& sink, RuntimeState* state,
30+
RuntimeProfile* profile) override;
31+
Status create_writer(std::unique_ptr<IPaimonWriter>* writer) override;
32+
Status close() override;
33+
PaimonBackendType type() const override { return PaimonBackendType::FFI; }
34+
};
35+
36+
} // namespace doris

0 commit comments

Comments
 (0)