Skip to content

Commit 0a4a71a

Browse files
[feature](external) Add generic connector write framework prerequisites (#68320)
### What problem does this PR solve? Issue Number: #65086 Related PR: #67395 Problem Summary: Forward-porting Paimon table writes exposes four gaps in the generic connector write framework. Implementing them as Paimon-specific branches in FE Core would duplicate the existing write pipeline and would also cause Paimon row-level changes to be planned with Iceberg position-delete semantics. This PR adds the missing connector-neutral framework capabilities: 1. **Write distribution.** A connector can describe routing columns, an opaque partition-function name and options, and writer-assignment semantics. FE Core only resolves columns and transports the description; BE performs the routing. Unsupported functions and unsupported BE execution versions fail explicitly. The existing reserved BE execution version 13 is reused, with no Paimon-specific execution version. 2. **Row-level DML.** Connectors declare whether changes use file-position deletes or changelog rows. Iceberg retains its position-delete plan. The generic changelog plan converts DELETE, UPDATE, MERGE, and DELETE USING into an operation marker plus the output row, while the connector supplies the marker encoding, primary keys, validation, and label prefix. Unsupported representations fail instead of falling through to an unrelated write path. 3. **Commit reports.** BE can return opaque connector commit fragments through the existing external-write report lifecycle. FE acknowledges only accepted reports, retries remain safe after a lost response, and duplicate reports do not add commit data twice. FE Core does not interpret connector-owned bytes. 4. **Variant writes.** FE type checking/construction and BE Arrow serialization support Binary Variant V2 as a generic capability required by connector writes. The implementation intentionally does not add Paimon-specific planner classes, hidden-column names, Thrift structures, or a new write system. Paimon-specific table rules, fixed-bucket hash implementation, writer, and transaction interpretation remain in the Paimon integration that will follow this prerequisite PR. The connector SPI surface changes once, so `connector.plugin.api.version` is increased from 10 to 11 once for the whole PR. ### Release note None ### Check List (For Author) - Test: - Targeted FE unit tests for connector SPI, row-level DML planning, commit report acknowledgement/serialization, Variant type handling, and connector write distribution - Relevant FE Maven reactors compiled successfully while running the targeted unit tests - BE formatting with clang-format 16 and `build-support/check-format.sh` - `build-support/check-build-hygiene.sh` - BE compilation and BE unit tests were not run for the latest distribution change - Behavior changed: Yes. Connectors can declare row-change representation and write distribution, return opaque acknowledged commit data, and write Binary Variant V2 through the shared framework. - Does this need documentation: No
1 parent 31c5836 commit 0a4a71a

118 files changed

Lines changed: 4635 additions & 491 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/data_type_variant_v2_serde.cpp‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#include "core/data_type_serde/data_type_variant_v2_serde.h"
1919

2020
#include <arrow/array/builder_binary.h>
21+
#include <arrow/array/builder_nested.h>
2122

2223
#include <algorithm>
2324
#include <cstring>
@@ -182,6 +183,133 @@ void preflight_json(const IColumn& column, size_t start, size_t end,
182183
});
183184
}
184185

186+
void validate_binary_variant_primitive(VariantPrimitiveId primitive_id) {
187+
switch (primitive_id) {
188+
case VariantPrimitiveId::NULL_VALUE:
189+
case VariantPrimitiveId::TRUE_VALUE:
190+
case VariantPrimitiveId::FALSE_VALUE:
191+
case VariantPrimitiveId::INT8:
192+
case VariantPrimitiveId::INT16:
193+
case VariantPrimitiveId::INT32:
194+
case VariantPrimitiveId::INT64:
195+
case VariantPrimitiveId::DOUBLE:
196+
case VariantPrimitiveId::DECIMAL4:
197+
case VariantPrimitiveId::DECIMAL8:
198+
case VariantPrimitiveId::DECIMAL16:
199+
case VariantPrimitiveId::DATE:
200+
case VariantPrimitiveId::TIMESTAMP_MICROS:
201+
case VariantPrimitiveId::TIMESTAMP_NTZ_MICROS:
202+
case VariantPrimitiveId::FLOAT:
203+
case VariantPrimitiveId::BINARY:
204+
case VariantPrimitiveId::STRING:
205+
case VariantPrimitiveId::UUID:
206+
return;
207+
case VariantPrimitiveId::TIME_NTZ_MICROS:
208+
case VariantPrimitiveId::TIMESTAMP_NANOS:
209+
case VariantPrimitiveId::TIMESTAMP_NTZ_NANOS:
210+
throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
211+
"Binary Variant V2 Arrow encoding does not support primitive id {}",
212+
static_cast<uint8_t>(primitive_id));
213+
}
214+
throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
215+
"Binary Variant V2 Arrow encoding does not support unknown primitive id {}",
216+
static_cast<uint8_t>(primitive_id));
217+
}
218+
219+
void validate_binary_variant_value(VariantRef value, uint32_t depth = 0) {
220+
if (depth > VARIANT_MAX_NESTING_DEPTH) {
221+
throw Exception(ErrorCode::CORRUPTION, "Variant value exceeds maximum nesting depth {}",
222+
VARIANT_MAX_NESTING_DEPTH);
223+
}
224+
const size_t encoded_size = value.value_size();
225+
if (encoded_size != value.value.size) {
226+
throw Exception(ErrorCode::CORRUPTION,
227+
"Variant value has {} trailing bytes after the encoded value",
228+
value.value.size - encoded_size);
229+
}
230+
231+
switch (value.basic_type()) {
232+
case VariantBasicType::PRIMITIVE:
233+
validate_binary_variant_primitive(value.primitive_id());
234+
return;
235+
case VariantBasicType::SHORT_STRING:
236+
return;
237+
case VariantBasicType::OBJECT:
238+
for (uint32_t i = 0; i < value.num_elements(); ++i) {
239+
uint32_t field_id = 0;
240+
VariantRef child = value.object_value_at(i, &field_id);
241+
value.metadata.key_at(field_id);
242+
validate_binary_variant_value(child, depth + 1);
243+
}
244+
return;
245+
case VariantBasicType::ARRAY:
246+
for (uint32_t i = 0; i < value.num_elements(); ++i) {
247+
validate_binary_variant_value(value.array_at(i), depth + 1);
248+
}
249+
return;
250+
}
251+
}
252+
253+
void require_variant_arrow_status(const arrow::Status& status) {
254+
if (!status.ok()) {
255+
throw Exception(ErrorCode::INTERNAL_ERROR, "Variant V2 Arrow append failed: {}",
256+
status.ToString());
257+
}
258+
}
259+
260+
Status write_binary_variant_arrow(const IColumn& column, const NullMap* null_map,
261+
arrow::StructBuilder& builder, size_t start, size_t end) {
262+
// StructBuilder::type() returns a shared_ptr by value. Keep that owner alive while using the
263+
// cast reference; otherwise the reference would dangle as soon as the temporary is destroyed.
264+
const auto builder_type = builder.type();
265+
const auto& struct_type = assert_cast<const arrow::StructType&>(*builder_type);
266+
if (struct_type.num_fields() != 2 || struct_type.field(0)->name() != "value" ||
267+
struct_type.field(1)->name() != "metadata" ||
268+
struct_type.field(0)->type()->id() != arrow::Type::BINARY ||
269+
struct_type.field(1)->type()->id() != arrow::Type::BINARY) {
270+
return Status::InvalidArgument(
271+
"Binary Variant V2 Arrow type must be "
272+
"struct<value: binary, metadata: binary>, got {}",
273+
struct_type.ToString());
274+
}
275+
auto* value_builder = dynamic_cast<arrow::BinaryBuilder*>(builder.field_builder(0));
276+
auto* metadata_builder = dynamic_cast<arrow::BinaryBuilder*>(builder.field_builder(1));
277+
if (value_builder == nullptr || metadata_builder == nullptr) {
278+
return Status::InvalidArgument("Binary Variant V2 Arrow child builders must be binary");
279+
}
280+
281+
// Consumers of the binary Variant V2 Arrow representation copy these two buffers without
282+
// inspecting them. Validate once at the serialization boundary so a writer cannot commit
283+
// malformed or unsupported bytes.
284+
const auto outer_nulls = forced_nulls(null_map);
285+
visit_variant_v2_values(
286+
column, start, end, outer_nulls,
287+
[&](size_t) { require_variant_arrow_status(builder.AppendNull()); },
288+
[&](size_t row, VariantRef value) {
289+
try {
290+
constexpr size_t BINARY_VARIANT_SIZE_LIMIT = 128 * 1024 * 1024;
291+
if (value.value.size > BINARY_VARIANT_SIZE_LIMIT ||
292+
value.metadata.size > BINARY_VARIANT_SIZE_LIMIT) {
293+
throw Exception(ErrorCode::INVALID_ARGUMENT,
294+
"exceeds the 128 MiB value/metadata limit");
295+
}
296+
value.metadata.validate();
297+
validate_binary_variant_value(value);
298+
} catch (const Exception& e) {
299+
throw Exception(e.code(), "Binary Variant V2 row {} is incompatible: {}", row,
300+
e.what());
301+
}
302+
require_variant_arrow_status(builder.Append());
303+
require_variant_arrow_status(
304+
value_builder->Append(reinterpret_cast<const uint8_t*>(value.value.data),
305+
cast_set<int32_t, size_t, false>(value.value.size)));
306+
require_variant_arrow_status(metadata_builder->Append(
307+
reinterpret_cast<const uint8_t*>(value.metadata.data),
308+
cast_set<int32_t, size_t, false>(value.metadata.size)));
309+
});
310+
return Status::OK();
311+
}
312+
185313
} // namespace
186314

187315
DataTypeVariantV2SerDe::DataTypeVariantV2SerDe(int nesting_level) : DataTypeSerDe(nesting_level) {}
@@ -553,6 +681,11 @@ Status DataTypeVariantV2SerDe::write_column_to_arrow(const IColumn& column, cons
553681
assert_cast<arrow::LargeStringBuilder&>(*array_builder), first, last,
554682
options);
555683
}
684+
if (array_builder->type()->id() == arrow::Type::STRUCT) {
685+
return write_binary_variant_arrow(column, null_map,
686+
assert_cast<arrow::StructBuilder&>(*array_builder),
687+
first, last);
688+
}
556689
return Status::InvalidArgument("Unsupported arrow type for variant column: {}",
557690
array_builder->type()->name());
558691
});

‎be/src/exec/operator/exchange_sink_operator.cpp‎

Lines changed: 51 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -28,13 +28,15 @@
2828
#include <random>
2929
#include <string>
3030

31+
#include "agent/be_exec_version_manager.h"
3132
#include "common/status.h"
3233
#include "core/column/column_const.h"
3334
#include "exec/exchange/exchange_writer.h"
3435
#include "exec/exchange/local_exchange_sink_operator.h"
3536
#include "exec/operator/exchange_sink_buffer.h"
3637
#include "exec/operator/operator.h"
3738
#include "exec/operator/sort_source_operator.h"
39+
#include "exec/partitioner/external/external_table_sink_hash_partitioner.h"
3840
#include "exec/pipeline/dependency.h"
3941
#include "exec/pipeline/pipeline_fragment_context.h"
4042
#include "exec/sink/scale_writer_partitioning_exchanger.hpp"
@@ -98,7 +100,8 @@ Status ExchangeSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& inf
98100
_part_type = p._part_type;
99101
// Shuffle the channels randomly
100102
if (_part_type == TPartitionType::UNPARTITIONED || _part_type == TPartitionType::RANDOM ||
101-
_part_type == TPartitionType::HIVE_TABLE_SINK_UNPARTITIONED) {
103+
_part_type == TPartitionType::HIVE_TABLE_SINK_UNPARTITIONED ||
104+
_part_type == TPartitionType::EXTERNAL_TABLE_SINK_UNPARTITIONED) {
102105
std::random_device rd;
103106
std::mt19937 g(rd());
104107
shuffle(channels.begin(), channels.end(), g);
@@ -175,6 +178,27 @@ Status ExchangeSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& inf
175178
RETURN_IF_ERROR(_partitioner->prepare(state, p._row_desc));
176179
custom_profile()->add_info_string(
177180
"Partitioner", fmt::format("ScaleWriterPartitioner({})", _partition_count));
181+
} else if (_part_type == TPartitionType::EXTERNAL_TABLE_SINK_HASH_PARTITIONED) {
182+
if (state->be_exec_version() < SUPPORT_EXTERNAL_TABLE_SINK_HASH_VERSION) {
183+
return Status::NotSupported(
184+
"External table sink hash exchange requires BE execution version {}, actual {}",
185+
SUPPORT_EXTERNAL_TABLE_SINK_HASH_VERSION, state->be_exec_version());
186+
}
187+
if (!p._has_external_table_sink_hash_partition_info) {
188+
return Status::InternalError("External table sink hash partition info is missing");
189+
}
190+
_partition_count = channels.size();
191+
const bool use_crc32c = _state->query_options().__isset.enable_new_shuffle_hash_method &&
192+
_state->query_options().enable_new_shuffle_hash_method;
193+
const ShuffleHashMethod hash_method =
194+
use_crc32c ? ShuffleHashMethod::CRC32C : ShuffleHashMethod::CRC32;
195+
_partitioner = std::make_unique<ExternalTableSinkHashPartitioner>(
196+
_partition_count, hash_method, p._external_table_sink_hash_partition_info);
197+
RETURN_IF_ERROR(_partitioner->init(p._texprs));
198+
RETURN_IF_ERROR(_partitioner->prepare(state, p._row_desc));
199+
custom_profile()->add_info_string(
200+
"Partitioner",
201+
fmt::format("ExternalTableSinkHashPartitioner({})", _partition_count));
178202
} else if (_part_type == TPartitionType::MERGE_PARTITIONED) {
179203
if (!p._has_merge_partition_info) {
180204
return Status::InternalError("Merge partition info is missing");
@@ -272,6 +296,7 @@ Status ExchangeSinkLocalState::open(RuntimeState* state) {
272296
if (_part_type == TPartitionType::HASH_PARTITIONED ||
273297
_part_type == TPartitionType::BUCKET_SHFFULE_HASH_PARTITIONED ||
274298
_part_type == TPartitionType::HIVE_TABLE_SINK_HASH_PARTITIONED ||
299+
_part_type == TPartitionType::EXTERNAL_TABLE_SINK_HASH_PARTITIONED ||
275300
_part_type == TPartitionType::OLAP_TABLE_SINK_HASH_PARTITIONED ||
276301
_part_type == TPartitionType::MERGE_PARTITIONED) {
277302
RETURN_IF_ERROR(_partitioner->open(state));
@@ -322,6 +347,8 @@ ExchangeSinkOperatorX::ExchangeSinkOperatorX(
322347
sink.output_partition.type == TPartitionType::BUCKET_SHFFULE_HASH_PARTITIONED ||
323348
sink.output_partition.type == TPartitionType::HIVE_TABLE_SINK_HASH_PARTITIONED ||
324349
sink.output_partition.type == TPartitionType::HIVE_TABLE_SINK_UNPARTITIONED ||
350+
sink.output_partition.type == TPartitionType::EXTERNAL_TABLE_SINK_HASH_PARTITIONED ||
351+
sink.output_partition.type == TPartitionType::EXTERNAL_TABLE_SINK_UNPARTITIONED ||
325352
sink.output_partition.type == TPartitionType::MERGE_PARTITIONED);
326353
#endif
327354
_name = "ExchangeSinkOperatorX";
@@ -333,6 +360,11 @@ ExchangeSinkOperatorX::ExchangeSinkOperatorX(
333360
_merge_partition_info = sink.output_partition.merge_partition_info;
334361
_has_merge_partition_info = true;
335362
}
363+
if (sink.output_partition.__isset.external_table_sink_hash_partition_info) {
364+
_external_table_sink_hash_partition_info =
365+
sink.output_partition.external_table_sink_hash_partition_info;
366+
_has_external_table_sink_hash_partition_info = true;
367+
}
336368

337369
if (_part_type != TPartitionType::UNPARTITIONED) {
338370
// if the destinations only one dest, we need to use broadcast
@@ -535,20 +567,16 @@ Status ExchangeSinkOperatorX::sink_impl(RuntimeState* state, Block* block, bool
535567
_part_type == TPartitionType::BUCKET_SHFFULE_HASH_PARTITIONED ||
536568
_part_type == TPartitionType::OLAP_TABLE_SINK_HASH_PARTITIONED ||
537569
_part_type == TPartitionType::HIVE_TABLE_SINK_HASH_PARTITIONED ||
570+
_part_type == TPartitionType::EXTERNAL_TABLE_SINK_HASH_PARTITIONED ||
538571
_part_type == TPartitionType::MERGE_PARTITIONED) {
539572
RETURN_IF_ERROR(local_state._writer->write(state, block, eos));
540-
} else if (_part_type == TPartitionType::HIVE_TABLE_SINK_UNPARTITIONED) {
573+
} else if (_part_type == TPartitionType::HIVE_TABLE_SINK_UNPARTITIONED ||
574+
_part_type == TPartitionType::EXTERNAL_TABLE_SINK_UNPARTITIONED) {
541575
// Control the number of channels according to the flow, thereby controlling the number of table sink writers.
576+
const auto block_bytes = block->bytes();
542577
RETURN_IF_ERROR(send_to_current_channel());
543-
_data_processed += block->bytes();
544-
if (_writer_count < local_state.channels.size()) {
545-
if (_data_processed >=
546-
_writer_count *
547-
config::table_sink_non_partition_write_scaling_data_processed_threshold) {
548-
_writer_count++;
549-
}
550-
}
551-
local_state.current_channel_idx = (local_state.current_channel_idx + 1) % _writer_count;
578+
const auto writer_count = _update_writer_scaling(block_bytes, local_state.channels.size());
579+
local_state.current_channel_idx = (local_state.current_channel_idx + 1) % writer_count;
552580
} else {
553581
// Range partition
554582
// 1. calculate range
@@ -587,6 +615,18 @@ Status ExchangeSinkOperatorX::sink_impl(RuntimeState* state, Block* block, bool
587615
return final_st;
588616
}
589617

618+
size_t ExchangeSinkOperatorX::_update_writer_scaling(size_t block_bytes, size_t max_writer_count) {
619+
LockGuard lock(_writer_scaling_mutex);
620+
_data_processed += block_bytes;
621+
if (_writer_count < max_writer_count &&
622+
_data_processed >=
623+
_writer_count *
624+
config::table_sink_non_partition_write_scaling_data_processed_threshold) {
625+
++_writer_count;
626+
}
627+
return _writer_count;
628+
}
629+
590630
void ExchangeSinkLocalState::register_channels(ExchangeSinkBuffer* buffer) {
591631
for (auto& channel : channels) {
592632
channel->set_exchange_buffer(buffer);

‎be/src/exec/operator/exchange_sink_operator.h‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,17 @@ class ExchangeSinkOperatorX MOCK_REMOVE(final) : public DataSinkOperatorX<Exchan
200200

201201
Status sink_impl(RuntimeState* state, Block* in_block, bool eos) override;
202202

203+
#ifdef BE_TEST
204+
size_t update_writer_scaling_for_test(size_t block_bytes, size_t max_writer_count) {
205+
return _update_writer_scaling(block_bytes, max_writer_count);
206+
}
207+
208+
std::pair<size_t, size_t> writer_scaling_state_for_test() {
209+
LockGuard lock(_writer_scaling_mutex);
210+
return {_data_processed, _writer_count};
211+
}
212+
#endif
213+
203214
bool is_serial_operator() const override { return true; }
204215
void set_low_memory_mode(RuntimeState* state) override {
205216
auto& local_state = get_local_state(state);
@@ -239,10 +250,13 @@ class ExchangeSinkOperatorX MOCK_REMOVE(final) : public DataSinkOperatorX<Exchan
239250
// or each ExchangeSinkLocalState can have its own sink buffer.
240251
std::shared_ptr<ExchangeSinkBuffer> _create_buffer(
241252
RuntimeState* state, const std::vector<InstanceLoId>& sender_ins_ids);
253+
size_t _update_writer_scaling(size_t block_bytes, size_t max_writer_count);
242254
std::shared_ptr<ExchangeSinkBuffer> _sink_buffer = nullptr;
243255
RuntimeState* _state = nullptr;
244256

245257
const std::vector<TExpr> _texprs;
258+
TExternalTableSinkHashPartitionInfo _external_table_sink_hash_partition_info;
259+
bool _has_external_table_sink_hash_partition_info = false;
246260
TMergePartitionInfo _merge_partition_info;
247261
bool _has_merge_partition_info = false;
248262

@@ -278,8 +292,9 @@ class ExchangeSinkOperatorX MOCK_REMOVE(final) : public DataSinkOperatorX<Exchan
278292

279293
// for external table sink random partition
280294
// Control the number of channels according to the flow, thereby controlling the number of table sink writers.
281-
size_t _data_processed = 0;
282-
int _writer_count = 1;
295+
AnnotatedMutex _writer_scaling_mutex;
296+
size_t _data_processed GUARDED_BY(_writer_scaling_mutex) = 0;
297+
size_t _writer_count GUARDED_BY(_writer_scaling_mutex) = 1;
283298
// If dest_is_merge is true, it indicates that the corresponding receiver is a VMERGING-EXCHANGE.
284299
// The receiver will sort the collected data, so the sender must ensure that the data sent is ordered.
285300
const bool _dest_is_merge;
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing, software
12+
// distributed under the License is distributed on an "AS IS" BASIS,
13+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
// See the License for the specific language governing permissions and
15+
// limitations under the License.
16+
17+
#include "exec/partitioner/external/external_partition_function_factory.h"
18+
19+
#include "common/status.h"
20+
21+
namespace doris {
22+
23+
Status create_external_partition_function(const TExternalTableSinkHashPartitionInfo& partition_info,
24+
PartitionerBase::HashValType logical_partition_count,
25+
ShuffleHashMethod hash_method,
26+
const std::vector<TExpr>& partition_exprs,
27+
std::unique_ptr<PartitionFunction>* partition_function) {
28+
if (partition_function == nullptr) {
29+
return Status::InvalidArgument("External partition function output is null");
30+
}
31+
if (partition_info.partition_function != "direct_hash") {
32+
return Status::NotSupported("Unsupported external sink partition function '{}'",
33+
partition_info.partition_function);
34+
}
35+
if (partition_info.__isset.partition_function_options &&
36+
!partition_info.partition_function_options.empty()) {
37+
return Status::InvalidArgument("Direct hash partition function does not accept options");
38+
}
39+
auto function = std::make_unique<HashPartitionFunction>(logical_partition_count, hash_method);
40+
RETURN_IF_ERROR(function->init(partition_exprs));
41+
*partition_function = std::move(function);
42+
return Status::OK();
43+
}
44+
45+
} // namespace doris

0 commit comments

Comments
 (0)