Skip to content

Commit b8c03bf

Browse files
authored
[feature](flight) Support native Variant V2 results for ADBC on branch-4.1 (apache#68667)
### What problem does this PR solve? Flight SQL exposes VARIANT as UTF8, preventing ADBC clients from receiving native binary Variant V2 values. On branch-4.1, return Variant V2 through the `arrow.parquet.variant` extension by default, with `struct<metadata: binary not null, value: binary not null>` storage. SQL NULL is a null struct; Variant null is an encoded non-null value. Clients without a Variant extension implementation can access the physical struct and field metadata. Support **Variant V2 only**. Legacy Variant returns an unsupported error, including nested legacy fields, constants, all-null columns and empty results. Reject legacy types during BE Flight schema conversion and defensively in the legacy SerDe. Remove the output-format session variable and result-sink flag entirely. Users who need text can explicitly `CAST(... AS STRING)`. Keep execution, GetTables, GetSchema and prepared-statement schemas consistent, including nested Variant fields. Native Variant output does not depend on a heartbeat capability or a cluster-wide BE support check. Retain the Variant V2 type requirement and reject non-native execution schemas. Reuse the V2 binary representation and compact dictionaries per selected row, preserving physical scalar types and decimal scales without repeatedly copying unrelated dictionary keys. Keep the value helper private to `data_type_variant_v2_serde.cpp`. Native encoding accepts up to 128 nested levels. Iceberg output keeps its existing behavior. Document the V2 requirement and client decoding limitations in the Python Flight sample README; parsing JSON does not bypass the configured Variant representation. ### Release note Arrow Flight SQL / ADBC returns Variant V2 as native Arrow Variant binary by default, without a session switch. Legacy Variant is unsupported. Use an explicit SQL cast to STRING for text output. ### Validation - The native-output update passed all 46 selected Arrow/Flight tests using a focused ASAN binary; the binary conversion code is unchanged by the capability removal. - For the capability removal, regenerated the heartbeat Thrift bindings, compiled the affected FE sources and BE heartbeat translation unit, and passed all 6 focused FE tests covering native schemas, legacy/UTF8 rejection and the existing Paimon heartbeat capability. - FE Checkstyle, clang-format 16 and diff checks passed. - The updated Groovy regression script and Python sample syntax were checked in the preceding update. - A full production build and live cluster regression/ADBC execution were not rerun for this update; CI validation is requested. ### Check List (For Author) - Test - [x] Regression test updated - [x] Unit Test - Behavior changed: - [x] Yes. Variant V2 uses native binary by default; legacy Variant requires an explicit text cast. - Does this need documentation? - [x] Yes, usage, V2-only scope and client decoding limitations are documented in the Python Flight sample README.
1 parent 3d78b6b commit b8c03bf

20 files changed

Lines changed: 1238 additions & 42 deletions

File tree

‎be/src/core/data_type_serde/data_type_variant_serde.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,12 @@ Status DataTypeVariantSerDe::write_column_to_arrow(const IColumn& column, const
157157
int64_t start, int64_t end,
158158
const cctz::time_zone& ctz) const {
159159
const auto* var = check_and_get_column<ColumnVariant>(column);
160+
if (array_builder->type()->id() == arrow::Type::STRUCT) {
161+
// Native Flight output must not reinterpret legacy storage as Variant V2.
162+
return Status::NotSupported(
163+
"Native Arrow Flight output only supports Variant V2, not legacy Variant; "
164+
"cast the result to STRING for text output");
165+
}
160166
if (array_builder->type()->id() == arrow::Type::LARGE_STRING) {
161167
auto& builder = assert_cast<arrow::LargeStringBuilder&>(*array_builder);
162168
return write_variant_column_to_arrow_impl(column, *var, null_map, builder, start, end, ctz);

‎be/src/core/data_type_serde/data_type_variant_v2_serde.cpp‎

Lines changed: 71 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
#include <algorithm>
2525
#include <cstring>
2626
#include <limits>
27+
#include <optional>
2728
#include <orc/Vector.hh>
2829
#include <span>
2930
#include <utility>
@@ -53,6 +54,48 @@
5354
namespace doris {
5455
namespace {
5556

57+
// ColumnVariantV2 already validates its encoded dictionaries and value structure. Only walk
58+
// the selected row here: validating every unused dictionary key per row is quadratic for
59+
// shared dictionaries, including when a nested ARRAY invokes this writer on one-row slices.
60+
Status append_flight_variant_value(VariantRef value, VariantBatchBuilder::Row& output,
61+
size_t depth = 0) {
62+
const auto basic_type = value.basic_type();
63+
if (depth > VARIANT_MAX_NESTING_DEPTH) {
64+
return Status::NotSupported(
65+
"Native Arrow Variant nesting exceeds {}; "
66+
"cast the result to STRING for text output",
67+
VARIANT_MAX_NESTING_DEPTH);
68+
}
69+
if (value.value_size() != value.value.size) {
70+
throw Exception(ErrorCode::CORRUPTION,
71+
"Native Arrow Variant contains trailing value bytes");
72+
}
73+
if (basic_type == VariantBasicType::OBJECT) {
74+
auto object = output.start_object();
75+
auto fields = value.object_view();
76+
for (uint32_t i = 0; i < fields.size(); ++i) {
77+
uint32_t field_id;
78+
auto child = fields.value_at(i, &field_id);
79+
object.add_key(value.metadata.key_at(field_id));
80+
RETURN_IF_ERROR(append_flight_variant_value(child, output, depth + 1));
81+
}
82+
object.finish();
83+
} else if (basic_type == VariantBasicType::ARRAY) {
84+
auto array = output.start_array();
85+
for (uint32_t i = 0; i < value.num_elements(); ++i) {
86+
RETURN_IF_ERROR(append_flight_variant_value(value.array_at(i), output, depth + 1));
87+
}
88+
array.finish();
89+
} else {
90+
// Primitives never reference dictionary keys. Reuse physical import to retain widths,
91+
// decimal scales and non-JSON types; canonical equality encoding normalizes those away.
92+
static constexpr char empty_metadata[] = {0x11, 0, 0};
93+
value.metadata = {empty_metadata, sizeof(empty_metadata)};
94+
output.add_value(value);
95+
}
96+
return Status::OK();
97+
}
98+
5699
using MetaIdsColumn = ColumnVector<TYPE_UINT32>;
57100

58101
std::span<const NullMap::value_type> forced_nulls(const NullMap* null_map) {
@@ -625,13 +668,14 @@ Status write_paimon_variant(const IColumn& column, const NullMap* null_map,
625668
return status;
626669
}
627670

628-
Status write_iceberg_variant(const IColumn& column, const NullMap* null_map,
629-
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end) {
671+
Status write_parquet_variant_arrow(const IColumn& column, const NullMap* null_map,
672+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
673+
bool compact_metadata) {
630674
if (start < 0 || end < start) {
631-
return Status::InvalidArgument("Invalid Iceberg Variant row range [{}, {})", start, end);
675+
return Status::InvalidArgument("Invalid Variant Arrow row range [{}, {})", start, end);
632676
}
633677
if (array_builder->type()->id() != arrow::Type::STRUCT) {
634-
return Status::InvalidArgument("Iceberg Variant writer requires a struct builder, got {}",
678+
return Status::InvalidArgument("Variant Arrow writer requires a struct builder, got {}",
635679
array_builder->type()->ToString());
636680
}
637681
auto& builder = assert_cast<arrow::StructBuilder&>(*array_builder);
@@ -640,7 +684,7 @@ Status write_iceberg_variant(const IColumn& column, const NullMap* null_map,
640684
type->field(1)->name() != "value" || type->field(0)->type()->id() != arrow::Type::BINARY ||
641685
type->field(1)->type()->id() != arrow::Type::BINARY) {
642686
return Status::InvalidArgument(
643-
"Iceberg Variant writer requires struct<metadata: binary, value: binary>, got {}",
687+
"Variant Arrow writer requires struct<metadata: binary, value: binary>, got {}",
644688
type->ToString());
645689
}
646690
auto& metadata_builder = assert_cast<arrow::BinaryBuilder&>(*builder.field_builder(0));
@@ -657,10 +701,27 @@ Status write_iceberg_variant(const IColumn& column, const NullMap* null_map,
657701
if (!status.ok()) {
658702
return;
659703
}
704+
std::optional<VariantBatchBuilder> compacted;
705+
if (compact_metadata) {
706+
const auto keys = value.metadata.dict_size();
707+
// An empty dictionary or an object using every key already has row-local metadata.
708+
if (keys != 0 && (value.basic_type() != VariantBasicType::OBJECT ||
709+
value.num_elements() != keys)) {
710+
VariantBatchBuilder encoder;
711+
auto row = encoder.begin_row();
712+
status = append_flight_variant_value(value, row);
713+
if (!status.ok()) {
714+
return;
715+
}
716+
row.finish();
717+
compacted.emplace(encoder.finish_batch());
718+
value = compacted->value_at(0);
719+
}
720+
}
660721
if (value.metadata.size > std::numeric_limits<int32_t>::max() ||
661722
value.value.size > std::numeric_limits<int32_t>::max()) {
662723
status = Status::InvalidArgument(
663-
"Iceberg Variant metadata/value exceeds Arrow binary size limit");
724+
"Variant Arrow metadata/value exceeds Arrow binary size limit");
664725
return;
665726
}
666727
status = checkArrowStatus(builder.Append(), column, builder);
@@ -733,6 +794,9 @@ Status DataTypeVariantV2SerDe::write_column_to_arrow(const IColumn& column, cons
733794
options.timezone = &ctz;
734795
const size_t first = checked_row(start);
735796
const size_t last = checked_row(end);
797+
if (array_builder->type()->id() == arrow::Type::STRUCT) {
798+
return write_parquet_variant_arrow(column, null_map, array_builder, start, end, true);
799+
}
736800
if (array_builder->type()->id() == arrow::Type::STRING) {
737801
return write_arrow(column, null_map, assert_cast<arrow::StringBuilder&>(*array_builder),
738802
first, last, options);
@@ -759,7 +823,7 @@ Status DataTypeVariantV2SerDe::write_column_to_iceberg_arrow(
759823
const std::shared_ptr<const IDataType>&, const IColumn& column, const NullMap* null_map,
760824
const std::shared_ptr<arrow::Field>&, arrow::ArrayBuilder* array_builder, int64_t start,
761825
int64_t end, const cctz::time_zone&) const {
762-
return write_iceberg_variant(column, null_map, array_builder, start, end);
826+
return write_parquet_variant_arrow(column, null_map, array_builder, start, end, false);
763827
}
764828

765829
Status DataTypeVariantV2SerDe::write_column_to_orc(const std::string&, const IColumn& column,

‎be/src/format/arrow/arrow_block_convertor.cpp‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -473,6 +473,28 @@ Status ArrowBlockConvertor::init() {
473473
return Status::OK();
474474
}
475475

476+
Status ArrowFlightArrowBlockConvertor::write_column(const std::shared_ptr<const IDataType>& type,
477+
const DataTypeSerDe& serde,
478+
const IColumn& column, const NullMap* null_map,
479+
const std::shared_ptr<arrow::Field>& field,
480+
arrow::ArrayBuilder* array_builder,
481+
int64_t start, int64_t end,
482+
const cctz::time_zone& ctz) const {
483+
if (contains_extension_type(field->type())) {
484+
std::shared_ptr<arrow::DataType> native_type;
485+
RETURN_IF_ERROR(
486+
ArrowFlightSchemaConvertor(ctz.name()).convert_to_arrow_type(type, &native_type));
487+
// Check the extension identity and its complete nested shape before allowing the
488+
// Variant SerDe to write binary storage. An arbitrary STRUCT is not a Variant binding.
489+
// Timestamp labels may differ for equivalent fixed offsets, including inside containers.
490+
if (is_declared_plain_arrow_binding(type, native_type, field->type())) {
491+
return serde.write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
492+
}
493+
}
494+
return DorisArrowBlockConvertor::write_column(type, serde, column, null_map, field,
495+
array_builder, start, end, ctz);
496+
}
497+
476498
Status ArrowFlightArrowBlockConvertor::convert_to_arrow(const Block& block, arrow::MemoryPool* pool,
477499
std::shared_ptr<arrow::RecordBatch>* result,
478500
size_t start_row, size_t end_row) const {

‎be/src/format/arrow/arrow_block_convertor.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,13 @@ class ArrowFlightArrowBlockConvertor final : public DorisArrowBlockConvertor {
116116
Status convert_to_arrow(const Block& block, arrow::MemoryPool* pool,
117117
std::shared_ptr<arrow::RecordBatch>* result, size_t start_row = 0,
118118
size_t end_row = 0) const override;
119+
120+
protected:
121+
Status write_column(const std::shared_ptr<const IDataType>& type, const DataTypeSerDe& serde,
122+
const IColumn& column, const NullMap* null_map,
123+
const std::shared_ptr<arrow::Field>& field,
124+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
125+
const cctz::time_zone& ctz) const override;
119126
};
120127

121128
class PythonArrowBlockConvertor final : public DorisArrowBlockConvertor {

‎be/src/format/arrow/arrow_row_batch.cpp‎

Lines changed: 48 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#include "format/arrow/arrow_row_batch.h"
1919

2020
#include <arrow/buffer.h>
21+
#include <arrow/extension/parquet_variant.h>
2122
#include <arrow/io/memory.h>
2223
#include <arrow/ipc/writer.h>
2324
#include <arrow/record_batch.h>
@@ -40,14 +41,30 @@
4041
#include "core/data_type/data_type_array.h"
4142
#include "core/data_type/data_type_map.h"
4243
#include "core/data_type/data_type_struct.h"
44+
#include "core/data_type/data_type_variant_v2.h"
4345
#include "core/data_type/define_primitive_type.h"
4446
#include "exprs/vexpr.h"
4547
#include "exprs/vexpr_context.h"
4648
#include "format/arrow/arrow_block_convertor.h"
49+
#include "format/arrow/arrow_utils.h"
4750
#include "runtime/descriptors.h"
4851

4952
namespace doris {
5053

54+
Status register_arrow_variant_extension() {
55+
// Remote Flight readers must restore the extension before decoding the result schema.
56+
static const auto status = [] {
57+
if (arrow::GetExtensionType("arrow.parquet.variant") != nullptr) {
58+
return arrow::Status::OK();
59+
}
60+
return arrow::RegisterExtensionType(
61+
std::static_pointer_cast<arrow::ExtensionType>(arrow::extension::variant(
62+
arrow::struct_({arrow::field("metadata", arrow::binary(), false),
63+
arrow::field("value", arrow::binary(), false)}))));
64+
}();
65+
return status.ok() ? Status::OK() : Status::InternalError(status.ToString());
66+
}
67+
5168
Status DorisArrowSchemaConvertor::convert_to_arrow_type(
5269
const DataTypePtr& origin_type, std::shared_ptr<arrow::DataType>* result) const {
5370
auto type = get_serialized_type(origin_type);
@@ -157,10 +174,9 @@ Status DorisArrowSchemaConvertor::convert_to_arrow_type(
157174
*result = std::make_shared<arrow::StructType>(fields);
158175
break;
159176
}
160-
case TYPE_VARIANT: {
177+
case TYPE_VARIANT:
161178
*result = arrow::utf8();
162179
break;
163-
}
164180
case TYPE_QUANTILE_STATE:
165181
case TYPE_BITMAP:
166182
case TYPE_HLL: {
@@ -229,6 +245,25 @@ std::string DorisArrowSchemaConvertor::timestamp_timezone(PrimitiveType) const {
229245
return _timezone == "Z" ? "UTC" : _timezone;
230246
}
231247

248+
Status ArrowFlightSchemaConvertor::convert_to_arrow_type(
249+
const DataTypePtr& type, std::shared_ptr<arrow::DataType>* result) const {
250+
// Flight always uses native Variant, including recursively converted children.
251+
if (type->get_primitive_type() == TYPE_VARIANT) {
252+
// Reject by type before reading rows, including empty results and nested legacy leaves.
253+
if (dynamic_cast<const DataTypeVariantV2*>(remove_nullable(type).get()) == nullptr) {
254+
return Status::NotSupported(
255+
"Native Arrow Flight output only supports Variant V2, not legacy Variant; "
256+
"cast the result to STRING for text output");
257+
}
258+
RETURN_IF_ERROR(register_arrow_variant_extension());
259+
*result = arrow::extension::variant(
260+
arrow::struct_({arrow::field("metadata", arrow::binary(), false),
261+
arrow::field("value", arrow::binary(), false)}));
262+
return Status::OK();
263+
}
264+
return DorisArrowSchemaConvertor::convert_to_arrow_type(type, result);
265+
}
266+
232267
std::string ArrowFlightSchemaConvertor::timestamp_timezone(PrimitiveType type) const {
233268
// DATETIMEV2 is wall-clock time; TIMESTAMPTZ must still describe an instant.
234269
return type == TYPE_DATETIMEV2 ? "" : DorisArrowSchemaConvertor::timestamp_timezone(type);
@@ -323,13 +358,17 @@ Status serialize_record_batch(const arrow::RecordBatch& record_batch, std::strin
323358
}
324359

325360
Status serialize_arrow_schema(std::shared_ptr<arrow::Schema>* schema, std::string* result) {
326-
auto make_empty_result = arrow::RecordBatch::MakeEmpty(*schema);
327-
if (!make_empty_result.ok()) {
328-
return Status::InternalError("serialize_arrow_schema failed, reason: {}",
329-
make_empty_result.status().ToString());
330-
}
331-
auto batch = make_empty_result.ValueOrDie();
332-
return serialize_record_batch(*batch, result);
361+
// Schema RPC readers only consume the IPC schema. Building an empty batch would require
362+
// nested extension builders, which Arrow does not provide for ARRAY/MAP/STRUCT<VARIANT>.
363+
std::shared_ptr<arrow::io::BufferOutputStream> sink;
364+
RETURN_DORIS_STATUS_IF_RESULT_ERROR(sink, arrow::io::BufferOutputStream::Create());
365+
std::shared_ptr<arrow::ipc::RecordBatchWriter> writer;
366+
RETURN_DORIS_STATUS_IF_RESULT_ERROR(writer, arrow::ipc::MakeStreamWriter(sink.get(), *schema));
367+
RETURN_DORIS_STATUS_IF_ERROR(writer->Close());
368+
std::shared_ptr<arrow::Buffer> buffer;
369+
RETURN_DORIS_STATUS_IF_RESULT_ERROR(buffer, sink->Finish());
370+
*result = buffer->ToString();
371+
return Status::OK();
333372
}
334373

335374
} // namespace doris

‎be/src/format/arrow/arrow_row_batch.h‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,8 +63,8 @@ class DorisArrowSchemaConvertor : public ArrowSchemaConvertor {
6363
std::shared_ptr<arrow::Schema>* result) const;
6464
Status get_arrow_schema_from_expr_ctxs(const VExprContextSPtrs& output_vexpr_ctxs,
6565
std::shared_ptr<arrow::Schema>* result) const;
66-
Status convert_to_arrow_type(const DataTypePtr& type,
67-
std::shared_ptr<arrow::DataType>* result) const;
66+
virtual Status convert_to_arrow_type(const DataTypePtr& type,
67+
std::shared_ptr<arrow::DataType>* result) const;
6868

6969
protected:
7070
virtual std::string timestamp_timezone(PrimitiveType type) const;
@@ -82,7 +82,13 @@ class DorisArrowSchemaConvertor : public ArrowSchemaConvertor {
8282

8383
class ArrowFlightSchemaConvertor : public DorisArrowSchemaConvertor {
8484
public:
85-
using DorisArrowSchemaConvertor::DorisArrowSchemaConvertor;
85+
explicit ArrowFlightSchemaConvertor(std::string timezone)
86+
: DorisArrowSchemaConvertor(std::move(timezone)) {}
87+
ArrowFlightSchemaConvertor(const Block& header, std::string timezone)
88+
: DorisArrowSchemaConvertor(header, std::move(timezone)) {}
89+
90+
Status convert_to_arrow_type(const DataTypePtr& type,
91+
std::shared_ptr<arrow::DataType>* result) const override;
8692

8793
protected:
8894
std::string timestamp_timezone(PrimitiveType type) const override;
@@ -103,6 +109,8 @@ class LegacyArrowFlightSchemaConvertor final : public ArrowFlightSchemaConvertor
103109
PrimitiveType primitive) const override;
104110
};
105111

112+
Status register_arrow_variant_extension();
113+
106114
std::shared_ptr<arrow::Field> create_arrow_field_with_metadata(
107115
const std::string& field_name, const std::shared_ptr<arrow::DataType>& arrow_type,
108116
bool is_nullable, PrimitiveType primitive_type);

‎be/src/service/arrow_flight/arrow_flight_batch_reader.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,7 @@ arrow::Status ArrowFlightBatchRemoteReader::_fetch_schema() {
298298
st = Status::create(callback->response_->status());
299299
ARROW_RETURN_NOT_OK(to_arrow_status(st));
300300

301+
ARROW_RETURN_NOT_OK(to_arrow_status(register_arrow_variant_extension()));
301302
if (callback->response_->has_schema() && !callback->response_->schema().empty()) {
302303
auto input =
303304
arrow::io::BufferReader::FromString(std::string(callback->response_->schema()));

0 commit comments

Comments
 (0)