Skip to content

Commit c070334

Browse files
committed
[refactor](arrow) Make format convertors explicit and schema-owned
### What problem does this PR solve? Related PR: apache#67784. This is the second split, based on the primitives merged in apache#68297. Arrow batch conversion mixes protocol serialization with table-specific UUID and Variant handling, while writers construct schemas separately. Introduce explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon convertors, instantiated with their own schema parameters and timezone. Move table schema construction/decoding into the convertors and route nested SerDe writes through the selected format. Separate Parquet, Hive and Iceberg writers and migrate all callers. Preserve current Parquet timestamp encoding, UUID/Variant layouts and external type mappings. Keep Python's numeric conversion timezone consistent with its declared Arrow schema, and reject incompatible nested target schemas before casts or child access. No FE, Thrift, ORC timestamp, or binary computation changes are included. ### Release note Fix Python UDF timestamp conversion to preserve wall-clock values when the Arrow protocol declares a fixed-offset timezone. ### Check List (For Author) - Test - [x] Unit Test: schema ownership/isolation, slices, invalid schemas, nested/null values, UUID/Variant bytes, fixed-offset Python batches, and Parquet timestamp representation. - [x] Regression test: add Python UDF/UDTF/UDAF timezone assertions and extract the corresponding existing Python snapshot corrections from apache#67784. - Behavior changed: - [x] Yes: align Python UDF conversion with its Arrow timezone declaration; report invalid nested schema bindings as errors. - Does this need documentation? - [x] No. Validation: ASAN BE build and 396 selected tests passed; the final incremental build and 285 focused tests also passed. clang-format 16 and Groovy/embedded Python syntax checks passed. Python SQL and external-catalog end-to-end regressions were not run locally and remain for CI. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent a334145 commit c070334

51 files changed

Lines changed: 2738 additions & 719 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/data_type_array_serde.cpp‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,77 @@ Status DataTypeArraySerDe::write_column_to_arrow(const IColumn& column, const Nu
321321
return Status::OK();
322322
}
323323

324+
namespace {
325+
326+
template <typename WriteNested>
327+
Status write_array_column_to_target(const IColumn& column, const NullMap* null_map,
328+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
329+
WriteNested&& write_nested) {
330+
const auto& array_column = assert_cast<const ColumnArray&>(column);
331+
const auto& offsets = array_column.get_offsets();
332+
const auto& nested_data = array_column.get_data();
333+
auto& builder = assert_cast<arrow::ListBuilder&>(*array_builder);
334+
auto* nested_builder = builder.value_builder();
335+
for (size_t array_idx = start; array_idx < end; ++array_idx) {
336+
if (null_map != nullptr && (*null_map)[array_idx]) {
337+
RETURN_IF_ERROR(checkArrowStatus(builder.AppendNull(), column, *array_builder));
338+
continue;
339+
}
340+
RETURN_IF_ERROR(checkArrowStatus(builder.Append(), column, *array_builder));
341+
RETURN_IF_ERROR(write_nested(nested_data, nested_builder, offsets[array_idx - 1],
342+
offsets[array_idx]));
343+
}
344+
return Status::OK();
345+
}
346+
347+
} // namespace
348+
349+
Status DataTypeArraySerDe::write_column_to_paimon_arrow(
350+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
351+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
352+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
353+
const cctz::time_zone& ctz) const {
354+
// Reject an incompatible target before casting its nested schema or builder.
355+
if (field->type()->id() != arrow::Type::LIST ||
356+
array_builder->type()->id() != arrow::Type::LIST) {
357+
return Status::InvalidArgument("Paimon array writer requires an Arrow list field");
358+
}
359+
const auto& array_type = assert_cast<const DataTypeArray&>(*type);
360+
const auto& list_type = assert_cast<const arrow::ListType&>(*field->type());
361+
const auto& nested_field = list_type.value_field();
362+
return write_array_column_to_target(
363+
column, null_map, array_builder, start, end,
364+
[&](const IColumn& nested_data, arrow::ArrayBuilder* nested_builder,
365+
int64_t nested_start, int64_t nested_end) {
366+
return nested_serde->write_column_to_paimon_arrow(
367+
array_type.get_nested_type(), nested_data, nullptr, nested_field,
368+
nested_builder, nested_start, nested_end, ctz);
369+
});
370+
}
371+
372+
Status DataTypeArraySerDe::write_column_to_iceberg_arrow(
373+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
374+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
375+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
376+
const cctz::time_zone& ctz) const {
377+
// Reject an incompatible target before casting its nested schema or builder.
378+
if (field->type()->id() != arrow::Type::LIST ||
379+
array_builder->type()->id() != arrow::Type::LIST) {
380+
return Status::InvalidArgument("Iceberg array writer requires an Arrow list field");
381+
}
382+
const auto& array_type = assert_cast<const DataTypeArray&>(*type);
383+
const auto& list_type = assert_cast<const arrow::ListType&>(*field->type());
384+
const auto& nested_field = list_type.value_field();
385+
return write_array_column_to_target(
386+
column, null_map, array_builder, start, end,
387+
[&](const IColumn& nested_data, arrow::ArrayBuilder* nested_builder,
388+
int64_t nested_start, int64_t nested_end) {
389+
return nested_serde->write_column_to_iceberg_arrow(
390+
array_type.get_nested_type(), nested_data, nullptr, nested_field,
391+
nested_builder, nested_start, nested_end, ctz);
392+
});
393+
}
394+
324395
Status DataTypeArraySerDe::read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array,
325396
int64_t start, int64_t end,
326397
const cctz::time_zone& ctz) const {

‎be/src/core/data_type_serde/data_type_array_serde.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,16 @@ class DataTypeArraySerDe : public DataTypeSerDe {
9292
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
9393
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
9494
const cctz::time_zone& ctz) const override;
95+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
96+
const IColumn& column, const NullMap* null_map,
97+
const std::shared_ptr<arrow::Field>& field,
98+
arrow::ArrayBuilder* array_builder, int64_t start,
99+
int64_t end, const cctz::time_zone& ctz) const override;
100+
Status write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>& type,
101+
const IColumn& column, const NullMap* null_map,
102+
const std::shared_ptr<arrow::Field>& field,
103+
arrow::ArrayBuilder* array_builder, int64_t start,
104+
int64_t end, const cctz::time_zone& ctz) const override;
95105
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
96106
int64_t end, const cctz::time_zone& ctz) const override;
97107
Status read_column_from_orc(IColumn& column, const OrcDecodedColumnView& view) const override;

‎be/src/core/data_type_serde/data_type_map_serde.cpp‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "core/column/column.h"
2727
#include "core/column/column_const.h"
2828
#include "core/column/column_map.h"
29+
#include "core/data_type/data_type_map.h"
2930
#include "core/data_type_serde/arrow_validation.h"
3031
#include "core/data_type_serde/complex_type_deserialize_util.h"
3132
#include "core/data_type_serde/orc_serde_utils.h"
@@ -440,6 +441,108 @@ Status DataTypeMapSerDe::write_column_to_arrow(const IColumn& column, const Null
440441
return Status::OK();
441442
}
442443

444+
namespace {
445+
446+
template <typename WriteKey, typename WriteValue>
447+
Status write_map_column_to_target(const IColumn& column, const NullMap* null_map,
448+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
449+
WriteKey&& write_key, WriteValue&& write_value) {
450+
auto& builder = assert_cast<arrow::MapBuilder&>(*array_builder);
451+
const auto& map_column = assert_cast<const ColumnMap&>(column);
452+
const IColumn& nested_keys_column = map_column.get_keys();
453+
const IColumn& nested_values_column = map_column.get_values();
454+
DCHECK(nested_keys_column.is_nullable());
455+
DCHECK(nested_values_column.is_nullable());
456+
const auto* keys_nullmap_data =
457+
check_and_get_column<ColumnNullable>(nested_keys_column)->get_null_map_data().data();
458+
const auto& offsets = map_column.get_offsets();
459+
auto* key_builder = builder.key_builder();
460+
auto* value_builder = builder.item_builder();
461+
462+
for (size_t row = start; row < end; ++row) {
463+
if (null_map != nullptr && (*null_map)[row]) {
464+
RETURN_IF_ERROR(checkArrowStatus(builder.AppendNull(), column, *array_builder));
465+
continue;
466+
}
467+
if (simd::contain_one(keys_nullmap_data + offsets[row - 1],
468+
offsets[row] - offsets[row - 1])) {
469+
return Status::Error(ErrorCode::INVALID_ARGUMENT,
470+
"Can not write null value of map key to arrow.");
471+
}
472+
RETURN_IF_ERROR(checkArrowStatus(builder.Append(), column, *array_builder));
473+
RETURN_IF_ERROR(write_key(nested_keys_column, key_builder, offsets[row - 1], offsets[row]));
474+
RETURN_IF_ERROR(
475+
write_value(nested_values_column, value_builder, offsets[row - 1], offsets[row]));
476+
}
477+
return Status::OK();
478+
}
479+
480+
} // namespace
481+
482+
Status DataTypeMapSerDe::write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
483+
const IColumn& column,
484+
const NullMap* null_map,
485+
const std::shared_ptr<arrow::Field>& field,
486+
arrow::ArrayBuilder* array_builder,
487+
int64_t start, int64_t end,
488+
const cctz::time_zone& ctz) const {
489+
// Reject an incompatible target before casting its nested schema or builder.
490+
if (field->type()->id() != arrow::Type::MAP ||
491+
array_builder->type()->id() != arrow::Type::MAP) {
492+
return Status::InvalidArgument("Paimon map writer requires an Arrow map field");
493+
}
494+
const auto& map_type = assert_cast<const DataTypeMap&>(*type);
495+
const auto& arrow_map_type = assert_cast<const arrow::MapType&>(*field->type());
496+
const auto& key_field = arrow_map_type.key_field();
497+
const auto& value_field = arrow_map_type.item_field();
498+
return write_map_column_to_target(
499+
column, null_map, array_builder, start, end,
500+
[&](const IColumn& nested_keys, arrow::ArrayBuilder* key_builder, int64_t nested_start,
501+
int64_t nested_end) {
502+
return key_serde->write_column_to_paimon_arrow(map_type.get_key_type(), nested_keys,
503+
nullptr, key_field, key_builder,
504+
nested_start, nested_end, ctz);
505+
},
506+
[&](const IColumn& nested_values, arrow::ArrayBuilder* value_builder,
507+
int64_t nested_start, int64_t nested_end) {
508+
return value_serde->write_column_to_paimon_arrow(
509+
map_type.get_value_type(), nested_values, nullptr, value_field,
510+
value_builder, nested_start, nested_end, ctz);
511+
});
512+
}
513+
514+
Status DataTypeMapSerDe::write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>& type,
515+
const IColumn& column,
516+
const NullMap* null_map,
517+
const std::shared_ptr<arrow::Field>& field,
518+
arrow::ArrayBuilder* array_builder,
519+
int64_t start, int64_t end,
520+
const cctz::time_zone& ctz) const {
521+
// Reject an incompatible target before casting its nested schema or builder.
522+
if (field->type()->id() != arrow::Type::MAP ||
523+
array_builder->type()->id() != arrow::Type::MAP) {
524+
return Status::InvalidArgument("Iceberg map writer requires an Arrow map field");
525+
}
526+
const auto& map_type = assert_cast<const DataTypeMap&>(*type);
527+
const auto& arrow_map_type = assert_cast<const arrow::MapType&>(*field->type());
528+
const auto& key_field = arrow_map_type.key_field();
529+
const auto& value_field = arrow_map_type.item_field();
530+
return write_map_column_to_target(
531+
column, null_map, array_builder, start, end,
532+
[&](const IColumn& nested_keys, arrow::ArrayBuilder* key_builder, int64_t nested_start,
533+
int64_t nested_end) {
534+
return key_serde->write_column_to_iceberg_arrow(
535+
map_type.get_key_type(), nested_keys, nullptr, key_field, key_builder,
536+
nested_start, nested_end, ctz);
537+
},
538+
[&](const IColumn& nested_values, arrow::ArrayBuilder* value_builder,
539+
int64_t nested_start, int64_t nested_end) {
540+
return value_serde->write_column_to_iceberg_arrow(
541+
map_type.get_value_type(), nested_values, nullptr, value_field,
542+
value_builder, nested_start, nested_end, ctz);
543+
});
544+
}
545+
443546
Status DataTypeMapSerDe::read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array,
444547
int64_t start, int64_t end,
445548
const cctz::time_zone& ctz) const {

‎be/src/core/data_type_serde/data_type_map_serde.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,16 @@ class DataTypeMapSerDe : public DataTypeSerDe {
8383
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8484
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
8585
const cctz::time_zone& ctz) const override;
86+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
87+
const IColumn& column, const NullMap* null_map,
88+
const std::shared_ptr<arrow::Field>& field,
89+
arrow::ArrayBuilder* array_builder, int64_t start,
90+
int64_t end, const cctz::time_zone& ctz) const override;
91+
Status write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>& type,
92+
const IColumn& column, const NullMap* null_map,
93+
const std::shared_ptr<arrow::Field>& field,
94+
arrow::ArrayBuilder* array_builder, int64_t start,
95+
int64_t end, const cctz::time_zone& ctz) const override;
8696
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
8797
int64_t end, const cctz::time_zone& ctz) const override;
8898
Status read_column_from_orc(IColumn& column, const OrcDecodedColumnView& view) const override;

‎be/src/core/data_type_serde/data_type_nullable_serde.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
#include "core/column/column_const.h"
3232
#include "core/column/column_nullable.h"
3333
#include "core/column/column_vector.h"
34+
#include "core/data_type/data_type_nullable.h"
3435
#include "core/data_type_serde/arrow_validation.h"
3536
#include "core/data_type_serde/data_type_serde.h"
3637
#include "core/data_type_serde/data_type_string_serde.h"
@@ -385,6 +386,28 @@ Status DataTypeNullableSerDe::write_column_to_arrow(const IColumn& column, const
385386
start, end, ctz);
386387
}
387388

389+
Status DataTypeNullableSerDe::write_column_to_paimon_arrow(
390+
const std::shared_ptr<const IDataType>& type, const IColumn& column, const NullMap*,
391+
const std::shared_ptr<arrow::Field>& field, arrow::ArrayBuilder* array_builder,
392+
int64_t start, int64_t end, const cctz::time_zone& ctz) const {
393+
const auto& nullable_type = assert_cast<const DataTypeNullable&>(*type);
394+
const auto& column_nullable = assert_cast<const ColumnNullable&>(column);
395+
return nested_serde->write_column_to_paimon_arrow(
396+
nullable_type.get_nested_type(), column_nullable.get_nested_column(),
397+
&column_nullable.get_null_map_data(), field, array_builder, start, end, ctz);
398+
}
399+
400+
Status DataTypeNullableSerDe::write_column_to_iceberg_arrow(
401+
const std::shared_ptr<const IDataType>& type, const IColumn& column, const NullMap*,
402+
const std::shared_ptr<arrow::Field>& field, arrow::ArrayBuilder* array_builder,
403+
int64_t start, int64_t end, const cctz::time_zone& ctz) const {
404+
const auto& nullable_type = assert_cast<const DataTypeNullable&>(*type);
405+
const auto& column_nullable = assert_cast<const ColumnNullable&>(column);
406+
return nested_serde->write_column_to_iceberg_arrow(
407+
nullable_type.get_nested_type(), column_nullable.get_nested_column(),
408+
&column_nullable.get_null_map_data(), field, array_builder, start, end, ctz);
409+
}
410+
388411
Status DataTypeNullableSerDe::read_column_from_arrow(IColumn& column,
389412
const arrow::Array* arrow_array, int64_t start,
390413
int64_t end,

‎be/src/core/data_type_serde/data_type_nullable_serde.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,16 @@ class DataTypeNullableSerDe : public DataTypeSerDe {
8787
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8888
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
8989
const cctz::time_zone& ctz) const override;
90+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
91+
const IColumn& column, const NullMap* null_map,
92+
const std::shared_ptr<arrow::Field>& field,
93+
arrow::ArrayBuilder* array_builder, int64_t start,
94+
int64_t end, const cctz::time_zone& ctz) const override;
95+
Status write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>& type,
96+
const IColumn& column, const NullMap* null_map,
97+
const std::shared_ptr<arrow::Field>& field,
98+
arrow::ArrayBuilder* array_builder, int64_t start,
99+
int64_t end, const cctz::time_zone& ctz) const override;
90100
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
91101
int64_t end, const cctz::time_zone& ctz) const override;
92102
Status read_column_from_decoded_values(IColumn& column,

‎be/src/core/data_type_serde/data_type_serde.h‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
namespace arrow {
3838
class ArrayBuilder;
3939
class Array;
40+
class Field;
4041
} // namespace arrow
4142
namespace cctz {
4243
class time_zone;
@@ -498,6 +499,23 @@ class DataTypeSerDe {
498499
virtual Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
499500
arrow::ArrayBuilder* array_builder, int64_t start,
500501
int64_t end, const cctz::time_zone& ctz) const = 0;
502+
// Most scalar types deliberately share their physical Arrow encoding across these protocols.
503+
// Target-specific SerDes override the corresponding method; callers never retry another
504+
// protocol method after an error.
505+
virtual Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>&,
506+
const IColumn& column, const NullMap* null_map,
507+
const std::shared_ptr<arrow::Field>&,
508+
arrow::ArrayBuilder* array_builder, int64_t start,
509+
int64_t end, const cctz::time_zone& ctz) const {
510+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
511+
}
512+
virtual Status write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>&,
513+
const IColumn& column, const NullMap* null_map,
514+
const std::shared_ptr<arrow::Field>&,
515+
arrow::ArrayBuilder* array_builder, int64_t start,
516+
int64_t end, const cctz::time_zone& ctz) const {
517+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
518+
}
501519
virtual Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array,
502520
int64_t start, int64_t end,
503521
const cctz::time_zone& ctz) const = 0;

0 commit comments

Comments
 (0)