Skip to content

Commit a01cc89

Browse files
authored
[fix](external) Preserve external binary, UUID and timestamp mappings on master (apache#68786)
### What problem does this PR solve? Related PR: apache#68532. Supersedes apache#68603. Preserve external binary, UUID, and timestamp semantics across connector schemas, JDBC transport, file readers, and external writes. The tables below list the type mappings changed by this PR, including mappings that previously depended on an opt-in property. ### Behavior changes - The binary mappings listed below now expose `VARBINARY` and preserve raw bytes, including invalid UTF-8, embedded NULs, and trailing zeros. File TVFs also apply this rule to nested binary fields. `DESCRIBE` and query results therefore expose binary types and hexadecimal values instead of decoded text. - External logical UUID types map to native Doris `UUID`, including nested fields, independently of binary mapping options. Native UUID inputs and canonical/compact UUID text can be written to Iceberg UUID columns; invalid text fails explicitly. UUID file reads preserve network byte order and can load directly into native Doris UUID columns. - Timestamps representing an instant map to `TIMESTAMPTZ`; timestamps representing local wall-clock fields map to `DATETIMEV2`. Instant semantics also apply to MySQL `TIMESTAMP` and ClickHouse `DateTime`/`DateTime64` without an explicit timezone in the type name. `TIMESTAMPTZ` preserves the instant, not the source timezone identifier or original offset spelling. - Catalog properties `enable.mapping.varbinary` / `enable.mapping.timestamp_tz` and file TVF properties `enable_mapping_varbinary` / `enable_mapping_timestamp_tz` remain accepted but no longer select the mappings below. Absent, `false`, and `true` settings produce the same types. Recursive mappings apply to supported ARRAY/MAP/STRUCT fields. - JDBC reads and writes preserve UTC instants and supported fractional precision independently of JVM/session timezone. PostgreSQL infinite/out-of-range `timestamptz` values become SQL NULL, with nullable metadata propagated even for source NOT NULL columns. MySQL retains its configured zero-date handling. Predicates mixing instant and wall-clock timestamps stay local when remote evaluation would change their meaning. - Binary partition values retain byte identity and distinguish SQL NULL from literal text. Iceberg UUID writes preserve UUID semantics through ARRAY/MAP/STRUCT leaves, including map keys and values; fixed-width binary writes validate the actual byte count. Paimon legacy ORC routing uses the split's owning branch and historical schema so instant columns use the appropriate reader. - MySQL query TVF wrapping distinguishes `--1` arithmetic from comments and honors the remote connection’s `NO_BACKSLASH_ESCAPES` mode. SQL Server UUID range predicates stay local because its GUID ordering differs from Doris. Iceberg UUID value predicates also stay local because file bounds can otherwise prune matching UUID rows. - Ordinary views retain binary types and declared VARBINARY bounds. Native-table materialization remains subject to Doris storage restrictions and needs explicit conversion where the target cannot store VARBINARY. Unsupported VARBINARY comparisons/functions remain unsupported; FE validation prevents implicit text coercion for the guarded operations. CDC JSON ingestion retains its existing transport conversions. ### Changed read mappings: external system → Doris “Before” describes the previous default (mapping options absent/false). Where an opt-in mapping already existed, this PR makes that mapping unconditional. `p` denotes the effective fractional precision supported by the mapper, capped at 6 (HMS uses the catalog timestamp precision); fixed precisions are shown explicitly. `n` denotes the supported declared byte bound; unqualified `VARBINARY` uses Doris's default maximum bound. Unchanged source-type mappings are omitted. | External system / access path | External type | Before | After | | --- | --- | --- | --- | | Hive / HMS | `BINARY` | `STRING` | `VARBINARY` | | Hive / HMS | `TIMESTAMP WITH LOCAL TIME ZONE` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Iceberg | `binary` | `STRING` | `VARBINARY` | | Iceberg | `fixed(n)` | `CHAR(n)` | `VARBINARY(n)` | | Iceberg | `uuid` | `STRING` | `UUID` | | Iceberg | `timestamptz` (`shouldAdjustToUTC=true`) | `DATETIMEV2(6)` | `TIMESTAMPTZ(6)` | | Paimon | `BINARY(n)`, `VARBINARY(n)` / `BYTES` | `STRING` | `VARBINARY(n)` / `VARBINARY` | | Paimon | `TIMESTAMP WITH LOCAL TIME ZONE` / `TIMESTAMP_LTZ(p)` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Fluss | `BINARY(n)`, `BYTES` | `STRING` | `VARBINARY(n)`, `VARBINARY` respectively | | Fluss | `TIMESTAMP_LTZ(p)` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Hudi / Avro logical schema | Logical `uuid` on `string` | `STRING` | `UUID` | | Hudi / Avro logical schema | `timestamp-millis`, `timestamp-micros` | `DATETIMEV2(3)`, `DATETIMEV2(6)` | `TIMESTAMPTZ(3)`, `TIMESTAMPTZ(6)` respectively | | Hudi / Avro logical schema | `local-timestamp-millis`, `local-timestamp-micros` | `BIGINT` | `DATETIMEV2(3)`, `DATETIMEV2(6)` respectively | | MaxCompute | `TIMESTAMP` | `DATETIMEV2(6)` | `TIMESTAMPTZ(6)` | | Trino Connector | `UUID` | Unsupported by the type mapper | `UUID` | | Trino Connector | `VARBINARY` | `STRING` | `VARBINARY` | | Trino Connector | `TIMESTAMP(p) WITH TIME ZONE` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | MySQL / MariaDB JDBC | `BINARY`, `VARBINARY`, `TINYBLOB`, `BLOB`, `MEDIUMBLOB`, `LONGBLOB` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | MySQL / MariaDB JDBC | `TIMESTAMP(p)` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | PostgreSQL JDBC | `uuid`, including array elements | `STRING` | `UUID` | | PostgreSQL JDBC | `bytea` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | PostgreSQL JDBC | `timestamptz` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Oracle JDBC | `BLOB` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | Oracle JDBC | `TIMESTAMP(p) WITH LOCAL TIME ZONE` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Oracle JDBC | `TIMESTAMP(p) WITH TIME ZONE` | `STRING` in the connector; unsupported in the legacy client | `TIMESTAMPTZ(p)` | | SQL Server JDBC | `uniqueidentifier` | `STRING` | `UUID` | | SQL Server JDBC | `binary`, `varbinary`, `image` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | SQL Server JDBC | `datetimeoffset(p)` | `STRING` | `TIMESTAMPTZ(p)` | | DB2 JDBC | `BLOB`, `BINARY`, `VARBINARY` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | GBase JDBC | JDBC `BINARY`, `VARBINARY`, `LONGVARBINARY`, `BLOB` | `STRING` | `VARBINARY` with the supported JDBC-reported byte bound | | ClickHouse JDBC | `DateTime`, `DateTime('zone')` | `DATETIMEV2(0)` | `TIMESTAMPTZ(0)` | | ClickHouse JDBC | `DateTime64(p[, 'zone'])` | `DATETIMEV2(p)` | `TIMESTAMPTZ(p)` | | Trino / Presto JDBC | `uuid`, including array elements | Unsupported by the type mapper | `UUID` | | Trino / Presto JDBC | `TIMESTAMP(p) WITH TIME ZONE`, `TIMESTAMP WITH TIME ZONE` | `DATETIMEV2` / `STRING`, or a metadata precision-parsing failure depending on spelling/path | `TIMESTAMPTZ(p)`; precision defaults to 6 when absent | | Trino / Presto JDBC connector | Unparameterized `TIMESTAMP` metadata | `STRING` | `DATETIMEV2(6)` | | Parquet file TVFs | Raw `BYTE_ARRAY`, including binary fallback for legacy ENUM/BSON annotations | `STRING` | `VARBINARY` | | Parquet file TVFs | UUID logical annotation on `FIXED_LEN_BYTE_ARRAY(16)` | `STRING` during legacy schema inference; scanner V2 already supported native UUID; opt-in binary mapping exposed `VARBINARY(16)` | `UUID` consistently | | Parquet file TVFs | Logical `TIMESTAMP` with `isAdjustedToUTC=true` | `DATETIMEV2(3/6)` | `TIMESTAMPTZ(3/6)` for millis / micros or nanos | | ORC file TVFs | `BINARY` | `STRING` | `VARBINARY` | | ORC file TVFs | `BINARY` with `iceberg.binary-type=UUID` | `STRING` | `UUID` | | ORC file TVFs | `TIMESTAMP_INSTANT` | `DATETIMEV2(6)` | `TIMESTAMPTZ(6)` | OceanBase JDBC delegates to the MySQL or Oracle mapper according to its database mode and inherits the corresponding changed mappings above. The file TVF rows apply to schema inference through supported file sources such as S3, HDFS, LOCAL, and HTTP. Existing unzoned timestamp mappings (for example MySQL `DATETIME`, PostgreSQL `timestamp`, Iceberg `timestamp`, and Paimon/Fluss `TIMESTAMP`) remain `DATETIMEV2` and are omitted from the table. Precision beyond microseconds is truncated. Physical binary storage with a recognized STRING/UTF8/DECIMAL annotation retains that logical type. ### Changed external schema mappings: Doris → external system | External target | Doris type | Before | After | | --- | --- | --- | --- | | Iceberg | `UUID`, including nested fields | Unsupported by the reverse schema mapper | Iceberg `uuid` | | Hive / HMS | `VARBINARY` | Unsupported by the reverse schema mapper | Hive `binary`, including nested fields | | Paimon | `DATETIMEV2(p)` | Paimon `TIMESTAMP(6)`; source precision discarded | Paimon `TIMESTAMP(p)` | | Paimon | `TIMESTAMPTZ(p)` | Unsupported by the reverse schema mapper | Paimon `TIMESTAMP_LTZ(p)` | | MaxCompute | `TIMESTAMPTZ` | Unsupported by the reverse schema mapper | MaxCompute `TIMESTAMP`, with UTC microsecond write transport | ### Release note The external binary, UUID, and timestamp mappings listed above are determined by source semantics. The former mapping options no longer opt out of VARBINARY or TIMESTAMPTZ. Applications reading these columns may observe changed schemas, binary result formatting, and timestamp semantics. External UUIDs retain native UUID semantics rather than mapping to string or binary. ### Validation - FE and Java plugin package build passed with 339 targeted unit tests, plus 59 JDBC planning/binding tests. FE Checkstyle passed. - 202 BE unit tests passed, covering UUID byte order and nullability, Parquet/ORC data and equality-delete combinations, missing-column defaults, ORC writes, and UUID partitions. Changed C++ lines passed clang-tidy; clang-format 16 and build hygiene checks passed. - 13 local regression suites passed: seven UUID file suites, PostgreSQL native UUID and all-types suites, MySQL query TVFs under both SQL modes, Iceberg nested UUID writes, cross-format reads/deletes, and initial defaults. Result files are generated by actual execution and compared in a separate run. - Iceberg write coverage includes canonical/compact text, native UUID INSERT SELECT, identity partitions, ARRAY/MAP/STRUCT nesting, NULL and empty containers, invalid inputs, and volatile map expressions whose key/value pairing must be retained. A non-pushable reference verifies UUID equality predicates do not lose rows to file pruning. - SQL Server, Trino, and Hudi UUID paths have unit coverage; their external integration suites were not run in this update. - Previous revision validation also covered the affected HDFS, Paimon, and HTTP TVF suites. ### Check List (For Author) - [x] Unit and regression coverage included. - [x] User-visible behavior and changed type mappings described. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note. - [ ] Confirm test cases. - [ ] Confirm document. - [ ] Add branch pick label if needed.
1 parent f40c80d commit a01cc89

350 files changed

Lines changed: 12581 additions & 3471 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/data_type_datetimev2_serde.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -638,6 +638,29 @@ Status DataTypeDateTimeV2SerDe::write_column_to_arrow(const IColumn& column,
638638
return Status::OK();
639639
}
640640

641+
Status DataTypeDateTimeV2SerDe::write_column_to_paimon_arrow(
642+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
643+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
644+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
645+
const cctz::time_zone& ctz) const {
646+
if (field->type()->id() != arrow::Type::TIMESTAMP) {
647+
return Status::InvalidArgument(
648+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
649+
type->get_name(), field->ToString());
650+
}
651+
// The pinned Paimon schema must agree with the declared precision; do not silently rescale.
652+
const auto& timestamp = assert_cast<const arrow::TimestampType&>(*field->type());
653+
const arrow::TimeUnit::type expected_unit = type->get_scale() > 3 ? arrow::TimeUnit::MICRO
654+
: type->get_scale() > 0 ? arrow::TimeUnit::MILLI
655+
: arrow::TimeUnit::SECOND;
656+
if (timestamp.unit() != expected_unit) {
657+
return Status::InvalidArgument(
658+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
659+
type->get_name(), field->ToString());
660+
}
661+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
662+
}
663+
641664
Status DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
642665
const arrow::Array* arrow_array,
643666
int64_t start, int64_t end,

‎be/src/core/data_type_serde/data_type_datetimev2_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,11 @@ class DataTypeDateTimeV2SerDe : public DataTypeNumberSerDe<PrimitiveType::TYPE_D
8888
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8989
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
9090
const cctz::time_zone& ctz) const override;
91+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
92+
const IColumn& column, const NullMap* null_map,
93+
const std::shared_ptr<arrow::Field>& field,
94+
arrow::ArrayBuilder* array_builder, int64_t start,
95+
int64_t end, const cctz::time_zone& ctz) const override;
9196
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
9297
int64_t end, const cctz::time_zone& ctz) const override;
9398
Status read_column_from_decoded_values(IColumn& column,

‎be/src/core/data_type_serde/data_type_string_serde.cpp‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -568,7 +568,12 @@ Status DataTypeStringSerDeBase<ColumnType>::write_column_to_iceberg_arrow(
568568
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
569569
const cctz::time_zone& ctz) const {
570570
if (!is_iceberg_uuid_field(field)) {
571-
// Keep the existing CHAR/STRING fixed-binary binding until external type mappings change.
571+
// Iceberg fixed(N) is a byte binding, so text padding must not change its physical value.
572+
if (array_builder->type()->id() == arrow::Type::FIXED_SIZE_BINARY) {
573+
return Status::InvalidArgument(
574+
"Iceberg fixed writer requires Doris VARBINARY, got {} for Arrow field {}",
575+
type->get_name(), field->ToString());
576+
}
572577
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
573578
}
574579
if (!is_string_type(type->get_primitive_type()) ||

‎be/src/core/data_type_serde/data_type_timestamptz_serde.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -399,6 +399,29 @@ Status DataTypeTimeStampTzSerDe::write_column_to_arrow(const IColumn& column,
399399
return Status::OK();
400400
}
401401

402+
Status DataTypeTimeStampTzSerDe::write_column_to_paimon_arrow(
403+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
404+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
405+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
406+
const cctz::time_zone& ctz) const {
407+
if (field->type()->id() != arrow::Type::TIMESTAMP) {
408+
return Status::InvalidArgument(
409+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
410+
type->get_name(), field->ToString());
411+
}
412+
// The pinned Paimon schema must agree with the declared precision; do not silently rescale.
413+
const auto& timestamp = assert_cast<const arrow::TimestampType&>(*field->type());
414+
const arrow::TimeUnit::type expected_unit = type->get_scale() > 3 ? arrow::TimeUnit::MICRO
415+
: type->get_scale() > 0 ? arrow::TimeUnit::MILLI
416+
: arrow::TimeUnit::SECOND;
417+
if (timestamp.unit() != expected_unit) {
418+
return Status::InvalidArgument(
419+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
420+
type->get_name(), field->ToString());
421+
}
422+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
423+
}
424+
402425
/**
403426
* Reads an Arrow timestamp array into a TIMESTAMPTZ column.
404427
*

‎be/src/core/data_type_serde/data_type_timestamptz_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,11 @@ class DataTypeTimeStampTzSerDe : public DataTypeNumberSerDe<PrimitiveType::TYPE_
6767
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
6868
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
6969
const cctz::time_zone& ctz) const override;
70+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
71+
const IColumn& column, const NullMap* null_map,
72+
const std::shared_ptr<arrow::Field>& field,
73+
arrow::ArrayBuilder* array_builder, int64_t start,
74+
int64_t end, const cctz::time_zone& ctz) const override;
7075

7176
// Overridden rather than inherited: DataTypeNumberSerDe's reader would memcpy Arrow's epoch
7277
// integers over this column's packed values, same width and no error. See the definition.

‎be/src/core/data_type_serde/data_type_varbinary_serde.cpp‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
#include "common/cast_set.h"
2424
#include "common/config.h"
2525
#include "core/column/column_varbinary.h"
26+
#include "core/data_type/data_type_varbinary.h"
2627
#include "core/data_type_serde/arrow_validation.h"
2728
#include "core/data_type_serde/orc_serde_utils.h"
2829
#include "core/data_type_serde/parquet_decode_source.h"
@@ -194,6 +195,16 @@ Status DataTypeVarbinarySerDe::write_column_to_arrow(const IColumn& column, cons
194195
return Status::OK();
195196
}
196197

198+
Status DataTypeVarbinarySerDe::write_column_to_iceberg_arrow(
199+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
200+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
201+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
202+
const cctz::time_zone& ctz) const {
203+
// VARBINARY expression lengths are metadata, not fixed byte widths; casts can leave them unbounded.
204+
// The Arrow writer checks every non-null value against the remote fixed/UUID width instead.
205+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
206+
}
207+
197208
Status DataTypeVarbinarySerDe::read_column_from_arrow(IColumn& column,
198209
const arrow::Array* arrow_array,
199210
int64_t start, int64_t end,

‎be/src/core/data_type_serde/data_type_varbinary_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,11 @@ class DataTypeVarbinarySerDe : public DataTypeSerDe {
8787
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8888
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
8989
const cctz::time_zone& ctz) const override;
90+
Status write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>& type,
91+
const IColumn& column, const NullMap* null_map,
92+
const std::shared_ptr<arrow::Field>& field,
93+
arrow::ArrayBuilder* array_builder, int64_t start,
94+
int64_t end, const cctz::time_zone& ctz) const override;
9095

9196
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
9297
int64_t end, const cctz::time_zone& ctz) const override;

‎be/src/exec/sink/writer/iceberg/partition_transformers.cpp‎

Lines changed: 57 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,6 @@ const std::chrono::sys_days PartitionColumnTransformUtils::EPOCH = std::chrono::
4646
std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
4747
const doris::iceberg::PartitionField& field, const DataTypePtr& source_type) {
4848
auto& transform = field.transform();
49-
// Identity/void only carry values; computed binary partition transforms are unsupported.
50-
if (source_type->get_primitive_type() == TYPE_VARBINARY && transform != "identity" &&
51-
transform != "void") {
52-
throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
53-
"VARBINARY partition transform {} is not supported", transform);
54-
}
5549
static const std::regex has_width(R"((\w+)\[(\d+)\])");
5650
std::smatch width_match;
5751

@@ -75,6 +69,10 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
7569
return std::make_unique<StringTruncatePartitionColumnTransform>(source_type,
7670
parsed_width);
7771
}
72+
case TYPE_VARBINARY: {
73+
return std::make_unique<BinaryTruncatePartitionColumnTransform>(source_type,
74+
parsed_width);
75+
}
7876
case TYPE_DECIMALV2: {
7977
return std::make_unique<DecimalTruncatePartitionColumnTransform<TYPE_DECIMALV2>>(
8078
source_type, parsed_width);
@@ -117,14 +115,23 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
117115
return std::make_unique<StringBucketPartitionColumnTransform>(source_type,
118116
parsed_width);
119117
}
118+
case TYPE_UUID:
119+
return std::make_unique<ByteBucketPartitionColumnTransform<ColumnUUID>>(
120+
source_type, parsed_width);
121+
case TYPE_VARBINARY: {
122+
return std::make_unique<BinaryBucketPartitionColumnTransform>(source_type,
123+
parsed_width);
124+
}
120125
case TYPE_DATEV2: {
121126
return std::make_unique<DateBucketPartitionColumnTransform>(source_type,
122127
parsed_width);
123128
}
124-
case TYPE_DATETIMEV2: {
125-
return std::make_unique<TimestampBucketPartitionColumnTransform>(source_type,
126-
parsed_width);
127-
}
129+
case TYPE_TIMESTAMPTZ:
130+
return std::make_unique<TimestampBucketPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
131+
source_type, parsed_width);
132+
case TYPE_DATETIMEV2:
133+
return std::make_unique<TimestampBucketPartitionColumnTransform<TYPE_DATETIMEV2>>(
134+
source_type, parsed_width);
128135
case TYPE_DECIMALV2: {
129136
return std::make_unique<DecimalBucketPartitionColumnTransform<TYPE_DECIMALV2>>(
130137
source_type, parsed_width);
@@ -161,9 +168,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
161168
case TYPE_DATEV2: {
162169
return std::make_unique<DateYearPartitionColumnTransform>(source_type);
163170
}
164-
case TYPE_DATETIMEV2: {
165-
return std::make_unique<TimestampYearPartitionColumnTransform>(source_type);
166-
}
171+
case TYPE_TIMESTAMPTZ:
172+
return std::make_unique<TimestampYearPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
173+
source_type);
174+
case TYPE_DATETIMEV2:
175+
return std::make_unique<TimestampYearPartitionColumnTransform<TYPE_DATETIMEV2>>(
176+
source_type);
167177
default: {
168178
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
169179
"Unsupported type {} for partition column transform {}",
@@ -175,9 +185,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
175185
case TYPE_DATEV2: {
176186
return std::make_unique<DateMonthPartitionColumnTransform>(source_type);
177187
}
178-
case TYPE_DATETIMEV2: {
179-
return std::make_unique<TimestampMonthPartitionColumnTransform>(source_type);
180-
}
188+
case TYPE_TIMESTAMPTZ:
189+
return std::make_unique<TimestampMonthPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
190+
source_type);
191+
case TYPE_DATETIMEV2:
192+
return std::make_unique<TimestampMonthPartitionColumnTransform<TYPE_DATETIMEV2>>(
193+
source_type);
181194
default: {
182195
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
183196
"Unsupported type {} for partition column transform {}",
@@ -189,9 +202,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
189202
case TYPE_DATEV2: {
190203
return std::make_unique<DateDayPartitionColumnTransform>(source_type);
191204
}
192-
case TYPE_DATETIMEV2: {
193-
return std::make_unique<TimestampDayPartitionColumnTransform>(source_type);
194-
}
205+
case TYPE_TIMESTAMPTZ:
206+
return std::make_unique<TimestampDayPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
207+
source_type);
208+
case TYPE_DATETIMEV2:
209+
return std::make_unique<TimestampDayPartitionColumnTransform<TYPE_DATETIMEV2>>(
210+
source_type);
195211
default: {
196212
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
197213
"Unsupported type {} for partition column transform {}",
@@ -200,9 +216,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
200216
}
201217
} else if (transform == "hour") {
202218
switch (source_type->get_primitive_type()) {
203-
case TYPE_DATETIMEV2: {
204-
return std::make_unique<TimestampHourPartitionColumnTransform>(source_type);
205-
}
219+
case TYPE_TIMESTAMPTZ:
220+
return std::make_unique<TimestampHourPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
221+
source_type);
222+
case TYPE_DATETIMEV2:
223+
return std::make_unique<TimestampHourPartitionColumnTransform<TYPE_DATETIMEV2>>(
224+
source_type);
206225
default: {
207226
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
208227
"Unsupported type {} for partition column transform {}",
@@ -252,11 +271,22 @@ std::string PartitionColumnTransform::get_partition_value(const DataTypePtr type
252271
case TYPE_DOUBLE: {
253272
return floating_point_partition_value_to_string(std::any_cast<Float64>(value));
254273
}
274+
case TYPE_UUID:
255275
case TYPE_VARCHAR:
256276
case TYPE_CHAR:
257277
case TYPE_STRING: {
258278
return std::any_cast<std::string>(value);
259279
}
280+
case TYPE_VARBINARY: {
281+
// Commit values are typed hexadecimal, never a potentially invalid UTF-8 string.
282+
const auto& bytes = std::any_cast<const std::string&>(value);
283+
std::string encoded = "0x";
284+
for (unsigned char byte : bytes) {
285+
encoded += "0123456789abcdef"[byte >> 4];
286+
encoded += "0123456789abcdef"[byte & 15];
287+
}
288+
return encoded;
289+
}
260290
case TYPE_DATE: {
261291
char buf[64];
262292
char* pos = std::any_cast<VecDateTimeValue>(value).to_string(buf);
@@ -272,6 +302,11 @@ std::string PartitionColumnTransform::get_partition_value(const DataTypePtr type
272302
char* pos = std::any_cast<DateV2Value<DateV2ValueType>>(value).to_string(buf);
273303
return std::string(buf, pos - buf - 1);
274304
}
305+
case TYPE_TIMESTAMPTZ: {
306+
// Commit an explicit UTC offset so FE session settings cannot reinterpret the instant.
307+
return std::any_cast<TimestampTzValue>(value).to_string(cctz::utc_time_zone(),
308+
type->get_scale());
309+
}
275310
case TYPE_DATETIMEV2: {
276311
char buf[64];
277312
char* pos = std::any_cast<DateV2Value<DateTimeV2ValueType>>(value).to_string(

0 commit comments

Comments
 (0)