Skip to content

Commit f7332ed

Browse files
authored
[fix](external) Preserve external binary and timestamp mappings (apache#68532)
### What problem does this PR solve? Related PRs: apache#67784, apache#68441 This combines splits 4 and 5 of apache#67784 on branch-4.1. External binary columns retain their bytes as VARBINARY, and external instant timestamps retain their time semantics as TIMESTAMPTZ instead of being exposed as unzoned DATETIME. Iceberg and Paimon write bindings and partition metadata preserve microseconds, negative epochs, and both instants in a DST overlap. - Unify Hive, Iceberg, Paimon and JDBC binary mappings, including fixed-width and UUID values. Preserve the existing text contract for Hive binary partition columns. - Map instant types consistently across Hive, Iceberg, Paimon, JDBC, Hudi, Trino connector and MaxCompute; local timestamp types remain DATETIMEV2. Align Java/JNI carriers, nested values, JDBC projections and predicates, and MaxCompute microsecond writes with these declared types. Bind JDBC instant writes with dialect-aware zoned parameters, including a supported NULL bind for Trino/Presto. Project Trino and Presto scalar and nested-array instant timestamps to UTC before JDBC decoding. PostgreSQL instants outside the supported range map to NULL, including source NOT NULL columns; predicates involving these timestamps remain local so filtering observes the decoded values. - Normalize the legacy catalog mapping properties on CREATE/ALTER. After master metadata replay, journal ordinary ALTER records for old missing/false markers before query readiness; replay and migration are idempotent. This changes persisted catalog metadata: rolling back FE binaries alone does not restore the old properties. External files are not rewritten. During a mixed-version FE upgrade, old and new FEs can expose different external schemas before a new master journals this migration; compatibility within that rolling-upgrade window is outside this change. Old-FE/new-BE compatibility for Iceberg FIXED writes, Hudi/MySQL timestamp reads, and MaxCompute timestamp writes is also intentionally outside this PR; those review findings are acknowledged and not fixed. - Retain Iceberg/Paimon write validation, binary-safe partition transport, and dynamic/static/hybrid overwrite behavior. Support Iceberg binary bucket hashing and byte-counted truncate partitions, preserving NULLs and raw binary prefixes. Route legacy Paimon ORC LTZ files through the SDK when native decoding cannot preserve the logical instant. Reject static Paimon LTZ overwrite keys that are ambiguous in the FE JVM time zone. - Permit VARBINARY in ordinary views while retaining native OLAP, CTAS and MTMV restrictions and binary computation rejections. File TVF binary mapping remains an explicit opt-in with default false; file timestamp mapping follows logical annotations. - Update affected regression schemas/results and use explicit supported text representations where fixtures require binary computation. Preserve CDC's JSON carriers: binary values use Base64 text and timestamps retain the existing unzoned representation, including array elements. - Reject VARBINARY arguments to IN/NOT IN and least/greatest/nullif during analysis while preserving CASE/IF/COALESCE value selection. Existing binary aggregate support and legality gaps remain outside this change. Validation: - Binary partition review follow-up: reproduced both unsupported-transform failures, then passed 34 related ASAN BE tests and clang-format 16. Added Parquet/ORC append/overwrite regressions with Spark byte and partition-pruning checks; Groovy parsing passed, external execution pending CI. Bucket expectations were checked against Iceberg Java transforms. - JDBC write review follow-up: 25 local unit tests and FE/JDBC Checkstyle passed after six expected failures were reproduced. Added per-dialect JNI binding checks and five opt-in server write round trips; 77 real-driver encoding checks passed. Thirteen server integration tests were skipped without configured connections. Corrected a PostgreSQL regression COUNT expectation to JDBC Long; the numeric result was already correct. - JDBC review fixes: reproduced four failing FE test methods, then passed 48 FE tests, 17 JDBC unit tests, and FE/JDBC Checkstyle. Added PostgreSQL SQL regressions for decoded NULL filtering, LIMIT, null-safe equality, NOT NULL metadata and COUNT; added Presto scalar/nested/TVF projection tests and optional driver integration coverage. A real PrestoSQL JDBC 350 parser check passed 18 cases across three JVM zones. Seven server integration tests were skipped because connections were not configured; external SQL execution remains pending CI. - Earlier follow-up: corrected three Iceberg FE UT expectations for mandatory logical mappings while retaining generation/authenticator checks; 152 related FE tests, 17 JDBC unit tests and FE/JDBC Checkstyle passed. - A real Trino JDBC 435 parser check passed 18 cases across three JVM zones; live-server Trino integration coverage was added but not executed locally. - Review follow-up: 59 targeted FE tests and FE Checkstyle passed after reproducing four failing test methods before the fix. Coverage includes CDC binary schemas and binary comparison rejection under both coercion modes. - 507 related FE unit tests passed across the main run and targeted rerun after correcting the view assertion; FE and connector Checkstyle passed. - 34 Java connector unit tests passed, covering timezone mismatches, nested timestamps, nulls, negative epochs, DST overlaps, and microsecond writes. - ASAN BE build and 265 related BE tests passed, including UUID partition-byte transport. - clang-format 16 passed for all 15 affected C/C++ files; AST parsing passed for all 43 newly affected Groovy suites. Existing split-4 SQL tests retain Spark epoch/byte comparisons. - External SQL regression and real-server JDBC integration execution are pending CI. Golden results were reviewed, not regenerated against external services locally. ### Release note Preserve binary and instant timestamp types across external catalogs and connectors, migrate legacy catalog mapping properties, and align Iceberg/Paimon writes and partition metadata. Existing external result schemas can change from STRING/DATETIME to VARBINARY/TIMESTAMPTZ. ### Check List (For Author) - [x] Regression tests added and affected expectations updated (external integration execution pending CI). - [x] Unit tests. - [x] Behavior changed: external type mapping and persisted catalog properties, connector timestamp handling, and external read/write bindings. - [x] Compatibility and rollback impact described above. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note. - [ ] Confirm test cases. - [ ] Confirm document. - [ ] Add branch pick label if needed.
1 parent 54c1d1a commit f7332ed

203 files changed

Lines changed: 7312 additions & 2429 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/data_type_datetimev2_serde.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -627,6 +627,29 @@ Status DataTypeDateTimeV2SerDe::write_column_to_arrow(const IColumn& column,
627627
return Status::OK();
628628
}
629629

630+
Status DataTypeDateTimeV2SerDe::write_column_to_paimon_arrow(
631+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
632+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
633+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
634+
const cctz::time_zone& ctz) const {
635+
if (field->type()->id() != arrow::Type::TIMESTAMP) {
636+
return Status::InvalidArgument(
637+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
638+
type->get_name(), field->ToString());
639+
}
640+
// The pinned Paimon schema must agree with the declared precision; do not silently rescale.
641+
const auto& timestamp = assert_cast<const arrow::TimestampType&>(*field->type());
642+
const arrow::TimeUnit::type expected_unit = type->get_scale() > 3 ? arrow::TimeUnit::MICRO
643+
: type->get_scale() > 0 ? arrow::TimeUnit::MILLI
644+
: arrow::TimeUnit::SECOND;
645+
if (timestamp.unit() != expected_unit) {
646+
return Status::InvalidArgument(
647+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
648+
type->get_name(), field->ToString());
649+
}
650+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
651+
}
652+
630653
Status DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
631654
const arrow::Array* arrow_array,
632655
int64_t start, int64_t end,

‎be/src/core/data_type_serde/data_type_datetimev2_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,11 @@ class DataTypeDateTimeV2SerDe : public DataTypeNumberSerDe<PrimitiveType::TYPE_D
8686
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8787
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
8888
const cctz::time_zone& ctz) const override;
89+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
90+
const IColumn& column, const NullMap* null_map,
91+
const std::shared_ptr<arrow::Field>& field,
92+
arrow::ArrayBuilder* array_builder, int64_t start,
93+
int64_t end, const cctz::time_zone& ctz) const override;
8994
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
9095
int64_t end, const cctz::time_zone& ctz) const override;
9196
Status read_column_from_decoded_values(IColumn& column,

‎be/src/core/data_type_serde/data_type_string_serde.cpp‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -568,7 +568,12 @@ Status DataTypeStringSerDeBase<ColumnType>::write_column_to_iceberg_arrow(
568568
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
569569
const cctz::time_zone& ctz) const {
570570
if (!is_iceberg_uuid_field(field)) {
571-
// Keep the existing CHAR/STRING fixed-binary binding until external type mappings change.
571+
// Iceberg fixed(N) is a byte binding, so text padding must not change its physical value.
572+
if (array_builder->type()->id() == arrow::Type::FIXED_SIZE_BINARY) {
573+
return Status::InvalidArgument(
574+
"Iceberg fixed writer requires Doris VARBINARY, got {} for Arrow field {}",
575+
type->get_name(), field->ToString());
576+
}
572577
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
573578
}
574579
if (!is_string_type(type->get_primitive_type()) ||

‎be/src/core/data_type_serde/data_type_timestamptz_serde.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -399,6 +399,29 @@ Status DataTypeTimeStampTzSerDe::write_column_to_arrow(const IColumn& column,
399399
return Status::OK();
400400
}
401401

402+
Status DataTypeTimeStampTzSerDe::write_column_to_paimon_arrow(
403+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
404+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
405+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
406+
const cctz::time_zone& ctz) const {
407+
if (field->type()->id() != arrow::Type::TIMESTAMP) {
408+
return Status::InvalidArgument(
409+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
410+
type->get_name(), field->ToString());
411+
}
412+
// The pinned Paimon schema must agree with the declared precision; do not silently rescale.
413+
const auto& timestamp = assert_cast<const arrow::TimestampType&>(*field->type());
414+
const arrow::TimeUnit::type expected_unit = type->get_scale() > 3 ? arrow::TimeUnit::MICRO
415+
: type->get_scale() > 0 ? arrow::TimeUnit::MILLI
416+
: arrow::TimeUnit::SECOND;
417+
if (timestamp.unit() != expected_unit) {
418+
return Status::InvalidArgument(
419+
"Paimon timestamp writer has no binding for Doris type {} and Arrow field {}",
420+
type->get_name(), field->ToString());
421+
}
422+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
423+
}
424+
402425
/**
403426
* Reads an Arrow timestamp array into a TIMESTAMPTZ column.
404427
*

‎be/src/core/data_type_serde/data_type_timestamptz_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,11 @@ class DataTypeTimeStampTzSerDe : public DataTypeNumberSerDe<PrimitiveType::TYPE_
6767
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
6868
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
6969
const cctz::time_zone& ctz) const override;
70+
Status write_column_to_paimon_arrow(const std::shared_ptr<const IDataType>& type,
71+
const IColumn& column, const NullMap* null_map,
72+
const std::shared_ptr<arrow::Field>& field,
73+
arrow::ArrayBuilder* array_builder, int64_t start,
74+
int64_t end, const cctz::time_zone& ctz) const override;
7075

7176
// Overridden rather than inherited: DataTypeNumberSerDe's reader would memcpy Arrow's epoch
7277
// integers over this column's packed values, same width and no error. See the definition.

‎be/src/core/data_type_serde/data_type_varbinary_serde.cpp‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222

2323
#include "common/config.h"
2424
#include "core/column/column_varbinary.h"
25+
#include "core/data_type/data_type_varbinary.h"
2526
#include "core/data_type_serde/arrow_validation.h"
2627
#include "core/data_type_serde/parquet_decode_source.h"
2728
#include "exprs/function/string_hex_util.h"
@@ -192,6 +193,25 @@ Status DataTypeVarbinarySerDe::write_column_to_arrow(const IColumn& column, cons
192193
return Status::OK();
193194
}
194195

196+
Status DataTypeVarbinarySerDe::write_column_to_iceberg_arrow(
197+
const std::shared_ptr<const IDataType>& type, const IColumn& column,
198+
const NullMap* null_map, const std::shared_ptr<arrow::Field>& field,
199+
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
200+
const cctz::time_zone& ctz) const {
201+
if (array_builder->type()->id() != arrow::Type::FIXED_SIZE_BINARY) {
202+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
203+
}
204+
// Fixed-width bindings describe bytes, including the declared width of nested fields.
205+
const auto& fixed = assert_cast<const arrow::FixedSizeBinaryType&>(*field->type());
206+
const auto& varbinary = assert_cast<const DataTypeVarbinary&>(*type);
207+
if (varbinary.len() != fixed.byte_width()) {
208+
return Status::InvalidArgument(
209+
"Iceberg fixed width does not match Doris VARBINARY length: expected {}, got {}",
210+
fixed.byte_width(), varbinary.len());
211+
}
212+
return write_column_to_arrow(column, null_map, array_builder, start, end, ctz);
213+
}
214+
195215
Status DataTypeVarbinarySerDe::read_column_from_arrow(IColumn& column,
196216
const arrow::Array* arrow_array,
197217
int64_t start, int64_t end,

‎be/src/core/data_type_serde/data_type_varbinary_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,11 @@ class DataTypeVarbinarySerDe : public DataTypeSerDe {
8888
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8989
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
9090
const cctz::time_zone& ctz) const override;
91+
Status write_column_to_iceberg_arrow(const std::shared_ptr<const IDataType>& type,
92+
const IColumn& column, const NullMap* null_map,
93+
const std::shared_ptr<arrow::Field>& field,
94+
arrow::ArrayBuilder* array_builder, int64_t start,
95+
int64_t end, const cctz::time_zone& ctz) const override;
9196

9297
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
9398
int64_t end, const cctz::time_zone& ctz) const override;

‎be/src/exec/sink/writer/iceberg/partition_transformers.cpp‎

Lines changed: 53 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,6 @@ const std::chrono::sys_days PartitionColumnTransformUtils::EPOCH = std::chrono::
4646
std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
4747
const doris::iceberg::PartitionField& field, const DataTypePtr& source_type) {
4848
auto& transform = field.transform();
49-
// Identity/void only carry values; computed binary partition transforms are unsupported.
50-
if (source_type->get_primitive_type() == TYPE_VARBINARY && transform != "identity" &&
51-
transform != "void") {
52-
throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
53-
"VARBINARY partition transform {} is not supported", transform);
54-
}
5549
static const std::regex has_width(R"((\w+)\[(\d+)\])");
5650
std::smatch width_match;
5751

@@ -75,6 +69,10 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
7569
return std::make_unique<StringTruncatePartitionColumnTransform>(source_type,
7670
parsed_width);
7771
}
72+
case TYPE_VARBINARY: {
73+
return std::make_unique<BinaryTruncatePartitionColumnTransform>(source_type,
74+
parsed_width);
75+
}
7876
case TYPE_DECIMALV2: {
7977
return std::make_unique<DecimalTruncatePartitionColumnTransform<TYPE_DECIMALV2>>(
8078
source_type, parsed_width);
@@ -117,14 +115,20 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
117115
return std::make_unique<StringBucketPartitionColumnTransform>(source_type,
118116
parsed_width);
119117
}
118+
case TYPE_VARBINARY: {
119+
return std::make_unique<BinaryBucketPartitionColumnTransform>(source_type,
120+
parsed_width);
121+
}
120122
case TYPE_DATEV2: {
121123
return std::make_unique<DateBucketPartitionColumnTransform>(source_type,
122124
parsed_width);
123125
}
124-
case TYPE_DATETIMEV2: {
125-
return std::make_unique<TimestampBucketPartitionColumnTransform>(source_type,
126-
parsed_width);
127-
}
126+
case TYPE_TIMESTAMPTZ:
127+
return std::make_unique<TimestampBucketPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
128+
source_type, parsed_width);
129+
case TYPE_DATETIMEV2:
130+
return std::make_unique<TimestampBucketPartitionColumnTransform<TYPE_DATETIMEV2>>(
131+
source_type, parsed_width);
128132
case TYPE_DECIMALV2: {
129133
return std::make_unique<DecimalBucketPartitionColumnTransform<TYPE_DECIMALV2>>(
130134
source_type, parsed_width);
@@ -161,9 +165,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
161165
case TYPE_DATEV2: {
162166
return std::make_unique<DateYearPartitionColumnTransform>(source_type);
163167
}
164-
case TYPE_DATETIMEV2: {
165-
return std::make_unique<TimestampYearPartitionColumnTransform>(source_type);
166-
}
168+
case TYPE_TIMESTAMPTZ:
169+
return std::make_unique<TimestampYearPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
170+
source_type);
171+
case TYPE_DATETIMEV2:
172+
return std::make_unique<TimestampYearPartitionColumnTransform<TYPE_DATETIMEV2>>(
173+
source_type);
167174
default: {
168175
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
169176
"Unsupported type {} for partition column transform {}",
@@ -175,9 +182,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
175182
case TYPE_DATEV2: {
176183
return std::make_unique<DateMonthPartitionColumnTransform>(source_type);
177184
}
178-
case TYPE_DATETIMEV2: {
179-
return std::make_unique<TimestampMonthPartitionColumnTransform>(source_type);
180-
}
185+
case TYPE_TIMESTAMPTZ:
186+
return std::make_unique<TimestampMonthPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
187+
source_type);
188+
case TYPE_DATETIMEV2:
189+
return std::make_unique<TimestampMonthPartitionColumnTransform<TYPE_DATETIMEV2>>(
190+
source_type);
181191
default: {
182192
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
183193
"Unsupported type {} for partition column transform {}",
@@ -189,9 +199,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
189199
case TYPE_DATEV2: {
190200
return std::make_unique<DateDayPartitionColumnTransform>(source_type);
191201
}
192-
case TYPE_DATETIMEV2: {
193-
return std::make_unique<TimestampDayPartitionColumnTransform>(source_type);
194-
}
202+
case TYPE_TIMESTAMPTZ:
203+
return std::make_unique<TimestampDayPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
204+
source_type);
205+
case TYPE_DATETIMEV2:
206+
return std::make_unique<TimestampDayPartitionColumnTransform<TYPE_DATETIMEV2>>(
207+
source_type);
195208
default: {
196209
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
197210
"Unsupported type {} for partition column transform {}",
@@ -200,9 +213,12 @@ std::unique_ptr<PartitionColumnTransform> PartitionColumnTransforms::create(
200213
}
201214
} else if (transform == "hour") {
202215
switch (source_type->get_primitive_type()) {
203-
case TYPE_DATETIMEV2: {
204-
return std::make_unique<TimestampHourPartitionColumnTransform>(source_type);
205-
}
216+
case TYPE_TIMESTAMPTZ:
217+
return std::make_unique<TimestampHourPartitionColumnTransform<TYPE_TIMESTAMPTZ>>(
218+
source_type);
219+
case TYPE_DATETIMEV2:
220+
return std::make_unique<TimestampHourPartitionColumnTransform<TYPE_DATETIMEV2>>(
221+
source_type);
206222
default: {
207223
throw doris::Exception(doris::ErrorCode::INTERNAL_ERROR,
208224
"Unsupported type {} for partition column transform {}",
@@ -257,6 +273,16 @@ std::string PartitionColumnTransform::get_partition_value(const DataTypePtr type
257273
case TYPE_STRING: {
258274
return std::any_cast<std::string>(value);
259275
}
276+
case TYPE_VARBINARY: {
277+
// Commit values are typed hexadecimal, never a potentially invalid UTF-8 string.
278+
const auto& bytes = std::any_cast<const std::string&>(value);
279+
std::string encoded = "0x";
280+
for (unsigned char byte : bytes) {
281+
encoded += "0123456789abcdef"[byte >> 4];
282+
encoded += "0123456789abcdef"[byte & 15];
283+
}
284+
return encoded;
285+
}
260286
case TYPE_DATE: {
261287
char buf[64];
262288
char* pos = std::any_cast<VecDateTimeValue>(value).to_string(buf);
@@ -272,6 +298,11 @@ std::string PartitionColumnTransform::get_partition_value(const DataTypePtr type
272298
char* pos = std::any_cast<DateV2Value<DateV2ValueType>>(value).to_string(buf);
273299
return std::string(buf, pos - buf - 1);
274300
}
301+
case TYPE_TIMESTAMPTZ: {
302+
// Commit an explicit UTC offset so FE session settings cannot reinterpret the instant.
303+
return std::any_cast<TimestampTzValue>(value).to_string(cctz::utc_time_zone(),
304+
type->get_scale());
305+
}
275306
case TYPE_DATETIMEV2: {
276307
char buf[64];
277308
char* pos = std::any_cast<DateV2Value<DateTimeV2ValueType>>(value).to_string(

0 commit comments

Comments
 (0)