Skip to content

Commit 575c5ac

Browse files
authored
[feat](datalake) Support nanosecond timestamps in Iceberg and Paimon (apache#68934)
### What problem does this PR solve? Iceberg `timestamp_ns` was unsupported, and Paimon `timestamp(9)` was exposed as `DATETIMEV2(6)`, losing the final three fractional digits. Map these unzoned types bidirectionally to Doris `TIMESTAMP_NS` and preserve all nine digits through reads, inserts and CTAS, including arrays, maps and structs. - Represent Iceberg `timestamp`, `timestamptz`, `timestamp_ns`, and `timestamptz_ns` with four distinct BE classes and type IDs. Arrow unit/timezone conversion and nested Parquet nanosecond detection dispatch directly on these IDs. - Add nanosecond materialization to format V2 Parquet/ORC readers and Iceberg writers. Format V1 readers remain unchanged. Nanosecond Iceberg schemas use Parquet 2.6; ORC uses a UTC wall-clock carrier to avoid historical timezone-offset differences at the signed nanosecond boundaries, preserving negative-fraction correction across external stripe writer timezones. - Support Iceberg identity, calendar and bucket partitions, including date-only static partitions and exact nanosecond overwrite/conflict filters. Support nanosecond predicate pushdown and filtered rewrites, ADD COLUMN defaults, and statement-scoped write defaults. Omit unsafe ORC timestamp bounds and derived MTMV RANGE boundaries. - Select Paimon's JNI reader for projected or filter-only high-precision fields, including nested fields, and use nanosecond Arrow transport for writes. Preserve physical metadata columns when the SDK can expose real file/row positions. Paimon precisions 7 and 8 also map to `TIMESTAMP_NS`. Iceberg nanosecond schemas require format version 3 and the format V2 reader (`enable_file_scanner_v2=true`). High-precision Paimon reads use JNI, which may cost more than native scans. The existing ORC-645 rejection of ambiguous pre-epoch fractions is retained; Parquet supports those values. ### Release note Support reading and writing Iceberg `timestamp_ns` and Paimon `timestamp(9)` as Doris `TIMESTAMP_NS`, preserving nanosecond precision. ### Validation - External regression follow-up: `paimon_timestamp_types`, `test_paimon_minio`, and `test_paimon_table_properties` all passed locally after regenerating their goldens from actual execution. Coverage retains scanner-V2-disabled JNI reads, both S3/MinIO property aliases, and SHOW CREATE TABLE for precision-7/8/9 columns. Current review fixes: - 270 focused FE tests passed, including real Iceberg schema/transaction commits, rewrite partition pruning, and Paimon ORC/Parquet scan planning. Connector packages and FE Checkstyle passed. - 263 BE tests passed under ASAN. External ORC writer fixtures cover GMT, Asia/Shanghai and America/New_York, epoch crossings, fractional precision, rounding and sparse reads. - `test_iceberg_timestamp_ns` and `test_paimon_timestamp_ns` passed locally with rebuilt FE connector plugins. New assertions cover filtered rewrites, explicit/omitted-column inserts with defaults, date-only static overwrite, and nanosecond columns with physical metadata. The Iceberg suite uses the existing newer REST fixture because the baseline 1.10 server incorrectly rescales nanosecond schema defaults. ORC decoding changes were validated separately by the ASAN tests above. - All nine changed C++ files passed clang-format 16, clang-tidy and build hygiene checks. Earlier feature validation: - Full FE and ASAN BE builds; FE Checkstyle, clang-format 16 and clang-tidy over all 25 changed C++ files. - 2,112 relevant Java tests: zero failures, five existing skips. - 117 relevant BE tests under ASAN, including independent Arrow/ORC file read-back, signed bounds, dictionary decoding and overflow handling. - Three end-to-end regression suites passed against the generated goldens: `test_iceberg_timestamp_ns`, `test_paimon_timestamp_ns`, and `paimon_timestamp_types`. Coverage includes Iceberg format V2 reads, Paimon JNI reads with both scanner settings, session time zones, nested values, filters, signed bounds, CTAS, Iceberg partition transforms/overwrite, and Paimon primary-key upsert. - Independent Flink read-back of four Doris-written Paimon tables: 28 rows across Parquet/ORC and append/primary-key tables, with exact nanosecond bounds and upserts. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test (independent Flink read-back described above) - Behavior changed: - [x] Yes. External unzoned nanosecond timestamps retain nine fractional digits. - Does this need documentation? - [x] Yes. The external type-mapping tables should describe `TIMESTAMP_NS`. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent b99a2a0 commit 575c5ac

91 files changed

Lines changed: 4386 additions & 238 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/data_type_datetimev2_serde.cpp‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
#include "core/data_type_serde/arrow_validation.h"
3333
#include "core/data_type_serde/decoded_column_view.h"
3434
#include "core/data_type_serde/orc_serde_utils.h"
35+
#include "core/data_type_serde/orc_timestamp_decode_context.h"
3536
#include "core/data_type_serde/parquet_decode_source.h"
3637
#include "core/data_type_serde/parquet_timestamp.h"
3738
#include "core/types.h"
@@ -52,8 +53,7 @@ static const int64_t micro_to_nano_second = 1000;
5253

5354
namespace {
5455

55-
Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColumnView& orc_view,
56-
const cctz::time_zone& timezone) {
56+
Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColumnView& orc_view) {
5757
const auto* orc_batch = dynamic_cast<const ::orc::TimestampVectorBatch*>(orc_view.batch);
5858
if (orc_batch == nullptr) {
5959
return Status::InternalError("Unexpected ORC timestamp batch type {}",
@@ -72,9 +72,10 @@ Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColum
7272
}
7373
auto& value =
7474
reinterpret_cast<DateV2Value<DateTimeV2ValueType>&>(data[old_data_size + row]);
75+
const int64_t seconds = orc_batch->data[source_row];
7576
orc_serde_utils::RoundedOrcTimestamp timestamp;
7677
auto status = orc_serde_utils::round_orc_timestamp_to_microseconds(
77-
orc_batch->data[source_row], orc_batch->nanoseconds[source_row], &timestamp);
78+
seconds, orc_batch->nanoseconds[source_row], &timestamp);
7879
if (!status.ok()) {
7980
data.resize(old_data_size);
8081
return status;
@@ -84,8 +85,9 @@ Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColum
8485
// Instant carry precedes timezone conversion; plain TIMESTAMP carry stays in civil time
8586
// so rounding cannot jump across a daylight-saving gap or fold.
8687
status = orc_serde_utils::orc_timestamp_to_datetime(
87-
is_timestamp_instant ? timestamp.seconds : orc_batch->data[source_row],
88-
timestamp.microseconds, timezone, !is_timestamp_instant && timestamp.carry, &value);
88+
is_timestamp_instant ? timestamp.seconds : seconds, timestamp.microseconds,
89+
orc_view.timestamp_context->timezone_for(*orc_view.file_type, seconds),
90+
!is_timestamp_instant && timestamp.carry, &value);
8991
if (!status.ok()) {
9092
data.resize(old_data_size);
9193
return status;
@@ -1045,11 +1047,11 @@ Status DataTypeDateTimeV2SerDe::read_column_from_orc(IColumn& column,
10451047
DORIS_CHECK(view.batch != nullptr);
10461048
const auto kind = view.file_type->getKind();
10471049
DORIS_CHECK(kind == ::orc::TypeKind::TIMESTAMP || kind == ::orc::TypeKind::TIMESTAMP_INSTANT);
1048-
DORIS_CHECK(view.timezone != nullptr);
1050+
DORIS_CHECK(view.timestamp_context != nullptr);
10491051
if (orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows) == 0) {
10501052
return Status::OK();
10511053
}
1052-
return decode_timestamp_orc_values(column, view, *view.timezone);
1054+
return decode_timestamp_orc_values(column, view);
10531055
}
10541056

10551057
} // namespace doris

‎be/src/core/data_type_serde/data_type_serde.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,13 +118,15 @@ struct FieldInfo {
118118
int precision = 0;
119119
};
120120

121+
struct OrcTimestampDecodeContext;
122+
121123
struct OrcDecodedColumnView {
122124
const orc::Type* file_type = nullptr;
123125
const orc::Type* selected_type = nullptr;
124126
const orc::ColumnVectorBatch* batch = nullptr;
125127
size_t rows = 0;
126128
const std::vector<size_t>* selected_rows = nullptr;
127-
const cctz::time_zone* timezone = nullptr;
129+
const OrcTimestampDecodeContext* timestamp_context = nullptr;
128130
bool enable_mapping_timestamp_tz = false;
129131
};
130132

‎be/src/core/data_type_serde/data_type_timestamp_ns_serde.cpp‎

Lines changed: 184 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@
2828
#include "core/column/column_nullable.h"
2929
#include "core/column/column_string.h"
3030
#include "core/data_type_serde/arrow_validation.h"
31+
#include "core/data_type_serde/decoded_column_view.h"
32+
#include "core/data_type_serde/orc_serde_utils.h"
33+
#include "core/data_type_serde/orc_timestamp_decode_context.h"
34+
#include "core/data_type_serde/parquet_timestamp.h"
3135
#include "core/value/vdatetime_value.h"
3236
#include "exprs/function/cast/cast_to_timestamp_ns.h"
3337
#include "util/mysql_row_buffer.h"
@@ -37,6 +41,56 @@ namespace doris {
3741

3842
namespace {
3943

44+
class TimestampNsParquetConsumer final : public ParquetFixedValueConsumer {
45+
public:
46+
TimestampNsParquetConsumer(IColumn& column, const ParquetDecodeContext& context,
47+
ParquetMaterializationState* state = nullptr)
48+
: _data(assert_cast<ColumnTimeStampNs&>(column).get_data()),
49+
_context(context),
50+
_state(state) {}
51+
52+
Status consume(const uint8_t* values, size_t count, size_t width) override {
53+
const auto old_size = _data.size();
54+
for (size_t i = 0; i < count; ++i) {
55+
int64_t nanos;
56+
Status status;
57+
if (_context.physical_type == ParquetPhysicalType::INT96) {
58+
DORIS_CHECK_EQ(width, sizeof(ParquetInt96Timestamp));
59+
status = parquet_int96_timestamp_nanos(
60+
unaligned_load<ParquetInt96Timestamp>(values + i * width), &nanos);
61+
} else if (_context.physical_type == ParquetPhysicalType::INT64) {
62+
DORIS_CHECK_EQ(width, sizeof(int64_t));
63+
status = parquet_timestamp_nanos(
64+
_context.time_unit, unaligned_load<int64_t>(values + i * width), &nanos);
65+
} else {
66+
return Status::NotSupported("TIMESTAMP_NS expects Parquet INT64 or INT96");
67+
}
68+
if (!status.ok()) {
69+
if (_state != nullptr && _state->mark_conversion_failure(_data.size())) {
70+
_data.emplace_back();
71+
continue;
72+
}
73+
_data.resize(old_size);
74+
return status;
75+
}
76+
_data.emplace_back(nanos);
77+
}
78+
return Status::OK();
79+
}
80+
81+
private:
82+
ColumnTimeStampNs::Container& _data;
83+
const ParquetDecodeContext& _context;
84+
ParquetMaterializationState* _state;
85+
};
86+
87+
class RejectTimestampNsBinaryConsumer final : public ParquetBinaryValueConsumer {
88+
public:
89+
Status consume(const StringRef*, size_t) override {
90+
return Status::NotSupported("Binary Parquet values cannot be materialized as TIMESTAMP_NS");
91+
}
92+
};
93+
4094
Status get_nanos_per_arrow_timestamp_unit(arrow::TimeUnit::type unit, int64_t* nanos_per_unit) {
4195
switch (unit) {
4296
case arrow::TimeUnit::SECOND:
@@ -345,6 +399,113 @@ Status DataTypeTimeStampNsSerDe::deserialize_column_from_jsonb_vector(
345399
source_null_map);
346400
}
347401

402+
Status DataTypeTimeStampNsSerDe::read_parquet_dictionary(
403+
IColumn& column, ParquetDecodeSource& source, const ParquetDecodeContext& context) const {
404+
TimestampNsParquetConsumer consumer(column, context);
405+
RejectTimestampNsBinaryConsumer binary;
406+
return source.decode_dictionary(consumer, binary);
407+
}
408+
409+
Status DataTypeTimeStampNsSerDe::read_column_from_parquet(
410+
IColumn& column, ParquetDecodeSource& source, const ParquetDecodeContext& context,
411+
size_t num_values, ParquetMaterializationState& state) const {
412+
TimestampNsParquetConsumer consumer(column, context, &state);
413+
if (context.encoding != ParquetValueEncoding::DICTIONARY) {
414+
return source.decode_fixed_values(num_values, consumer);
415+
}
416+
if (state.dictionary_generation != source.dictionary_generation()) {
417+
state.typed_dictionary = column.clone_empty();
418+
auto* null_map = state.begin_dictionary_conversion(source.dictionary_size());
419+
TimestampNsParquetConsumer dictionary(*state.typed_dictionary, context, &state);
420+
RejectTimestampNsBinaryConsumer binary;
421+
auto status = source.decode_dictionary(dictionary, binary);
422+
state.end_dictionary_conversion(null_map);
423+
RETURN_IF_ERROR(status);
424+
DORIS_CHECK_EQ(state.typed_dictionary->size(), source.dictionary_size());
425+
state.dictionary_generation = source.dictionary_generation();
426+
}
427+
return state.materialize_dictionary(column, source, num_values);
428+
}
429+
430+
Status DataTypeTimeStampNsSerDe::read_column_from_decoded_values(
431+
IColumn& column, const DecodedColumnView& view) const {
432+
if (view.value_kind != DecodedValueKind::INT64 && view.value_kind != DecodedValueKind::INT96) {
433+
return Status::NotSupported("TIMESTAMP_NS expects decoded INT64 or INT96 values");
434+
}
435+
if (view.values == nullptr && decoded_column_view_has_non_null_value(view)) {
436+
return Status::Corruption("Decoded TIMESTAMP_NS value buffer is null");
437+
}
438+
auto unit = ParquetTimeUnit::MICROS;
439+
if (view.time_unit == DecodedTimeUnit::MILLIS) {
440+
unit = ParquetTimeUnit::MILLIS;
441+
} else if (view.time_unit == DecodedTimeUnit::NANOS) {
442+
unit = ParquetTimeUnit::NANOS;
443+
}
444+
auto& data = assert_cast<ColumnTimeStampNs&>(column).get_data();
445+
const auto old_size = data.size();
446+
for (int64_t row = 0; row < view.row_count; ++row) {
447+
if (decoded_column_view_row_is_null(view, row)) {
448+
data.emplace_back();
449+
continue;
450+
}
451+
int64_t nanos;
452+
auto status =
453+
view.value_kind == DecodedValueKind::INT96
454+
? parquet_int96_timestamp_nanos(
455+
unaligned_load<ParquetInt96Timestamp>(
456+
view.values + row * sizeof(ParquetInt96Timestamp)),
457+
&nanos)
458+
: parquet_timestamp_nanos(
459+
unit,
460+
unaligned_load<int64_t>(view.values + row * sizeof(int64_t)),
461+
&nanos);
462+
if (!status.ok()) {
463+
if (decoded_column_view_can_null_on_conversion_failure(view)) {
464+
decoded_column_view_insert_null_on_conversion_failure(column, view, row);
465+
continue;
466+
}
467+
data.resize(old_size);
468+
return status;
469+
}
470+
data.emplace_back(nanos);
471+
}
472+
return Status::OK();
473+
}
474+
475+
Status DataTypeTimeStampNsSerDe::read_column_from_orc(IColumn& column,
476+
const OrcDecodedColumnView& view) const {
477+
const auto& batch = assert_cast<const orc::TimestampVectorBatch&>(*view.batch);
478+
auto& data = assert_cast<ColumnTimeStampNs&>(column).get_data();
479+
const auto old_size = data.size();
480+
const auto rows = orc_serde_utils::orc_decode_row_count(view.rows, view.selected_rows);
481+
DORIS_CHECK(view.file_type != nullptr);
482+
DORIS_CHECK(view.timestamp_context != nullptr);
483+
for (size_t row = 0; row < rows; ++row) {
484+
const auto source = orc_serde_utils::orc_source_row_at(row, view.selected_rows);
485+
if (orc_serde_utils::orc_row_is_null(batch, source)) {
486+
data.emplace_back();
487+
continue;
488+
}
489+
const auto nanos = batch.nanoseconds[source];
490+
if (nanos < 0 || nanos >= TimeStampNsValue::NANOS_PER_SECOND) {
491+
data.resize(old_size);
492+
return Status::DataQualityError("Invalid ORC timestamp nanoseconds: {}", nanos);
493+
}
494+
DateV2Value<DateTimeV2ValueType> civil;
495+
const int64_t seconds = batch.data[source];
496+
const auto timezone = view.timestamp_context->timezone_for(*view.file_type, seconds);
497+
auto status = orc_serde_utils::orc_timestamp_to_datetime(seconds, nanos / 1000, timezone,
498+
false, &civil);
499+
TimeStampNsValue value;
500+
if (!status.ok() || !value.from_datetime(civil, nanos % 1000)) {
501+
data.resize(old_size);
502+
return Status::DataQualityError("ORC timestamp is outside the TIMESTAMP_NS range");
503+
}
504+
data.push_back(value);
505+
}
506+
return Status::OK();
507+
}
508+
348509
Status DataTypeTimeStampNsSerDe::write_column_to_arrow(const IColumn& column,
349510
const NullMap* null_map,
350511
arrow::ArrayBuilder* array_builder,
@@ -438,7 +599,29 @@ Status DataTypeTimeStampNsSerDe::write_column_to_orc(const std::string& timezone
438599
orc::ColumnVectorBatch* orc_col_batch,
439600
int64_t start, int64_t end, Arena& arena,
440601
const FormatOptions& options) const {
441-
return Status::NotSupported("DataTypeTimeStampNsSerDe::write_column_to_orc");
602+
const auto& data = assert_cast<const ColumnTimeStampNs&>(column).get_data();
603+
auto& batch = assert_cast<orc::TimestampVectorBatch&>(*orc_col_batch);
604+
for (int64_t row = start; row < end; ++row) {
605+
if (batch.notNull[row] == 0) {
606+
continue;
607+
}
608+
int64_t seconds;
609+
auto civil = data[row].to_datetime();
610+
if (!civil.unix_timestamp(&seconds, timezone)) {
611+
return Status::DataQualityError("Cannot convert TIMESTAMP_NS to ORC timestamp");
612+
}
613+
const auto nanos = data[row].nanosecond();
614+
// Match the other timestamp writers: ORC-645 aliases these fractions across the epoch.
615+
if (seconds == -1 && nanos >= 1000000) {
616+
return Status::NotSupported(
617+
"ORC cannot represent pre-epoch timestamp fractions in [-0.999, 0) "
618+
"seconds without data loss; use Parquet for these values");
619+
}
620+
batch.data[row] = seconds;
621+
batch.nanoseconds[row] = nanos;
622+
}
623+
batch.numElements = end - start;
624+
return Status::OK();
442625
}
443626

444627
void DataTypeTimeStampNsSerDe::write_one_cell_to_binary(const IColumn& src_column,

‎be/src/core/data_type_serde/data_type_timestamp_ns_serde.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,15 @@ class DataTypeTimeStampNsSerDe final
7878
ColumnNullable& column_to, const ColumnString& column_from, CastParameters& cast_params,
7979
const NullMap::value_type* source_null_map = nullptr) const override;
8080

81+
Status read_column_from_parquet(IColumn& column, ParquetDecodeSource& source,
82+
const ParquetDecodeContext& context, size_t num_values,
83+
ParquetMaterializationState& state) const override;
84+
Status read_parquet_dictionary(IColumn& column, ParquetDecodeSource& source,
85+
const ParquetDecodeContext& context) const override;
86+
Status read_column_from_decoded_values(IColumn& column,
87+
const DecodedColumnView& view) const override;
88+
Status read_column_from_orc(IColumn& column, const OrcDecodedColumnView& view) const override;
89+
8190
Status write_column_to_arrow(const IColumn& column, const NullMap* null_map,
8291
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
8392
const cctz::time_zone& ctz) const override;

‎be/src/core/data_type_serde/orc_serde_utils.cpp‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,23 @@
2020
#include "common/cast_set.h"
2121
#include "common/check.h"
2222
#include "core/column/column_array.h"
23+
#include "core/data_type_serde/orc_timestamp_decode_context.h"
2324

2425
namespace doris::orc_serde_utils {
2526

27+
bool has_nanosecond_timestamp(const ::orc::Type& type) {
28+
if (type.getKind() == ::orc::TIMESTAMP && type.hasAttributeKey("iceberg.timestamp-unit") &&
29+
type.getAttributeValue("iceberg.timestamp-unit") == "NANOS") {
30+
return true;
31+
}
32+
for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
33+
if (has_nanosecond_timestamp(*type.getSubtype(child))) {
34+
return true;
35+
}
36+
}
37+
return false;
38+
}
39+
2640
size_t orc_decode_row_count(size_t rows, const std::vector<size_t>* selected_rows) {
2741
if (selected_rows == nullptr) {
2842
return rows;
@@ -93,7 +107,9 @@ DecodedColumnView make_orc_decoded_view(const OrcDecodedColumnView& orc_view,
93107
DecodedColumnView view;
94108
view.value_kind = value_kind;
95109
view.row_count = cast_set<int64_t>(orc_decode_row_count(orc_view.rows, orc_view.selected_rows));
96-
view.timezone = orc_view.timezone;
110+
if (orc_view.timestamp_context != nullptr) {
111+
view.timezone = &orc_view.timestamp_context->session_timezone;
112+
}
97113
return view;
98114
}
99115

‎be/src/core/data_type_serde/orc_serde_utils.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#pragma once
1919

2020
#include <cstring>
21+
#include <orc/Type.hh>
2122
#include <orc/Vector.hh>
2223
#include <vector>
2324

@@ -29,6 +30,8 @@
2930
namespace doris {
3031
namespace orc_serde_utils {
3132

33+
bool has_nanosecond_timestamp(const ::orc::Type& type);
34+
3235
size_t orc_decode_row_count(size_t rows, const std::vector<size_t>* selected_rows);
3336
size_t orc_source_row_at(size_t row, const std::vector<size_t>* selected_rows);
3437
bool orc_row_is_null(const ::orc::ColumnVectorBatch& batch, size_t row);
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "core/data_type_serde/orc_timestamp_decode_context.h"
19+
20+
#include <orc/Type.hh>
21+
22+
#include "common/check.h"
23+
24+
namespace doris {
25+
26+
cctz::time_zone OrcTimestampDecodeContext::timezone_for(const orc::Type& file_type,
27+
int64_t seconds) const {
28+
const auto kind = file_type.getKind();
29+
DORIS_CHECK(kind == orc::TIMESTAMP || kind == orc::TIMESTAMP_INSTANT);
30+
// Plain timestamps reconstruct the SDK reader's civil value. Instant-to-DATETIME conversion
31+
// uses the session zone, which must not follow a plain sibling's stripe writer timezone.
32+
if (kind == orc::TIMESTAMP && writer_timezone) {
33+
return writer_timezone(seconds);
34+
}
35+
return kind == orc::TIMESTAMP ? reader_timezone : session_timezone;
36+
}
37+
38+
} // namespace doris

0 commit comments

Comments
 (0)