Skip to content

Commit 388e935

Browse files
authored
[fix](file-format) Align Parquet and ORC timestamp semantics (#68441)
### What problem does this PR solve? Related PR: #67784. This is the third split, following #68297 and #68381, targeting `branch-4.1`. Parquet reads and writes can conflate wall-clock timestamps with UTC instants. ORC decoding truncates nanoseconds while pruning and writer statistics use different precision, which can shift values or discard matching rows. - Keep unannotated INT96 as DATETIMEV2 and carry an explicit `hive.parquet.time-zone` contract through schema inference, scanning, nested/filter-only projections and Hive writes. An unset or empty property selects wall-clock INT96; a named zone decodes values normalized in that zone. Scanner selection and condition-cache keys preserve this interpretation. - Default Parquet OUTFILE/EXPORT to INT64 logical timestamps: DATETIMEV2 has `isAdjustedToUTC=false`, while TIMESTAMPTZ retains instant semantics. Keep explicit `enable_int96_timestamps=true`, validate its boolean value, and pass it through EXPORT. Hive writes retain INT96. - Round ORC nanoseconds to microseconds consistently, distinguish civil-time carry from instant carry at DST transitions, and reject invalid nanoseconds or years before narrowing into packed storage. Align aggregate statistics, conservative SARG bounds and writer bounds; reject the pre-epoch fractional interval that ORC cannot write losslessly. - Preserve Paimon timestamp semantics recursively on the FE wire and apply the effective file types before TableReader builds mappings and blocks. Keep nested overrides for full and predicate-only projections, and isolate Paimon from Hive INT96 timezone properties. Advertise native Parquet in hybrid Paimon/Hudi scans, including deferred Hudi splits, so scanner selection honors the contract. This split preserves existing external type-mapping defaults. It includes the Paimon read-side prerequisites for the INT96 change; remaining Iceberg/Paimon bindings stay in the next split. It adds no binary computation support. Unannotated raw INT96 intentionally remains DATETIMEV2. This PR does not restore the old mapping override or add a mixed-version writer compatibility gate. ### Release note Parquet OUTFILE/EXPORT now defaults to INT64 logical timestamps; set `enable_int96_timestamps=true` when INT96 output is required. `hive.parquet.time-zone` explicitly controls INT96 normalization. ORC timestamp rounding and pruning now agree at microsecond and DST boundaries; lossy pre-epoch writes fail explicitly. ### Validation - Before the review fixes: built the official ASAN BE unit-test target and passed 622 selected tests; passed 43 selected FE tests and parsed 21 changed regression Groovy scripts. - Review-fix validation: all 446 selected BE tests in 18 suites passed under ASAN, including real INT96 decoding through PaimonReader/TableReader, hidden nested filters, NULLs and both timestamp-mapping settings. All 104 FE tests in five classes passed, including recursive Thrift round trips and eager/deferred hybrid scan planning. - FE Checkstyle and clang-format 16 checks passed for all affected files. - Regression evidence: five new FE tests and four BE tests failed before their fixes. The full native Paimon read test additionally reproduced a nested predicate-only column type mismatch before fixing semantic propagation from the annotated file schema. - External SQL/Hive/HDFS regressions were not executed locally; full regression validation is pending CI. ### External regression follow-up Updated timestamp goldens and scanner assertions for native ORC rounding, raw INT96 reads, and required V2 dispatch. The INT64 export cases now read their own output files. The malformed timestamp case explicitly selects strict conversion and restores the previous setting. Existing exact pruning counters and ordered result checks remain covered. Validation: 360 related ASAN BE tests passed, all eight changed Groovy scripts parsed successfully, and updated CI result rows and golden structure were checked. The Parquet profile regression now checks the same exact V2 pruning counters across all four legacy min/max and lazy-read switch combinations: versioned Parquet requires V2, which always applies safe metadata pruning and lazy materialization. Follow-up validation passed all 93 selected Parquet reader/statistics ASAN tests, Groovy compilation, and 28 profile checks against recorded CI metrics (not a full SQL regression run). Full external regression reruns are pending CI. The remaining Lance ranking, object-storage timeout, OBS dependency and large-string memory failures also occur on independent PR builds and are outside this change. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - Behavior changed: - [x] Yes. Timestamp representation, interpretation and precision changes are described above. - Does this need documentation? - [x] Yes. The INT64 default and `hive.parquet.time-zone` should be reflected in the file-format documentation. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent e0b5cf2 commit 388e935

123 files changed

Lines changed: 3839 additions & 770 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/data_type_datetimev2_serde.cpp‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,24 @@ Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColum
7373
}
7474
auto& value =
7575
reinterpret_cast<DateV2Value<DateTimeV2ValueType>&>(data[old_data_size + row]);
76-
value.from_unixtime(orc_batch->data[source_row], timezone);
77-
value.set_microsecond(cast_set<uint64_t>(orc_batch->nanoseconds[source_row] / 1000));
76+
orc_serde_utils::RoundedOrcTimestamp timestamp;
77+
auto status = orc_serde_utils::round_orc_timestamp_to_microseconds(
78+
orc_batch->data[source_row], orc_batch->nanoseconds[source_row], &timestamp);
79+
if (!status.ok()) {
80+
data.resize(old_data_size);
81+
return status;
82+
}
83+
const bool is_timestamp_instant =
84+
orc_view.file_type->getKind() == ::orc::TypeKind::TIMESTAMP_INSTANT;
85+
// Instant carry precedes timezone conversion; plain TIMESTAMP carry stays in civil time
86+
// so rounding cannot jump across a daylight-saving gap or fold.
87+
status = orc_serde_utils::orc_timestamp_to_datetime(
88+
is_timestamp_instant ? timestamp.seconds : orc_batch->data[source_row],
89+
timestamp.microseconds, timezone, !is_timestamp_instant && timestamp.carry, &value);
90+
if (!status.ok()) {
91+
data.resize(old_data_size);
92+
return status;
93+
}
7894
}
7995
return Status::OK();
8096
}
@@ -838,6 +854,13 @@ Status DataTypeDateTimeV2SerDe::write_column_to_orc(const std::string& timezone,
838854
return Status::InternalError("get unix timestamp error.");
839855
}
840856

857+
// ORC-645 aliases this pre-epoch fraction to a positive timestamp on disk.
858+
// Keep the same fail-fast contract as TIMESTAMPTZ instead of writing a wrong value.
859+
if (timestamp == -1 && datetime_val.microsecond() >= 1000) {
860+
return Status::NotSupported(
861+
"ORC cannot represent pre-epoch timestamp fractions in [-0.999, 0) seconds "
862+
"without data loss; use Parquet for these values");
863+
}
841864
cur_batch->data[row_id] = timestamp;
842865
cur_batch->nanoseconds[row_id] = datetime_val.microsecond() * micro_to_nano_second;
843866
}

‎be/src/core/data_type_serde/data_type_timestamptz_serde.cpp‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,21 @@ Status decode_timestamp_tz_orc_values(IColumn& nested_column,
5656
continue;
5757
}
5858
auto& value = data[old_data_size + row];
59-
value.from_unixtime(orc_batch->data[source_row], utc_time_zone);
60-
value.set_microsecond(cast_set<uint64_t>(orc_batch->nanoseconds[source_row] / 1000));
59+
orc_serde_utils::RoundedOrcTimestamp timestamp;
60+
auto status = orc_serde_utils::round_orc_timestamp_to_microseconds(
61+
orc_batch->data[source_row], orc_batch->nanoseconds[source_row], &timestamp);
62+
if (!status.ok()) {
63+
data.resize(old_data_size);
64+
return status;
65+
}
66+
DateV2Value<DateTimeV2ValueType> datetime;
67+
status = orc_serde_utils::orc_timestamp_to_datetime(
68+
timestamp.seconds, timestamp.microseconds, utc_time_zone, false, &datetime);
69+
if (!status.ok()) {
70+
data.resize(old_data_size);
71+
return status;
72+
}
73+
value = TimestampTzValue(datetime);
6174
}
6275
return Status::OK();
6376
}
@@ -492,6 +505,13 @@ Status DataTypeTimeStampTzSerDe::write_column_to_orc(const std::string& timezone
492505
int64_t timestamp = 0;
493506
col_data[row_id].unix_timestamp(&timestamp, UTC);
494507

508+
// ORC-645 increments this second to zero, making it indistinguishable from a
509+
// positive instant. Reject it before the writer can silently corrupt the value.
510+
if (timestamp == -1 && col_data[row_id].microsecond() >= 1000) {
511+
return Status::NotSupported(
512+
"ORC cannot represent pre-epoch timestamp fractions in [-0.999, 0) seconds "
513+
"without data loss; use Parquet for these values");
514+
}
495515
cur_batch->data[row_id] = timestamp;
496516
cur_batch->nanoseconds[row_id] = col_data[row_id].microsecond() * micro_to_nano_second;
497517
}

‎be/src/core/data_type_serde/orc_serde_utils.cpp‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,53 @@ bool orc_row_is_null(const ::orc::ColumnVectorBatch& batch, size_t row) {
4141
return batch.hasNulls && !batch.notNull[row];
4242
}
4343

44+
Status round_orc_timestamp_to_microseconds(int64_t seconds, int64_t nanoseconds,
45+
RoundedOrcTimestamp* result) {
46+
constexpr int64_t NANOS_PER_SECOND = 1000000000;
47+
constexpr int64_t NANOS_PER_MICROSECOND = 1000;
48+
constexpr int64_t MICROS_PER_SECOND = 1000000;
49+
DORIS_CHECK(result != nullptr);
50+
// Nanoseconds come from an external file, so malformed input must fail the scan rather than
51+
// terminate the BE process.
52+
if (nanoseconds < 0 || nanoseconds >= NANOS_PER_SECOND) {
53+
return Status::DataQualityError("Invalid ORC timestamp nanoseconds: {}", nanoseconds);
54+
}
55+
// Doris stores six fractional digits, so use half-up rounding and carry 999999500ns into the
56+
// next second instead of silently truncating the ORC value.
57+
const auto rounded_microseconds =
58+
(nanoseconds + NANOS_PER_MICROSECOND / 2) / NANOS_PER_MICROSECOND;
59+
// Validate the carry here, but validate Doris' calendar range after timezone conversion:
60+
// a valid year-zero local timestamp may have a UTC epoch before year zero.
61+
if (__builtin_add_overflow(seconds, rounded_microseconds / MICROS_PER_SECOND,
62+
&result->seconds)) {
63+
return Status::DataQualityError("ORC timestamp overflows after microsecond rounding");
64+
}
65+
result->microseconds = cast_set<uint64_t>(rounded_microseconds % MICROS_PER_SECOND);
66+
result->carry = rounded_microseconds >= MICROS_PER_SECOND;
67+
return Status::OK();
68+
}
69+
70+
Status orc_timestamp_to_datetime(int64_t seconds, uint64_t microseconds,
71+
const cctz::time_zone& timezone, bool carry_in_civil_time,
72+
DateV2Value<DateTimeV2ValueType>* value) {
73+
auto civil = cctz::convert(cctz::time_point<cctz::seconds>(cctz::seconds(seconds)), timezone);
74+
if (carry_in_civil_time) {
75+
++civil;
76+
}
77+
// Validate before packing: narrowing the year to uint16_t can wrap an invalid year to 0-9999.
78+
if (civil.year() < 0 || civil.year() > 9999) {
79+
return Status::DataQualityError(
80+
"Decoded ORC timestamp is outside the target timezone range");
81+
}
82+
// Narrow only after the year check; cctz normalizes the other civil fields and ORC rounding
83+
// keeps microseconds below one second.
84+
value->unchecked_set_time(cast_set<uint16_t>(civil.year()), cast_set<uint8_t>(civil.month()),
85+
cast_set<uint8_t>(civil.day()), cast_set<uint8_t>(civil.hour()),
86+
cast_set<uint8_t>(civil.minute()), cast_set<uint16_t>(civil.second()),
87+
cast_set<uint32_t>(microseconds));
88+
return Status::OK();
89+
}
90+
4491
DecodedColumnView make_orc_decoded_view(const OrcDecodedColumnView& orc_view,
4592
DecodedValueKind value_kind) {
4693
DecodedColumnView view;

‎be/src/core/data_type_serde/orc_serde_utils.h‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
#include "core/arena.h"
2525
#include "core/column/column_array.h"
2626
#include "core/data_type_serde/data_type_serde.h"
27+
#include "core/value/vdatetime_value.h"
2728

2829
namespace doris {
2930
namespace orc_serde_utils {
@@ -32,6 +33,19 @@ size_t orc_decode_row_count(size_t rows, const std::vector<size_t>* selected_row
3233
size_t orc_source_row_at(size_t row, const std::vector<size_t>* selected_rows);
3334
bool orc_row_is_null(const ::orc::ColumnVectorBatch& batch, size_t row);
3435

36+
struct RoundedOrcTimestamp {
37+
int64_t seconds;
38+
uint64_t microseconds;
39+
bool carry;
40+
};
41+
42+
Status round_orc_timestamp_to_microseconds(int64_t seconds, int64_t nanoseconds,
43+
RoundedOrcTimestamp* result);
44+
45+
Status orc_timestamp_to_datetime(int64_t seconds, uint64_t microseconds,
46+
const cctz::time_zone& timezone, bool carry_in_civil_time,
47+
DateV2Value<DateTimeV2ValueType>* value);
48+
3549
DecodedColumnView make_orc_decoded_view(const OrcDecodedColumnView& orc_view,
3650
DecodedValueKind value_kind);
3751

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
#include "exec/scan/file_scanner_v2.h"
3434
#include "exec/scan/scanner_context.h"
3535
#include "format/format_common.h"
36+
#include "format/table/iceberg_scan_semantics.h"
3637
#include "storage/storage_engine.h"
3738
#include "storage/tablet/tablet_manager.h"
3839

@@ -147,9 +148,26 @@ bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions& query_o
147148
const bool is_transactional_hive =
148149
scan_params.__isset.table_format_params &&
149150
scan_params.table_format_params.table_format_type == "transactional_hive";
150-
return query_options.__isset.enable_file_scanner_v2 && query_options.enable_file_scanner_v2 &&
151-
!is_load && scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP &&
152-
!is_transactional_hive;
151+
// Hybrid scans advertise native Parquet at scan level because ranges can arrive after the
152+
// scanner is selected; FORMAT_JNI alone does not imply that every range uses JNI.
153+
const bool is_hybrid_native_parquet = scan_params.format_type == TFileFormatType::FORMAT_JNI &&
154+
scan_params.__isset.contains_native_parquet &&
155+
scan_params.contains_native_parquet;
156+
// Version 1 introduces the explicit wall-clock/instant contract that scanner V1 cannot honor.
157+
const bool requires_parquet_timestamp_contract =
158+
(scan_params.format_type == TFileFormatType::FORMAT_PARQUET ||
159+
is_hybrid_native_parquet || supports_iceberg_scan_semantics_v1(&scan_params)) &&
160+
(scan_params.__isset.hive_parquet_time_zone ||
161+
(scan_params.__isset.parquet_timestamp_semantics_version &&
162+
scan_params.parquet_timestamp_semantics_version >= 1));
163+
const bool scanner_v2_requested = (query_options.__isset.enable_file_scanner_v2 &&
164+
query_options.enable_file_scanner_v2) ||
165+
requires_parquet_timestamp_contract;
166+
// Iceberg's default write format does not describe retained files, and remote splits arrive
167+
// after scanner construction. Keep versioned Iceberg scans on V2 without eagerly listing
168+
// every file.
169+
return scanner_v2_requested && !is_load &&
170+
scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP && !is_transactional_hive;
153171
}
154172

155173
bool FileScanLocalState::_can_generate_physical_splits(const TQueryOptions& query_options,

‎be/src/exec/sink/writer/vhive_partition_writer.cpp‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,11 @@ VHivePartitionWriter::VHivePartitionWriter(const TDataSink& t_sink, std::string
5050
_file_format_type(file_format_type),
5151
_hive_compress_type(hive_compress_type),
5252
_hive_serde_properties(hive_serde_properties),
53-
_hadoop_conf(hadoop_conf) {}
53+
_hadoop_conf(hadoop_conf) {
54+
if (t_sink.hive_table_sink.__isset.hive_parquet_time_zone) {
55+
_hive_parquet_time_zone = t_sink.hive_table_sink.hive_parquet_time_zone;
56+
}
57+
}
5458

5559
Status VHivePartitionWriter::open(RuntimeState* state, RuntimeProfile* operator_profile) {
5660
_state = state;
@@ -97,6 +101,12 @@ Status VHivePartitionWriter::open(RuntimeState* state, RuntimeProfile* operator_
97101
// changing the default Hive parquet timestamp encoding to standard logical types.
98102
ParquetFileOptions parquet_options = {parquet_compression_type,
99103
TParquetVersion::PARQUET_1_0, false, true};
104+
// Match the scan contract: empty means wall-clock, not the insert session zone.
105+
// Keep the session fallback only for old FE requests that omit the field entirely.
106+
if (_hive_parquet_time_zone.has_value()) {
107+
parquet_options.int96_timezone =
108+
_hive_parquet_time_zone->empty() ? "UTC" : *_hive_parquet_time_zone;
109+
}
100110
_file_format_transformer = std::make_unique<VHiveParquetWriter>(
101111
state, _file_writer.get(), _write_output_expr_ctxs, _write_column_names, false,
102112
parquet_options);

‎be/src/exec/sink/writer/vhive_partition_writer.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919

2020
#include <gen_cpp/DataSinks_types.h>
2121

22+
#include <optional>
23+
2224
#include "core/column/column.h"
2325
#include "exprs/vexpr_fwd.h"
2426
#include "format/transformer/vfile_format_transformer.h"
@@ -101,6 +103,7 @@ class VHivePartitionWriter {
101103
TFileCompressType::type _hive_compress_type;
102104
const THiveSerDeProperties* _hive_serde_properties;
103105
const std::map<std::string, std::string>& _hadoop_conf;
106+
std::optional<std::string> _hive_parquet_time_zone;
104107

105108
std::shared_ptr<io::FileSystem> _fs = nullptr;
106109

‎be/src/format/parquet/parquet_arrow_block_convertor.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,11 @@ Status ParquetArrowBlockConvertor::init() {
3131
std::vector<std::shared_ptr<arrow::Field>> fields;
3232
fields.reserve(_types.size());
3333
// Retain the declared Arrow timezone label; cctz's fixed-offset name is internal.
34-
// Preserve the existing timezone-bearing DATETIMEV2 schema for both Parquet encodings.
34+
// INT96 normalization and schema construction must use the same instance's timezone.
3535
for (size_t i = 0; i < _types.size(); ++i) {
3636
std::shared_ptr<arrow::DataType> type;
37-
RETURN_IF_ERROR(convert_to_arrow_type(_types[i], &type, _timezone_name));
37+
RETURN_IF_ERROR(
38+
convert_to_arrow_type(_types[i], &type, _timezone_name, !_enable_int96_timestamps));
3839
fields.emplace_back(arrow::field(_names[i], type, _types[i]->is_nullable()));
3940
}
4041
_arrow_schema = arrow::schema(std::move(fields));

‎be/src/format/parquet/parquet_arrow_block_convertor.h‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,13 @@ namespace doris {
2424
class ParquetArrowBlockConvertor : public ArrowBlockConvertor {
2525
public:
2626
ParquetArrowBlockConvertor(DataTypes types, std::vector<std::string> names,
27-
std::string timezone_name, const cctz::time_zone& timezone)
27+
std::string timezone_name, const cctz::time_zone& timezone,
28+
bool enable_int96_timestamps)
2829
: ArrowBlockConvertor(nullptr, timezone),
2930
_types(std::move(types)),
3031
_names(std::move(names)),
31-
_timezone_name(std::move(timezone_name)) {}
32+
_timezone_name(std::move(timezone_name)),
33+
_enable_int96_timestamps(enable_int96_timestamps) {}
3234
Status init() override;
3335

3436
protected:
@@ -41,6 +43,7 @@ class ParquetArrowBlockConvertor : public ArrowBlockConvertor {
4143
DataTypes _types;
4244
std::vector<std::string> _names;
4345
std::string _timezone_name;
46+
bool _enable_int96_timestamps;
4447
};
4548

4649
} // namespace doris

‎be/src/format/parquet/schema_desc.cpp‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -232,15 +232,10 @@ std::pair<DataTypePtr, bool> FieldDescriptor::get_doris_type(
232232
ans.first = DataTypeFactory::instance().create_data_type(TYPE_BIGINT, nullable);
233233
break;
234234
case tparquet::Type::INT96:
235-
if (_enable_mapping_timestamp_tz) {
236-
// treat INT96 as TIMESTAMPTZ
237-
ans.first = DataTypeFactory::instance().create_data_type(TYPE_TIMESTAMPTZ, nullable,
238-
0, 6);
239-
} else {
240-
// in most cases, it's a nano timestamp
241-
ans.first = DataTypeFactory::instance().create_data_type(TYPE_DATETIMEV2, nullable,
242-
0, 6);
243-
}
235+
// INT96 has no timezone annotation. A table's logical schema may override this
236+
// for known instant columns, but a catalog option cannot supply missing semantics.
237+
ans.first =
238+
DataTypeFactory::instance().create_data_type(TYPE_DATETIMEV2, nullable, 0, 6);
244239
break;
245240
case tparquet::Type::FLOAT:
246241
ans.first = DataTypeFactory::instance().create_data_type(TYPE_FLOAT, nullable);

0 commit comments

Comments
 (0)