Skip to content

Commit 980f576

Browse files
morningmanclaude
andcommitted
[fix](orc) decode ORC DATE statistics with the row decoder's calendar
### What problem does this PR solve? Issue Number: close apache#67366 Related PR: apache#67449 Problem Summary: Review follow-up on apache#67449. The two commits before this one moved the DATE row decoders onto the proleptic Gregorian ordinal but left three consumers of the same ordinal behind, and the Iceberg transforms kept one pre-existing defect. format_v2's ORC reader still converted DATE stripe statistics through `date_day_offset_dict`, whose out-of-table fallback is the old `1900-01-01 + (value + 25567)` arithmetic. Those zone maps have exactly one consumer, `OrcReader::get_aggregate_result`, which answers a pushed-down MIN/MAX from statistics without reading a row, so on the default path (`enable_file_scanner_v2` and `enable_push_down_no_group_agg`) `MIN(d)` reported 1900-01-01 for a file whose smallest row is 0000-01-01 -- a value present in no row at all. Decode the bounds with the same conversion the rows use and return false when a bound has no Doris DATE, which makes the reader fall back to a normal row scan. `ReadMapDecimalDateWithCenturyBoundary` pinned that dictionary fallback: it wrote the ORC ordinal -719530 and expected 1900-01-01, which the new decoder rejects. Point the fixture at 0000-01-01, the smallest representable DATE, and add a dedicated test that the three unrepresentable shapes (-719530, the proleptic-only -719469 and 2932897) fail the scan. Iceberg's `bucket` over a timestamp hashes the full microsecond value (iceberg-api `Bucket.BucketLong` over `BucketUtil.hash(long)`), but Doris hashed whole seconds times a million, so a DATETIME(6) row landed in a different bucket than the same row written by Spark and bucket pruning skipped the Doris-written file. `human_year()` did not zero-pad, so a year-zero row's partition directory was `..._year=0` where Spark writes `..._year=0000`. On the FE side, `IcebergPartitionUtils` rendered MTMV partition-range bounds with the pattern letter `y` (year-of-era), so the ordinal -719528 the `day` transform now emits for 0000-01-01 rendered as 0001-01-01 and collided with the range of the real 0001-01-01 partition. The `ExpressionEstimation` hunk from the first commit is reverted: DATE/DATETIME column statistics are `yyyyMMddHHmmss` numbers, but `getDatetimeFromLong()` reads them as epoch seconds, so every value is >= 101000000 and the year-zero correction could never fire. The real defect (the encoding mismatch) predates this PR and belongs in its own change with its own test. Also in this commit: the shared out-of-range message no longer says "Parquet" for an ORC file and now carries the offending value, the ORC and Arrow callers prepend the column name, the epoch-day constants are derived from the daynr domain instead of typed out, the Arrow date64 branch spells its contract like the date32 one, and `epoch_date()`/`epoch_datetime()` plus their two cast includes are deleted (no caller left). ### Release note - A pushed-down `MIN(date_col)` / `MAX(date_col)` over an ORC file now agrees with the rows. It previously answered from stripe statistics decoded in Doris's own calendar, so a file containing 0000-01-01 .. 0000-02-28 reported a bound that no row holds (1900-01-01 for 0000-01-01). - Iceberg `bucket(n, ts)` now hashes the full microsecond value, as the Iceberg specification requires. A `DATETIME` column with a non-zero sub-second part routes to a different (and now Spark-compatible) bucket than before; existing files keep their old bucket values until they are rewritten. - An Iceberg `year` partition directory is zero-padded (`..._year=0000`), which is what Spark writes. Only the directory name changes; the manifest value is unaffected. - The MTMV partition-range view of an Iceberg table with year-zero `day`/`hour` partitions renders 0000-01-01 instead of 0001-01-01. - An unrepresentable DATE in a Parquet or ORC file is now reported with the offending value and the column name, and no longer names the wrong format. ### Check List (For Author) - Test: Regression test + Unit Test - BE UT: orc_reader_test (year-zero rows, a null row, the three rejected ordinals, MIN/MAX against the row decode, and the statistics fallback), data_type_datev2_serde_calendar_test (the Parquet/ORC decoded-value entry point, strict and null-on-failure), partition_transformers_test (microsecond buckets, the zero-padded human year), vdatetime_value_test - FE UT: IcebergPartitionUtilsTest (year-zero day and hour ranges) - Regression: the ORC half of external_table_p0/iceberg/write/test_iceberg_write_partition_epoch_boundary compares a pushed-down MIN/MAX with the rows of the same file - Behavior changed: Yes (see the Release note) - Does this need documentation: No Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
1 parent 3fd801d commit 980f576

19 files changed

Lines changed: 641 additions & 193 deletions

File tree

‎be/src/core/data_type_serde/data_type_datev2_serde.cpp‎

Lines changed: 30 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@
2121
#include <cctz/time_zone.h>
2222
#include <fmt/core.h>
2323

24-
#include <algorithm>
2524
#include <cstdint>
2625
#include <limits>
2726
#include <vector>
@@ -63,24 +62,42 @@ Status decode_date_orc_values(const DataTypeSerDe& serde, IColumn& column,
6362
date_values.resize(output_rows);
6463
for (size_t row = 0; row < output_rows; ++row) {
6564
const auto source_row = orc_serde_utils::orc_source_row_at(row, orc_view.selected_rows);
65+
// The payload of a null slot is undefined, so it is zeroed rather than range-checked;
66+
// `read_column_from_decoded_values` never decodes it.
67+
if (view.null_map != nullptr && view.null_map[row] != 0) {
68+
date_values[row] = 0;
69+
continue;
70+
}
6671
// ORC DATE is days since 1970-01-01 in the proleptic Gregorian calendar, the same encoding
67-
// `decode_date_orc_values`'s consumer expects. Clamp instead of narrowing blindly so a
68-
// corrupt file surfaces as a range error in `decode_parquet_date()` rather than wrapping.
72+
// `decode_epoch_days()` expects, but ORC hands it over in an int64 batch. Reject a value
73+
// that does not fit the int32 view here, while the real file value is still available for
74+
// the message, instead of narrowing it blindly.
6975
const int64_t file_days = orc_batch->data[source_row];
70-
date_values[row] = static_cast<int32_t>(
71-
std::clamp<int64_t>(file_days, std::numeric_limits<int32_t>::min(),
72-
std::numeric_limits<int32_t>::max()));
76+
if (file_days < std::numeric_limits<int32_t>::min() ||
77+
file_days > std::numeric_limits<int32_t>::max()) {
78+
return Status::DataQualityError(
79+
"DATE value {} is outside the Doris DATE range (0000-01-01..9999-12-31, "
80+
"0000-02-29 excluded)",
81+
file_days);
82+
}
83+
date_values[row] = static_cast<int32_t>(file_days);
7384
}
7485
view.values = reinterpret_cast<const uint8_t*>(date_values.data());
7586
RETURN_IF_ERROR(orc_serde_utils::read_decoded_values(serde, column, &view));
7687
return Status::OK();
7788
}
7889

79-
Status decode_parquet_date(int32_t encoded_date, DateV2Value<DateV2ValueType>* value) {
90+
// Shared by every "days since 1970-01-01" source: Parquet, ORC and Arrow date32/date64. The
91+
// message must therefore not name one format, and it carries the value so a broken file can be
92+
// identified without re-reading it.
93+
Status decode_epoch_days(int64_t encoded_date, DateV2Value<DateV2ValueType>* value) {
8094
DORIS_CHECK(value != nullptr);
8195
const int64_t day_number = epoch_days_to_daynr(encoded_date);
8296
if (day_number == 0 || !value->get_date_from_daynr(static_cast<uint64_t>(day_number))) {
83-
return Status::DataQualityError("Parquet DATE value is out of range");
97+
return Status::DataQualityError(
98+
"DATE value {} is outside the Doris DATE range (0000-01-01..9999-12-31, "
99+
"0000-02-29 excluded)",
100+
encoded_date);
84101
}
85102
return Status::OK();
86103
}
@@ -100,7 +117,7 @@ class DateV2ParquetConsumer final : public ParquetFixedValueConsumer {
100117
_data.resize(old_size + num_values);
101118
for (size_t row = 0; row < num_values; ++row) {
102119
DateV2Value<DateV2ValueType> value;
103-
const auto status = decode_parquet_date(
120+
const auto status = decode_epoch_days(
104121
unaligned_load<int32_t>(values + row * sizeof(int32_t)), &value);
105122
if (!status.ok()) {
106123
if (_state != nullptr && _state->mark_conversion_failure(old_size + row)) {
@@ -257,10 +274,9 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
257274
"Arrow Date64 value must contain whole days: row={}, milliseconds={}",
258275
value_i, milliseconds);
259276
}
260-
const int64_t daynr = epoch_days_to_daynr(milliseconds / MILLISECONDS_PER_DAY);
261277
DateV2Value<DateV2ValueType> value;
262-
if (daynr <= 0 || daynr > DATE_MAX_DAYNR ||
263-
!value.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
278+
if (const auto status = decode_epoch_days(milliseconds / MILLISECONDS_PER_DAY, &value);
279+
!status.ok()) {
264280
return Status::InvalidArgument(
265281
"Arrow Date64 value is outside the Doris DATE range: "
266282
"row={}, milliseconds={}",
@@ -280,9 +296,8 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
280296
const uint8_t* raw_byte_ptr = base_ptr + value_i * element_size;
281297
auto date_value = unaligned_load<int32_t>(raw_byte_ptr);
282298

283-
const int64_t daynr = epoch_days_to_daynr(date_value);
284299
DateV2Value<DateV2ValueType> v;
285-
if (daynr == 0 || !v.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
300+
if (const auto status = decode_epoch_days(date_value, &v); !status.ok()) {
286301
return Status::InvalidArgument(
287302
"Arrow Date32 value is outside the Doris DATE range: row={}, days={}",
288303
value_i, date_value);
@@ -314,7 +329,7 @@ Status DataTypeDateV2SerDe::read_column_from_decoded_values(IColumn& column,
314329
continue;
315330
}
316331
DateV2Value<DateV2ValueType> date_v2;
317-
const auto status = decode_parquet_date(values[row], &date_v2);
332+
const auto status = decode_epoch_days(values[row], &date_v2);
318333
if (!status.ok()) {
319334
// Decoded values back both metadata conversion and native materialization. Preserve
320335
// strict errors while allowing nullable non-strict scans to mark only the bad row.

‎be/src/core/value/vdatetime_value.h‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1694,8 +1694,6 @@ class date_day_offset_dict {
16941694

16951695
static constexpr int START_YEAR = 1900; // 1900-01-01
16961696
static constexpr int END_YEAR = 2039; // 2039-10-24
1697-
static constexpr int DAY_OFFSET_CAL_START_POINT_DAYNR =
1698-
719528; // 1970-01-01 (start from 0000-01-01, 0000-01-01 is day 1, returns 1)
16991697

17001698
static std::array<DateV2Value<DateV2ValueType>, DICT_DAYS> DATE_DAY_OFFSET_ITEMS;
17011699
static std::array<std::array<std::array<int, 31>, 12>, 140> DATE_DAY_OFFSET_DICT;
@@ -1710,6 +1708,9 @@ class date_day_offset_dict {
17101708
date_day_offset_dict& operator=(const date_day_offset_dict&) = default;
17111709

17121710
public:
1711+
static constexpr int DAY_OFFSET_CAL_START_POINT_DAYNR =
1712+
719528; // 1970-01-01 (start from 0000-01-01, 0000-01-01 is day 1, returns 1)
1713+
17131714
static bool can_speed_up_calc_daynr(int year) { return year >= START_YEAR && year <= END_YEAR; }
17141715

17151716
static int get_offset_by_daynr(int daynr) { return daynr - DAY_OFFSET_CAL_START_POINT_DAYNR; }
@@ -1777,18 +1778,29 @@ inline uint32_t calc_daynr(uint16_t year, uint8_t month, uint8_t day) {
17771778
// numberings coincide from 0000-03-01 (daynr 60) onwards, so the whole difference is the missing
17781779
// 0000-02-29. Every conversion between a Doris DATE and an external "days since 1970-01-01" value
17791780
// must go through the two helpers below instead of adding/subtracting the epoch daynr directly.
1780-
inline constexpr int64_t DAYNR_OF_UNIX_EPOCH = 719528; // calc_daynr(1970, 1, 1)
1781-
inline constexpr int64_t DAYNR_OF_0000_03_01 = 60; // first daynr shared by both calendars
1782-
inline constexpr int64_t EPOCH_DAYS_MIN = -719528; // 0000-01-01, the smallest Doris DATE
1783-
inline constexpr int64_t EPOCH_DAYS_0000_02_29 = -719469; // exists in proleptic Gregorian only
1784-
inline constexpr int64_t EPOCH_DAYS_MAX = 2932896; // 9999-12-31, the largest Doris DATE
1781+
inline constexpr int64_t DAYNR_OF_UNIX_EPOCH =
1782+
date_day_offset_dict::DAY_OFFSET_CAL_START_POINT_DAYNR;
1783+
inline constexpr int64_t DAYNR_OF_0000_03_01 = 60; // first daynr shared by both calendars
17851784

17861785
// Doris daynr -> days since 1970-01-01 in the proleptic Gregorian calendar.
17871786
inline constexpr int32_t daynr_to_epoch_days(int64_t daynr) {
17881787
return static_cast<int32_t>(daynr - DAYNR_OF_UNIX_EPOCH -
17891788
(daynr < DAYNR_OF_0000_03_01 ? 1 : 0));
17901789
}
17911790

1791+
// The three boundaries of the external ordinal domain, derived from the daynr domain rather than
1792+
// typed out, so a change to either calendar constant cannot leave them behind.
1793+
inline constexpr int64_t EPOCH_DAYS_MIN = daynr_to_epoch_days(1); // 0000-01-01, smallest Doris DATE
1794+
inline constexpr int64_t EPOCH_DAYS_0000_02_29 =
1795+
daynr_to_epoch_days(DAYNR_OF_0000_03_01) - 1; // exists in proleptic Gregorian only
1796+
inline constexpr int64_t EPOCH_DAYS_MAX =
1797+
daynr_to_epoch_days(DATE_MAX_DAYNR); // 9999-12-31, the largest Doris DATE
1798+
1799+
static_assert(DAYNR_OF_UNIX_EPOCH == 719528, "calc_daynr(1970, 1, 1)");
1800+
static_assert(EPOCH_DAYS_MIN == -719528);
1801+
static_assert(EPOCH_DAYS_0000_02_29 == -719469);
1802+
static_assert(EPOCH_DAYS_MAX == 2932896);
1803+
17921804
// Inverse of `daynr_to_epoch_days()`. Returns 0, which is never a valid daynr, when the value has
17931805
// no Doris DATE representation: before 0000-01-01, after 9999-12-31, or the proleptic-only
17941806
// 0000-02-29. Callers must reject 0 instead of feeding it to `get_date_from_daynr()`.

‎be/src/exec/sink/writer/iceberg/partition_transformers.h‎

Lines changed: 12 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,10 @@
2020
#include "core/column/column.h"
2121
#include "core/column/column_nullable.h"
2222
#include "core/data_type/data_type_factory.hpp"
23+
#include "core/data_type/data_type_number.h"
24+
#include "core/data_type/data_type_string.h"
2325
#include "exec/common/stringop_substring.h"
24-
#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
25-
#include "exprs/function/cast/cast_to_datev2_impl.hpp"
26+
#include "exprs/function/function_helpers.h"
2627
#include "util/bit_util.h"
2728

2829
namespace doris {
@@ -49,33 +50,11 @@ inline constexpr int ICEBERG_EPOCH_YEAR = 1970;
4950

5051
class PartitionColumnTransformUtils {
5152
public:
52-
static DateV2Value<DateV2ValueType>& epoch_date() {
53-
static DateV2Value<DateV2ValueType> epoch_date;
54-
static bool initialized = false;
55-
if (!initialized) {
56-
CastParameters params;
57-
DORIS_CHECK((CastToDateV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
58-
{"1970-01-01 00:00:00", 19}, epoch_date, nullptr, params)));
59-
initialized = true;
60-
}
61-
return epoch_date;
62-
}
63-
64-
static DateV2Value<DateTimeV2ValueType>& epoch_datetime() {
65-
static DateV2Value<DateTimeV2ValueType> epoch_datetime;
66-
static bool initialized = false;
67-
if (!initialized) {
68-
CastParameters params;
69-
DORIS_CHECK((CastToDatetimeV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
70-
{"1970-01-01 00:00:00", 19}, epoch_datetime, nullptr, -1, params)));
71-
initialized = true;
72-
}
73-
return epoch_datetime;
74-
}
75-
7653
static std::string human_year(int year_ordinal) {
7754
auto ymd = std::chrono::year_month_day {EPOCH} + std::chrono::years(year_ordinal);
78-
return std::to_string(static_cast<int>(ymd.year()));
55+
// iceberg-api's TransformUtil.humanYear is String.format("%04d", ...): year 0 has to be
56+
// "0000", not "0", or the partition directory differs from the one Spark writes.
57+
return fmt::format("{:04d}", static_cast<int>(ymd.year()));
7958
}
8059

8160
static std::string human_month(int month_ordinal) {
@@ -691,7 +670,12 @@ class TimestampBucketPartitionColumnTransform : public PartitionColumnTransform
691670
LOG(WARNING) << "Failed to call unix_timestamp :" << value.debug_string();
692671
timestamp = 0;
693672
}
694-
Int64 long_value = static_cast<Int64>(timestamp) * 1000000;
673+
// Iceberg hashes the full microsecond value of a timestamp (spec: Partition
674+
// Transforms, `bucket` over `timestamp`), so the sub-second part must be carried
675+
// along; dropping it puts a DATETIME(6) row in a different bucket than Spark does.
676+
// `unix_timestamp()` returns whole seconds floored towards negative infinity, so
677+
// adding the wall-clock microseconds is exact before the epoch as well.
678+
Int64 long_value = static_cast<Int64>(timestamp) * 1000000 + value.microsecond();
695679
uint32_t hash_value = HashUtil::murmur_hash3_32(&long_value, sizeof(long_value), 0);
696680

697681
*p_out = (hash_value & INT32_MAX) % _bucket_num;

‎be/src/format/arrow/arrow_stream_reader.cpp‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,10 +114,15 @@ Status ArrowStreamReader::_do_get_next_block(Block* block, size_t* read_rows, bo
114114
column_name_in_block, column_name);
115115
}
116116

117-
RETURN_IF_ERROR(
117+
auto status =
118118
columns_guard.get_datatype_by_position(c)
119119
->get_serde()
120-
->read_column_from_arrow(*columns[c], column, 0, num_rows, _ctzz));
120+
->read_column_from_arrow(*columns[c], column, 0, num_rows, _ctzz);
121+
if (!status.ok()) {
122+
// The SerDe sees only a value buffer, so it cannot name the column itself.
123+
return status.prepend(
124+
fmt::format("Failed to read arrow column '{}': ", column_name));
125+
}
121126
} catch (Exception& e) {
122127
return Status::InternalError("Failed to convert from arrow to block: {}", e.what());
123128
}

‎be/src/format/table/remote_doris_reader.cpp‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -86,10 +86,15 @@ Status RemoteDorisReader::_do_get_next_block(Block* block, size_t* read_rows, bo
8686

8787
try {
8888
auto block_pos = (*_col_name_to_block_idx)[column_name];
89-
RETURN_IF_ERROR(columns_guard.get_datatype_by_position(block_pos)
90-
->get_serde()
91-
->read_column_from_arrow(*columns[block_pos], column, 0,
92-
num_rows, _ctzz));
89+
auto status = columns_guard.get_datatype_by_position(block_pos)
90+
->get_serde()
91+
->read_column_from_arrow(*columns[block_pos], column, 0, num_rows,
92+
_ctzz);
93+
if (!status.ok()) {
94+
// The SerDe sees only a value buffer, so it cannot name the column itself.
95+
return status.prepend(
96+
fmt::format("Failed to read arrow column '{}': ", column_name));
97+
}
9398
} catch (Exception& e) {
9499
return Status::InternalError(
95100
"Failed to convert from arrow to block, column_name: {}, e: {}", column_name,

‎be/src/format_v2/orc/orc_reader.cpp‎

Lines changed: 29 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -486,10 +486,26 @@ bool set_date_zone_map(const ::orc::ColumnStatistics& statistics, segment_v2::Zo
486486
!date_statistics->hasMaximum()) {
487487
return false;
488488
}
489-
auto& date_dict = date_day_offset_dict::get();
490-
return set_validated_zone_map(
491-
Field::create_field<TYPE_DATEV2>(date_dict[date_statistics->getMinimum()]),
492-
Field::create_field<TYPE_DATEV2>(date_dict[date_statistics->getMaximum()]), zone_map);
489+
// ORC DATE statistics are proleptic-Gregorian day ordinals, the same domain the row decoder
490+
// (DataTypeDateV2SerDe::read_column_from_orc) interprets. Converting them through
491+
// `date_day_offset_dict` instead would put the year-zero window one day off the rows and let a
492+
// pushed-down MIN/MAX report a value no row holds. A bound with no Doris DATE disables the
493+
// statistics, so MIN/MAX falls back to scanning rows.
494+
const auto to_date = [](int64_t epoch_days) -> std::optional<DateV2Value<DateV2ValueType>> {
495+
const int64_t daynr = epoch_days_to_daynr(epoch_days);
496+
DateV2Value<DateV2ValueType> value;
497+
if (daynr == 0 || !value.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
498+
return std::nullopt;
499+
}
500+
return value;
501+
};
502+
const auto min_value = to_date(date_statistics->getMinimum());
503+
const auto max_value = to_date(date_statistics->getMaximum());
504+
if (!min_value.has_value() || !max_value.has_value()) {
505+
return false;
506+
}
507+
return set_validated_zone_map(Field::create_field<TYPE_DATEV2>(*min_value),
508+
Field::create_field<TYPE_DATEV2>(*max_value), zone_map);
493509
}
494510

495511
std::optional<DateV2Value<DateTimeV2ValueType>> datetime_v2_from_orc_millis(
@@ -2247,8 +2263,15 @@ Status OrcReader::_decode_column_into_block(const ::orc::StructVectorBatch& stru
22472263
const auto* selected_type = _state->selected_type->getSubtype(selected_batch_idx);
22482264
DORIS_CHECK(selected_type != nullptr);
22492265
auto column = file_block->get_by_position(block_position.value()).column->assert_mutable();
2250-
RETURN_IF_ERROR(_decode_column(*type, *selected_type, *struct_batch.fields[selected_batch_idx],
2251-
column, rows, selected_rows));
2266+
auto status = _decode_column(*type, *selected_type, *struct_batch.fields[selected_batch_idx],
2267+
column, rows, selected_rows);
2268+
if (!status.ok()) {
2269+
// The decoders work on a bare value buffer and cannot name the column themselves; without
2270+
// this the user only learns that some DATE/TIMESTAMP in some file is unrepresentable.
2271+
return status.prepend(fmt::format(
2272+
"Failed to decode ORC column '{}': ",
2273+
_state->root_type->getFieldName(static_cast<uint64_t>(file_column_id.value()))));
2274+
}
22522275
file_block->replace_by_position(block_position.value(), std::move(column));
22532276
return Status::OK();
22542277
}

0 commit comments

Comments
 (0)