Skip to content

Commit 6714c7f

Browse files
committed
[fix](be) Reject Arrow Date32 values outside the Doris DATE range
### What problem does this PR solve? Issue Number: None Related PR: #69581, #68649 Problem Summary: Reproduction: read an Arrow Date32 array holding a day outside 0000-01-01..9999-12-31 into a Doris DATE column, for example 2932897 (10000-01-01) written by PyArrow into a Lance dataset, or the same value returned by an ADBC driver, an Arrow stream load or the paimon-rust reader. The row comes back as a non-null `0000-00-00`, also inside ARRAY, STRUCT and MAP values. Root cause: the DATE32 branch of the shared `DataTypeDateV2SerDe::read_column_from_arrow` ignores the bool returned by `get_date_from_daynr()` (dropped since #48944) and appends the default-constructed value. It also adds the epoch offset in 32 bits, so INT32_MAX overflows, and it decodes the payload of NULL slots. The DATE64 branch next to it already rejects out-of-range days. Fix: port the DATE32 range part of the branch-4.1/4.2 fix #68649. NULL slots append the default value without being decoded, the day number is computed in 64 bits, and a day number <= 0, above DATE_MAX_DAYNR or rejected by `get_date_from_daynr()` returns `InvalidArgument("Arrow Date32 value is outside the Doris DATE range: row=..., days=...")` with the 4.x wording. ARRAY/STRUCT/MAP leaves get the same check through their element SerDes. `write_column_to_arrow` and the year-zero calendar mapping (the other part of #68649, and master draft #67449) are unchanged; the new tests avoid the days whose meaning that mapping changes. Before: out-of-range days read as `0000-00-00`. After: the read fails with the error above; in-range values and NULLs read as before. The Lance reader ported in #69581 (from the branch-4.2 snapshot, which already contains #68649) relies on this SerDe check. ### Release note Reading Arrow DATE32 values outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) now fails instead of returning 0000-00-00. ### Check List (For Author) - Test: Unit Test - New: `DataTypeDateTimeV2SerDeTest.RejectsInvalidDate32Values`, `LanceTableReaderTypeTest.RejectsDate32OutsideDorisDateRange`, `LanceTableReaderTypeTest.ReadsLastDorisDateAndIgnoresNullDate32Payload`. - All pass on macOS arm64 (ASAN), together with `DataTypeDateTimeV2SerDeTest.*`, `DataTypeSerDeArrowTest.*`, `DataTypeSerDeArrowValidationTest.*`, `ArrowColumnToDorisColumnTest.*`, `ParquetExprTest.*`, `IcebergV2ReaderTest.*` and the Lance suites. With master's DATE32 branch restored, the first two fail. - `DataTypeDateTimeV2SerDeTest` aborts in its setup under macOS ASAN on master: `load_data_from_csv()` in `be/test/testutil/test_util.cpp` dereferences `std::set::end()`. It ran here with that assertion removed locally; that fix goes in a separate PR. - Behavior changed: Yes. An Arrow DATE32 day outside 0000-01-01..9999-12-31 used to be returned as a non-null 0000-00-00; the scan now fails with an InvalidArgument error, as Arrow DATE64 values already did. - Does this need documentation: No Co-authored-by: Gabriel <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent a2914ae commit 6714c7f

3 files changed

Lines changed: 158 additions & 1 deletion

File tree

‎be/src/core/data_type_serde/data_type_datev2_serde.cpp‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -277,11 +277,27 @@ Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
277277
const auto* base_ptr = reinterpret_cast<const uint8_t*>(concrete_array->raw_values());
278278
const size_t element_size = sizeof(int32_t);
279279
for (auto value_i = start; value_i < end; ++value_i) {
280+
// A null slot has no calendar value and its payload may be outside the DATE range.
281+
if (concrete_array->IsNull(value_i)) {
282+
col_data.emplace_back(DateV2Value<DateV2ValueType>());
283+
continue;
284+
}
280285
const uint8_t* raw_byte_ptr = base_ptr + value_i * element_size;
281286
auto date_value = unaligned_load<int32_t>(raw_byte_ptr);
282287

288+
// Arrow allows any int32 day, Doris DATE only day numbers 1..DATE_MAX_DAYNR
289+
// (0000-01-01..9999-12-31). For example, 2932897 is 10000-01-01: its day number
290+
// 3652425 makes get_date_from_daynr() fail, and ignoring that result used to append
291+
// a non-null 0000-00-00. Add in 64 bits so INT32_MAX cannot overflow, and reject
292+
// the value like the DATE64 branch above.
293+
const int64_t daynr = static_cast<int64_t>(date_value) + date_threshold;
283294
DateV2Value<DateV2ValueType> v;
284-
v.get_date_from_daynr(date_value + date_threshold);
295+
if (daynr <= 0 || daynr > DATE_MAX_DAYNR ||
296+
!v.get_date_from_daynr(static_cast<uint64_t>(daynr))) {
297+
return Status::InvalidArgument(
298+
"Arrow Date32 value is outside the Doris DATE range: row={}, days={}",
299+
value_i, date_value);
300+
}
285301
col_data.emplace_back(v);
286302
}
287303
} else {

‎be/test/core/data_type_serde/data_type_serde_datetime_v2_test.cpp‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,64 @@ TEST_F(DataTypeDateTimeV2SerDeTest, RejectsInvalidDate64Values) {
388388
expect_invalid(out_of_range, "outside the Doris DATE range");
389389
}
390390

391+
TEST_F(DataTypeDateTimeV2SerDeTest, RejectsInvalidDate32Values) {
392+
const auto make_date32 = [](int32_t days) {
393+
arrow::Date32Builder builder;
394+
EXPECT_TRUE(builder.Append(days).ok());
395+
std::shared_ptr<arrow::Array> array;
396+
EXPECT_TRUE(builder.Finish(&array).ok());
397+
return array;
398+
};
399+
cctz::time_zone tz;
400+
401+
// 1970-01-01 + 2932896 days is 9999-12-31, the last Doris DATE.
402+
auto last_date = ColumnDateV2::create();
403+
auto status =
404+
serde_date_v2->read_column_from_arrow(*last_date, make_date32(2932896).get(), 0, 1, tz);
405+
ASSERT_TRUE(status.ok()) << status.to_string();
406+
ASSERT_EQ(1, last_date->size());
407+
EXPECT_EQ(9999, last_date->get_data()[0].year());
408+
EXPECT_EQ(12, last_date->get_data()[0].month());
409+
EXPECT_EQ(31, last_date->get_data()[0].day());
410+
411+
// 10000-01-01, -719529 (the day before 0000-01-01 in Arrow's proleptic Gregorian calendar)
412+
// and both int32 extremes used to read as a non-null 0000-00-00; INT32_MAX also overflowed
413+
// the 32-bit day number.
414+
for (const int32_t days : {2932897, -719529, std::numeric_limits<int32_t>::min(),
415+
std::numeric_limits<int32_t>::max()}) {
416+
SCOPED_TRACE(days);
417+
auto column = ColumnDateV2::create();
418+
status = serde_date_v2->read_column_from_arrow(*column, make_date32(days).get(), 0, 1, tz);
419+
EXPECT_FALSE(status.ok());
420+
EXPECT_NE(std::string::npos,
421+
status.to_string().find("Arrow Date32 value is outside the Doris DATE range"))
422+
<< status.to_string();
423+
}
424+
425+
// A NULL slot keeps whatever payload its writer left; it must not be decoded. Validity
426+
// bitmap 0b10 marks row 0 NULL over the out-of-range payload and row 1 valid.
427+
arrow::Date32Builder builder;
428+
ASSERT_TRUE(builder.AppendValues({2932897, 0}).ok());
429+
std::shared_ptr<arrow::Array> payload_array;
430+
ASSERT_TRUE(builder.Finish(&payload_array).ok());
431+
auto data = payload_array->data()->Copy();
432+
data->buffers[0] = arrow::Buffer::FromVector(std::vector<uint8_t> {0x02});
433+
data->SetNullCount(1);
434+
const auto nullable_array = arrow::MakeArray(data);
435+
DataTypeNullableSerDe nullable_serde(serde_date_v2);
436+
auto nullable_column = ColumnNullable::create(ColumnDateV2::create(), ColumnUInt8::create());
437+
status =
438+
nullable_serde.read_column_from_arrow(*nullable_column, nullable_array.get(), 0, 2, tz);
439+
ASSERT_TRUE(status.ok()) << status.to_string();
440+
ASSERT_EQ(2, nullable_column->size());
441+
EXPECT_TRUE(nullable_column->is_null_at(0));
442+
EXPECT_FALSE(nullable_column->is_null_at(1));
443+
const auto& dates = assert_cast<const ColumnDateV2&>(nullable_column->get_nested_column());
444+
EXPECT_EQ(1970, dates.get_data()[1].year());
445+
EXPECT_EQ(1, dates.get_data()[1].month());
446+
EXPECT_EQ(1, dates.get_data()[1].day());
447+
}
448+
391449
TEST_F(DataTypeDateTimeV2SerDeTest, ArrowTimeToTimeV2) {
392450
const auto read_time = [](arrow::ArrayBuilder* builder, int scale,
393451
const std::vector<double>& expected) {

‎be/test/format_v2/table/lance_reader_test.cpp‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2893,6 +2893,89 @@ TEST(LanceTableReaderTypeTest, ReadsDurationBoundaryValues) {
28932893
EXPECT_TRUE(reader.close().ok());
28942894
}
28952895

2896+
// Verifies Date32 days after 9999-12-31 fail instead of reading as a non-null 0000-00-00, also
2897+
// when ARRAY, STRUCT, and MAP leaves reach the DATE SerDe through their element SerDes.
2898+
TEST(LanceTableReaderTypeTest, RejectsDate32OutsideDorisDateRange) {
2899+
// 1970-01-01 + 2932897 days is 10000-01-01.
2900+
arrow::Date32Builder date_builder;
2901+
ASSERT_TRUE(date_builder.Append(2932897).ok());
2902+
std::shared_ptr<arrow::Array> dates;
2903+
ASSERT_TRUE(date_builder.Finish(&dates).ok());
2904+
arrow::Int32Builder offsets_builder;
2905+
ASSERT_TRUE(offsets_builder.AppendValues({0, 1}).ok());
2906+
std::shared_ptr<arrow::Int32Array> offsets;
2907+
ASSERT_TRUE(offsets_builder.Finish(&offsets).ok());
2908+
arrow::StringBuilder keys_builder;
2909+
ASSERT_TRUE(keys_builder.Append("k").ok());
2910+
std::shared_ptr<arrow::StringArray> keys;
2911+
ASSERT_TRUE(keys_builder.Finish(&keys).ok());
2912+
2913+
auto list_result = arrow::ListArray::FromArrays(
2914+
arrow::list(arrow::field("item", arrow::date32())), *offsets, *dates);
2915+
ASSERT_TRUE(list_result.ok()) << list_result.status().ToString();
2916+
auto struct_result = arrow::StructArray::Make({dates}, {arrow::field("d", arrow::date32())});
2917+
ASSERT_TRUE(struct_result.ok()) << struct_result.status().ToString();
2918+
auto map_result = arrow::MapArray::FromArrays(arrow::map(arrow::utf8(), arrow::date32()),
2919+
offsets, keys, dates);
2920+
ASSERT_TRUE(map_result.ok()) << map_result.status().ToString();
2921+
2922+
const auto date_type = nullable_type(TYPE_DATEV2);
2923+
const std::vector<std::pair<std::shared_ptr<arrow::Array>, DataTypePtr>> cases {
2924+
{dates, date_type},
2925+
{std::move(list_result).ValueUnsafe(),
2926+
make_nullable(std::make_shared<DataTypeArray>(date_type))},
2927+
{std::move(struct_result).ValueUnsafe(),
2928+
make_nullable(std::make_shared<DataTypeStruct>(DataTypes {date_type}, Strings {"d"}))},
2929+
{std::move(map_result).ValueUnsafe(),
2930+
make_nullable(std::make_shared<DataTypeMap>(nullable_type(TYPE_STRING), date_type))},
2931+
};
2932+
for (const auto& [array, doris_type] : cases) {
2933+
SCOPED_TRACE(array->type()->ToString());
2934+
const auto schema = arrow::schema({arrow::field("value", array->type())});
2935+
const auto record_batch = arrow::RecordBatch::Make(schema, 1, {array});
2936+
const Columns columns {projected_column("value", doris_type)};
2937+
TQueryGlobals query_globals;
2938+
RuntimeState state(TQueryOptions(), query_globals);
2939+
Block block;
2940+
add_output_columns(&block, columns);
2941+
size_t rows = 0;
2942+
const auto status =
2943+
convert_record_batch_for_test(&state, columns, record_batch, &block, &rows);
2944+
EXPECT_FALSE(status.ok());
2945+
EXPECT_NE(std::string::npos,
2946+
status.to_string().find("Arrow Date32 value is outside the Doris DATE range"))
2947+
<< status.to_string();
2948+
}
2949+
}
2950+
2951+
// Verifies the last Doris DATE reads exactly and the payload under a NULL slot is not decoded.
2952+
TEST(LanceTableReaderTypeTest, ReadsLastDorisDateAndIgnoresNullDate32Payload) {
2953+
arrow::Date32Builder builder;
2954+
ASSERT_TRUE(builder.AppendValues({2932896, 2932897}).ok());
2955+
std::shared_ptr<arrow::Array> payload;
2956+
ASSERT_TRUE(builder.Finish(&payload).ok());
2957+
// Validity bitmap 0b01: row 0 holds 9999-12-31, row 1 is NULL over 10000-01-01.
2958+
auto data = payload->data()->Copy();
2959+
data->buffers[0] = arrow::Buffer::FromVector(std::vector<uint8_t> {0x01});
2960+
data->SetNullCount(1);
2961+
const auto schema = arrow::schema({arrow::field("value", arrow::date32())});
2962+
const auto record_batch = arrow::RecordBatch::Make(schema, 2, {arrow::MakeArray(data)});
2963+
2964+
const auto date_type = nullable_type(TYPE_DATEV2);
2965+
const Columns columns {projected_column("value", date_type)};
2966+
TQueryGlobals query_globals;
2967+
RuntimeState state(TQueryOptions(), query_globals);
2968+
Block block;
2969+
add_output_columns(&block, columns);
2970+
size_t rows = 0;
2971+
const auto status = convert_record_batch_for_test(&state, columns, record_batch, &block, &rows);
2972+
ASSERT_TRUE(status.ok()) << status.to_string();
2973+
ASSERT_EQ(2, rows);
2974+
const auto& values = assert_cast<const ColumnNullable&>(*block.get_by_position(0).column);
2975+
EXPECT_EQ((ColumnUInt8::Container {0, 1}), values.get_null_map_data());
2976+
EXPECT_EQ("9999-12-31", date_type->to_string(values, 0));
2977+
}
2978+
28962979
// Verifies ordinary nested columns bypass Arrow reconstruction when no BFloat16 exists.
28972980
TEST(LanceTableReaderTypeTest, KeepsOrdinaryNestedArrayUnchangedDuringNormalization) {
28982981
const auto nested_type = arrow::struct_({

0 commit comments

Comments
 (0)