Repository navigation
Commit 6714c7f
committed
[fix](be) Reject Arrow Date32 values outside the Doris DATE range
### What problem does this PR solve?
Issue Number: None
Related PR: #69581, #68649
Problem Summary:
Reproduction: read an Arrow Date32 array holding a day outside
0000-01-01..9999-12-31 into a Doris DATE column, for example 2932897
(10000-01-01) written by PyArrow into a Lance dataset, or the same value
returned by an ADBC driver, an Arrow stream load or the paimon-rust reader.
The row comes back as a non-null `0000-00-00`, also inside ARRAY, STRUCT and
MAP values.
Root cause: the DATE32 branch of the shared
`DataTypeDateV2SerDe::read_column_from_arrow` ignores the bool returned by
`get_date_from_daynr()` (dropped since #48944) and appends the
default-constructed value. It also adds the epoch offset in 32 bits, so
INT32_MAX overflows, and it decodes the payload of NULL slots. The DATE64
branch next to it already rejects out-of-range days.
Fix: port the DATE32 range part of the branch-4.1/4.2 fix #68649. NULL slots
append the default value without being decoded, the day number is computed
in 64 bits, and a day number <= 0, above DATE_MAX_DAYNR or rejected by
`get_date_from_daynr()` returns `InvalidArgument("Arrow Date32 value is
outside the Doris DATE range: row=..., days=...")` with the 4.x wording.
ARRAY/STRUCT/MAP leaves get the same check through their element SerDes.
`write_column_to_arrow` and the year-zero calendar mapping (the other part of
#68649, and master draft #67449) are unchanged; the new tests avoid the days
whose meaning that mapping changes.
Before: out-of-range days read as `0000-00-00`. After: the read fails with
the error above; in-range values and NULLs read as before.
The Lance reader ported in #69581 (from the branch-4.2 snapshot, which
already contains #68649) relies on this SerDe check.
### Release note
Reading Arrow DATE32 values outside 0000-01-01..9999-12-31 (ADBC, Arrow
stream load, paimon-rust, Lance) now fails instead of returning 0000-00-00.
### Check List (For Author)
- Test: Unit Test
- New: `DataTypeDateTimeV2SerDeTest.RejectsInvalidDate32Values`,
`LanceTableReaderTypeTest.RejectsDate32OutsideDorisDateRange`,
`LanceTableReaderTypeTest.ReadsLastDorisDateAndIgnoresNullDate32Payload`.
- All pass on macOS arm64 (ASAN), together with `DataTypeDateTimeV2SerDeTest.*`,
`DataTypeSerDeArrowTest.*`, `DataTypeSerDeArrowValidationTest.*`,
`ArrowColumnToDorisColumnTest.*`, `ParquetExprTest.*`,
`IcebergV2ReaderTest.*` and the Lance suites. With master's DATE32 branch
restored, the first two fail.
- `DataTypeDateTimeV2SerDeTest` aborts in its setup under macOS ASAN on
master: `load_data_from_csv()` in `be/test/testutil/test_util.cpp`
dereferences `std::set::end()`. It ran here with that assertion removed
locally; that fix goes in a separate PR.
- Behavior changed: Yes. An Arrow DATE32 day outside 0000-01-01..9999-12-31
used to be returned as a non-null 0000-00-00; the scan now fails with an
InvalidArgument error, as Arrow DATE64 values already did.
- Does this need documentation: No
Co-authored-by: Gabriel <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent a2914ae commit 6714c7f
3 files changed
Lines changed: 158 additions & 1 deletion
File tree
- be
- src/core/data_type_serde
- test
- core/data_type_serde
- format_v2/table
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
277 | 277 | | |
278 | 278 | | |
279 | 279 | | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
280 | 285 | | |
281 | 286 | | |
282 | 287 | | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
283 | 294 | | |
284 | | - | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
285 | 301 | | |
286 | 302 | | |
287 | 303 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
388 | 388 | | |
389 | 389 | | |
390 | 390 | | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
391 | 449 | | |
392 | 450 | | |
393 | 451 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2893 | 2893 | | |
2894 | 2894 | | |
2895 | 2895 | | |
| 2896 | + | |
| 2897 | + | |
| 2898 | + | |
| 2899 | + | |
| 2900 | + | |
| 2901 | + | |
| 2902 | + | |
| 2903 | + | |
| 2904 | + | |
| 2905 | + | |
| 2906 | + | |
| 2907 | + | |
| 2908 | + | |
| 2909 | + | |
| 2910 | + | |
| 2911 | + | |
| 2912 | + | |
| 2913 | + | |
| 2914 | + | |
| 2915 | + | |
| 2916 | + | |
| 2917 | + | |
| 2918 | + | |
| 2919 | + | |
| 2920 | + | |
| 2921 | + | |
| 2922 | + | |
| 2923 | + | |
| 2924 | + | |
| 2925 | + | |
| 2926 | + | |
| 2927 | + | |
| 2928 | + | |
| 2929 | + | |
| 2930 | + | |
| 2931 | + | |
| 2932 | + | |
| 2933 | + | |
| 2934 | + | |
| 2935 | + | |
| 2936 | + | |
| 2937 | + | |
| 2938 | + | |
| 2939 | + | |
| 2940 | + | |
| 2941 | + | |
| 2942 | + | |
| 2943 | + | |
| 2944 | + | |
| 2945 | + | |
| 2946 | + | |
| 2947 | + | |
| 2948 | + | |
| 2949 | + | |
| 2950 | + | |
| 2951 | + | |
| 2952 | + | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
| 2956 | + | |
| 2957 | + | |
| 2958 | + | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
| 2969 | + | |
| 2970 | + | |
| 2971 | + | |
| 2972 | + | |
| 2973 | + | |
| 2974 | + | |
| 2975 | + | |
| 2976 | + | |
| 2977 | + | |
| 2978 | + | |
2896 | 2979 | | |
2897 | 2980 | | |
2898 | 2981 | | |
| |||
0 commit comments