Repository navigation
Commit 6c1fd8f
[fix](be) Keep a runtime calendar check after packing Arrow timestamps
### What problem does this PR solve?
Issue Number: None
Related PR: #69697
Problem Summary: The range check added before packing an Arrow timestamp
does not make every instant inside the range a Doris date. cctz counts
year 0 as a leap year and Doris does not (is_leap(0) in util/time_lut.h is
false), so timestamp(us) -62162121600000000, 0000-02-29 00:00:00 UTC, is
inside 0000-01-01Z..9999-12-31Z but packs into a value that
is_valid_date() rejects. For TIMESTAMPTZ this PR had turned the runtime
is_valid_date() check after packing into a DCHECK, so that input aborted
an ASAN or debug BE and appended an invalid TIMESTAMPTZ in a release
build, where master returns DataQualityError. The DATETIMEV2 Arrow path
checks only the civil year, so the same date at the session time zone was
stored as an invalid DATETIMEV2, as it was before this PR.
Both paths now check is_valid_date() at run time after packing and return
DataQualityError for such a value. The ORC path
(orc_timestamp_to_datetime) has the same year-only check and is not
changed here.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- BE UT RejectsArrowTimestampsOutsideTimestampTzRange and
RejectsArrowTimestampsOutsideDateTimeV2Range read
-62162121600000000 and expect DataQualityError; both fail when the
new checks skip that value. The serde, Arrow round-trip, nested
serde and Lance reader suites (168 tests) pass.
- Behavior changed: Yes. An Arrow timestamp on 0000-02-29 is rejected
instead of being stored as an invalid date.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>1 parent 3f336d2 commit 6c1fd8f
4 files changed
Lines changed: 24 additions & 1 deletion
File tree
- be
- src/core/data_type_serde
- test/core/data_type_serde
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
739 | 739 | | |
740 | 740 | | |
741 | 741 | | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
742 | 749 | | |
743 | 750 | | |
744 | 751 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
106 | | - | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
107 | 113 | | |
108 | 114 | | |
109 | 115 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
626 | 626 | | |
627 | 627 | | |
628 | 628 | | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
629 | 634 | | |
630 | 635 | | |
631 | 636 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
428 | 428 | | |
429 | 429 | | |
430 | 430 | | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
431 | 436 | | |
432 | 437 | | |
0 commit comments