You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3fd801d
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](parquet) accept year-zero DATETIME on the Parquet read path
### What problem does this PR solve?
Issue Number: closeapache#67447
Related PR: apache#67449
Problem Summary:
Doris DATETIME starts at 0000-01-01 00:00:00, but the Parquet reader gated every
timestamp at 0001-01-01, so a value Doris accepts, stores and exports could not be read
back out of Doris's own file. Two independent defects sat behind that.
MIN_DORIS_TIMESTAMP_MICROS was a whole year narrower than the type it materialises into:
0000-01-01 is 366 days below 0001-01-01, not 365, because year 0 IS a leap year in the
proleptic Gregorian calendar that Parquet timestamps are defined in. The bound was also
applied to the raw instant, before the timezone offset, so it lost representable values
at both ends -- east of UTC a local 0001-01-01 00:00:00 is an instant below the civil
minimum, and west of it a local 9999-12-31 23:59:59.999999 is one above the maximum.
What stays in the shared helper is only a coarse guard (one day of slack, more than any
real offset), keeping the unit-overflow and malformed-INT96 checks format-level; the
exact civil range is now enforced per target type after the conversion.
append_datetimev2_from_epoch_micros() then added calc_daynr(1970, 1, 1) straight to a
proleptic Gregorian day ordinal. Doris follows MySQL's calendar, in which year 0 is not
a leap year, so the two numberings disagree over 0000-01-01 .. 0000-02-28: 0000-01-01
computed daynr 0 and was rejected outright, and the rest of that window would have
decoded one day early. Route it through epoch_days_to_daynr(), the helper the DATE
reader already uses, which also rejects the proleptic-only 0000-02-29 rather than
colliding it with 0000-02-28.
TIMESTAMPTZ shares the same storage and the same helper, and both its error text and the
type's own comment still advertised a year-one floor.
The legacy scanner's zero-date compatibility for genuinely unrepresentable values is left
alone: after this change year zero no longer reaches that path at all.
A new SerDe test checks every representable day of year zero against cctz rather than
against Doris's own day arithmetic, plus the two offset edges, INT96, the millis unit,
plain, dictionary, decoded-value and raw-predicate materialization, and TIMESTAMPTZ.
Reverting the three source files turns 12 of these red and leaves the two
"still rejects out-of-range" cases green, so they cannot pass vacuously.
End to end on a local cluster: the extended Iceberg suite now compares timestamp values,
not just partition tuples, against Spark 4.0.0 in both directions (Doris-written read by
both engines, Spark-written read by Doris) -- Iceberg `timestamp` is not adjusted to UTC,
so it exercises the civil path. The OUTFILE round trip was verified over file:// plus
local(), the reproduction from the report, across three session timezones including the
two that straddle the range edges, plus the counts, the ORC control and INSERT ... SELECT.
That run also showed the legacy scanner (enable_file_scanner_v2=false) returns NULL for
any pre-epoch timestamp with a sub-second part, year zero or not, because it truncates a
negative epoch value towards zero instead of flooring it. That is a separate defect in a
path this change does not touch, so the new suite deliberately does not assert the two
scanners agree.
### Release note
Doris can now read back the year-zero DATETIME values it accepts, stores and
exports:
- The Parquet reader accepted no timestamp below 0001-01-01, so a DATETIME in
0000-01-01 .. 0000-12-31 written by Doris itself came back as NULL (or failed
in strict mode). It is now read correctly, and the civil range is enforced
after the timezone offset instead of before it, which also recovers the values
at both ends of the range that a non-UTC session timezone used to lose.
- The 0000-01-01 .. 0000-02-28 window decoded one day early and 0000-01-01 was
rejected outright, because a proleptic Gregorian day ordinal was added to
Doris's MySQL day number; both now decode correctly, and the proleptic-only
0000-02-29 is rejected instead of colliding with 0000-02-28.
- TIMESTAMPTZ shares the same storage and helper and gains the same range.
### Check List (For Author)
- Test: Regression test + Unit Test
- BE UT: data_type_datetimev2_serde_calendar_test (every representable day of
year zero against cctz, both offset edges, INT96, the millis unit, plain,
dictionary, decoded-value and raw-predicate materialization, TIMESTAMPTZ),
data_type_serde_parquet_test
- Regression: export_p0/outfile/test_outfile_datetime_year_zero,
external_table_p0/iceberg/write/test_iceberg_write_partition_epoch_boundary
- Manual: local cluster, OUTFILE round trip over file:// and local(), three
session timezones, INSERT ... SELECT from the exported file
- Behavior changed: Yes (see the Release note: year-zero DATETIME/TIMESTAMPTZ
values that used to read back as NULL are now returned)
- Does this need documentation: No (no document states the DATETIME lower bound
of the Parquet read path)
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
0 commit comments