You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a5b8251
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](parquet) accept year-zero DATETIME on the Parquet read path
Doris DATETIME starts at 0000-01-01 00:00:00, but the Parquet reader gated every
timestamp at 0001-01-01, so a value Doris accepts, stores and exports could not be read
back out of Doris's own file. Two independent defects sat behind that.
MIN_DORIS_TIMESTAMP_MICROS was a whole year narrower than the type it materialises into:
0000-01-01 is 366 days below 0001-01-01, not 365, because year 0 IS a leap year in the
proleptic Gregorian calendar that Parquet timestamps are defined in. The bound was also
applied to the raw instant, before the timezone offset, so it lost representable values
at both ends -- east of UTC a local 0001-01-01 00:00:00 is an instant below the civil
minimum, and west of it a local 9999-12-31 23:59:59.999999 is one above the maximum.
What stays in the shared helper is only a coarse guard (one day of slack, more than any
real offset), keeping the unit-overflow and malformed-INT96 checks format-level; the
exact civil range is now enforced per target type after the conversion.
append_datetimev2_from_epoch_micros() then added calc_daynr(1970, 1, 1) straight to a
proleptic Gregorian day ordinal. Doris follows MySQL's calendar, in which year 0 is not
a leap year, so the two numberings disagree over 0000-01-01 .. 0000-02-28: 0000-01-01
computed daynr 0 and was rejected outright, and the rest of that window would have
decoded one day early. Route it through epoch_days_to_daynr(), the helper the DATE
reader already uses, which also rejects the proleptic-only 0000-02-29 rather than
colliding it with 0000-02-28.
TIMESTAMPTZ shares the same storage and the same helper, and both its error text and the
type's own comment still advertised a year-one floor.
The legacy scanner's zero-date compatibility for genuinely unrepresentable values is left
alone: after this change year zero no longer reaches that path at all.
A new SerDe test checks every representable day of year zero against cctz rather than
against Doris's own day arithmetic, plus the two offset edges, INT96, the millis unit,
plain, dictionary, decoded-value and raw-predicate materialization, and TIMESTAMPTZ.
Reverting the three source files turns 12 of these red and leaves the two
"still rejects out-of-range" cases green, so they cannot pass vacuously.
End to end on a local cluster: the extended Iceberg suite now compares timestamp values,
not just partition tuples, against Spark 4.0.0 in both directions (Doris-written read by
both engines, Spark-written read by Doris) -- Iceberg `timestamp` is not adjusted to UTC,
so it exercises the civil path. The OUTFILE round trip was verified over file:// plus
local(), the reproduction from the report, across three session timezones including the
two that straddle the range edges, plus the counts, the ORC control and INSERT ... SELECT.
That run also showed the legacy scanner (enable_file_scanner_v2=false) returns NULL for
any pre-epoch timestamp with a sub-second part, year zero or not, because it truncates a
negative epoch value towards zero instead of flooring it. That is a separate defect in a
path this change does not touch, so the new suite deliberately does not assert the two
scanners agree.
Fixesapache#67447
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
0 commit comments