You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 980f576
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](orc) decode ORC DATE statistics with the row decoder's calendar
### What problem does this PR solve?
Issue Number: closeapache#67366
Related PR: apache#67449
Problem Summary:
Review follow-up on apache#67449. The two commits before this one moved the DATE row
decoders onto the proleptic Gregorian ordinal but left three consumers of the
same ordinal behind, and the Iceberg transforms kept one pre-existing defect.
format_v2's ORC reader still converted DATE stripe statistics through
`date_day_offset_dict`, whose out-of-table fallback is the old
`1900-01-01 + (value + 25567)` arithmetic. Those zone maps have exactly one
consumer, `OrcReader::get_aggregate_result`, which answers a pushed-down
MIN/MAX from statistics without reading a row, so on the default path
(`enable_file_scanner_v2` and `enable_push_down_no_group_agg`) `MIN(d)`
reported 1900-01-01 for a file whose smallest row is 0000-01-01 -- a value
present in no row at all. Decode the bounds with the same conversion the rows
use and return false when a bound has no Doris DATE, which makes the reader fall
back to a normal row scan.
`ReadMapDecimalDateWithCenturyBoundary` pinned that dictionary fallback: it wrote
the ORC ordinal -719530 and expected 1900-01-01, which the new decoder rejects.
Point the fixture at 0000-01-01, the smallest representable DATE, and add a
dedicated test that the three unrepresentable shapes (-719530, the proleptic-only
-719469 and 2932897) fail the scan.
Iceberg's `bucket` over a timestamp hashes the full microsecond value
(iceberg-api `Bucket.BucketLong` over `BucketUtil.hash(long)`), but Doris hashed
whole seconds times a million, so a DATETIME(6) row landed in a different bucket
than the same row written by Spark and bucket pruning skipped the Doris-written
file. `human_year()` did not zero-pad, so a year-zero row's partition directory
was `..._year=0` where Spark writes `..._year=0000`.
On the FE side, `IcebergPartitionUtils` rendered MTMV partition-range bounds with
the pattern letter `y` (year-of-era), so the ordinal -719528 the `day` transform
now emits for 0000-01-01 rendered as 0001-01-01 and collided with the range of
the real 0001-01-01 partition.
The `ExpressionEstimation` hunk from the first commit is reverted: DATE/DATETIME
column statistics are `yyyyMMddHHmmss` numbers, but `getDatetimeFromLong()` reads
them as epoch seconds, so every value is >= 101000000 and the year-zero
correction could never fire. The real defect (the encoding mismatch) predates
this PR and belongs in its own change with its own test.
Also in this commit: the shared out-of-range message no longer says "Parquet" for
an ORC file and now carries the offending value, the ORC and Arrow callers
prepend the column name, the epoch-day constants are derived from the daynr
domain instead of typed out, the Arrow date64 branch spells its contract like the
date32 one, and `epoch_date()`/`epoch_datetime()` plus their two cast includes
are deleted (no caller left).
### Release note
- A pushed-down `MIN(date_col)` / `MAX(date_col)` over an ORC file now agrees
with the rows. It previously answered from stripe statistics decoded in Doris's
own calendar, so a file containing 0000-01-01 .. 0000-02-28 reported a bound
that no row holds (1900-01-01 for 0000-01-01).
- Iceberg `bucket(n, ts)` now hashes the full microsecond value, as the Iceberg
specification requires. A `DATETIME` column with a non-zero sub-second part
routes to a different (and now Spark-compatible) bucket than before; existing
files keep their old bucket values until they are rewritten.
- An Iceberg `year` partition directory is zero-padded (`..._year=0000`), which
is what Spark writes. Only the directory name changes; the manifest value is
unaffected.
- The MTMV partition-range view of an Iceberg table with year-zero `day`/`hour`
partitions renders 0000-01-01 instead of 0001-01-01.
- An unrepresentable DATE in a Parquet or ORC file is now reported with the
offending value and the column name, and no longer names the wrong format.
### Check List (For Author)
- Test: Regression test + Unit Test
- BE UT: orc_reader_test (year-zero rows, a null row, the three rejected
ordinals, MIN/MAX against the row decode, and the statistics fallback),
data_type_datev2_serde_calendar_test (the Parquet/ORC decoded-value entry
point, strict and null-on-failure), partition_transformers_test
(microsecond buckets, the zero-padded human year), vdatetime_value_test
- FE UT: IcebergPartitionUtilsTest (year-zero day and hour ranges)
- Regression: the ORC half of
external_table_p0/iceberg/write/test_iceberg_write_partition_epoch_boundary
compares a pushed-down MIN/MAX with the rows of the same file
- Behavior changed: Yes (see the Release note)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01LPwhYhsSio1HYk2kFnx7KY
0 commit comments