Repository navigation
chore: Update to arrow/parquet 60.1.0 - #26867
Conversation
|
|
||
| fn value_offsets(&self, row: usize) -> (i64, i64) { | ||
| let start = self.value_offset(row) as i64; | ||
| let start = self.value_offset_at(row) as i64; |
There was a problem hiding this comment.
| // while spill_file_count remains 1 (since we're writing to the same file) | ||
| in_progress_file.append_batch(&batch1)?; | ||
| verify_metrics(&in_progress_file, 1, 440, 3)?; | ||
| verify_metrics(&in_progress_file, 1, 424, 3)?; |
There was a problem hiding this comment.
| .map(|row| (input * rows_per_input + row) % dictionary_len), | ||
| ); | ||
| take(&dictionary, &indices, None) | ||
| let taken = take(&dictionary, &indices, None)?; |
There was a problem hiding this comment.
ce310c5 to
5a12062
Compare
Co-Authored-By: Claude Fable 5.1 <[email protected]>
| ## all. Every route into a named zone -- a column cast, `AT TIME ZONE`, a | ||
| ## string literal, or `::timestamptz` under a session zone -- fails with an | ||
| ## error, where PostgreSQL resolves the value. | ||
| ## Casting a naive timestamp into a named zone resolves an ambiguous or |
There was a problem hiding this comment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
| 01)ProjectionExec: expr=[count(Int64(1))@0 as count(*)], metrics=[<slt:ignore>] | ||
| 02)--AggregateExec: mode=Single, gby=[], aggr=[count(Int64(1))], metrics=[<slt:ignore>] | ||
| 03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=8, input_rows=516, <slt:ignore>] | ||
| 03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=6, input_rows=387, <slt:ignore>] |
There was a problem hiding this comment.
Due to
The right side is replayed once per left chunk. Spilled batches are now read back without validity buffers for null-free columns, so they take less memory and the build side fits in three chunks instead of four.
| SELECT approx_distinct(c14) AS a, approx_distinct(c15) AS b, approx_distinct(arrow_cast(c15, 'Date64')), approx_distinct(arrow_cast(c15, 'Time32(Second)')) as c, approx_distinct(arrow_cast(c15, 'Time64(Nanosecond)')) AS d FROM aggregate_test_100 | ||
| ---- | ||
| 18 60 60 60 60 | ||
| 18 60 7 60 60 |
There was a problem hiding this comment.
Due to
Casting a Timestamp to Date64 now truncates to whole days (previously it kept the time of day), so only 7 distinct dates remain.
| FROM (VALUES (arrow_cast(9223372036854775807, 'Timestamp(Second, None)')), (NULL)) AS t(ts) | ||
| ---- | ||
| NULL NULL NULL NULL | ||
| 7 NULL NULL NULL |
There was a problem hiding this comment.
Due to
second is now computed directly from the stored value for timestamps without a timezone,
i64::MAX seconds yields 7 (i64::MAX % 60) instead of overflowing to NULL.
The sub-second parts still go through chrono and remain NULL. The nullability reproducer further down now uses millisecond so it still produces a NULL from a non-null input.
| /// Arrays that each hold views into one shared dictionary buffer, as the | ||
| /// Parquet reader produces for a dictionary-encoded column, combined the | ||
| /// way a sort combines its input batches. Each output row cycles through | ||
| /// way a sort combines its input batches: the combined array lists the |
There was a problem hiding this comment.
note this is test code
| /// Parquet reader produces for a dictionary-encoded column, combined the | ||
| /// way a sort combines its input batches. Each output row cycles through | ||
| /// Parquet reader produces for a dictionary-encoded column, combined with | ||
| /// `concat` as a sort combines its buffered batches. The rows cycle through |
There was a problem hiding this comment.
interleave no longer duplicates the buffers
|
Ok,let's get this in |
## Which issue does this PR close? - Closes apache#26981. - Related to apache#26451, the same cause for timestamp → time (fixed on its own by apache#26452). ## Rationale for this change Parquet statistics pruning can drop row groups that contain matching rows when the predicate casts the column. With `i` in (-1, 0, 1, 255, 256) and naive timestamps `ts` at 00:00, 01:00, 02:30, 02:45 and 03:00 on 2024-03-10, all in one row group: | Query | In-memory table | Parquet | |---|---|---| | `SELECT count(*) FROM t WHERE CAST(i AS BOOLEAN) <= false` | 1 | 0 | | `SELECT count(*) FROM t WHERE CAST(i AS BYTEA) < X'05'` | 3 | 0 | | `SELECT count(*) FROM t WHERE ts > TIMESTAMPTZ '2024-03-10 07:15:00+00:00'` with `time_zone = 'America/New_York'` | 2 | 0 | Pruning casts the min/max statistics and compares them with the literal, which is only correct when the cast keeps the order of the values. Pruning decides that with its own check: any cast is allowed unless exactly one side is a string. So int → boolean, int → binary (little-endian bytes) and named-timezone timestamp → date all pass, although none of them keep order. The last row is new with arrow 60.1.0 (apache#26867). A naive timestamp compared with a named timezone is read as a local time, and arrow now moves a local time that a transition skips forward instead of rejecting it: 02:30, which New York skipped, becomes 07:30 UTC, after 03:00 (07:00 UTC). When such a value is the minimum of a row group, the cast bounds are inverted and the query fails with `Internal error: Interval's lower bound ... is greater than the upper bound`. Sort elimination already asks the same question through `is_order_preserving_cast`. The two checks disagree in both directions: pruning accepts the casts above, and `is_order_preserving_cast` rejects casts that do keep order and that pruning relies on today (integer ↔ date/timestamp/duration, string and binary re-encodings, date → timestamp with a timezone). Arrow 60.1.0 also made one of its own entries wrong: a named-timezone timestamp → `Date64` now takes the local date, as `Date32` does, so a sort on that cast can be removed although the output is out of order. ## What changes are included in this PR? 1. `is_order_preserving_cast` (now public in `datafusion_physical_expr::expressions`) accepts the casts that keep order and that only pruning knew about: - string ↔ string and binary ↔ binary re-encodings (including `FixedSizeBinary`) - numbers ↔ temporal types stored as a count (date, timestamp, time, duration) - `Date32` → timestamp with any timezone (a skipped midnight moves forward by at most a day, so it never passes the next one), and `Date64` → timestamp with a fixed offset - `Time64` → `Time32` and `Time64(µs)` → `Time64(ns)`, which Arrow 60.1.0 checks for overflow 2. It rejects named-timezone timestamp → `Date64`, as it already did for `Date32`. 3. Pruning (`verify_support_type_for_prune`) asks `is_order_preserving_cast` instead of its own string check. 4. A test that checks `is_order_preserving_cast` against the strict cast that `CastExpr` runs (`ColumnarValue::cast_to`): every pair it accepts must keep the order of sample values that convert. The samples include values around midnight, a DST gap and overlap, a timezone that moved the date backwards, and a day that Pacific/Apia skipped. A positive control asserts the samples expose the casts the rule rejects. Pruning keeps every other cast it used before (compared over all castable pairs of the numeric, string, binary and temporal types) and stops using: | Cast | Why | |---|---| | integer/float → boolean | -1 and 1 are true, 0 is false | | integer → binary | little-endian bytes do not sort like the integers | | named-timezone timestamp → `Date32`/`Date64` | a transition can move the local date backwards | | naive timestamp or `Date64` → named-timezone timestamp | arrow moves a skipped local time forward, past the local times just after the gap | | casts from or to `Null`, to list/struct, interval casts | the rule has no entry for them | ## What is the testing strategy for this PR? - `test_order_preserving_cast_matches_cast_kernel` (physical-expr): checks the rule against the strict cast, with a positive control. Making named-timezone timestamp → date return `true` fails it with `562129259 -> 1987-10-25, but 562129260 -> 1987-10-24`. - `row_group_predicate_cast_must_preserve_order` (pruning): casts that must not prune and casts that must keep pruning, for both `CAST` and `TRY_CAST`. - `push_down_filter_parquet.slt`: the three queries above against an in-memory table and Parquet, plus an `EXPLAIN` showing that the same timestamp comparison under a fixed-offset session timezone still prunes. - `cast.slt`: the sort on `arrow_cast(i, 'Timestamp(Second, None)')` is removed when the input is ordered by `i` (`EXPLAIN` and the row order), and the sort on a named-timezone timestamp cast to `Date64` is kept. - The new pruning unit test cases and Parquet queries fail with the old pruning check. The `Date64` sort case and the New York Parquet query return wrong results on main. - `bloom_filter_pruning_error_retains_row_group_without_match` relied on the int → `FixedSizeBinary` cast failing at evaluation. Pruning now rejects that cast up front, so the test compares `c1` with `CAST('abc' AS Int32)` instead, which still fails at evaluation. ## Are there any user-facing changes? - Queries that filter Parquet through the casts in the table above return the correct rows; those predicates no longer prune row groups. With a named session timezone, comparing a naive timestamp column with a timezone-aware timestamp no longer prunes; with a fixed offset it still does. - Sort elimination treats the newly accepted casts as order-preserving under `CAST` (not `TRY_CAST`), so a sort on, for example, `CAST(int_col AS TIMESTAMP)` over input ordered by `int_col` can be removed. A sort on a named-timezone timestamp cast to `Date64` is no longer removed. - New public function `datafusion_physical_expr::expressions::is_order_preserving_cast` (addition only).
Which issue does this PR close?
60.1.0(October 2026) arrow-rs#10525Rationale for this change
Update to the latest arrow/parquet release,
60.1.0. See the upstream CHANGELOG for details.What changes are included in this PR?
60.1.0What is the testing strategy for this PR?
Existing CI. This release candidate was also tested in #26543.
Are there any user-facing changes?
New dependency versions.