Skip to content

chore: Update to arrow/parquet 60.1.0 - #26867

Merged
alamb merged 12 commits into
apache:mainfrom
alamb:alamb/arrow_60.1.0
Nov 1, 2026
Merged

alamb merged 12 commits into
apache:mainfrom
alamb:alamb/arrow_60.1.0

Conversation

@alamb

@alamb alamb commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Update to the latest arrow/parquet release, 60.1.0. See the upstream CHANGELOG for details.

What changes are included in this PR?

  1. Update to arrow/parquet 60.1.0
  2. Adjust code for upstream API differences (see inline comments)
  3. Update test expectations for upstream behavior changes

What is the testing strategy for this PR?

Existing CI. This release candidate was also tested in #26543.

Are there any user-facing changes?

New dependency versions.


fn value_offsets(&self, row: usize) -> (i64, i64) {
let start = self.value_offset(row) as i64;
let start = self.value_offset_at(row) as i64;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// while spill_file_count remains 1 (since we're writing to the same file)
in_progress_file.append_batch(&batch1)?;
verify_metrics(&in_progress_file, 1, 440, 3)?;
verify_metrics(&in_progress_file, 1, 424, 3)?;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size is smaller 🎉

.map(|row| (input * rows_per_input + row) % dictionary_len),
);
take(&dictionary, &indices, None)
let taken = take(&dictionary, &indices, None)?;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alamb
alamb force-pushed the alamb/arrow_60.1.0 branch from ce310c5 to 5a12062 Compare October 29, 2026 16:42
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Oct 29, 2026
## all. Every route into a named zone -- a column cast, `AT TIME ZONE`, a
## string literal, or `::timestamptz` under a session zone -- fails with an
## error, where PostgreSQL resolves the value.
## Casting a naive timestamp into a named zone resolves an ambiguous or

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

01)ProjectionExec: expr=[count(Int64(1))@0 as count(*)], metrics=[<slt:ignore>]
02)--AggregateExec: mode=Single, gby=[], aggr=[count(Int64(1))], metrics=[<slt:ignore>]
03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=8, input_rows=516, <slt:ignore>]
03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=6, input_rows=387, <slt:ignore>]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to

The right side is replayed once per left chunk. Spilled batches are now read back without validity buffers for null-free columns, so they take less memory and the build side fits in three chunks instead of four.

SELECT approx_distinct(c14) AS a, approx_distinct(c15) AS b, approx_distinct(arrow_cast(c15, 'Date64')), approx_distinct(arrow_cast(c15, 'Time32(Second)')) as c, approx_distinct(arrow_cast(c15, 'Time64(Nanosecond)')) AS d FROM aggregate_test_100
----
18 60 60 60 60
18 60 7 60 60

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to

Casting a Timestamp to Date64 now truncates to whole days (previously it kept the time of day), so only 7 distinct dates remain.

FROM (VALUES (arrow_cast(9223372036854775807, 'Timestamp(Second, None)')), (NULL)) AS t(ts)
----
NULL NULL NULL NULL
7 NULL NULL NULL

@alamb alamb Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to

second is now computed directly from the stored value for timestamps without a timezone,

i64::MAX seconds yields 7 (i64::MAX % 60) instead of overflowing to NULL.

The sub-second parts still go through chrono and remain NULL. The nullability reproducer further down now uses millisecond so it still produces a NULL from a non-null input.

/// Arrays that each hold views into one shared dictionary buffer, as the
/// Parquet reader produces for a dictionary-encoded column, combined the
/// way a sort combines its input batches. Each output row cycles through
/// way a sort combines its input batches: the combined array lists the

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note this is test code

@alamb
alamb marked this pull request as ready for review October 29, 2026 17:33
/// Parquet reader produces for a dictionary-encoded column, combined the
/// way a sort combines its input batches. Each output row cycles through
/// Parquet reader produces for a dictionary-encoded column, combined with
/// `concat` as a sort combines its buffered batches. The rows cycle through

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interleave no longer duplicates the buffers

Comment thread datafusion-testing Resolved
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 1, 2026
@alamb

alamb commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor Author

@alamb
alamb added this pull request to the merge queue Nov 1, 2026
@alamb

alamb commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Ok,let's get this in

Merged via the queue into apache:main with commit 6a9a129 Nov 1, 2026
45 of 46 checks passed
@alamb
alamb deleted the alamb/arrow_60.1.0 branch November 1, 2026 13:07
KonaeAkira pushed a commit to KonaeAkira/datafusion that referenced this pull request Nov 7, 2026
## Which issue does this PR close?

- Closes apache#26981.
- Related to apache#26451, the same cause for timestamp → time (fixed on its
own by apache#26452).

## Rationale for this change

Parquet statistics pruning can drop row groups that contain matching
rows when the predicate casts the column. With `i` in (-1, 0, 1, 255,
256) and naive timestamps `ts` at 00:00, 01:00, 02:30, 02:45 and 03:00
on 2024-03-10, all in one row group:

| Query | In-memory table | Parquet |
|---|---|---|
| `SELECT count(*) FROM t WHERE CAST(i AS BOOLEAN) <= false` | 1 | 0 |
| `SELECT count(*) FROM t WHERE CAST(i AS BYTEA) < X'05'` | 3 | 0 |
| `SELECT count(*) FROM t WHERE ts > TIMESTAMPTZ '2024-03-10
07:15:00+00:00'` with `time_zone = 'America/New_York'` | 2 | 0 |

Pruning casts the min/max statistics and compares them with the literal,
which is only correct when the cast keeps the order of the values.
Pruning decides that with its own check: any cast is allowed unless
exactly one side is a string. So int → boolean, int → binary
(little-endian bytes) and named-timezone timestamp → date all pass,
although none of them keep order.

The last row is new with arrow 60.1.0 (apache#26867). A naive timestamp
compared with a named timezone is read as a local time, and arrow now
moves a local time that a transition skips forward instead of rejecting
it: 02:30, which New York skipped, becomes 07:30 UTC, after 03:00 (07:00
UTC). When such a value is the minimum of a row group, the cast bounds
are inverted and the query fails with `Internal error: Interval's lower
bound ... is greater than the upper bound`.

Sort elimination already asks the same question through
`is_order_preserving_cast`. The two checks disagree in both directions:
pruning accepts the casts above, and `is_order_preserving_cast` rejects
casts that do keep order and that pruning relies on today (integer ↔
date/timestamp/duration, string and binary re-encodings, date →
timestamp with a timezone). Arrow 60.1.0 also made one of its own
entries wrong: a named-timezone timestamp → `Date64` now takes the local
date, as `Date32` does, so a sort on that cast can be removed although
the output is out of order.

## What changes are included in this PR?

1. `is_order_preserving_cast` (now public in
`datafusion_physical_expr::expressions`) accepts the casts that keep
order and that only pruning knew about:
- string ↔ string and binary ↔ binary re-encodings (including
`FixedSizeBinary`)
- numbers ↔ temporal types stored as a count (date, timestamp, time,
duration)
- `Date32` → timestamp with any timezone (a skipped midnight moves
forward by at most a day, so it never passes the next one), and `Date64`
→ timestamp with a fixed offset
- `Time64` → `Time32` and `Time64(µs)` → `Time64(ns)`, which Arrow
60.1.0 checks for overflow
2. It rejects named-timezone timestamp → `Date64`, as it already did for
`Date32`.
3. Pruning (`verify_support_type_for_prune`) asks
`is_order_preserving_cast` instead of its own string check.
4. A test that checks `is_order_preserving_cast` against the strict cast
that `CastExpr` runs (`ColumnarValue::cast_to`): every pair it accepts
must keep the order of sample values that convert. The samples include
values around midnight, a DST gap and overlap, a timezone that moved the
date backwards, and a day that Pacific/Apia skipped. A positive control
asserts the samples expose the casts the rule rejects.

Pruning keeps every other cast it used before (compared over all
castable pairs of the numeric, string, binary and temporal types) and
stops using:

| Cast | Why |
|---|---|
| integer/float → boolean | -1 and 1 are true, 0 is false |
| integer → binary | little-endian bytes do not sort like the integers |
| named-timezone timestamp → `Date32`/`Date64` | a transition can move
the local date backwards |
| naive timestamp or `Date64` → named-timezone timestamp | arrow moves a
skipped local time forward, past the local times just after the gap |
| casts from or to `Null`, to list/struct, interval casts | the rule has
no entry for them |

## What is the testing strategy for this PR?

- `test_order_preserving_cast_matches_cast_kernel` (physical-expr):
checks the rule against the strict cast, with a positive control. Making
named-timezone timestamp → date return `true` fails it with `562129259
-> 1987-10-25, but 562129260 -> 1987-10-24`.
- `row_group_predicate_cast_must_preserve_order` (pruning): casts that
must not prune and casts that must keep pruning, for both `CAST` and
`TRY_CAST`.
- `push_down_filter_parquet.slt`: the three queries above against an
in-memory table and Parquet, plus an `EXPLAIN` showing that the same
timestamp comparison under a fixed-offset session timezone still prunes.
- `cast.slt`: the sort on `arrow_cast(i, 'Timestamp(Second, None)')` is
removed when the input is ordered by `i` (`EXPLAIN` and the row order),
and the sort on a named-timezone timestamp cast to `Date64` is kept.
- The new pruning unit test cases and Parquet queries fail with the old
pruning check. The `Date64` sort case and the New York Parquet query
return wrong results on main.
- `bloom_filter_pruning_error_retains_row_group_without_match` relied on
the int → `FixedSizeBinary` cast failing at evaluation. Pruning now
rejects that cast up front, so the test compares `c1` with `CAST('abc'
AS Int32)` instead, which still fails at evaluation.

## Are there any user-facing changes?

- Queries that filter Parquet through the casts in the table above
return the correct rows; those predicates no longer prune row groups.
With a named session timezone, comparing a naive timestamp column with a
timezone-aware timestamp no longer prunes; with a fixed offset it still
does.
- Sort elimination treats the newly accepted casts as order-preserving
under `CAST` (not `TRY_CAST`), so a sort on, for example, `CAST(int_col
AS TIMESTAMP)` over input ordered by `int_col` can be removed. A sort on
a named-timezone timestamp cast to `Date64` is no longer removed.
- New public function
`datafusion_physical_expr::expressions::is_order_preserving_cast`
(addition only).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants