You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 26b40dd
Browse filesBrowse the repository at this point in the historyBrowse files
## Why are the changes needed?
### Which issue does this PR close?
Part of apache#10586. This fixes the unsigned string/binary byte-array case;
it does not claim to resolve every Parquet ordering issue. It is also
the correctness prerequisite for apache#24526.
### Rationale for this change
A Parquet scan can skip a file, row group, or page when its statistics
prove that no row can satisfy the filter. If those statistics use a
different comparison order from the query, that proof is invalid:
DataFusion can silently discard a row that should be returned.
For example, suppose a Parquet row group contains `'aé'`, `'az'`, and
`'b'`:
```sql
SELECT s FROM t WHERE s = 'az';
-- Expected: one row containing 'az'
```
Parquet's deprecated byte-array `min`/`max` fields use signed
comparison, whereas Arrow compares strings using unsigned UTF-8 bytes.
The same values therefore have two different orders:
| Comparison | Values in ascending order | Minimum / maximum |
| --- | --- | --- |
| Legacy signed-byte order | `aé < az < b` | `aé` / `b` |
| Arrow's unsigned-byte order | `az < aé < b` | `az` / `b` |
The first UTF-8 byte of `é` is `0xC3`: it sorts before `z`'s `0x7A` as a
signed byte, but after it as an unsigned byte. If DataFusion interprets
the legacy interval `['aé', 'b']` using Arrow's ordering, it sees `'az'
< 'aé'` and can incorrectly skip the row group. Merely checking that
`min <= max` does not help: the two reported endpoints are still in
ascending order.
The newer `min_value`/`max_value` fields and page-index bounds also need
an ordering declaration that the reader understands. A missing or
unknown `column_orders` entry is not enough to justify assuming Arrow's
ordering. The [Parquet statistics
definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift)
and [logical-type ordering
rules](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)
describe these distinctions.
This bug is independently observable with an ordinary equality filter;
it does not require a large `IN` list.
## What changes were proposed in this PR?
### What changes are included in this PR?
The change makes a recognized comparison order a prerequisite for using
unsigned byte-array bounds. At the point where Parquet metadata becomes
Arrow statistics, DataFusion checks whether the column's
physical/logical type and footer establish the expected unsigned order.
It also rejects row-group bounds taken from the deprecated signed-order
fields. When that evidence is missing, min/max is reported as unknown,
so pruning keeps potentially matching data instead of guessing.
The rule is applied at the granularity where the statistics are used. An
unsafe row group prevents DataFusion from claiming a trustworthy bound
for the whole file, but it does not make other row groups' valid bounds
unusable. Page-index bounds are checked against the footer
independently. The same safeguards cover static and runtime row-group
pruning, as well as the inverse predicates used to decide whether every
row already satisfies a filter. Null counts and unrelated columns'
statistics remain available.
The row-group pruning API gains a metadata-aware entry point so callers
can supply the footer needed for this decision. The existing entry point
remains source-compatible and behaves conservatively when that
information is unavailable. Signed logical types such as decimals retain
their existing behavior.
### Are there any user-facing changes?
Queries no longer discard matching data because of these untrustworthy
byte-array bounds. Older files, or files with an unrecognized ordering,
may require more scanning. Modern files with trustworthy bounds retain
min/max pruning. There is no file-format change or breaking public API
change.
## How was this PR tested?
### Are these changes tested?
The regression uses actual serialized Parquet files containing the
example above, with modern, deprecated, missing-order, and unknown-order
metadata. It checks that the matching `az` row survives file, row-group,
runtime, and page pruning, while valid statistics can still eliminate
unrelated data. On unchanged Apache `f1f0449a`, the adapted equality
regression fails because the deprecated-order case loses `az`; the
modern-statistics control passes.
The dedicated Parquet-crate run passed 231 unit tests and four doctests.
Its seven focused ordering tests also cover mixed safe/unsafe row
groups, null counts, fixed-length binary and UUID, signed decimal, and
logical types with undefined ordering. Formatting,
all-targets/all-features Clippy with warnings denied, and
`./dev/rust_lint.sh` passed. The extended workspace run passed 10,666
Rust tests, with eight ignored, and all 503 SQL-logic files.
The existing metadata benchmark was run on Apache `f1f0449a` and this
patch using the same valid modern-footer fixture. Across nine
full-statistics cases there was no material regression; the largest
case, with 256 columns and 128 row groups, measured 1.330 ms before and
1.332 ms after. Both runs used Rust 1.97.0, `release-nonlto`, 20
samples, and separate build directories on an Apple M5 Max.
<details>
<summary>Validation commands</summary>
```sh
cargo test --locked --profile ci -p datafusion-datasource-parquet
cargo bench --locked --profile release-nonlto -p datafusion-datasource-parquet \
--bench parquet_metadata_statistics -- \
metadata_full --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot
RUST_BACKTRACE=1 cargo test --locked --profile ci \
--exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \
--workspace --lib --tests --bins \
--features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```
</details>
0 commit comments