You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 96eb0df
Browse filesBrowse the repository at this point in the historyBrowse files
fix: validate parquet encoding when configured (apache#25784)
## Which issue does this PR close?
Part of apache#17498. This PR validates the global Parquet encoding setting;
the issue also tracks other configuration values.
## Rationale for this change
`SET datafusion.execution.parquet.encoding = 'invalid'` currently
succeeds. The error appears only when a Parquet write starts, so the
invalid setting is harder to diagnose.
## What changes are included in this PR?
- Add `DFParquetEncoding` to validate supported names when the global
setting is assigned, following the approach used for compression in
apache#25626.
- Preserve the previous value after an invalid `SET`, and display valid
values in canonical lowercase form.
- Use the validated value when building Parquet writer properties while
retaining column-specific encoding behavior.
- Validate the global setting in both Proto conversion paths and add
regression tests.
## What is the testing strategy for this PR?
Added unit tests for encoding parsing and conversion and Proto
conversion, plus SQL logic coverage for `SET`, `SHOW`, invalid `SET`,
and `RESET`.
Validation with Rust 1.99.0:
- Before removing the optimizer lint annotations, the extended workspace
test command in `AGENTS.md` passed: 12,678 Rust tests, 8 ignored, and
all 534 SQL logic test files.
- `cargo fmt --all --check`
- `./dev/rust_lint.sh` (before removing the optimizer lint annotations)
- After removing the annotations: `cargo fmt --all` and
`./ci/scripts/rust_clippy.sh` passed.
There is no relevant local benchmark for configuration parsing or
writer-property setup.
## Are there any user-facing changes?
Invalid global Parquet encodings now fail at `SET` time instead of at
write time. `SHOW` displays valid encodings in lowercase.
The public Rust field `ParquetOptions.encoding` changes from
`Option<String>` to `Option<DFParquetEncoding>`. Please apply the `api
change` label.
---------
Co-authored-by: Andrew Lamb <[email protected]>
0 commit comments