Repository navigation
Commit bcf75da
fix: validate parquet.compression when it is set (apache#25626)
## Which issue does this PR close?
- Part of apache#17498. This covers `datafusion.execution.parquet.compression`
only.
## Rationale for this change
The option is stored as a plain string, so `SET
datafusion.execution.parquet.compression = 'zstdd(3)'` succeeds, `SHOW`
reports the bad value, and the error only appears later when a Parquet
file is written. The same goes for a codec without its level (`zstd`), a
level on a codec that takes none (`snappy(2)`), and a level out of range
(`zstd(23)`). The issue is about catching these at `SET` time, and this
option was converted in the first attempt (apache#17697) and planned in apache#19498
but never landed.
## What changes are included in this PR?
- `DFParquetCompression` in `parquet_config.rs`, an enum of the seven
supported codecs with the level carried by `Gzip`, `Brotli` and `Zstd`.
`FromStr` does the validation, `Display` produces the same strings the
option stored before (`zstd(3)`, `lz4_raw`), and the `ConfigField` impls
follow `DFParquetStatistics`, including the `Option` handling that
leaves an unset value unset when a bad value is rejected.
- `ParquetOptions.compression` becomes `Option<DFParquetCompression>`,
with the same default of `zstd(3)`. The per-column
`ParquetColumnOptions.compression` stays a string, matching the scope of
apache#24642.
- `parse_compression_string` is now a two-line wrapper over the new
type, so there is one parser. Its error messages are unchanged.
- Proto conversion parses on the way in, so an invalid value is rejected
there too.
- The level ranges are mirrored from the parquet crate as constants,
with a test that fails if they ever differ from what `GzipLevel`,
`BrotliLevel` and `ZstdLevel` accept. That test caught my first copy of
the zstd range, which came from parquet 59; the workspace uses parquet
60, where `zstd(0)` is accepted, and it still is.
Behavior notes:
- A malformed closing parenthesis such as `zstd(3x` used to be read as
`zstd(3)` and is now rejected.
- An out-of-range level gets a DataFusion message naming the valid range
instead of the parquet crate's message.
- The error for a bad value set through `COPY ... OPTIONS` now arrives
when the option is set, wrapped in "Error setting config ...", which one
existing test in `listing/table.rs` asserted on and now expects.
This changes the type of a public field, so it should carry the `api
change` label. Two call sites in this repository that assigned a string
needed to parse instead.
## What is the testing strategy for this PR?
- `parquet_config.rs`: parsing and display for every codec, nine
rejected inputs with their messages, and the range-drift test above.
- `config.rs`: `test_parquet_compression_validation`, mirroring the
statistics test, including that a rejected value leaves the previous one
in place and that an unset value stays unset.
- `proto-common`: a round trip and a rejected value from proto.
- `set_variable.slt`: set, show, three rejected values with their full
error text, and that the setting is unchanged afterwards.
- Run locally: `datafusion-common` with all features (628 unit tests)
and without the parquet feature, `proto-common` and `proto-models`,
`datasource-parquet` (271), the `set_variable` and `information_schema`
sqllogictest files, the core listing, parquet and dataframe unit tests
(114) and `parquet_integration` (242). `cargo clippy --workspace
--all-targets --all-features -- -D warnings`, `cargo fmt --check`, typos
and `dev/update_config_docs.sh` are clean; the generated docs did not
change because the default displays the same.
## Are there any user-facing changes?
Yes. An invalid `datafusion.execution.parquet.compression` value is
rejected by `SET` and by `COPY ... OPTIONS` with a message, instead of
being accepted and failing at write time. The field type in
`ParquetOptions` changes from `Option<String>` to
`Option<DFParquetCompression>`.1 parent f39e8d2 commit bcf75da
11 files changed
Lines changed: 482 additions & 107 deletions
File tree
- datafusion
- common/src
- file_options
- core/src
- dataframe
- datasource/listing
- execution/context
- datasource-parquet/src
- proto-common/src
- from_proto
- to_proto
- proto-models/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| |||
1440 | 1442 | | |
1441 | 1443 | | |
1442 | 1444 | | |
1443 | | - | |
| 1445 | + | |
1444 | 1446 | | |
1445 | 1447 | | |
1446 | 1448 | | |
| |||
4684 | 4686 | | |
4685 | 4687 | | |
4686 | 4688 | | |
| 4689 | + | |
| 4690 | + | |
| 4691 | + | |
| 4692 | + | |
| 4693 | + | |
| 4694 | + | |
| 4695 | + | |
| 4696 | + | |
| 4697 | + | |
| 4698 | + | |
| 4699 | + | |
| 4700 | + | |
| 4701 | + | |
| 4702 | + | |
| 4703 | + | |
| 4704 | + | |
| 4705 | + | |
| 4706 | + | |
| 4707 | + | |
| 4708 | + | |
| 4709 | + | |
| 4710 | + | |
| 4711 | + | |
| 4712 | + | |
| 4713 | + | |
| 4714 | + | |
| 4715 | + | |
| 4716 | + | |
| 4717 | + | |
| 4718 | + | |
| 4719 | + | |
| 4720 | + | |
| 4721 | + | |
| 4722 | + | |
| 4723 | + | |
| 4724 | + | |
| 4725 | + | |
| 4726 | + | |
| 4727 | + | |
| 4728 | + | |
| 4729 | + | |
| 4730 | + | |
| 4731 | + | |
| 4732 | + | |
| 4733 | + | |
| 4734 | + | |
| 4735 | + | |
| 4736 | + | |
| 4737 | + | |
| 4738 | + | |
| 4739 | + | |
| 4740 | + | |
| 4741 | + | |
| 4742 | + | |
| 4743 | + | |
| 4744 | + | |
| 4745 | + | |
| 4746 | + | |
| 4747 | + | |
| 4748 | + | |
| 4749 | + | |
| 4750 | + | |
| 4751 | + | |
| 4752 | + | |
| 4753 | + | |
| 4754 | + | |
| 4755 | + | |
| 4756 | + | |
| 4757 | + | |
| 4758 | + | |
| 4759 | + | |
| 4760 | + | |
| 4761 | + | |
| 4762 | + | |
| 4763 | + | |
| 4764 | + | |
| 4765 | + | |
| 4766 | + | |
| 4767 | + | |
4687 | 4768 | | |
4688 | 4769 | | |
4689 | 4770 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | | - | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
285 | | - | |
| 285 | + | |
286 | 286 | | |
287 | 287 | | |
288 | 288 | | |
| |||
320 | 320 | | |
321 | 321 | | |
322 | 322 | | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | 323 | | |
364 | 324 | | |
365 | 325 | | |
366 | 326 | | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
| 327 | + | |
411 | 328 | | |
412 | 329 | | |
413 | 330 | | |
| |||
433 | 350 | | |
434 | 351 | | |
435 | 352 | | |
436 | | - | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
437 | 356 | | |
438 | 357 | | |
439 | 358 | | |
| |||
471 | 390 | | |
472 | 391 | | |
473 | 392 | | |
474 | | - | |
| 393 | + | |
475 | 394 | | |
476 | 395 | | |
477 | 396 | | |
| |||
597 | 516 | | |
598 | 517 | | |
599 | 518 | | |
600 | | - | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
601 | 525 | | |
602 | 526 | | |
603 | 527 | | |
| |||
770 | 694 | | |
771 | 695 | | |
772 | 696 | | |
773 | | - | |
| 697 | + | |
774 | 698 | | |
775 | 699 | | |
776 | 700 | | |
| |||
0 commit comments