Skip to content

Commit bcf75da

Browse files
akashjainndiegoQuinas
authored andcommitted
fix: validate parquet.compression when it is set (apache#25626)
## Which issue does this PR close? - Part of apache#17498. This covers `datafusion.execution.parquet.compression` only. ## Rationale for this change The option is stored as a plain string, so `SET datafusion.execution.parquet.compression = 'zstdd(3)'` succeeds, `SHOW` reports the bad value, and the error only appears later when a Parquet file is written. The same goes for a codec without its level (`zstd`), a level on a codec that takes none (`snappy(2)`), and a level out of range (`zstd(23)`). The issue is about catching these at `SET` time, and this option was converted in the first attempt (apache#17697) and planned in apache#19498 but never landed. ## What changes are included in this PR? - `DFParquetCompression` in `parquet_config.rs`, an enum of the seven supported codecs with the level carried by `Gzip`, `Brotli` and `Zstd`. `FromStr` does the validation, `Display` produces the same strings the option stored before (`zstd(3)`, `lz4_raw`), and the `ConfigField` impls follow `DFParquetStatistics`, including the `Option` handling that leaves an unset value unset when a bad value is rejected. - `ParquetOptions.compression` becomes `Option<DFParquetCompression>`, with the same default of `zstd(3)`. The per-column `ParquetColumnOptions.compression` stays a string, matching the scope of apache#24642. - `parse_compression_string` is now a two-line wrapper over the new type, so there is one parser. Its error messages are unchanged. - Proto conversion parses on the way in, so an invalid value is rejected there too. - The level ranges are mirrored from the parquet crate as constants, with a test that fails if they ever differ from what `GzipLevel`, `BrotliLevel` and `ZstdLevel` accept. That test caught my first copy of the zstd range, which came from parquet 59; the workspace uses parquet 60, where `zstd(0)` is accepted, and it still is. Behavior notes: - A malformed closing parenthesis such as `zstd(3x` used to be read as `zstd(3)` and is now rejected. - An out-of-range level gets a DataFusion message naming the valid range instead of the parquet crate's message. - The error for a bad value set through `COPY ... OPTIONS` now arrives when the option is set, wrapped in "Error setting config ...", which one existing test in `listing/table.rs` asserted on and now expects. This changes the type of a public field, so it should carry the `api change` label. Two call sites in this repository that assigned a string needed to parse instead. ## What is the testing strategy for this PR? - `parquet_config.rs`: parsing and display for every codec, nine rejected inputs with their messages, and the range-drift test above. - `config.rs`: `test_parquet_compression_validation`, mirroring the statistics test, including that a rejected value leaves the previous one in place and that an unset value stays unset. - `proto-common`: a round trip and a rejected value from proto. - `set_variable.slt`: set, show, three rejected values with their full error text, and that the setting is unchanged afterwards. - Run locally: `datafusion-common` with all features (628 unit tests) and without the parquet feature, `proto-common` and `proto-models`, `datasource-parquet` (271), the `set_variable` and `information_schema` sqllogictest files, the core listing, parquet and dataframe unit tests (114) and `parquet_integration` (242). `cargo clippy --workspace --all-targets --all-features -- -D warnings`, `cargo fmt --check`, typos and `dev/update_config_docs.sh` are clean; the generated docs did not change because the default displays the same. ## Are there any user-facing changes? Yes. An invalid `datafusion.execution.parquet.compression` value is rejected by `SET` and by `COPY ... OPTIONS` with a message, instead of being accepted and failing at write time. The field type in `ParquetOptions` changes from `Option<String>` to `Option<DFParquetCompression>`.
1 parent f39e8d2 commit bcf75da

11 files changed

Lines changed: 482 additions & 107 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 83 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@ use arrow_ipc::CompressionType;
2323
use crate::encryption::{FileDecryptionProperties, FileEncryptionProperties};
2424
use crate::error::{_config_datafusion_err, _config_err};
2525
use crate::format::{ExplainAnalyzeCategories, ExplainFormat, MetricType};
26-
use crate::parquet_config::{DFParquetStatistics, DFParquetWriterVersion};
26+
use crate::parquet_config::{
27+
DFParquetCompression, DFParquetStatistics, DFParquetWriterVersion,
28+
};
2729
use crate::parsers::{CompressionTypeVariant, CsvQuoteStyle};
2830
use crate::utils::get_available_parallelism;
2931
use crate::{DataFusionError, Result};
@@ -1440,7 +1442,7 @@ config_namespace! {
14401442
///
14411443
/// Note that this default setting is not the same as
14421444
/// the default parquet writer setting.
1443-
pub compression: Option<String>, transform = str::to_lowercase, default = Some("zstd(3)".into())
1445+
pub compression: Option<DFParquetCompression>, default = Some(DFParquetCompression::Zstd(3))
14441446

14451447
/// (writing) Sets if dictionary encoding is enabled. If NULL, uses
14461448
/// default parquet writer setting
@@ -4684,6 +4686,85 @@ mod tests {
46844686
assert_eq!(scalar, DFParquetStatistics::Page);
46854687
}
46864688

4689+
#[cfg(feature = "parquet")]
4690+
#[test]
4691+
fn test_parquet_compression_validation() {
4692+
use crate::{config::ConfigOptions, parquet_config::DFParquetCompression};
4693+
4694+
let mut config = ConfigOptions::default();
4695+
assert_eq!(
4696+
config.execution.parquet.compression,
4697+
Some(DFParquetCompression::Zstd(3))
4698+
);
4699+
4700+
for (value, expected) in [
4701+
("snappy", DFParquetCompression::Snappy),
4702+
("GZIP(6)", DFParquetCompression::Gzip(6)),
4703+
("'zstd(22)'", DFParquetCompression::Zstd(22)),
4704+
] {
4705+
config
4706+
.set("datafusion.execution.parquet.compression", value)
4707+
.unwrap();
4708+
assert_eq!(config.execution.parquet.compression, Some(expected));
4709+
}
4710+
4711+
for (value, message) in [
4712+
(
4713+
"zstdd(3)",
4714+
"Unknown or unsupported parquet compression: zstdd(3)",
4715+
),
4716+
(
4717+
"zstd",
4718+
"zstd compression requires specifying a level such as zstd(4)",
4719+
),
4720+
(
4721+
"snappy(2)",
4722+
"Compression snappy does not support specifying a level",
4723+
),
4724+
("zstd(23)", "Invalid compression level 23 for zstd"),
4725+
] {
4726+
let err = config
4727+
.set("datafusion.execution.parquet.compression", value)
4728+
.unwrap_err();
4729+
assert_contains!(err.to_string(), message);
4730+
// A rejected value leaves the previous one in place.
4731+
assert_eq!(
4732+
config.execution.parquet.compression,
4733+
Some(DFParquetCompression::Zstd(22))
4734+
);
4735+
}
4736+
4737+
// An unset value can arise from deserialization. An invalid update must
4738+
// leave that state unchanged rather than inserting the default.
4739+
config.execution.parquet.compression = None;
4740+
assert!(
4741+
config
4742+
.set("datafusion.execution.parquet.compression", "zstd")
4743+
.is_err()
4744+
);
4745+
assert_eq!(config.execution.parquet.compression, None);
4746+
4747+
config.execution.parquet.compression = Some(DFParquetCompression::Lz4);
4748+
assert!(
4749+
config
4750+
.set("datafusion.execution.parquet.compression.typo", "snappy")
4751+
.is_err()
4752+
);
4753+
assert!(
4754+
config
4755+
.reset("datafusion.execution.parquet.compression.typo")
4756+
.is_err()
4757+
);
4758+
assert_eq!(
4759+
config.execution.parquet.compression,
4760+
Some(DFParquetCompression::Lz4)
4761+
);
4762+
4763+
let mut scalar = DFParquetCompression::Lz4;
4764+
assert!(ConfigField::set(&mut scalar, "typo", "snappy").is_err());
4765+
assert_eq!(scalar, DFParquetCompression::Lz4);
4766+
}
4767+
46874768
#[cfg(feature = "parquet")]
46884769
#[test]
46894770
fn set_cdc_enabled_flag() {

‎datafusion/common/src/file_options/parquet_writer.rs‎

Lines changed: 14 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,13 @@ use std::sync::Arc;
2222
use crate::{
2323
_internal_datafusion_err, DataFusionError, Result,
2424
config::{ParquetCdcOptions, ParquetOptions, TableParquetOptions},
25+
parquet_config::DFParquetCompression,
2526
};
2627

2728
use arrow::datatypes::Schema;
2829
use parquet::arrow::encode_arrow_schema;
2930
use parquet::{
3031
arrow::ARROW_SCHEMA_META_KEY,
31-
basic::{BrotliLevel, GzipLevel, ZstdLevel},
3232
file::{
3333
metadata::KeyValue,
3434
properties::{
@@ -282,7 +282,7 @@ impl ParquetOptions {
282282
// We do not have access to default ColumnProperties set in Arrow.
283283
// Therefore, only overwrite if these settings exist.
284284
if let Some(compression) = compression {
285-
builder = builder.set_compression(parse_compression_string(compression)?);
285+
builder = builder.set_compression((*compression).into());
286286
}
287287
if let Some(encoding) = encoding {
288288
builder = builder.set_encoding(parse_encoding_string(encoding)?);
@@ -320,94 +320,11 @@ pub(crate) fn parse_encoding_string(
320320
}
321321
}
322322

323-
/// Splits compression string into compression codec and optional compression_level
324-
/// I.e. gzip(2) -> gzip, 2
325-
fn split_compression_string(str_setting: &str) -> Result<(String, Option<u32>)> {
326-
// ignore string literal chars passed from sqlparser i.e. remove single quotes
327-
let str_setting = str_setting.replace('\'', "");
328-
let split_setting = str_setting.split_once('(');
329-
330-
match split_setting {
331-
Some((codec, rh)) => {
332-
let level = &rh[..rh.len() - 1].parse::<u32>().map_err(|_| {
333-
DataFusionError::Configuration(format!(
334-
"Could not parse compression string. \
335-
Got codec: {codec} and unknown level from {str_setting}"
336-
))
337-
})?;
338-
Ok((codec.to_owned(), Some(*level)))
339-
}
340-
None => Ok((str_setting.to_owned(), None)),
341-
}
342-
}
343-
344-
/// Helper to ensure compression codecs which don't support levels
345-
/// don't have one set. E.g. snappy(2) is invalid.
346-
fn check_level_is_none(codec: &str, level: Option<&u32>) -> Result<()> {
347-
if level.is_some() {
348-
return Err(DataFusionError::Configuration(format!(
349-
"Compression {codec} does not support specifying a level"
350-
)));
351-
}
352-
Ok(())
353-
}
354-
355-
/// Helper to ensure compression codecs which require a level
356-
/// do have one set. E.g. zstd is invalid, zstd(3) is valid
357-
fn require_level(codec: &str, level: Option<u32>) -> Result<u32> {
358-
level.ok_or(DataFusionError::Configuration(format!(
359-
"{codec} compression requires specifying a level such as {codec}(4)"
360-
)))
361-
}
362-
363323
/// Parses datafusion.execution.parquet.compression String to a parquet::basic::Compression
364324
pub fn parse_compression_string(
365325
str_setting: &str,
366326
) -> Result<parquet::basic::Compression> {
367-
let str_setting_lower: &str = &str_setting.to_lowercase();
368-
let (codec, level) = split_compression_string(str_setting_lower)?;
369-
let codec = codec.as_str();
370-
match codec {
371-
"uncompressed" => {
372-
check_level_is_none(codec, level.as_ref())?;
373-
Ok(parquet::basic::Compression::UNCOMPRESSED)
374-
}
375-
"snappy" => {
376-
check_level_is_none(codec, level.as_ref())?;
377-
Ok(parquet::basic::Compression::SNAPPY)
378-
}
379-
"gzip" => {
380-
let level = require_level(codec, level)?;
381-
Ok(parquet::basic::Compression::GZIP(GzipLevel::try_new(
382-
level,
383-
)?))
384-
}
385-
"brotli" => {
386-
let level = require_level(codec, level)?;
387-
Ok(parquet::basic::Compression::BROTLI(BrotliLevel::try_new(
388-
level,
389-
)?))
390-
}
391-
"lz4" => {
392-
check_level_is_none(codec, level.as_ref())?;
393-
Ok(parquet::basic::Compression::LZ4)
394-
}
395-
"zstd" => {
396-
let level = require_level(codec, level)?;
397-
Ok(parquet::basic::Compression::ZSTD(ZstdLevel::try_new(
398-
level as i32,
399-
)?))
400-
}
401-
"lz4_raw" => {
402-
check_level_is_none(codec, level.as_ref())?;
403-
Ok(parquet::basic::Compression::LZ4_RAW)
404-
}
405-
_ => Err(DataFusionError::Configuration(format!(
406-
"Unknown or unsupported parquet compression: \
407-
{str_setting}. Valid values are: uncompressed, snappy, gzip(level), \
408-
brotli(level), lz4, zstd(level), and lz4_raw."
409-
))),
410-
}
327+
Ok(str_setting.parse::<DFParquetCompression>()?.into())
411328
}
412329

413330
pub(crate) fn parse_statistics_string(str_setting: &str) -> Result<EnabledStatistics> {
@@ -433,7 +350,9 @@ mod tests {
433350
MaxRowGroupBytes, ParquetCdcOptions, ParquetColumnOptions,
434351
ParquetEncryptionOptions, ParquetOptions,
435352
};
436-
use crate::parquet_config::{DFParquetStatistics, DFParquetWriterVersion};
353+
use crate::parquet_config::{
354+
DFParquetCompression, DFParquetStatistics, DFParquetWriterVersion,
355+
};
437356
use parquet::basic::Compression;
438357
use parquet::file::properties::{
439358
BloomFilterProperties, DEFAULT_BLOOM_FILTER_FPP, DEFAULT_BLOOM_FILTER_NDV,
@@ -471,7 +390,7 @@ mod tests {
471390
data_pagesize_limit: 42,
472391
write_batch_size: 42,
473392
writer_version,
474-
compression: Some("zstd(22)".into()),
393+
compression: Some(DFParquetCompression::Zstd(22)),
475394
dictionary_enabled: Some(!defaults.dictionary_enabled.unwrap_or(false)),
476395
dictionary_page_size_limit: 43,
477396
statistics_enabled: Some(DFParquetStatistics::Chunk),
@@ -597,7 +516,12 @@ mod tests {
597516

598517
// global options which set the default column props
599518
encoding: default_col_props.encoding,
600-
compression: default_col_props.compression,
519+
compression: match props.compression(&default_col) {
520+
Compression::ZSTD(level) => {
521+
Some(DFParquetCompression::Zstd(level.compression_level() as u32))
522+
}
523+
_ => None,
524+
},
601525
dictionary_enabled: default_col_props.dictionary_enabled,
602526
statistics_enabled: Some(props.statistics_enabled(&default_col).into()),
603527
bloom_filter_on_write: default_col_props
@@ -770,7 +694,7 @@ mod tests {
770694
let mut from_extern_parquet =
771695
session_config_from_writer_props(&default_writer_props);
772696
from_extern_parquet.global.created_by = same_created_by;
773-
from_extern_parquet.global.compression = Some("zstd(3)".into());
697+
from_extern_parquet.global.compression = Some(DFParquetCompression::Zstd(3));
774698
from_extern_parquet.global.skip_arrow_metadata = true;
775699

776700
assert_eq!(

0 commit comments

Comments
 (0)