You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d89a85d
Browse filesBrowse the repository at this point in the historyBrowse files
fix: validate parquet.coerce_int96_tz when it is set. (#26420)
## Which issue does this PR close?
- Closes#26362.
## Rationale for this change
`SET datafusion.execution.parquet.coerce_int96_tz = 'Not/A_Timezone'`
succeeds today, and the error only shows up on the next Parquet read.
After #25628 the unit option next to it is checked when it is set, so
this does the same for the timezone.
## What changes are included in this PR?
`ParquetOptions::coerce_int96_tz` is now `Option<ConfigTimeZone>`, the
type `execution.time_zone` already uses, so it accepts exactly what it
accepted before. `parse_coerce_int96_tz_string` is removed, and both
protobuf decoders parse the value.
An invalid value is rejected even when `coerce_int96` is unset, as the
issue asked.
## What is the testing strategy for this PR?
- `set_variable.slt`: SET/SHOW keeps the spelling (`+08`), a bad value
is rejected and leaves the old one, RESET goes back to NULL, and bad
table options fail at CREATE with an inferred and an explicit schema.
These fail on main.
- `test_parquet_coerce_int96_tz_validation` in `config.rs`.
- A rejection test for each protobuf decoder, and the existing
round-trip tests use the typed value.
- The existing timezone read tests in `parquet.slt` pass unchanged.
## Are there any user-facing changes?
Yes. Invalid timezones are rejected when set. The public field type
changes, so this needs `api change`. I added a note to the 56.0.0
upgrade guide.
datafusion.execution.parquet.bloom_filter_on_read true (reading) Use any available bloom filters when reading parquet files
404
404
datafusion.execution.parquet.bloom_filter_on_write false (writing) Write bloom filters for all columns when creating parquet files
405
405
datafusion.execution.parquet.coerce_int96 NULL (reading) If set, parquet reader will read columns of physical type int96 as originating from this resolution instead of nanosecond. Valid values are: ns, us, ms, and s. This is useful for reading data from systems like Spark which store microsecond resolution timestamps in an int96, allowing them to write values with a larger date range than 64-bit timestamps with nanosecond resolution.
406
-
datafusion.execution.parquet.coerce_int96_tz NULL (reading) Optional timezone applied to INT96 columns when `coerce_int96` is set. When `Some`, INT96 columns coerce to `Timestamp(<coerce_int96>, Some(<tz>))` instead of the default `Timestamp(<coerce_int96>, None)`. Spark and other systems write INT96 values as UTC-adjusted instants, so callers that need the resulting Arrow type to be timezone-aware (e.g. for Spark `TimestampType` semantics) should set this to `"UTC"`. No effect when `coerce_int96` is `None`.
406
+
datafusion.execution.parquet.coerce_int96_tz NULL (reading) Optional timezone applied to INT96 columns when `coerce_int96` is set. When `Some`, INT96 columns coerce to `Timestamp(<coerce_int96>, Some(<tz>))` instead of the default `Timestamp(<coerce_int96>, None)`. Spark and other systems write INT96 values as UTC-adjusted instants, so callers that need the resulting Arrow type to be timezone-aware (e.g. for Spark `TimestampType` semantics) should set this to `"UTC"`. No effect when `coerce_int96` is `None`, but the value is still validated when it is set.
407
407
datafusion.execution.parquet.column_index_truncate_length 64 (writing) Sets column index truncate length
408
408
datafusion.execution.parquet.compression zstd(3) (writing) Sets default parquet compression codec. Valid values are: uncompressed, snappy, gzip(level), brotli(level), lz4, zstd(level), and lz4_raw. These values are not case sensitive. If NULL, uses default parquet writer setting Note that this default setting is not the same as the default parquet writer setting.
409
409
datafusion.execution.parquet.content_defined_chunking.enabled false (writing) EXPERIMENTAL: Enable content-defined chunking (CDC) when writing parquet files. When enabled, parallel writing is automatically disabled since the chunker state must persist across row groups.
0 commit comments