You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Partition values are not URL-decoded when extracted from Hive-style paths #19650
When using Hive-style partitioned tables where partition values contain URL-encoded characters (like / encoded as %2F or spaces as %20), DataFusion returns the literal encoded string instead of the decoded value.
For example, given a file at: s3://bucket/table/category=foo%2Fbar/file.parquet
The partition column category returns the literal value foo%2Fbar instead of the expected decoded value foo/bar.
Related Issues
This is a follow-up to #7877, which was partially addressed by #8012.
While #8012 fixed URL decoding for the Table URL (ListingTableUrl::parse()), it did not apply decoding to the extracted partition values from the actual file paths within parse_partitions_for_path().
Partition values should be URL-decoded, consistent with how ListingTableUrl handles URL-encoded paths. This matches the behavior of Apache Spark and Apache Hive.
Additional context
The fix involves updating parse_partitions_for_path in datafusion/catalog-listing/src/helpers.rs to use percent-encoding.
Because decoding creates a new string, the function signature needs to change from Option<Vec<&str>> to Option<Vec<String>>.
This affects users storing data in Hive-partitioned layouts on object stores (S3/GCS/Azure) where special characters in paths are standard.
Activity
Decode Hive partition values in listing tables (apache#23226)
Decode Hive partition values in listing tables (apache#23226)