Skip to content

fix(spark): match Spark 4 encode charset behavior - #27260

Open
Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/spark-encode-spark4-compat
Open

Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/spark-encode-spark4-compat

Conversation

@Subodh-17

Copy link
Copy Markdown

Summary

Addresses apache/datafusion-spark#46 by adding selectable Spark 4-compatible behavior to the datafusion-spark encode expression.

  • Adds datafusion.spark.legacy_java_charsets and datafusion.spark.legacy_coding_error_action, both defaulting to false.
  • Enforces Spark 4's seven canonical charset names by default while retaining legacy aliases when legacy_java_charsets is enabled.
  • Raises MALFORMED_CHARACTER_CODING (SQLSTATE 22000) for unmappable characters by default.
  • Restores legacy replacement behavior when legacy_coding_error_action is enabled.
  • Adds unit tests and SQLLogicTests for charset handling, configuration combinations, error behavior, and settings exposure.
  • Documents both configuration options.

Validation

  • cargo check -p datafusion-spark -p datafusion-common
  • cargo clippy -p datafusion-spark -p datafusion-common -- -D warnings
  • cargo fmt --check
  • Focused encode unit tests
  • Full datafusion-spark library tests (333 passed)
  • Focused encode.slt SQLLogicTest
  • Focused information_schema.slt SQLLogicTest
  • git diff HEAD^ HEAD --check

Compatibility note

The existing empty-input UTF-16 behavior is unchanged: DataFusion emits the UTF-16 BOM, whereas Spark returns an empty byte array. This pre-existing divergence is documented and retained.

Closes apache/datafusion-spark#46.

@github-actions github-actions Bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) common Related to common crate spark Changes to datafusion-spark crate labels Nov 9, 2026
@andygrove

Copy link
Copy Markdown
Member

Heads-up: as discussed in apache/datafusion-spark#87, we plan to move the datafusion-spark crate, including its sqllogictest files and git history, to the new https://github.com/apache/datafusion-spark repo in the next day or two. After that, changes to datafusion-spark will go to the new repo.

This PR changes datafusion-spark files. If it hasn't merged before the move, please open a new PR with those changes against apache/datafusion-spark (GitHub can't transfer PRs between repos). Any changes to other DataFusion crates stay here. We'll post instructions for porting open PRs in apache/datafusion-spark#87 once the code is in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate documentation Improvements or additions to documentation spark Changes to datafusion-spark crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] encode accepts charsets and unmappable characters that Spark 4 rejects

2 participants