You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] unbase64 rejects input that Spark's MIME decoder accepts #42
Spark's UnBase64 decodes with java.util.Base64.getMimeDecoder (stringExpressions.scala#L2897-L2918). That decoder does not require padding and ignores characters outside the Base64 alphabet. unbase64 in datafusion-spark is rewritten to the core decode(..., 'base64pad') (base64.rs#L169-L172), and the existing test expects an error for input that Spark decodes: base64.slt#L109 has unbase64('123') failing with "Failed to decode value using base64", and #L112 does the same for the binary variant.
To Reproduce
Spark 4.0.0, the same result with spark.sql.ansi.enabled on or off for '123' and 'YWJj!!':
SELECT hex(unbase64('123'))
-- D76DSELECT hex(unbase64('12'))
-- D7SELECT hex(unbase64('YQ'))
-- 61SELECT hex(unbase64('YW Jj'))
-- 616263SELECT hex(unbase64('YWJj!!'))
-- 616263SELECT hex(unbase64('!!!!'))
-- (empty)SELECT hex(unbase64('Y'))
-- error: IllegalArgumentException: Last unit does not have enough valid bits
Only unbase64('123') was run on DataFusion, through the existing test. The other inputs are listed for Spark's behavior and were not run.
Expected behavior
Decode like the MIME decoder: accept missing padding, skip characters outside the alphabet, and keep an error for a dangling single character. Update the expectations at base64.slt#L109 and #L112.
Additional context
Found by running the queries in datafusion/sqllogictest/test_files/spark on a local Spark and comparing the results with the expected values in the files, which is the kind of check proposed in #7. Spark results come from Apple's build of Spark 4.0.0 (4.0.0.65-apple) with a UTC session and spark.sql.ansi.enabled=false unless stated. The Spark source links point at the v4.0.0 tag. DataFusion results are the sqllogictest expectations at 9dec77584, which CI enforces. Anything marked as not run was derived from reading the code.
Activity
Wali05 commentedon Oct 30, 2026
take
fix(spark): decode unbase64 like Spark's MIME decoder (apache#84)