Skip to content

[Bug] unbase64 rejects input that Spark's MIME decoder accepts #42

Closed

Description

@comphead

Describe the bug

Spark's UnBase64 decodes with java.util.Base64.getMimeDecoder (stringExpressions.scala#L2897-L2918). That decoder does not require padding and ignores characters outside the Base64 alphabet. unbase64 in datafusion-spark is rewritten to the core decode(..., 'base64pad') (base64.rs#L169-L172), and the existing test expects an error for input that Spark decodes: base64.slt#L109 has unbase64('123') failing with "Failed to decode value using base64", and #L112 does the same for the binary variant.

To Reproduce

Spark 4.0.0, the same result with spark.sql.ansi.enabled on or off for '123' and 'YWJj!!':

SELECT hex(unbase64('123'))
-- D76D
SELECT hex(unbase64('12'))
-- D7
SELECT hex(unbase64('YQ'))
-- 61
SELECT hex(unbase64('YW Jj'))
-- 616263
SELECT hex(unbase64('YWJj!!'))
-- 616263
SELECT hex(unbase64('!!!!'))
-- (empty)
SELECT hex(unbase64('Y'))
-- error: IllegalArgumentException: Last unit does not have enough valid bits

Only unbase64('123') was run on DataFusion, through the existing test. The other inputs are listed for Spark's behavior and were not run.

Expected behavior

Decode like the MIME decoder: accept missing padding, skip characters outside the alphabet, and keep an error for a dangling single character. Update the expectations at base64.slt#L109 and #L112.

Additional context

Found by running the queries in datafusion/sqllogictest/test_files/spark on a local Spark and comparing the results with the expected values in the files, which is the kind of check proposed in #7. Spark results come from Apple's build of Spark 4.0.0 (4.0.0.65-apple) with a UTC session and spark.sql.ansi.enabled=false unless stated. The Spark source links point at the v4.0.0 tag. DataFusion results are the sqllogictest expectations at 9dec77584, which CI enforces. Anything marked as not run was derived from reading the code.

Activity

  1. Wali05 commented on Oct 30, 2026

    @Wali05
    Contributor

    take

  2. added a commit that references this issue on Nov 13, 2026
    95e0837
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions