Skip to content

Native lpad and rpad fail when the string argument evaluates to a scalar, such as a scalar subquery inside IF #7245

Open

Description

@andygrove

Describe the bug

lpad and rpad fail natively when their string argument references a column but evaluates to a scalar for the whole batch. CometStringLPad and CometStringRPad only keep a Literal string on Spark, so IF(id > 0, (SELECT max(s) FROM strs), s) is serialized. When every row in the batch takes the subquery branch, the conditional returns a scalar, and the native read_side_padding kernel only accepts an array as its first argument:

org.apache.comet.CometNativeException: Unsupported arguments [Scalar(Utf8("cd")), Scalar(Int32(5)), Scalar(Utf8("0"))] for function rpad/lpad/read_side_padding.

Spark returns the padded value. Default configs are enough to hit this.

Steps to reproduce

Seq((1, "ab"), (2, "cd")).toDF("id", "s").repartition(1).write.parquet("/tmp/strs")
spark.read.parquet("/tmp/strs").createOrReplaceTempView("strs")
sql("SELECT id, lpad(IF(id > 0, (SELECT max(s) FROM strs), s), 5, '0') FROM strs").collect()

rpad, and CASE WHEN id > 0 THEN (SELECT max(s) FROM strs) ELSE s END as the string argument, fail the same way.

Reproduced on main at c4a0b1f with the default Spark 4.1 profile.

Expected behavior

Spark's rows: 000cd for both rows with lpad, and cd000 with rpad.

Additional context

This is the padding half of the scalar-argument problem in #3336. #3336 says hour, minute, second and unix_timestamp need ConstantFolding disabled to fail, but a plain scalar subquery reaches them with default configs too. SELECT hour((SELECT max(ts) FROM t)) FROM t fails with hour(scalar) should be fold in Spark JVM side., and unix_timestamp fails the same way. #5889 carries fixes for both halves that don't depend on its struct support.

Activity

  1. added
    bugSomething isn't working
    priority:mediumFunctional bugs, performance regressions, broken features
    on Oct 26, 2026
  2. tlfua commented on Nov 3, 2026

    @tlfua

    take

  3. tlfua commented on Nov 3, 2026

    @tlfua

    Hi @LinSimon-901101, I've taken this issue and I'm working on the kernel-side fix: making read_side_padding (lpad/rpad) accept a runtime scalar string argument, as suggested in @sunchao's review on #5889 and @andygrove's split plan (padding fix to land before the struct support).

    Since you mentioned moving the independent fixes out of #5889 into separate PRs, just want to make sure we don't duplicate work. Please let me know if you've already started on the padding part. Thanks!

  4. LinSimon-901101 commented on Nov 3, 2026

    @LinSimon-901101
    Contributor

    Hi @tlfua , I haven’t started working on a separate fix for #7245, so please go ahead. Thanks for taking this on!

  5. added a commit that references this issue on Nov 7, 2026
    208b970
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions