You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native lpad and rpad fail when the string argument evaluates to a scalar, such as a scalar subquery inside IF #7245
lpad and rpad fail natively when their string argument references a column but evaluates to a scalar for the whole batch. CometStringLPad and CometStringRPad only keep a Literal string on Spark, so IF(id > 0, (SELECT max(s) FROM strs), s) is serialized. When every row in the batch takes the subquery branch, the conditional returns a scalar, and the native read_side_padding kernel only accepts an array as its first argument:
org.apache.comet.CometNativeException: Unsupported arguments [Scalar(Utf8("cd")), Scalar(Int32(5)), Scalar(Utf8("0"))] for function rpad/lpad/read_side_padding.
Spark returns the padded value. Default configs are enough to hit this.
Steps to reproduce
Seq((1, "ab"), (2, "cd")).toDF("id", "s").repartition(1).write.parquet("/tmp/strs")
spark.read.parquet("/tmp/strs").createOrReplaceTempView("strs")
sql("SELECT id, lpad(IF(id > 0, (SELECT max(s) FROM strs), s), 5, '0') FROM strs").collect()
rpad, and CASE WHEN id > 0 THEN (SELECT max(s) FROM strs) ELSE s END as the string argument, fail the same way.
Reproduced on main at c4a0b1f with the default Spark 4.1 profile.
Expected behavior
Spark's rows: 000cd for both rows with lpad, and cd000 with rpad.
Additional context
This is the padding half of the scalar-argument problem in #3336. #3336 says hour, minute, second and unix_timestamp need ConstantFolding disabled to fail, but a plain scalar subquery reaches them with default configs too. SELECT hour((SELECT max(ts) FROM t)) FROM t fails with hour(scalar) should be fold in Spark JVM side., and unix_timestamp fails the same way. #5889 carries fixes for both halves that don't depend on its struct support.
Hi @LinSimon-901101, I've taken this issue and I'm working on the kernel-side fix: making read_side_padding (lpad/rpad) accept a runtime scalar string argument, as suggested in @sunchao's review on #5889 and @andygrove's split plan (padding fix to land before the struct support).
Since you mentioned moving the independent fixes out of #5889 into separate PRs, just want to make sure we don't duplicate work. Please let me know if you've already started on the padding part. Thanks!
Activity
tlfua commentedon Nov 3, 2026
take
tlfua commentedon Nov 3, 2026
Hi @LinSimon-901101, I've taken this issue and I'm working on the kernel-side fix: making
read_side_padding(lpad/rpad) accept a runtime scalar string argument, as suggested in @sunchao's review on #5889 and @andygrove's split plan (padding fix to land before the struct support).Since you mentioned moving the independent fixes out of #5889 into separate PRs, just want to make sure we don't duplicate work. Please let me know if you've already started on the padding part. Thanks!
LinSimon-901101 commentedon Nov 3, 2026
Hi @tlfua , I haven’t started working on a separate fix for #7245, so please go ahead. Thanks for taking this on!
fix: support scalar string argument in native lpad/rpad