Repository navigation
Expand scalar async UDF results to the batch row count - #25828
Merged
Merged
Conversation
github-actions
Bot
added
physical-expr
hareshkh
commented
Sep 28, 2026
| // scalar applies to every row, so it must be expanded to the number of rows in | ||
| // each chunk rather than to a single row. | ||
| #[tokio::test] | ||
| async fn test_async_udf_with_scalar_result() -> Result<()> { |
Contributor
Author
There was a problem hiding this comment.
Without the fix fails with:
Error: ArrowError(InvalidArgumentError("all columns in a record batch must have the same length"), Some(""))
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25828 +/- ##
==========================================
- Coverage 82.59% 82.59% -0.01%
==========================================
Files 1144 1144
Lines 441854 441849 -5
Branches 441854 441849 -5
==========================================
- Hits 364965 364951 -14
- Misses 54809 54815 +6
- Partials 22080 22083 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
hareshkh
marked this pull request as ready for review
September 28, 2026 13:30
xanderbailey
approved these changes
Sep 28, 2026
xanderbailey
left a comment
xanderbailey
left a comment
Contributor
There was a problem hiding this comment.
Verified that the test fails before the fix. Thanks @hareshkh great find!
Contributor
Author
Jefffrey
approved these changes
Sep 29, 2026
Jefffrey
left a comment
Jefffrey
left a comment
Contributor
There was a problem hiding this comment.
its unfortunate that we cant exactly reap the benefits of ColumnarValue::Scalar here and are always forced to expand to arrays, though with how this code is structured (compared to its sync counterpart) i suppose it might be hard to do without a wider refactor
fix makes sense, thanks for the ping 👍
Jefffrey
enabled auto-merge
September 30, 2026 13:42
Contributor
|
thanks @hareshkh & @xanderbailey |
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
## Which issue does this PR close? - Closes apache#25875 ## Rationale for this change When an async UDF returns a `ColumnarValue::Scalar` (e.g. all arguments are literals), queries on more than one row fail with `all columns in a record batch must have the same length`. ## What changes are included in this PR? `AsyncFuncExpr::invoke_with_args` expands each result to its chunk's row count via `into_array_of_size`, instead of turning a scalar into a 1-row array. ## What is the testing strategy for this PR? Added `test_async_udf_with_scalar_result` in `user_defined_async_scalar_functions.rs`. It fails without the fix. ## Are there any user-facing changes? No.
This was referenced Nov 12, 2026
This was referenced Nov 12, 2026
alamb
pushed a commit
that referenced
this pull request
Nov 12, 2026
…25828) (#27372) - Backport of #25828 to `branch-54` - Part of #23805 Co-authored-by: Haresh Khanna <[email protected]>
alamb
pushed a commit
that referenced
this pull request
Nov 12, 2026
…25828) (#27373) - Backport of #25828 to `branch-55` - Part of #26868 Co-authored-by: Haresh Khanna <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
When an async UDF returns a
ColumnarValue::Scalar(e.g. all arguments are literals), queries on more than one row fail withall columns in a record batch must have the same length.What changes are included in this PR?
AsyncFuncExpr::invoke_with_argsexpands each result to its chunk's row count viainto_array_of_size, instead of turning a scalar into a 1-row array.What is the testing strategy for this PR?
Added
test_async_udf_with_scalar_resultinuser_defined_async_scalar_functions.rs. It fails without the fix.Are there any user-facing changes?
No.