You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Avoid use of Signature::user_defined for normal functions #12725
Before #12622 is completed, we can introduce signature like Numeric / String / Float / Integer and so on that is used in places more than 1, others use Signature::user_defined by default
The coerce_types() method is only invoked when the signature is TypeSignature::UserDefined. With Signature::any(1), our coerce_types() for dictionary unwrapping is never called, causing dictionary values to be passed directly to accumulators that can't handle them.
I was looking at min_max and found that min/max could be a valid use case for Signature::user_defined for now:
When I changed from Signature::user_defined to Signature::any(1):
* Basic min/max operations work fine
* Dictionary-typed columns fail (wrong results + panics)
The coerce_types() method is only invoked when the signature is TypeSignature::UserDefined. With Signature::any(1), our coerce_types() for dictionary unwrapping is never called, causing dictionary values to be passed directly to accumulators that can't handle them.
Activity
jayzhan211 commentedon Oct 3, 2024
Before #12622 is completed, we can introduce signature like Numeric / String / Float / Integer and so on that is used in places more than 1, others use
Signature::user_definedby defaultRefactor Spark bitshift signature (#18649)
Refactor Spark expm1 signature (#18655)
Refactor substr signature (#18653)
Refactor Spark crc32/sha1 signatures (#18662)
Refactor Spark crc32/sha1 signatures (apache#18662)
Refactor avg & sum signatures away from user defined (#18769)
Refactor spark `bit_get()` signature away from user defined (#18836)
Refactor Spark crc32/sha1 signatures (apache#18662)
Refactor avg & sum signatures away from user defined (apache#18769)
Refactor spark `bit_get()` signature away from user defined (apache#1…
Refactor `to_local_time()` signature away from user_defined (#18707)
Jefffrey commentedon Nov 27, 2025
Current status (as of main fb14d7c):
Should be doable with current API
Aiming to work through these.
power()signature away from user defined #18968sha2implementation #19475asciisignature away fromuser_defined#19513Further investigation (API changes?)
I think these are possible, hopefully extent of API changes needed are quite limited in scope.
Probably stay as user defined
Would be nice if below ones could be refactored away from user defined but likely needs wider API changes to make it possible?
codetyri0n commentedon Nov 27, 2025
can help out with a few here @Jefffrey
3 remaining items
kumarUjjawal commentedon Dec 3, 2025
@Jefffrey I have opened a PR for spark width_bucket.
refactor: Refactor spark make_interval signature away from user defin…
[datafusion-spark]: Refactor make_dt_interval's signature away from u…
codetyri0n commentedon Dec 6, 2025
are you working on hex as well? @Jefffrey - if not I can pick it up
Jefffrey commentedon Dec 6, 2025
I'm not currently looking at it, feel free to pick it up.
refactor: Refactor spark width bucket signature away from user defined (
[datafusion-spark]: Refactor hex's signature away from user_defined (#…
Refactor `power()` signature away from user defined (#18968)
Fix input handling for encoding functions & various refactors (#18754)
kumarUjjawal commentedon Dec 29, 2025
I was looking at min_max and found that min/max could be a valid use case for
Signature::user_definedfor now:When I changed from
Signature::user_definedtoSignature::any(1):The
coerce_types()method is only invoked when the signature isTypeSignature::UserDefined. WithSignature::any(1), ourcoerce_types()for dictionary unwrapping is never called, causing dictionary values to be passed directly to accumulators that can't handle them.@Jefffrey did you get the time to look at it?
Jefffrey commentedon Dec 29, 2025
Perhaps for simplicity we'll keep it as it us until we have better signature support for dictionaries; see: #19458
refactor: Spark `ascii` signature away from `user_defined` (#19513)
Simplify Spark `sha2` implementation (#19475)
Simplify Spark `sha2` implementation (apache#19475)