You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f5e4e84
Browse filesBrowse the repository at this point in the historyBrowse files
[SPARK-45746][PYTHON] Return specific error messages if UDTF 'analyze' or 'eval' method accepts or returns wrong values
### What changes were proposed in this pull request?
This PR adds checks to return specific error messages if any Python UDTF `analyze` or `eval` method accepts or returns wrong values.
Error messages improved include:
* If the `__init__` method takes more arguments than `self` and `analyze_result`.
* If the UDTF call passes more or fewer arguments than `analyze` or `eval` expects (not using `*args` or `**kwargs`).
* If the `analyze` method returns an object besides a `StructType` in the `AnalyzeResult` `schema` field.
* If there are extra optional `AnalyzeResult` fields relating to input table arguments (e.g. `with_single_partition`) but the `analyze` method received no input table argument.
* If the `analyze` method tries to return a list of strings for the `partition_by` optional field of the `AnalyzeResult` instead of a list of `PartitioningColumn` objects.
* If the `AnalyzeResult` is missing the `schema` argument entirely.
* If we use keyword arguments in the TVF call but the `analyze` or `eval` method does not accept arguments with those keyword(s) (or `**kwargs`).
### Why are the changes needed?
This helps users understand how to easily fix their user-defined table functions if they are malformed.
### Does this PR introduce _any_ user-facing change?
Yes, see above.
### How was this patch tested?
This PR adds test coverage.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#43611 from dtenedor/fix-more-udtf-errors.
Authored-by: Daniel Tenedorio <[email protected]>
Signed-off-by: Takuya UESHIN <[email protected]>
Copy file name to clipboardExpand all lines: python/pyspark/errors/error_classes.py
+15Lines changed: 15 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -798,6 +798,21 @@
798
798
"Cannot convert the output value of the column '<col_name>' with type '<col_type>' to the specified return type of the column: '<arrow_type>'. Please check if the data types match and try again."
"Failed to evaluate the user-defined table function '<name>' because its constructor is invalid: the function implements the 'analyze' method, but its constructor has more than two arguments (including the 'self' reference). Please update the table function so that its constructor accepts exactly one 'self' argument, or one 'self' argument plus another argument for the result of the 'analyze' method, and try the query again."
804
+
]
805
+
},
806
+
"UDTF_CONSTRUCTOR_INVALID_NO_ANALYZE_METHOD" : {
807
+
"message" : [
808
+
"Failed to evaluate the user-defined table function '<name>' because its constructor is invalid: the function does not implement the 'analyze' method, and its constructor has more than one argument (including the 'self' reference). Please update the table function so that its constructor accepts exactly one 'self' argument, and try the query again."
"Failed to evaluate the user-defined table function '<name>' because the function arguments did not match the expected signature of the 'eval' method (<reason>). Please update the query so that this table function call provides arguments matching the expected signature, or else update the table function so that its 'eval' method accepts the provided arguments, and then try the query again."
814
+
]
815
+
},
801
816
"UDTF_EXEC_ERROR" : {
802
817
"message" : [
803
818
"User defined table function encountered an error in the '<method_name>' method: <error>"
0 commit comments