You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Rust UDF adapter rebuilds the kernel impl and re-resolves the return type on every batch #5296
ImportedCScalarUdf::invoke_with_args does the following on every batch:
builds a fresh CometCScalarKernelImpl via new_impl,
re-encodes every argument Field to an FFI_ArrowSchema,
calls the kernel's init, which re-invokes its return_field,
decodes the resulting FFI_ArrowSchema back into a Field,
drops the impl, running its release callback.
Argument types do not change across batches for a given ScalarFunctionExpr, and return_type() already did the same work once at planning time, so steps 2 and 4 are pure repetition.
Two separable pieces of work:
Drop the return-type decode (small). DataFusion 54's ScalarFunctionArgs carries return_field, so Field::try_from(&out_schema) in invoke_with_args is unnecessary. Careful, though: today the output array is imported with the type the kernel reported for this batch, and switching to the plan-time type means from_ffi_and_data_type receives a type that can differ in nested nullability from what the kernel actually produced — the same asymmetry the declared-vs-actual check in planner.rs deliberately erases. Needs the complex-type matrix in CometRustUdfSuite re-run rather than a spot check.
Cache the impl across batches (larger, ABI decision). The per-batch impl lifecycle is currently documented in comet-udf-sdk, so a kernel author may assume a fresh impl per call. Reusing one across batches changes that contract and should be decided deliberately, alongside the thread-safety question in the Mutex issue.
Activity
docs: point the Rust UDF follow-up comments at their tracking issues
feat: custom Rust UDFs via arrow-ffi [experimental] (apache#4459)