Repository navigation
Closed as not planned
Labels
good first issueGood for newcomersGood for newcomers
Description
Sub-issue of #22418. Port ScalarFunctionExpr to own its protobuf (de)serialization via the try_to_proto / try_from_proto hooks added in #21929, removing it from the central downcast chain.
Reference: copy the Column migration in #21929. Mind the asymmetry — try_to_proto is a trait override, try_from_proto is an inherent fn (see steps).
Where ScalarFunctionExpr lives
datafusion/physical-expr/src/scalar_function.rs (crate datafusion-physical-expr).
Steps
- Add
try_to_protoinside theimpl PhysicalExpr for ScalarFunctionExprblock — it overrides a trait method, so it must live there, feature-gated#[cfg(feature = "proto")].
⚠️ Do not put it in an inherentimpl ScalarFunctionExprblock. The serializer calls it through&dyn PhysicalExpr, so an inherent method is silently never reached and serialization keeps using the old chain. This is the most common review bounce on these PRs. - Add
ScalarFunctionExpr::try_from_proto(node, ctx)as an inherent associated fn (#[cfg(feature = "proto")] impl ScalarFunctionExpr) and wire it into the matchingExprType::…arm indatafusion/proto/src/physical_plan/from_proto.rs. - In this same PR, delete the
ScalarFunctionExprarm fromdatafusion/proto/src/physical_plan/to_proto.rsand the oldExprType::…decode arm infrom_proto.rs. This is not cleanup-for-later — it is the only thing that proves your hook is actually reached. If the roundtrip tests still pass after deleting the arm, it works; if you leave the arm, the tests pass through dead code. - Keep the wire format byte-for-byte identical.
expr_idisNonehere (onlyDynamicFilterPhysicalExprsets it). - Keep
use datafusion_proto_models::protobuf;inside the method bodies (matches theColumnreference).
Tests
- Existing
roundtrip_physical_plan/roundtrip_physical_exprmust still pass:cargo test -p datafusion-proto --test proto_integration. - Add a direct hook test in
datafusion/physical-expr/src/scalar_function.rscovering both directions and a bad-input case (wrongexpr_typevariant / missing child → clean error, not panic). Reviewers have asked for this on every reviewed PR. - Put
#[cfg(test)] mod testsat the end of the file orclippy -D warningsfails (items-after-test-module).
Before you push
- Fill in every section of the PR template.
cargo fmt --allandcargo clippy --all-targets --all-features -- -D warningsmust pass.
Checklist
-
try_to_protoinimpl PhysicalExpr for ScalarFunctionExpr(not inherent),#[cfg(feature="proto")] -
ScalarFunctionExpr::try_from_protowired intofrom_proto.rs - central
to_proto.rs+from_proto.rsarms deleted in this PR - direct hook test incl. bad-input case; test module at end of file
- roundtrip tests pass;
fmt+clippy -D warningsclean; PR template filled in
Metadata
Metadata
Assignees
Labels
good first issueGood for newcomersGood for newcomers
Activity
adriangb commentedon May 28, 2026
Please base your PR on the helpers documented in #22418 (added in #22596). If you've already written this against the older inline pattern, the migration is small — see the second commit of #22596 for examples.
omer-cengel commentedon May 29, 2026
Hi @adriangb !
If it’s OK with you, I’d like to work on this issue.
adriangb commentedon May 29, 2026
There was some discussion already in #22418 (comment), lets make sure we are not duplicating work.
@ernestprovo23 do you plan on working on this?
hexbinoct commentedon Jun 1, 2026
Building on ernestprovo23's question and adriangb's answer above — one wrinkle for whoever implements it.
The proposed
decode_udf(...) -> Result<Arc<ScalarUDF>>andencode_udf(&ScalarUDF)nameScalarUDF, which lives indatafusion-expr. But the ctx traits are indatafusion-physical-expr-common, anddatafusion-expralready depends on that crate, so it can't depend back (even optionally) without a cycle — andScalarUDFisn't re-exported fromdatafusion-expr-common. So those two methods likely need some indirection: a type-erased handle, or keeping the UDF encode/decode on the proto-sidePhysicalExprEncode/PhysicalExprDecodeimpls (which live indatafusion-protoand can seeScalarUDF) rather than asScalarUDF-typed methods on the public ctx struct.config_options() -> Arc<ConfigOptions>is fine —ConfigOptionsis indatafusion-common.@ernestprovo23 looks set to take this; just flagging to save a compile-cycle surprise.
Olawoyin007 commentedon Jul 9, 2026
I'd like to pick this up - it looks like the earlier discussion stalled. I'll follow the helper pattern from #22596 with the Column migration in #21929 as the reference, and account for the decode-side UDF lookup wrinkle discussed above. Planning to have a PR up in the next few days.
feat: add typed function-codec and session-config capabilities to pro…
feat: port ScalarFunctionExpr to try_to_proto / try_from_proto
feat: add typed function-codec and session-config capabilities to pro…
feat: port ScalarFunctionExpr to try_to_proto / try_from_proto
adriangb commentedon Jul 10, 2026
After discussion / experimentation in the linked PRs we've come to the conclusion that this isn't worth doing. ScalarUDF will stay as a "special case" for now. If we need to support it more generically so that e.g. expressions wrapping a ScalarUDF can declare custom serialization we should probably go with #23421. Thanks all!