Skip to content

Port ScalarFunctionExpr to use try_to_proto / try_from_proto #22430

Closed as not planned

Description

@adriangb

Sub-issue of #22418. Port ScalarFunctionExpr to own its protobuf (de)serialization via the try_to_proto / try_from_proto hooks added in #21929, removing it from the central downcast chain.

Reference: copy the Column migration in #21929. Mind the asymmetry — try_to_proto is a trait override, try_from_proto is an inherent fn (see steps).

Where ScalarFunctionExpr lives

datafusion/physical-expr/src/scalar_function.rs (crate datafusion-physical-expr).

Steps

  1. Add try_to_proto inside the impl PhysicalExpr for ScalarFunctionExpr block — it overrides a trait method, so it must live there, feature-gated #[cfg(feature = "proto")].
    ⚠️ Do not put it in an inherent impl ScalarFunctionExpr block. The serializer calls it through &dyn PhysicalExpr, so an inherent method is silently never reached and serialization keeps using the old chain. This is the most common review bounce on these PRs.
  2. Add ScalarFunctionExpr::try_from_proto(node, ctx) as an inherent associated fn (#[cfg(feature = "proto")] impl ScalarFunctionExpr) and wire it into the matching ExprType::… arm in datafusion/proto/src/physical_plan/from_proto.rs.
  3. In this same PR, delete the ScalarFunctionExpr arm from datafusion/proto/src/physical_plan/to_proto.rs and the old ExprType::… decode arm in from_proto.rs. This is not cleanup-for-later — it is the only thing that proves your hook is actually reached. If the roundtrip tests still pass after deleting the arm, it works; if you leave the arm, the tests pass through dead code.
  4. Keep the wire format byte-for-byte identical. expr_id is None here (only DynamicFilterPhysicalExpr sets it).
  5. Keep use datafusion_proto_models::protobuf; inside the method bodies (matches the Column reference).

Tests

  • Existing roundtrip_physical_plan / roundtrip_physical_expr must still pass: cargo test -p datafusion-proto --test proto_integration.
  • Add a direct hook test in datafusion/physical-expr/src/scalar_function.rs covering both directions and a bad-input case (wrong expr_type variant / missing child → clean error, not panic). Reviewers have asked for this on every reviewed PR.
  • Put #[cfg(test)] mod tests at the end of the file or clippy -D warnings fails (items-after-test-module).

Before you push

  • Fill in every section of the PR template.
  • cargo fmt --all and cargo clippy --all-targets --all-features -- -D warnings must pass.

Checklist

  • try_to_proto in impl PhysicalExpr for ScalarFunctionExpr (not inherent), #[cfg(feature="proto")]
  • ScalarFunctionExpr::try_from_proto wired into from_proto.rs
  • central to_proto.rs + from_proto.rs arms deleted in this PR
  • direct hook test incl. bad-input case; test module at end of file
  • roundtrip tests pass; fmt + clippy -D warnings clean; PR template filled in

Activity

  1. adriangb commented on May 28, 2026

    @adriangb
    ContributorAuthor

    Please base your PR on the helpers documented in #22418 (added in #22596). If you've already written this against the older inline pattern, the migration is small — see the second commit of #22596 for examples.

  2. omer-cengel commented on May 29, 2026

    @omer-cengel

    Hi @adriangb !

    If it’s OK with you, I’d like to work on this issue.

  3. adriangb commented on May 29, 2026

    @adriangb
    ContributorAuthor

    There was some discussion already in #22418 (comment), lets make sure we are not duplicating work.

    @ernestprovo23 do you plan on working on this?

  4. hexbinoct commented on Jun 1, 2026

    @hexbinoct

    Building on ernestprovo23's question and adriangb's answer above — one wrinkle for whoever implements it.

    The proposed decode_udf(...) -> Result<Arc<ScalarUDF>> and encode_udf(&ScalarUDF) name ScalarUDF, which lives in datafusion-expr. But the ctx traits are in datafusion-physical-expr-common, and datafusion-expr already depends on that crate, so it can't depend back (even optionally) without a cycle — and ScalarUDF isn't re-exported from datafusion-expr-common. So those two methods likely need some indirection: a type-erased handle, or keeping the UDF encode/decode on the proto-side PhysicalExprEncode/PhysicalExprDecode impls (which live in datafusion-proto and can see ScalarUDF) rather than as ScalarUDF-typed methods on the public ctx struct.

    config_options() -> Arc<ConfigOptions> is fine — ConfigOptions is in datafusion-common.

    @ernestprovo23 looks set to take this; just flagging to save a compile-cycle surprise.

  5. Olawoyin007 commented on Jul 9, 2026

    @Olawoyin007

    I'd like to pick this up - it looks like the earlier discussion stalled. I'll follow the helper pattern from #22596 with the Column migration in #21929 as the reference, and account for the decode-side UDF lookup wrinkle discussed above. Planning to have a PR up in the next few days.

  6. adriangb commented on Jul 10, 2026

    @adriangb
    ContributorAuthor

    After discussion / experimentation in the linked PRs we've come to the conclusion that this isn't worth doing. ScalarUDF will stay as a "special case" for now. If we need to support it more generically so that e.g. expressions wrapping a ScalarUDF can declare custom serialization we should probably go with #23421. Thanks all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions