Repository navigation
Commit 198269d
chore(proto): deprecate
## Which issue does this PR close?
- Follow-up cleanup for the proto hook migration EPIC apache#23494. Does not
close it.
## Rationale for this change
Five public accessor methods on physical plan nodes exist for one reason
only: an older protobuf serialization PR needed to reach a private
struct field from outside the plan's own module. Each was introduced by
the serialization PR that needed it, and none was ever part of an API
anyone asked for.
Now that every one of these plans serializes itself through its own
`try_to_proto` hook — which lives in the same module and can read the
fields directly — the accessors have no callers inside DataFusion.
But "no caller inside DataFusion" is not the same as "no caller". Of the
five, **only one is deprecated** — the other four all turned out to have
a real, non-serialization consumer downstream:
| Method | Introduced by | Callers in DataFusion | Known downstream
caller | This PR |
| --- | --- | --- | --- | --- |
| `AsyncFuncExec::async_exprs` | "[Proto]: Serialization support for
`AsyncFuncExec`" (apache#19118) | none | none found | **deprecated** |
| `AnalyzeExec::verbose` | "Implement protobuf serialization for
AnalyzeExec" (apache#7574) | none | datafusion-distributed, openobserve | kept
as-is |
| `AnalyzeExec::show_statistics` | "Implement protobuf serialization for
AnalyzeExec" (apache#7574) | none | openobserve | kept as-is |
| `UnnestExec::list_column_indices` | "Support encoding and decoding
UnnestExec" (apache#12344) | none | goldsky streamling | kept as-is |
| `UnnestExec::struct_column_indices` | "Support encoding and decoding
UnnestExec" (apache#12344) | none | goldsky streamling | kept as-is |
An earlier revision of this PR deprecated four of the five.
@kumarUjjawal's
[review](apache#24168 (review))
pointed at two downstream projects I had not checked, which between them
use three of those four. Those three deprecations have been reverted;
see [below](#downstream-usage-check).
Every one of the four kept accessors is the same shape: downstream code
downcasts a planned node and needs to read its private fields in order
to rebuild it as its own node. That is a legitimate use, and the fact
that an accessor was *originally added* for proto doesn't make its
current use wrong. Deprecating them would push a warning onto downstream
projects for an API they have a real need for, with nothing to point
them at instead.
## What changes are included in this PR?
Adds `#[deprecated(since = "55.0.0", note = "...")]` to
`AsyncFuncExec::async_exprs`. Nothing is removed, no behavior changes,
and the other four accessors are untouched.
The `note` is honest that there is no replacement: `AsyncFuncExec`
serializes itself through `AsyncFuncExec::try_to_proto`, which reads the
field directly, so there is nothing to point users at. It follows the
existing phrasing used by the deprecated shims in
`datafusion/proto/src/physical_plan/mod.rs` ("unused by DataFusion;
...") combined with the repo's established no-replacement idiom ("please
open an issue if you have a use case for it").
`AsyncFuncExec::async_exprs` already had zero callers before apache#24166; its
`try_to_proto` hook was written against the field from the start.
## Are these changes tested?
There is no new behavior to test — the real verification is that the
compiler agrees the method is unused. Since `deprecated` is a warning
and CI builds with `-D warnings`, a clean lint over the whole workspace
*is* the proof that no internal caller remains.
Run locally on this branch:
- `cargo fmt --all`
- `cargo clippy --all-targets --workspace --features
avro,integration-tests,extended_tests -- -D warnings` (CI's exact
invocation) — clean across every crate, including `datafusion-cli`,
`benchmarks`, `datafusion-examples` and `substrait`
- `cargo test -p datafusion-proto --test proto_integration` — 219
passed, 0 failed
- `cargo test -p datafusion-physical-plan` — passed, 0 failed
## Downstream usage check
Before deprecating, I checked the three main downstream consumers at
their current `main` (2026-08-09), by cloning each repo and grepping for
all five accessor names plus every mention of `AnalyzeExec` /
`UnnestExec` / `AsyncFuncExec`. GitHub code search returned 503s and
silent empty results at the time, which is exactly why the survey missed
two projects — @kumarUjjawal caught both in review.
| Repo | Uses any of the five? | Which |
| --- | --- | --- |
|
[datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed)
(`45bd823`) | Yes | `AnalyzeExec::verbose` |
| [openobserve](https://github.com/openobserve/openobserve) (`575e8ea`)
| Yes | `AnalyzeExec::verbose`, `AnalyzeExec::show_statistics` |
| [goldsky streamling](https://github.com/goldsky-io/streamling)
(`8d85af9`) | Yes | `UnnestExec::list_column_indices`,
`UnnestExec::struct_column_indices` |
| [datafusion-comet](https://github.com/apache/datafusion-comet)
(`c706360`) | No | — |
| [datafusion-ballista](https://github.com/apache/datafusion-ballista)
(`06f8f1d`) | No | — |
**datafusion-distributed — `verbose`.** `src/explain_analyze.rs:34`
builds a `DistributedAnalyzeExec` from `analyze_exec.verbose()`, driven
by a planner that downcasts a planned `AnalyzeExec`
(`src/distributed_planner/distributed_query_planner.rs:96`). It does not
read `show_statistics` — `DistributedAnalyzeExec` doesn't carry that
flag.
**openobserve — `verbose` *and* `show_statistics`.**
[`src/search/src/datafusion/optimizer/physical_optimizer/distribute_analyze.rs:31-37`](https://github.com/openobserve/openobserve/blob/575e8ea4d5fd4b0e630aa013a1a31b1da56708e9/src/search/src/datafusion/optimizer/physical_optimizer/distribute_analyze.rs#L31-L37)
does the same rewrite as datafusion-distributed, but reads both flags:
```rust
if let Some(analyze) = plan.downcast_ref::<AnalyzeExec>() {
let distribute_analyze = Arc::new(DistributeAnalyzeExec::new(
analyze.verbose(),
analyze.show_statistics(),
analyze.input().clone(),
)) as Arc<dyn ExecutionPlan>;
```
**goldsky streamling — both unnest accessors.**
[`crates/streamling-core/src/operators/unnest.rs:85-99`](https://github.com/goldsky-io/streamling/blob/8d85af926b034085aece04233722c508ac80f2d0/crates/streamling-core/src/operators/unnest.rs#L85-L99),
in `StreamingUnnestExec::from_original`, rebuilds a DataFusion
`UnnestExec` as its own streaming operator and reads both index lists to
do it:
```rust
let list_column_indices = original_unnest
.list_column_indices()
.iter()
.map(|idx| ListUnnest { index_in_input_schema: idx.index_in_input_schema, depth: idx.depth })
.collect();
let struct_column_indices = original_unnest.struct_column_indices().to_vec();
```
**datafusion-comet — no usage.** It constructs `UnnestExec::new(...)` in
`native/core/src/execution/planner.rs:2081` but never reads the index
lists back out. Zero hits for any of the five names.
**datafusion-ballista — no usage.** Two near-misses, both false
positives: `ballista/core/src/planner.rs:148` reads `analyze.verbose`,
but that is the public field on the **logical** `LogicalPlan::Analyze`
node, not the physical accessor;
`ballista/scheduler/src/state/distributed_explain.rs:155` calls
`UnnestExec::new(...)`, construction only.
**`AsyncFuncExec::async_exprs`.** Code search for `async_exprs` and for
the literal `async_exprs()` across public Rust code returns hits only in
DataFusion itself and in forks/vendored copies of it
(`ClickHouse/rust_vendor`, `apache/datafusion-sandbox`,
`Epsio-Labs/hiring-datafusion`, `smartdu/datafusion`). The three
non-fork repos that mention `AsyncFuncExec` — `apache/sedona-db`,
`influxdata/datafusion-udf-wasm`, `goldmedal/datafusion-llm-function` —
have no calls to the accessor.
This still covers only what public code search and these five projects
show. If you know of a consumer of `AsyncFuncExec::async_exprs`, say so
and I'll drop the last deprecation too, on the same reasoning applied to
the other four.
## Are there any user-facing changes?
Yes, and the `api change` label applies.
Downstream users who call `AsyncFuncExec::async_exprs` will now see a
deprecation warning. Nothing breaks in this release — the method still
works exactly as before. Removal follows the normal deprecation window
described in the [API health
policy](https://github.com/apache/datafusion/blob/main/docs/source/contributor-guide/api-health.md)
(six major versions or six months, whichever is longer), consistent with
the plan in EPIC apache#23494.
`AnalyzeExec::verbose`, `AnalyzeExec::show_statistics`,
`UnnestExec::list_column_indices` and
`UnnestExec::struct_column_indices` are unchanged, so
datafusion-distributed, openobserve and goldsky streamling see no new
warning.
There is intentionally no replacement API for `async_exprs`. If you have
a use case for reading that field from outside the plan, please open an
issue — that is a real API request worth designing deliberately, rather
than something to leave standing by accident.
---------
Co-authored-by: Claude Opus 5 <[email protected]>AsyncFuncExec::async_exprs, which only existed for proto serialization (apache#24168)1 parent dc69242 commit 198269d
1 file changed
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
109 | 113 | | |
110 | 114 | | |
111 | 115 | | |
| |||
0 commit comments