Repository navigation
Commit 206813f
refactor(proto): destructure plan and proto structs in join serde hooks (apache#24164)
## Which issue does this PR close?
<!-- No dedicated issue; this is follow-up cleanup for the proto hook
migration EPIC. -->
- Part of apache#23494.
## Rationale for this change
EPIC apache#23494 moved every built-in `ExecutionPlan` off the central
`downcast_ref` chain
in `datafusion-proto` onto per-plan hooks: a `try_to_proto` override
inside
`impl ExecutionPlan for FooExec`, and an inherent
`FooExec::try_from_proto`.
Those hooks read plan state through **getters** (`self.filter()`,
`self.join_type()`, …).
That makes adding a field to a plan struct invisible to serialization —
nothing breaks
at compile time, the new field is just silently not serialized, and the
bug only shows
up as state quietly disappearing on a round-trip.
This is not hypothetical. `HashJoinExec.fetch` was dropped on round-trip
for exactly
this reason (fixed in apache#24165) — the field was added to the struct and
the encoder,
still calling getters, never learned about it.
Destructuring removes the failure mode. If `try_to_proto` starts with an
exhaustive
`let Self { … } = self;` with **no `..`**, adding a field to the plan
struct is a compile
error in the encoder. If `try_from_proto` destructures the
prost-generated node struct the
same way, adding a field to the `.proto` is a compile error in every
decoder. The prost
structs are plain, all-`pub`, and not `#[non_exhaustive]`, so this
compiles today.
## What changes are included in this PR?
Applies the pattern to the join plans, one commit per plan:
- `CrossJoinExec`
- `NestedLoopJoinExec`
- `SortMergeJoinExec`
- `SymmetricHashJoinExec`
- `HashExpr` / `HashTableLookupExpr`
(`joins/hash_join/partitioned_hash_eval.rs`)
- `HashJoinExec`
Fields that genuinely are not serialized bind to `_` and carry a short
comment saying
why — derived at construction, runtime state, or recomputed by
`new`/`try_new`/the builder
on decode. For example:
```rust
let Self {
left,
right,
on,
filter,
join_type,
sort_options,
null_equality,
// derived from the children's schemas by `try_new` on decode
schema: _,
// runtime metrics, not part of the plan
metrics: _,
// recomputed from `on` and `sort_options` by `try_new` on decode
left_sort_exprs: _,
// recomputed from `on` and `sort_options` by `try_new` on decode
right_sort_exprs: _,
// recomputed by `try_new` on decode
cache: _,
} = self;
```
`HashTableLookupExpr::try_to_proto` deliberately serializes none of its
state — it holds a
runtime `Arc<Map>` and is replaced with `lit(true)`. It gets a
destructure with every field
bound to `_`, so that adding a field there forces a decision instead of
passing unnoticed.
For `HashJoinExec`, the serialized bindings are `left`, `right`, `on`,
`filter`,
`join_type`, `mode`, `projection`, `null_equality`, `null_aware`,
`dynamic_filter` and
`fetch`. The six ignored fields are `join_schema`, `left_fut`,
`random_state`, `metrics`,
`column_indices` and `cache` — `random_state` is the fixed
`HASH_JOIN_SEED` constant that
the builder sets identically on decode, and the rest are runtime state
or recomputed by the
builder.
**No additional unserialized fields were found.** Every field on the
join plans that is
not written to the proto is genuinely derived, recomputed on decode, or
runtime-only.
**This is a pure refactor. The wire format is byte-for-byte unchanged
and no behavior changes.**
Every getter that was replaced returns exactly the field it was replaced
with. The only
incidental changes are in `SymmetricHashJoinExec::try_from_proto` and
`HashJoinExec::try_from_proto`, where four `internal_datafusion_err!`
calls switched to
inlined format args now that the field is bound locally; the message
text they produce is
identical.
## Are these changes tested?
Covered by the existing round-trip tests — that is the point of the
change: if the
destructures had drifted from the encoders, the plans would fail to
round-trip.
Run locally:
- `cargo test -p datafusion-proto --test proto_integration` — 215
passed, including all
five hash-join round-trip tests (`roundtrip_hash_join`,
`roundtrip_hash_join_fetch`,
`roundtrip_hash_join_projection_states`,
`test_hash_join_with_dynamic_filter_roundtrip`,
`roundtrip_sym_hash_join`)
- `cargo test -p datafusion-physical-plan` — 1641 passed
- `cargo clippy --all-targets --workspace --features
avro,integration-tests,extended_tests -- -D warnings`
- `cargo fmt --all`
## Are there any user-facing changes?
No.
---------
Co-authored-by: Claude Opus 5 <[email protected]>1 parent 5091b42 commit 206813f
6 files changed
Lines changed: 326 additions & 180 deletions
File tree
- datafusion/physical-plan/src/joins
- hash_join
- sort_merge_join
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
462 | 462 | | |
463 | 463 | | |
464 | 464 | | |
465 | | - | |
466 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
467 | 482 | | |
468 | 483 | | |
469 | 484 | | |
| |||
492 | 507 | | |
493 | 508 | | |
494 | 509 | | |
495 | | - | |
496 | | - | |
497 | | - | |
498 | | - | |
499 | | - | |
500 | | - | |
501 | | - | |
502 | | - | |
503 | | - | |
504 | | - | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
505 | 517 | | |
506 | 518 | | |
507 | 519 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1859 | 1859 | | |
1860 | 1860 | | |
1861 | 1861 | | |
1862 | | - | |
1863 | | - | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
| 1877 | + | |
| 1878 | + | |
| 1879 | + | |
| 1880 | + | |
| 1881 | + | |
| 1882 | + | |
| 1883 | + | |
| 1884 | + | |
| 1885 | + | |
| 1886 | + | |
| 1887 | + | |
| 1888 | + | |
| 1889 | + | |
| 1890 | + | |
| 1891 | + | |
| 1892 | + | |
1864 | 1893 | | |
1865 | | - | |
1866 | | - | |
| 1894 | + | |
1867 | 1895 | | |
1868 | 1896 | | |
1869 | 1897 | | |
| |||
1873 | 1901 | | |
1874 | 1902 | | |
1875 | 1903 | | |
1876 | | - | |
1877 | | - | |
1878 | | - | |
| 1904 | + | |
| 1905 | + | |
1879 | 1906 | | |
1880 | 1907 | | |
1881 | | - | |
| 1908 | + | |
1882 | 1909 | | |
1883 | 1910 | | |
1884 | 1911 | | |
1885 | 1912 | | |
1886 | 1913 | | |
1887 | | - | |
1888 | | - | |
| 1914 | + | |
| 1915 | + | |
1889 | 1916 | | |
1890 | 1917 | | |
1891 | 1918 | | |
1892 | | - | |
1893 | | - | |
1894 | | - | |
1895 | | - | |
1896 | | - | |
| 1919 | + | |
| 1920 | + | |
| 1921 | + | |
| 1922 | + | |
| 1923 | + | |
| 1924 | + | |
| 1925 | + | |
1897 | 1926 | | |
1898 | 1927 | | |
1899 | 1928 | | |
| |||
1914 | 1943 | | |
1915 | 1944 | | |
1916 | 1945 | | |
1917 | | - | |
| 1946 | + | |
1918 | 1947 | | |
1919 | 1948 | | |
1920 | 1949 | | |
1921 | 1950 | | |
1922 | | - | |
| 1951 | + | |
1923 | 1952 | | |
1924 | | - | |
| 1953 | + | |
1925 | 1954 | | |
1926 | 1955 | | |
1927 | 1956 | | |
| |||
1946 | 1975 | | |
1947 | 1976 | | |
1948 | 1977 | | |
1949 | | - | |
1950 | | - | |
1951 | | - | |
1952 | | - | |
1953 | | - | |
1954 | | - | |
1955 | | - | |
| 1978 | + | |
| 1979 | + | |
| 1980 | + | |
| 1981 | + | |
| 1982 | + | |
| 1983 | + | |
| 1984 | + | |
| 1985 | + | |
| 1986 | + | |
| 1987 | + | |
| 1988 | + | |
| 1989 | + | |
| 1990 | + | |
| 1991 | + | |
| 1992 | + | |
| 1993 | + | |
| 1994 | + | |
| 1995 | + | |
| 1996 | + | |
1956 | 1997 | | |
1957 | 1998 | | |
1958 | 1999 | | |
1959 | | - | |
1960 | | - | |
| 2000 | + | |
1961 | 2001 | | |
1962 | 2002 | | |
1963 | 2003 | | |
| |||
1976 | 2016 | | |
1977 | 2017 | | |
1978 | 2018 | | |
1979 | | - | |
1980 | | - | |
1981 | | - | |
1982 | | - | |
| 2019 | + | |
| 2020 | + | |
1983 | 2021 | | |
1984 | | - | |
| 2022 | + | |
1985 | 2023 | | |
1986 | 2024 | | |
1987 | 2025 | | |
1988 | 2026 | | |
1989 | | - | |
1990 | | - | |
1991 | | - | |
1992 | | - | |
1993 | | - | |
1994 | | - | |
1995 | | - | |
1996 | | - | |
1997 | | - | |
| 2027 | + | |
| 2028 | + | |
| 2029 | + | |
| 2030 | + | |
| 2031 | + | |
| 2032 | + | |
1998 | 2033 | | |
1999 | 2034 | | |
2000 | 2035 | | |
2001 | 2036 | | |
2002 | 2037 | | |
2003 | | - | |
2004 | | - | |
| 2038 | + | |
2005 | 2039 | | |
2006 | 2040 | | |
2007 | 2041 | | |
2008 | 2042 | | |
2009 | 2043 | | |
2010 | | - | |
| 2044 | + | |
2011 | 2045 | | |
2012 | 2046 | | |
2013 | 2047 | | |
| |||
2023 | 2057 | | |
2024 | 2058 | | |
2025 | 2059 | | |
2026 | | - | |
2027 | | - | |
| 2060 | + | |
2028 | 2061 | | |
2029 | 2062 | | |
2030 | 2063 | | |
| |||
2039 | 2072 | | |
2040 | 2073 | | |
2041 | 2074 | | |
2042 | | - | |
| 2075 | + | |
2043 | 2076 | | |
2044 | 2077 | | |
2045 | 2078 | | |
2046 | | - | |
| 2079 | + | |
2047 | 2080 | | |
2048 | 2081 | | |
2049 | 2082 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
208 | 208 | | |
209 | 209 | | |
210 | 210 | | |
211 | | - | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
212 | 221 | | |
213 | 222 | | |
214 | 223 | | |
215 | 224 | | |
216 | 225 | | |
217 | | - | |
218 | | - | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
219 | 230 | | |
220 | 231 | | |
221 | 232 | | |
| |||
243 | 254 | | |
244 | 255 | | |
245 | 256 | | |
246 | | - | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
247 | 266 | | |
248 | 267 | | |
249 | | - | |
250 | | - | |
| 268 | + | |
| 269 | + | |
251 | 270 | | |
252 | 271 | | |
253 | 272 | | |
| |||
394 | 413 | | |
395 | 414 | | |
396 | 415 | | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
397 | 428 | | |
398 | 429 | | |
399 | 430 | | |
| |||
0 commit comments