You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5a65e93
Browse filesBrowse the repository at this point in the historyBrowse files
Restore the From / TryFrom proto conversions dropped since 54.1.0 (apache#24205)
## Which issue does this PR close?
- Closesapache#24019.
- closesapache#23494
## Rationale for this change
`datafusion-proto` 54.1.0 publishes 39 `From` / `TryFrom` impls
converting between DataFusion types and their protobuf messages. On
`main` all of them were replaced by the crate-local `FromProto` /
`TryFromProto` traits introduced in apache#21929, so code written against the
released version stops compiling:
```rust
let proto = protobuf::PartitionedFile::try_from(&file)?; // no longer resolves on main
let frame = WindowFrame::try_from(proto_frame)?; // no longer resolves on main
```
That was collateral damage from the orphan-rule workaround, which was
needed during the migration but can now be unwound to result in no
breaking change across releases.
## What changes are included in this PR?
Each conversion moves to a crate that owns one side of it, and goes back
to being a plain `From` / `TryFrom` — the shape 54.1.0 published. Error
types are unchanged (`FromProtoError` decoding, `ToProtoError` encoding,
`DataFusionError` for the datasource types).
| Types | New home |
|---|---|
| `PartitionedFile`, `FileRange`, `FileGroup`, `JsonSink`, `CsvSink`,
`ParquetSink`, `FileSinkConfig` | already moved by apache#24006 / apache#23781 —
this PR just deletes the `TryFromProto` shims that delegated to them |
| `WindowFrame`, `WindowFrameBound`, `WindowFrameUnits`,
`MergeIntoClauseKind`, `NullTreatment` | `datafusion-expr`, behind a new
`proto` feature (optional `datafusion-proto-common` /
`datafusion-proto-models` deps, mirroring `datafusion-datasource`) |
| `UnnestOptions`, `TableReference`, `StringifiedPlan`, `JoinType`,
`JoinConstraint`, `NullEquality`, `CsvOptions`, `JsonOptions`, and the
parquet options types | `datafusion-proto-models`, on the local proto
type — their DataFusion side sits *below* that crate in the graph, the
same arrangement `datafusion-proto-common` already uses for
`ScalarValue` / `Statistics` |
| `CsvFormatFactory`, `JsonFormatFactory`, `ParquetFormatFactory` |
`datafusion-datasource-{csv,json,parquet}`, behind each crate's existing
`proto` feature |
| `Column` <-> `protobuf::PhysicalColumn` | `datafusion-physical-expr`;
`Column::try_to_proto` / `try_from_proto` now go through it instead of
building the message inline |
`TryFrom<&[PartitionedFile]> for protobuf::FileGroup` needed one extra
step. `datafusion-datasource` cannot host it — `&T` is `#[fundamental]`
but `[T]` is not, so `&[PartitionedFile]` counts as foreign there
(`error[E0117]: slices are always foreign`). But in
`datafusion-proto-models` the *self* type is local, which is all the
orphan rule needs, and staying generic over the element avoids naming
`PartitionedFile`, which sits above that crate in the graph:
```rust
impl<T> TryFrom<&[T]> for protobuf::FileGroup
where
for<'a> &'a T: TryInto<protobuf::PartitionedFile, Error = DataFusionError>,
{ ... }
```
The bound is satisfied by `TryFrom<&PartitionedFile> for
protobuf::PartitionedFile` in `datafusion-datasource`, so
`protobuf::FileGroup::try_from(&files[..])` resolves for callers exactly
as it did in 54.1.0.
Two items beyond the issue's checklist, both needed to reach zero
implementors:
- the parquet options conversions (`ParquetOptions`,
`TableParquetOptions`, `ParquetColumnOptions`, `ParquetCdcOptions`) —
the issue's table undercounts `file_formats.rs` because they live in a
private module, but trait impls are global, so they were public API too.
They return as `TryFrom`; `main` had already made them fallible, so an
exact restore of 54.1.0's infallible `From` isn't available.
- `From<&protobuf::PhysicalColumn> for Column`, which the issue's
evidence table counts but no work item names.
Not restored, and worth calling out: `From<protobuf::dml_node::Type> for
WriteOp` and its reverse. `main` replaced them with `parse_write_op` /
`serialize_write_op` because `MergeInto` carries a payload a `From` impl
cannot express. That is a separate, deliberate change.
Finally, `convert.rs` and `convert_required_proto!` are deleted. apache#21929
introduced `FromProto` / `TryFromProto` so the `datafusion-proto-models`
extraction could land without relocating ~39 conversions at the same
time, and flagged them there as "a known workaround, not the end state",
with dropping them listed under Future work. With every conversion moved
they have no implementors and no callers. Neither trait has ever shipped
in a release, so they are removed outright rather than deprecated —
there is nothing for downstream users to migrate off, and doing it now
keeps the workaround out of the released API entirely.
## Are these changes tested?
Yes.
- New `datafusion/proto/tests/cases/public_conversions.rs` coerces all
45 proto conversions in the touched crates to `fn` pointers (the 39 from
54.1.0 plus the ones added on `main`). This is the regression guard the
issue asks for: it fails to compile when an impl is removed, and stays
quiet when one merely moves between crates, which is exactly the case
`cargo-semver-checks` cannot see.
- New round-trip tests next to the moved impls in `datafusion-expr` and
`datafusion-proto-models` (window frames, table references, join enums,
unnest options, stringified plans).
- The `PartitionedFile` tests move from `datafusion-proto` to
`datafusion-datasource`, alongside the logic they cover; two that
duplicated existing coverage there are dropped.
- Existing round-trip suites (`roundtrip_logical_plan`,
`roundtrip_physical_plan`) pass unchanged, which is the real wire-format
check.
- Every moved impl body was diffed against `main`: 22 are byte-identical
modulo the trait rename, and the other 9 differ only by `Self::`
shorthand, error-type aliasing, and rustfmt reflow. No serialization
logic changed.
- `./dev/rust_lint.sh`, `cargo machete`, and the extended test suite all
pass at HEAD. Also checked: `datafusion-proto` without `parquet`,
`datafusion-expr` with `proto` off and `--no-default-features`, the
format crates without `proto`, and `json` on both proto crates.
## Are there any user-facing changes?
Yes, and they restore rather than break the released API.
- The 39 conversions removed since 54.1.0 compile again. Trait impls are
global, so `X::try_from(&proto)` / `proto.try_into()` resolve regardless
of which crate now hosts the impl — no import changes needed, and no
upgrade-guide entry for the moves.
- One genuine delta remains: the parquet options conversions are
`TryFrom` rather than 54.1.0's infallible `From`. That predates this PR
— `main` had already made them fallible — but it is a real 54.1.0 ->
55.0.0 break and was undocumented, so it is now in the 55.0.0 upgrade
guide with a migration snippet.
- `FromProto` / `TryFromProto` and `convert_required_proto!` are gone.
Not a breaking change: they exist only on `main` and appear nowhere in
54.0.0 or 54.1.0.
- `datafusion-expr` gains an off-by-default `proto` feature. Additive.
- `datafusion-proto-models` gains a direct `datafusion-common`
dependency (already present transitively) and two new public modules.
Keeping the `api change` label for the parquet options fallibility.
---------
Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
0 commit comments