You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b095406
Browse filesBrowse the repository at this point in the historyBrowse files
fix(proto): preserve Arrow IPC stream format (apache#24224)
## Which issue does this PR close?
- Closesapache#24196.
## Rationale for this change
`ArrowSource` distinguishes between Arrow IPC file and stream formats,
but this format was not serialized. As a result, a stream scan
round-tripped through protobuf as a file scan, selecting the wrong
opener and potentially allowing unsupported range-based repartitioning.
## What changes are included in this PR?
- Add an `ArrowIpcFormat` discriminator to `ArrowScanExecNode`.
- Serialize and restore the `ArrowSource` IPC format.
- Decode payloads without the new field as file format, preserving the
previous behavior.
- Regenerate the prost and pbjson models.
- Add regression coverage for file, stream, and older payloads without
the format field.
## Are these changes tested?
Yes:
- `cargo test -p datafusion-proto --test proto_integration
roundtrip_arrow`
- `cargo test -p datafusion-proto --test proto_integration
arrow_scan_without_format_field_decodes_as_file_format`
- `cargo fmt --all`
- Targeted all-feature clippy for `datafusion-datasource-arrow` and
`datafusion-proto`
## Are there any user-facing changes?
Arrow IPC stream scans now preserve their format across protobuf round
trips.
The binary protobuf change is additive. Older payloads continue to
decode as file format. Stream preservation requires both producer and
consumer to include this change because older versions do not carry or
read the discriminator. The generated Rust `ArrowScanExecNode` struct
gains a `format` field.
0 commit comments