Repository navigation
Open
Labels
dependenciesPull requests that update a dependency filePull requests that update a dependency filerustPull requests that update rust codePull requests that update rust code
Description
A node's worker decodes every plan stage a peer sends it. The release profile sets panic = "abort", so a decoder that panics on malformed bytes stops the whole process. A failed allocation aborts it whatever the profile says. #1680 finding 1 found the sites below. Each one gets a local guard, mostly in a vendored copy. This issue tracks the upstream fixes that let each vendored copy go back to the registry.
| Site | Crate, version locked | Upstream | Local guard |
|---|---|---|---|
an AvroScan node in a build without avro |
datafusion-proto 55.1.0 |
apache/datafusion#26998 | vendored, patch (1), on main |
an Unnest recursion with an unset input_column/output_column; an Explain plan with an unset plan_type |
datafusion-physical-plan 55.1.0 |
apache/datafusion#26999 | vendored, both sites fixed where they panic, on main (#1710) |
Decimal* value bytes of the wrong length; a nested ScalarValue batch with no column; the Constraint conversion; TableParquetOptions unwraps; CsvOptions indexing |
datafusion-proto-common 55.1.0 |
apache/datafusion#27004 (fixes 1, 2, 5, 6); the Constraint and TableParquetOptions sites are fixed on main by apache/datafusion#24726, in no 55.x release |
vendored and fixed (#1719) |
| buffer offset/length past the body | arrow-ipc 59.2.0 |
fixed on main by apache/arrow-rs#11881, in no 59.x release |
vendored, backporting the fix (in progress under #1680) |
lz4's unbounded read_to_end |
arrow-ipc 59.2.0 |
fixed on main by apache/arrow-rs#10959, in no 59.x release |
same, backporting the fix |
the sender-declared uncompressed length reserved up front (lz4 Vec::with_capacity, zstd bulk decompress) |
arrow-ipc 59.2.0, unchanged on main |
apache/arrow-rs#12030 (filed with #1721's red-run abort, memory allocation of 9223372036854775807 bytes failed); the parquet analogue is apache/arrow-rs#10734 |
same, refusing a length above the compressed size times the codec's expansion ratio |
create_worker_client decodes messages up to usize::MAX |
datafusion-distributed 4.0.0 |
datafusion-contrib/datafusion-distributed#828 | a client cap on main (#1715): the decode limit is the working set, floored at 64 KiB so a peer's typed refusal still decodes |
bincode 1 (datafusion-contrib/datafusion-distributed#829) is tracked in #1686.
Limits:
- Cells have run in CI for three rows: the
AvroScanrow, thedatafusion-physical-planrow (fix(exec-datafusion): refuse a peer's malformed Explain or Unnest #1710) and the client-cap row (fix(exec-datafusion): cap a peer reply's gRPC decode at the working set #1715). For thedatafusion-proto-commonrow, fix(exec-datafusion): refuse a peer's malformed stage, not panic #1719's test-only CI run observed one panic (constraint_mode not set); nextest's fail-fast cancelled the other four cells, so those four rest on reading the locked source. For thearrow-ipcrows, fix(flight): refuse a peer's Arrow IPC that panics or aborts a node #1721's test-only CI run (6a6012f) observed the out-of-bounds buffer panic (arrow-bufferimmutable.rs:288) and the lz4 abort (SIGABRT). It also observed a decimal bit-width panic (convert.rs:487, fixed onmainby fix(arrow-ipc): return an error instead of panicking on malformed IPC schema apache/arrow-rs#10647) and a negative fixed-size-binary width that is accepted, not refused (fixed onmainby fix(arrow-ipc): reject negative FixedSizeBinary byte width in IPC schema apache/arrow-rs#11883). Nextest's fail-fast cancelled the union and zstd cells, so those two rest on reading the source.
Blocked on
- datafusion-proto: decoding an AvroScan node without the avro feature panics apache/datafusion#26998
- physical-plan: decoding an Unnest or Explain node with an unset field panics apache/datafusion#26999
- proto-common: decoding a malformed scalar, union type or CSV options panics instead of erroring apache/datafusion#27004
- a
datafusion-proto-commonrelease with fix(proto): check integer conversions in common options and constraints apache/datafusion#24726 that this graph resolves - an
arrow-ipc59.x or 60 release with fix(arrow-ipc): return an error instead of panicking on out-of-bounds IPC buffers apache/arrow-rs#11881 and #10959 that this graph resolves - arrow-ipc: a sender-declared uncompressed length sizes an up-front allocation, aborting on i64::MAX apache/arrow-rs#12030 -
arrow-ipcup-front allocation from a sender-declared length - create_worker_client hard-codes usize::MAX message limits with no way to choose others datafusion-contrib/datafusion-distributed#828
- FederationOptimizerRule leaves a plan that is only a TableScan unfederated datafusion-contrib/datafusion-federation#190 (a root leaf is left unfederated). Fixed upstream by fix: Federate a plan that is only a table scan datafusion-contrib/datafusion-federation#192 and released in 0.5.9. The vendored copy was dropped in build(deps): take datafusion-federation 0.5.9 from the registry #1765.
Done when
Each fix is in a release the lockfile resolves, and the matching vendored crate or local patch is dropped, per its VENDOR.md removal condition.
Part of #1680
Metadata
Metadata
Assignees
Labels
dependenciesPull requests that update a dependency filePull requests that update a dependency filerustPull requests that update rust codePull requests that update rust code
Activity
fix(exec-datafusion): refuse a peer's malformed stage, not panic
geoHeil commentedon Nov 2, 2026
Status for the
datafusion-distributedrow:WorkerServiceClientis enough to set a decode limit. It is: aChannelResolvercan returnWorkerServiceClient::new(channel).max_decoding_message_size(limit). We replied that a re-export covers sutura's patch (2), and offered a PR with a doctest.Neither is merged, so patches (1) and (2) stay vendored.
geoHeil commentedon Nov 4, 2026
Upstream PRs opened for two rows:
WorkerServiceClientpublic datafusion-contrib/datafusion-distributed#849: re-exports the generated client asgrpc::WorkerServiceClient, so aChannelResolvercan setmax_decoding_message_size. A doctest covers it. Fork CI is green. Our client cap (fix(exec-datafusion): cap a peer reply's gRPC decode at the working set #1715) stays until a release with it is locked.The three apache rows (#26998, #26999, arrow-rs #12030) follow once their fork CI finishes.
geoHeil commentedon Nov 4, 2026
Upstream PRs opened for three more rows:
AvroScanarm and the two logicalListingScanarms returnnot_impl_err!instead of panicking when the needed feature is off.UnnestExecandExplainExecdecoding return an error naming the unset field. The same panics in the logicalFromimpls are left for a follow-up, because fixing them changes a public API.CompressionCodec::decompressrefuses a declared size above the compressed length times the codec's maximum expansion ratio.Each has a fork run where both tests fail without the fix and pass with it; the links are in each PR. Our vendored copies stay until a release with each fix is locked, per
VENDOR.md.