Skip to content

deps: plan-decode panics in datafusion and arrow-ipc wait on upstream fixes #1703

Open

Description

@geoHeil

A node's worker decodes every plan stage a peer sends it. The release profile sets panic = "abort", so a decoder that panics on malformed bytes stops the whole process. A failed allocation aborts it whatever the profile says. #1680 finding 1 found the sites below. Each one gets a local guard, mostly in a vendored copy. This issue tracks the upstream fixes that let each vendored copy go back to the registry.

Site Crate, version locked Upstream Local guard
an AvroScan node in a build without avro datafusion-proto 55.1.0 apache/datafusion#26998 vendored, patch (1), on main
an Unnest recursion with an unset input_column/output_column; an Explain plan with an unset plan_type datafusion-physical-plan 55.1.0 apache/datafusion#26999 vendored, both sites fixed where they panic, on main (#1710)
Decimal* value bytes of the wrong length; a nested ScalarValue batch with no column; the Constraint conversion; TableParquetOptions unwraps; CsvOptions indexing datafusion-proto-common 55.1.0 apache/datafusion#27004 (fixes 1, 2, 5, 6); the Constraint and TableParquetOptions sites are fixed on main by apache/datafusion#24726, in no 55.x release vendored and fixed (#1719)
buffer offset/length past the body arrow-ipc 59.2.0 fixed on main by apache/arrow-rs#11881, in no 59.x release vendored, backporting the fix (in progress under #1680)
lz4's unbounded read_to_end arrow-ipc 59.2.0 fixed on main by apache/arrow-rs#10959, in no 59.x release same, backporting the fix
the sender-declared uncompressed length reserved up front (lz4 Vec::with_capacity, zstd bulk decompress) arrow-ipc 59.2.0, unchanged on main apache/arrow-rs#12030 (filed with #1721's red-run abort, memory allocation of 9223372036854775807 bytes failed); the parquet analogue is apache/arrow-rs#10734 same, refusing a length above the compressed size times the codec's expansion ratio
create_worker_client decodes messages up to usize::MAX datafusion-distributed 4.0.0 datafusion-contrib/datafusion-distributed#828 a client cap on main (#1715): the decode limit is the working set, floored at 64 KiB so a peer's typed refusal still decodes

bincode 1 (datafusion-contrib/datafusion-distributed#829) is tracked in #1686.

Limits:

Blocked on

Done when

Each fix is in a release the lockfile resolves, and the matching vendored crate or local patch is dropped, per its VENDOR.md removal condition.

Part of #1680

Activity

  1. added
    dependenciesPull requests that update a dependency file
    rustPull requests that update rust code
    on Nov 1, 2026
  2. geoHeil commented on Nov 2, 2026

    @geoHeil
    MemberAuthor

    Status for the datafusion-distributed row:

    Neither is merged, so patches (1) and (2) stay vendored.

  3. geoHeil commented on Nov 4, 2026

    @geoHeil
    MemberAuthor

    Upstream PRs opened for two rows:

    The three apache rows (#26998, #26999, arrow-rs #12030) follow once their fork CI finishes.

  4. geoHeil commented on Nov 4, 2026

    @geoHeil
    MemberAuthor

    Upstream PRs opened for three more rows:

    Each has a fork run where both tests fail without the fix and pass with it; the links are in each PR. Our vendored copies stay until a release with each fix is locked, per VENDOR.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dependenciesPull requests that update a dependency filerustPull requests that update rust code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions