Stacked Borrows violation in PushDecoderStreamState::transition

Describe the bug

Miri detects a Stacked Borrows violation in datafusion-datasource-parquet/src/opener.rs at line 1258:

self.decoder.push_ranges(ranges, data)?;

A Unique retag (mutable borrow) of the decoder is created, then invalidated by a SharedReadOnly retag at an .await point in the unfold-based stream. When the future resumes, the original Unique tag no longer exists in the borrow stack.

This was found while testing DataFusion 54.0 with Apache DataFusion Comet. Comet runs Miri in CI and caught this: CI run. DataFusion does not currently run Miri in its own CI.

To Reproduce

Run Miri against any test that exercises the PushDecoderStreamState parquet stream path. In our case it was triggered by a test that calls scan.execute(...).collect().await on a parquet scan.

Expected behavior

No undefined behavior under Miri's Stacked Borrows model.

Miri output

error: Undefined Behavior: trying to retag from <28128923> for SharedReadWrite permission at alloc8513101[0x8], but that tag does not exist in the borrow stack for this location
    --> datafusion/datasource-parquet/src/opener.rs:1258:25
     |
1258 |                         self.decoder.push_ranges(ranges, data)?;
     |                         ^^^^^^^^^^^^ this error occurs as part of two-phase retag at alloc8513101[0x8..0x20]
     |
     = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
     = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <28128923> was created by a Unique retag at offsets [0x8..0x20]
help: <28128923> was later invalidated at offsets [0x0..0x138] by a SharedReadOnly retag

Additional context

The full backtrace points to PushDecoderStreamState::transition -> RowGroupsPrunedParquetOpen::build_stream closure -> futures::stream::Unfold poll. The aliasing violation occurs because the decoder is mutably re-borrowed across a yield point in the unfold stream.

fix: Remove nested async block causing Stacked Borrows violation in P…

@Dandandan

Reopening — the fix in #21663 removed the nested async block but miri still reports a Stacked Borrows violation at the same location (opener.rs:1267). The inlined code still reborrows &mut self across the .await yield point in get_byte_ranges, which conflicts with the subsequent self.decoder.push_ranges() call. Miri sees these as overlapping reborrows of the same &mut self even though they access disjoint fields.

Reproducer: running Comet's miri tests against current DF main (commit 2ef0217) triggers this on test_nested_types_extract_missing_struct_names_missing_field. CI run: https://github.com/apache/datafusion-comet/actions/runs/24518967597/job/71671004017?pr=3916

error: Undefined Behavior: trying to retag from <28193196> for SharedReadWrite permission at alloc8537353[0x8], but that tag does not exist in the borrow stack for this location
    --> datafusion/datasource-parquet/src/opener.rs:1267:45
     |
1267 | ...                   if let Err(e) = self.decoder.push_ranges(ranges, data) {
     |                                       ^^^^^^^^^^^^ this error occurs as part of two-phase retag at alloc8537353[0x8..0x20]

A possible fix would be to destructure self into its fields before the loop so that miri can see the disjoint borrows explicitly, e.g.:

let Self { reader, decoder, baseline_metrics, .. } = self;
// then use reader, decoder, baseline_metrics directly

CC @Dandandan

Draft PR with a fix while I test in Comet CI:

#21680
apache/datafusion-comet#3960

fix: try again to fix Miri in ParquetOpener (#21680)

fix: Remove nested async block causing Stacked Borrows violation in P…

fix: try again to fix Miri in ParquetOpener (apache#21680)

Navigation Menu

Repository navigation

Stacked Borrows violation in `PushDecoderStreamState::transition` #21662

Describe the bug

To Reproduce

Expected behavior

Miri output

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Stacked Borrows violation in PushDecoderStreamState::transition #21662

Description

Describe the bug

To Reproduce

Expected behavior

Miri output

Additional context

Activity

mbutrovich commented on Apr 16, 2026

mbutrovich commented on Apr 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Stacked Borrows violation in `PushDecoderStreamState::transition` #21662