You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2ec55f9
Browse filesBrowse the repository at this point in the historyBrowse files
## Which issue does this PR close?
- Backport of #22950 to `branch-54` (for 54.1.0, tracked in #22547).
- Fixes panic #22935 (reporter bisected to a regression between 53.1.0
and 54.0.0; introduced by #20047).
## Rationale for this change
Anonymous file reads can read the same path with different explicit
schemas in one session. The shared file-statistics cache did not
validate that cached statistics matched the read schema, so a narrower
cached entry could be reused for a later wider-schema read and panic
during statistics projection (`index out of bounds` in projection). No
API change, so it fits the backport criteria.
## What changes are included in this PR?
This is an **adapted** backport. The core fix (`table.rs`, `options.rs`,
`context/mod.rs`) is applied; two conflicts were resolved for
`branch-54`'s API shape:
- `ListingTable::statistics_cache()` returns `Option<&Arc<dyn
FileStatisticsCache>>` (on `branch-54`, `FileStatisticsCache` is a
trait, not a concrete type as on `main`).
- Kept `branch-54`'s import paths for `TableScopedPath` /
`FileStatisticsCache`.
- The regression test
`anonymous_parquet_stats_cache_with_explicit_wider_schema` computes
statistics via `plan.partition_statistics(None)` instead of
`statistics_with_args`, which does not exist on `branch-54`.
## Are these changes tested?
Yes. The regression test passes with the fix and, without it, reproduces
the exact reported panic (`index out of bounds: the len is 2 but the
index is 2`), confirming it exercises the regression.
## Are there any user-facing changes?
No API change. Fixes a panic on anonymous multi-schema reads of the same
path.
Co-authored-by: Kumar Ujjawal <[email protected]>
/// Returns whether the read schema was inferred or specified.
600
+
fnschema_source(&self) -> SchemaSource{
601
+
SchemaSource::Specified
602
+
}
603
+
598
604
/// helper function to reduce repetitive code. Infers the schema from sources if not provided. Infinite data sources not supported through this function.
599
605
asyncfn_get_resolved_schema(
600
606
&'aself,
@@ -653,6 +659,10 @@ impl ReadOptions<'_> for CsvReadOptions<'_> {
0 commit comments