[Feature Request] Opt-in lenient schema mode to restore DF 51 SchemaAdapter behavior for Parquet reading

Problem

After upgrading from DataFusion 51 to 52, we hit several hard errors reading Parquet files that worked fine in DF 51. The root cause is that DF 52 removed SchemaAdapter (PR #18998), which previously handled schema mismatches transparently via map_batch().

In real-world data lakes, Parquet files accumulated over years often have schema inconsistencies that are unavoidable:

Nullability mismatch: Files declare columns as REQUIRED (non-nullable) but actual data contains nulls — common with schema evolution or different writer implementations
List/Struct inner field differences: e.g. List(Field("conditions", Int32, false)) vs List(Field("element", Int32, true)) — different Parquet writers use different field naming and nullability conventions
Type evolution: Older files store columns as Utf8, newer table schema expects Date32

These are not bugs — they are a natural reality of long-lived data lakes with multiple writers. DF 51 handled all of these gracefully and silently.

How DF 51 handled this (worked perfectly)

SchemaAdapter::map_batch() ran after the Parquet reader produced each RecordBatch:

Rebuilt batches using the table schema, bypassing nullability validation entirely
Called arrow::compute::cast_with_options() on each column for type coercion (e.g. Utf8 → Date32)
Transparently handled missing columns, renamed fields, and nullability differences

This was a safe, battle-tested approach that made DataFusion robust for real-world Parquet files.

What breaks in DF 52

DF 52 replaced SchemaAdapter with PhysicalExprAdapterFactory, which only adapts filter/projection expressions — not batch data. Now:

RecordBatch::try_new_with_options (arrow-rs) strictly validates !field.is_nullable() && null_count > 0 and rejects batches. This happens inside the Parquet reader stream, before any user code can intercept.
ArrowReaderMetadata::try_new (parquet-rs) rejects schema overrides where nullability differs — even for safe widening (non-nullable → nullable):
```
Incompatible supplied Arrow schema: nullability mismatch for field X: expected true but found false
```
No escape hatch: There's no option to skip validation in RecordBatch, ArrowReaderMetadata, or ParquetRecordBatchStreamBuilder. We cannot work around this from the outside.

Concrete errors we hit

# Non-nullable column with null data
Arrow error: Invalid argument error: Column 'conditions' is declared as 
non-nullable but contains null values

# Schema override rejected when trying to force nullable
Parquet error: Arrow: Incompatible supplied Arrow schema: 
nullability mismatch for field location: expected true but found false

Workarounds attempted (all failed or insufficient)

Approach	Result
Force nullable in `TableSchema` passed to `ParquetSource`	Parquet reader still uses file's physical schema internally → `RecordBatch` validation fails before our code runs
Override schema via `ArrowReaderOptions::with_schema()`	`ArrowReaderMetadata::try_new` rejects nullable widening
Wrap stream to fix batches post-read	Error occurs inside stream, batch never emitted
Port `SchemaAdapter::map_batch()` logic into the opener	Partially works, but breaks on multiple layers — `ProjectionExprs`, `ArrowReaderMetadata`, `RecordBatch` all validate independently

Proposal: restore DF 51's automatic schema adaptation as an opt-in feature

We believe we are not the only downstream affected by this breaking change. Any data lake with historical Parquet files from different writers or schema evolution will hit similar issues.

Could DataFusion provide an opt-in option to restore DF 51's SchemaAdapter behavior? Specifically:

A ParquetSource or ParquetOpener option (e.g. with_lenient_schema(true) or with_schema_adapter(true)) that, when enabled:
- Allows nullable widening in ArrowReaderMetadata::try_new (non-nullable file → nullable table schema is always safe)
- Applies arrow::compute::cast() for type mismatches (like the old map_batch)
- Rebuilds RecordBatch with the table schema after reading, skipping strict nullability validation
This should be opt-in (default off) to preserve the current strict behavior for users who want it, while giving data lake users a migration path.
Alternatively, relaxing the ArrowReaderMetadata::try_new check in arrow-rs to allow nullable widening (non-nullable → nullable) would solve the most common case. This is always safe — non-nullable data is trivially valid in a nullable column. The current check treats widening and narrowing as equally invalid, which seems overly strict.

Impact on complex downstream systems

Our system is a large-scale data platform built on DataFusion with extensive customizations:

Custom physical operators (sort-preserving joins, cooperative execution, reverse scan, OneOf routing)
Custom ParquetFileReaderFactory with caching and instrumented I/O
Custom ExecutionPlanFactory that wraps ParquetSource/ReverseParquetSource with adapted schemas
Materialized Views with their own table schemas that propagate through ProjectionExec output

The removal of SchemaAdapter affects every layer that touches Parquet schema — not just the reader, but also projections, partition column injection, expression evaluation, and output schema validation. Because validation happens independently at multiple levels (ArrowReaderMetadata, RecordBatch, ProjectionExprs), fixing it in one place often exposes failures in another. This made the DF 51 → 52 upgrade extremely difficult and error-prone for us.

An upstream-supported compatibility strategy would be far more reliable than downstream workarounds, since the schema adaptation needs to be integrated at the core reader level where arrow-rs and DataFusion interact.

Without this, upgrading from DF 51 to DF 52 requires either:

Rewriting all historical Parquet files (impractical for petabyte-scale lakes)
Forking both DataFusion and arrow-rs to patch validation at multiple layers (what we're currently doing — fragile and hard to maintain)
Staying on DF 51 indefinitely

None of these are sustainable. We'd love to hear if others have encountered similar issues and what the community's recommended approach is.

Environment

DataFusion 51 → 52 upgrade
arrow-rs / parquet 57.x
Production data lake with ~5 years of Parquet files from multiple writers/pipelines

@adriangb

cc @adriangb @alamb @xudong963

The new system handles all of the cases the old system does. This is evidenced by all tests passing. We also have parquet files with null columns that are marked as non-nullable at the table level.

Are you able to give a MRE that worked with DF 51 but does not work with DF 52 (or main)?

@adriangb

Thanks @adriangb for reply:

In DF 51, SchemaAdapterFactory was built into ParquetOpener itself — meaning all users of ParquetSource got batch-level schema adaptation automatically, regardless of whether they went through ListingTable or created ParquetSource directly via a custom TableProvider/ExecutionPlanFactory.

In DF 52, SchemaAdapterFactory was removed from ParquetOpener and replaced with PhysicalExprAdapterFactory. The new adapter handles coercion at the expression level (e.g. inserting CAST in projections). However, this may not be fully wired up for all code paths — specifically when ParquetSource is created directly (not through ListingTable).

Our custom ParquetExecFactory creates ParquetSource::new(table_schema) directly. This worked in DF 51 because the opener's built-in SchemaAdapter handled everything. In DF 52, the same code path fails because:

The opener no longer has SchemaAdapter::map_batch() for batch-level coercion
PhysicalExprAdapterFactory may not be correctly initialized for the direct ParquetSource path

So this does appear to be a regression for anyone who uses ParquetSource directly rather than through ListingTable. The SchemaAdapterFactory was an implicit contract of the ParquetOpener that downstream code relied on.

Could you confirm whether PhysicalExprAdapterFactory is intended to cover all the same paths that SchemaAdapterFactory did? Or is there additional setup needed when creating ParquetSource directly?

Could you confirm whether PhysicalExprAdapterFactory is intended to cover all the same paths that SchemaAdapterFactory did? Or is there additional setup needed when creating ParquetSource directly?

Yes PhysicalExprAdapterFactory does cover all of the same cases that SchemaAdapterFactory did. There may be some differences in wiring when creating a ParquetSource directly (mostly due to refactorings not specifically this change), however I can confirm that the default PhysicalExprAdapterFactory is bound even if you don't specify a custom one:

datafusion/datafusion/datasource-parquet/src/source.rs

Lines 519 to 522 in 1416ed4

    
           let expr_adapter_factory = base_config 
        
               .expr_adapter_factory 
        
               .clone() 
        
               .unwrap_or_else(|| Arc::new(DefaultPhysicalExprAdapterFactory) as _);

I think this can be one of two things:

A bug in DataFusion that we can fix. We need an MRE to understand it.
Something wired up incorrectly (possibly because of code churn or confusing APIs/docs in DataFusion!) in your system. I'm happy to help try to figure out what it is but it might be hard to reproduce.

@adriangb

Thanks @adriangb for looking into this. After extensive testing, here's what I found:

DF 51 works, DF 52 fails — confirmed by reverting our codebase to the pre-DF-52-upgrade commit and running the same queries against the same data. All queries succeed on DF 51, several fail on DF 52.

However, I was unable to create a standalone MRE using vanilla DataFusion APIs that reproduces the difference. The scenarios I tested (missing non-nullable column, List inner field mismatch, nullable file data + non-nullable table schema) either fail on both versions or succeed on both.

Our system uses a custom ExecutionPlanFactory that creates ParquetSource directly and wires it through FileScanConfigBuilder::from(base_config).with_source(new_source). The interaction between SchemaAdapter::map_batch() (DF 51) and our custom path appears to be what handled the edge cases — but I can't isolate it into a minimal example because the behavior depends on our full FileScanTable → ParquetExecFactory → ParquetSource pipeline.

Given that PhysicalExprAdapterFactory does cover the same cases in vanilla DF, I think the issue is likely in how we're wiring things up after the migration. I'll continue debugging on our side.

One concrete thing that might help other DF 52 migrators: the replace_schema block in ParquetOpener (L600-617 in opener.rs) does RecordBatch::try_new_with_options(output_schema, arrays) without any column casting — it just swaps the schema. In DF 51, SchemaAdapter::map_batch() called arrow::compute::cast() on each column before creating the RecordBatch, which handled subtle differences like List inner field metadata. If anyone has a custom setup where the output schema doesn't exactly match the physical file schema (including inner field names/nullability), the replace_schema block will reject it.

Thanks again for the help. I'll close this if we determine it's fully on our side.

Makes sense, I'm sorry for the extra work this has caused on your end. Obviously I can't help debug that much, but maybe could you check that DefaultPhysicalExprAdapterFactory is getting called? You could try wrapping it with your own implementation with some debug statements and wire that up and then you can see what the input / output schemas and expressions are.

Thanks, i found the workaround in our custom system now, and working well.

Shall we close this issue then? Or is there something you think DataFusion could improve?

@adriangb

Closed it now @adriangb .

Navigation Menu

Repository navigation

[Feature Request] Opt-in lenient schema mode to restore DF 51 SchemaAdapter behavior for Parquet reading #21290

Problem

How DF 51 handled this (worked perfectly)

What breaks in DF 52

Concrete errors we hit

Workarounds attempted (all failed or insufficient)

Proposal: restore DF 51's automatic schema adaptation as an opt-in feature

Impact on complex downstream systems

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	let expr_adapter_factory = base_config
	.expr_adapter_factory
	.clone()
	.unwrap_or_else(\|\| Arc::new(DefaultPhysicalExprAdapterFactory) as _);

[Feature Request] Opt-in lenient schema mode to restore DF 51 SchemaAdapter behavior for Parquet reading #21290

Description

Problem

How DF 51 handled this (worked perfectly)

What breaks in DF 52

Concrete errors we hit

Workarounds attempted (all failed or insufficient)

Proposal: restore DF 51's automatic schema adaptation as an opt-in feature

Impact on complex downstream systems

Environment

Activity

zhuqi-lucas commented on Apr 1, 2026

adriangb commented on Apr 1, 2026

zhuqi-lucas commented on Apr 1, 2026

adriangb commented on Apr 1, 2026

zhuqi-lucas commented on Apr 1, 2026

adriangb commented on Apr 1, 2026

zhuqi-lucas commented on Apr 2, 2026

adriangb commented on Apr 4, 2026

zhuqi-lucas commented on Apr 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions