Should a DataFusion scan fail on a mismatched column that the query doesn't read? #11270

Problem or motivation

Right now, the Vortex opener checks every column of a file against the table schema before it scans the file, so a column that doesn't fit fails every query on that file, even a query that never reads the column. This happens because the opener computes the physical schema of the file with calculate_physical_schema over all of its columns, and any error there ends the scan.

This used to be rare, since only a layout mismatch fails there (e.g., a table field List over a FixedSizeList file column), and only a user-supplied table schema can produce one. Schema inference merges the file schemas with Schema::try_merge, which rejects columns whose types differ.

But #11268 adds an extension type check to the physical schema (X24), so a table field and its file column must now name the same extension type. This is subtle because an extension type lives in the field metadata, and Schema::try_merge merges metadata as a union instead of rejecting a difference. So schema inference now builds tables that fail:

a.vortex: { id: i32, value: vortex.json(utf8) }
b.vortex: { id: i32, value: utf8 }
table t:  { id: Int32, value: Utf8View + arrow.json }

SELECT value FROM t  -- fails on b.vortex (labeled its strings as Json before #11268)
SELECT id FROM t     -- fails on b.vortex (worked before #11268)

Failing SELECT value is the point of #11268. Whether SELECT id should fail is the question here. Note that DataFusion itself only adapts the columns that a query's expressions reference (through its PhysicalExprAdapter), so this behavior comes from the Vortex opener, not from DataFusion.

Proposed solution

There are 3 options, and they can be combined:

1. Keep failing every query on the file, as Require matching extension types in the DataFusion physical schema #11268 does. This is the simplest rule, and the error names the file and the column. But one file with a stray column makes the whole table unusable, for every query.
2. Check only the columns that a query reads. The opener would compute the physical schema against the table schema restricted to the columns that the projection and the filters reference, so an unread column gets its natural Arrow type and is never checked. This matches DataFusion, and it relaxes layout mismatches the same way. Of course, filters pushed into the Vortex scan need care, since they read columns that the output doesn't project.
3. Reject the mismatch during schema inference. VortexFormat::infer_schema could require every file to agree on the extension type of each column, so CREATE EXTERNAL TABLE fails up front and names the files. But a listing table infers its schema once, so this doesn't cover a file added later or a user-supplied schema, which means it only makes sense next to 1 or 2.

Open question: which policy should the DataFusion integration follow, for both layout and extension type mismatches?

Additional context

• Require matching extension types in the DataFusion physical schema #11268 (DS2 in Tracking Issue: Fix Extension Types #11114) adds the check, in both the ListingTable opener and the V2 VortexDataSourceBuilder. It builds on Scan extension columns over list, map, and string storage in DataFusion #11225, which makes extension columns over list, map, and string storage scannable.