You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
native_datafusion: ParquetSchemaConvert error does not include the file path #4316
When the native_datafusion scan adapter rejects an incompatible Parquet column read, the resulting SparkError::ParquetSchemaConvert carries an empty file_path. The JVM shim translates this to a SparkException whose message reads:
Parquet column cannot be converted in file . Column: [a], Expected: int, Found: BINARY.
(Note the empty path between in file and ..) Spark's vectorized reader populates this path via FileScanRDD's catch block (currentFile.urlEncodedPath), so its message reads e.g. ... in file file:/tmp/.../part-00000.parquet. Column: ....
This blocks several Spark SQL tests that extract the path from the message and re-open the file (e.g. ParquetSchemaSuite > schema mismatch failure error message for parquet vectorized reader).
Where the gap is
SparkPhysicalExprAdapter::replace_with_spark_cast and the deferred RejectOnNonEmpty expression build the error with file_path: String::new() because PhysicalExprAdapterFactory::create does not receive the file path. Fixing this likely requires either:
Capturing the file path when the per-file adapter is created (would need a DataFusion API extension), or
Catching ParquetSchemaConvert at a higher layer with file context (e.g. the parquet ScanExec/FileOpener wrapper) and re-raising with the path filled in.
Repro
./dev/diffs/3.4.3.diff has the test currently tagged with IgnoreCometNativeDataFusion pointing at this issue. Drop the tag and run:
Note that this is not a correctness issue and does not need to be fixed. It just means we need to continue ignore some tests and we use this issue as a reference from the ignored tests.
Activity
fix: re-ignore "schema mismatch failure error message" test, file apa…
andygrove commentedon May 13, 2026
Note that this is not a correctness issue and does not need to be fixed. It just means we need to continue ignore some tests and we use this issue as a reference from the ignored tests.
fix: reject BINARY-as-DECIMAL, re-ignore parquet-mr permissive tests (a…