Skip to content

Commit 692bfaa

Browse files
committed
docs: note Spark 3.x extra SparkException cause-chain layer (apache#4354)
1 parent fa7a9e2 commit 692bfaa

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

  • docs/source/user-guide/latest/compatibility

‎docs/source/user-guide/latest/compatibility/scans.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,16 @@ behavior.
120120
`Encountered error while reading file . Data type mismatches…` (note the empty path). Behavior is
121121
consistent across Spark versions. See
122122
[#4316](https://github.com/apache/datafusion-comet/issues/4316).
123+
- **Spark 3.x: extra `SparkException` layer in the cause chain**. The native error is translated to a
124+
`SparkException` whose cause is `SchemaColumnConvertNotSupportedException` (matching what Spark would
125+
throw); on Spark 3.x the executor / task error handling re-wraps this once more on the way back to
126+
the driver, producing a two-level chain (`SparkException → SparkException →
127+
SchemaColumnConvertNotSupportedException`) instead of the one-level chain Spark's own vectorized
128+
reader produces. Code that catches `SparkException` and inspects only the immediate cause via
129+
`e.getCause.isInstanceOf[SchemaColumnConvertNotSupportedException]` will see the inner
130+
`SparkException` instead. Walk the cause chain to recover the
131+
`SchemaColumnConvertNotSupportedException`. Spark 4.0+ produces a single-level chain, matching
132+
vanilla Spark. See [#4354](https://github.com/apache/datafusion-comet/issues/4354).
123133

124134
## `native_iceberg_compat` Limitations
125135

0 commit comments

Comments
 (0)