Repository navigation
Commit 170e5eb
committed
Fix the native Iceberg task schema at planning
`taskSchema` read the nested schema pruning config every time it was
called: during planning, where CometScanRule checks the schema, and
again when the scan ran and serialized it. A query planned with pruning
on and run with pruning off, such as a cached DataFrame, read the full
schema, which the nested-evolution check never walked. Make it a val,
built with the metadata during planning.
The struct fallback test plans a VERSION AS OF read with pruning on and
runs it with pruning off.1 parent 86eaa60 commit 170e5eb
2 files changed
Lines changed: 12 additions & 3 deletions
File tree
- spark/src
- main/scala/org/apache/comet/iceberg
- test/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2450 | 2450 | | |
2451 | 2451 | | |
2452 | 2452 | | |
2453 | | - | |
2454 | | - | |
| 2453 | + | |
| 2454 | + | |
| 2455 | + | |
| 2456 | + | |
2455 | 2457 | | |
2456 | | - | |
| 2458 | + | |
2457 | 2459 | | |
2458 | 2460 | | |
2459 | 2461 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1880 | 1880 | | |
1881 | 1881 | | |
1882 | 1882 | | |
| 1883 | + | |
| 1884 | + | |
| 1885 | + | |
| 1886 | + | |
| 1887 | + | |
| 1888 | + | |
| 1889 | + | |
1883 | 1890 | | |
1884 | 1891 | | |
1885 | 1892 | | |
| |||
0 commit comments