Repository navigation
Commit b295a67
authored
fix: infer a Parquet column as nullable when some files lack it (#26104)
## Which issue does this PR close?
- Closes #26103
## Rationale for this change
`ListingTable` schema inference over Parquet merges the per-file schemas
with `Schema::try_merge`, which widens nullability only for fields that
appear in more than one schema. A column that is `required` in some
files and absent from the others therefore comes out `NOT NULL`, but
reading the files without it fills it with nulls, and the scan fails:
```
Arrow error: Invalid argument error: Column 'c' is declared as non-nullable but contains null values
```
Pure SQL reproducer (on `main`):
```sql
COPY (SELECT 1 AS id, 10 AS c) TO '/tmp/evo/a.parquet';
COPY (SELECT 2 AS id) TO '/tmp/evo/b.parquet';
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION '/tmp/evo/';
SELECT * FROM t; -- fails; DESCRIBE shows c Int64 NO
```
Before DataFusion 52 this was masked by `SchemaAdapter::map_batch`,
which rebuilt batches under the table schema without validating
nullability (removed in #18998). The strict check is right; the inferred
schema is what is wrong.
## What changes are included in this PR?
In `ParquetFormat::infer_schema`, count in how many files each top-level
field appears and, after the merge, mark every field that is missing
from at least one file as nullable. Fields present in every file keep
whatever `try_merge` produced, so tables whose files all share a schema
are unchanged.
## Are these changes tested?
- New `schema_evolution.slt` case: two `COPY TO` files, `CREATE EXTERNAL
TABLE` without a schema, `DESCRIBE` shows the partial column as
nullable, `SELECT` returns the null. Fails on `main` with the error
above.
- New `schema_merge_marks_partially_present_columns_nullable` in
`core/tests/parquet/schema.rs`: three files, column in a different
position, an already-nullable partial column, and both `skip_metadata`
paths.
## Are there any user-facing changes?
An inferred Parquet table schema now reports a column as nullable when
some files do not contain it. Tables where every file has every column
are unaffected. Explicitly declared schemas are not touched.1 parent b40ef06 commit b295a67
3 files changed
Lines changed: 183 additions & 2 deletions
File tree
- datafusion
- core/tests/parquet
- datasource-parquet/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
| |||
185 | 187 | | |
186 | 188 | | |
187 | 189 | | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
188 | 255 | | |
189 | 256 | | |
190 | 257 | | |
| |||
214 | 281 | | |
215 | 282 | | |
216 | 283 | | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
217 | 311 | | |
218 | 312 | | |
219 | 313 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | | - | |
| 102 | + | |
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| |||
272 | 272 | | |
273 | 273 | | |
274 | 274 | | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
275 | 303 | | |
276 | 304 | | |
277 | 305 | | |
| |||
400 | 428 | | |
401 | 429 | | |
402 | 430 | | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
403 | 444 | | |
404 | 445 | | |
405 | 446 | | |
| |||
413 | 454 | | |
414 | 455 | | |
415 | 456 | | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
416 | 460 | | |
417 | 461 | | |
418 | 462 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
0 commit comments