Repository navigation
Commit 12ba4b7
fix: reject duplicate field names during CSV and Parquet schema inference (apache#24387)
## Which issue does this PR close?
- Closes apache#24381.
- Closes apache#12852.
## Rationale for this change
When a Parquet file has multiple columns with the same name and
compatible types, DataFusion merges them during schema inference. The
scan succeeds without warning but returns only one of those columns.
PyArrow can produce such a file via its public API:
```python
left = pa.table({"id": [1, 2, 3], "value": [10, 20, 30]})
right = pa.table({"id": [1, 2, 3], "value": [100, 200, 300]})
joined = left.join(right, keys="id") # left_suffix/right_suffix default to None
pq.write_table(joined, path) # writes id, value, value
```
DataFusion 54.0.0 returns ['id', 'value'] for this file; the [100, 200,
300] column is missing.
Other readers either preserve the data or reject the file:
| Reader | Result |
| --- | --- |
| `pyarrow` `ParquetFile.read()` | keeps all three columns |
| `pyarrow` `pq.read_table()` | `ArrowInvalid: Multiple matches for
FieldRef.Name(value)` |
| `pyarrow` dataset | `ArrowInvalid: Can't unify schema with duplicate
field names` |
| `duckdb` | keeps all three, renames the second to `value_1` |
| `polars` | `DuplicateError` |
| `datafusion` | **succeeds, missing column(s)** |
The same bug is behind two issues apache#24381 and apache#12852. CSV and Parquet
schema inference both pass each file's schema to `Schema::try_merge`,
which matches fields by name. Duplicate names in one file are merged
before the scan. Thus, column(s) disappear silently.
## What changes are included in this PR?
This PR adds `ensure_unique_field_names` to `datafusion-datasource`. It
is called for each inferred CSV and Parquet schema before the merge. It
uses the existing `SchemaError::DuplicateUnqualifiedField` error and
adds the file location to the message.
Duplicate names are rejected. The PR does not rename columns or change
how fields with unique names are merged across files.
## Are these changes tested?
Yes. Two regression tests in order to cover the CSV and Parquet paths:
- `datafusion/core/src/datasource/file_format/parquet.rs` —
`infer_schema_rejects_duplicate_field_names`
- `datafusion/core/src/datasource/file_format/csv.rs` —
`infer_schema_rejects_duplicate_header_names`
Both tests create their inputs in a temporary directory; no fixtures are
added. Both fail on `main` and pass with this change. On `main`, the CSV
test infers `Schema { fields: [id, value] }` from a three-column header,
matching the behavior reported in apache#12852.
## Are there any user-facing changes?
Yes. CSV and Parquet schema inference now returns a clear error when an
inferred
file schema repeats a field name. The error names the file and the
repeated
column instead of returning an incomplete schema.
---------
Co-authored-by: Andrew Lamb <[email protected]>1 parent 7c5a81b commit 12ba4b7
7 files changed
Lines changed: 125 additions & 5 deletions
File tree
- datafusion
- core
- src/datasource/file_format
- tests/data
- datasource-csv/src
- datasource-parquet/src
- datasource/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1628 | 1628 | | |
1629 | 1629 | | |
1630 | 1630 | | |
| 1631 | + | |
| 1632 | + | |
| 1633 | + | |
| 1634 | + | |
| 1635 | + | |
| 1636 | + | |
| 1637 | + | |
| 1638 | + | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
1631 | 1658 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1822 | 1822 | | |
1823 | 1823 | | |
1824 | 1824 | | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
1825 | 1859 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
| |||
397 | 398 | | |
398 | 399 | | |
399 | 400 | | |
400 | | - | |
| 401 | + | |
401 | 402 | | |
402 | 403 | | |
403 | 404 | | |
404 | 405 | | |
405 | 406 | | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
406 | 419 | | |
407 | 420 | | |
408 | 421 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
40 | | - | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
41 | 44 | | |
42 | 45 | | |
43 | 46 | | |
| |||
388 | 391 | | |
389 | 392 | | |
390 | 393 | | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
391 | 405 | | |
392 | 406 | | |
393 | 407 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
45 | 68 | | |
46 | 69 | | |
47 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
18 | 25 | | |
19 | 26 | | |
20 | 27 | | |
| |||
0 commit comments