Skip to content

Commit 12ba4b7

Browse files
sovsparrowalamb
andauthored
fix: reject duplicate field names during CSV and Parquet schema inference (apache#24387)
## Which issue does this PR close? - Closes apache#24381. - Closes apache#12852. ## Rationale for this change When a Parquet file has multiple columns with the same name and compatible types, DataFusion merges them during schema inference. The scan succeeds without warning but returns only one of those columns. PyArrow can produce such a file via its public API: ```python left = pa.table({"id": [1, 2, 3], "value": [10, 20, 30]}) right = pa.table({"id": [1, 2, 3], "value": [100, 200, 300]}) joined = left.join(right, keys="id") # left_suffix/right_suffix default to None pq.write_table(joined, path) # writes id, value, value ``` DataFusion 54.0.0 returns ['id', 'value'] for this file; the [100, 200, 300] column is missing. Other readers either preserve the data or reject the file: | Reader | Result | | --- | --- | | `pyarrow` `ParquetFile.read()` | keeps all three columns | | `pyarrow` `pq.read_table()` | `ArrowInvalid: Multiple matches for FieldRef.Name(value)` | | `pyarrow` dataset | `ArrowInvalid: Can't unify schema with duplicate field names` | | `duckdb` | keeps all three, renames the second to `value_1` | | `polars` | `DuplicateError` | | `datafusion` | **succeeds, missing column(s)** | The same bug is behind two issues apache#24381 and apache#12852. CSV and Parquet schema inference both pass each file's schema to `Schema::try_merge`, which matches fields by name. Duplicate names in one file are merged before the scan. Thus, column(s) disappear silently. ## What changes are included in this PR? This PR adds `ensure_unique_field_names` to `datafusion-datasource`. It is called for each inferred CSV and Parquet schema before the merge. It uses the existing `SchemaError::DuplicateUnqualifiedField` error and adds the file location to the message. Duplicate names are rejected. The PR does not rename columns or change how fields with unique names are merged across files. ## Are these changes tested? Yes. Two regression tests in order to cover the CSV and Parquet paths: - `datafusion/core/src/datasource/file_format/parquet.rs` — `infer_schema_rejects_duplicate_field_names` - `datafusion/core/src/datasource/file_format/csv.rs` — `infer_schema_rejects_duplicate_header_names` Both tests create their inputs in a temporary directory; no fixtures are added. Both fail on `main` and pass with this change. On `main`, the CSV test infers `Schema { fields: [id, value] }` from a three-column header, matching the behavior reported in apache#12852. ## Are there any user-facing changes? Yes. CSV and Parquet schema inference now returns a clear error when an inferred file schema repeats a field name. The error names the file and the repeated column instead of returning an incomplete schema. --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent 7c5a81b commit 12ba4b7

7 files changed

Lines changed: 125 additions & 5 deletions

File tree

‎datafusion/core/src/datasource/file_format/csv.rs‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1628,4 +1628,31 @@ mod tests {
16281628

16291629
Ok(())
16301630
}
1631+
1632+
#[tokio::test]
1633+
async fn infer_schema_rejects_duplicate_header_names() -> Result<()> {
1634+
let directory = tempfile::tempdir()?;
1635+
let path = directory.path().join("duplicate_header.csv");
1636+
std::fs::write(&path, "id,value,value\n1,10,100\n")?;
1637+
1638+
let store = Arc::new(LocalFileSystem::new()) as _;
1639+
let meta = crate::test::object_store::local_unpartitioned_file(&path);
1640+
1641+
let ctx = SessionContext::new().state();
1642+
let error = CsvFormat::default()
1643+
.with_has_header(true)
1644+
.infer_schema(&ctx, &store, std::slice::from_ref(&meta))
1645+
.await
1646+
.expect_err("duplicate header names must not infer a schema")
1647+
.to_string();
1648+
1649+
assert!(
1650+
error.contains("duplicate unqualified field name")
1651+
&& error.contains("value")
1652+
&& error.contains("duplicate_header.csv"),
1653+
"unexpected error: {error}"
1654+
);
1655+
1656+
Ok(())
1657+
}
16311658
}

‎datafusion/core/src/datasource/file_format/parquet.rs‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1822,4 +1822,38 @@ mod tests {
18221822

18231823
Ok(())
18241824
}
1825+
1826+
#[tokio::test]
1827+
async fn infer_schema_rejects_duplicate_field_names() -> Result<()> {
1828+
let schema = Arc::new(Schema::new(vec![
1829+
Field::new("id", DataType::Int64, false),
1830+
Field::new("value", DataType::Int64, false),
1831+
Field::new("value", DataType::Int64, false),
1832+
]));
1833+
let batch = RecordBatch::try_new(
1834+
schema,
1835+
vec![
1836+
Arc::new(Int64Array::from(vec![1, 2, 3])) as ArrayRef,
1837+
Arc::new(Int64Array::from(vec![10, 20, 30])) as ArrayRef,
1838+
Arc::new(Int64Array::from(vec![100, 200, 300])) as ArrayRef,
1839+
],
1840+
)?;
1841+
1842+
let store = Arc::new(LocalFileSystem::new()) as _;
1843+
let (meta, _files) = store_parquet(vec![batch], false).await?;
1844+
1845+
let ctx = SessionContext::new().state();
1846+
let error = ParquetFormat::default()
1847+
.infer_schema(&ctx, &store, &meta)
1848+
.await
1849+
.expect_err("duplicate field names must not infer a schema")
1850+
.to_string();
1851+
1852+
assert!(
1853+
error.contains("duplicate unqualified field name") && error.contains("value"),
1854+
"unexpected error: {error}"
1855+
);
1856+
1857+
Ok(())
1858+
}
18251859
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
id,value,value
2+
1,10,100

‎datafusion/datasource-csv/src/file_format.rs‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ use datafusion_datasource::file::FileSource;
4141
use datafusion_datasource::file_compression_type::FileCompressionType;
4242
use datafusion_datasource::file_format::{
4343
DEFAULT_SCHEMA_INFER_MAX_RECORD, FileFormat, FileFormatFactory,
44+
ensure_unique_field_names,
4445
};
4546
use datafusion_datasource::file_scan_config::{FileScanConfig, FileScanConfigBuilder};
4647
use datafusion_datasource::file_sink_config::{FileSink, FileSinkConfig};
@@ -397,12 +398,24 @@ impl FileFormat for CsvFormat {
397398
)
398399
})?;
399400
records_to_read -= records_read;
400-
schemas.push(schema);
401+
schemas.push((&object.location, schema));
401402
if records_to_read == 0 {
402403
break;
403404
}
404405
}
405406

407+
let mut seen = HashSet::new();
408+
for (location, schema) in &schemas {
409+
ensure_unique_field_names(schema, &mut seen).map_err(|err| {
410+
DataFusionError::Context(
411+
format!("Error when processing CSV file {location}"),
412+
Box::new(err),
413+
)
414+
})?;
415+
}
416+
drop(seen);
417+
418+
let schemas = schemas.into_iter().map(|(_, schema)| schema);
406419
let merged_schema = Schema::try_merge(schemas)?;
407420
Ok(Arc::new(merged_schema))
408421
}

‎datafusion/datasource-parquet/src/file_format.rs‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
//! [`ParquetFormat`]: Parquet [`FileFormat`] abstractions
1919
20+
use std::collections::HashSet;
2021
use std::fmt;
2122
use std::fmt::Debug;
2223
use std::ops::Range;
@@ -37,7 +38,9 @@ use datafusion_datasource::TableSchema;
3738
use datafusion_datasource::file_compression_type::FileCompressionType;
3839
use datafusion_datasource::file_sink_config::FileSinkConfig;
3940

40-
use datafusion_datasource::file_format::{FileFormat, FileFormatFactory};
41+
use datafusion_datasource::file_format::{
42+
FileFormat, FileFormatFactory, ensure_unique_field_names,
43+
};
4144

4245
use datafusion_common::Statistics;
4346
use datafusion_common::config::{ConfigField, ConfigFileType, TableParquetOptions};
@@ -388,6 +391,17 @@ impl FileFormat for ParquetFormat {
388391
schemas
389392
.sort_unstable_by(|(location1, _), (location2, _)| location1.cmp(location2));
390393

394+
let mut seen = HashSet::new();
395+
for (location, schema) in &schemas {
396+
ensure_unique_field_names(schema, &mut seen).map_err(|err| {
397+
DataFusionError::Context(
398+
format!("Error when processing Parquet file {location}"),
399+
Box::new(err),
400+
)
401+
})?;
402+
}
403+
drop(seen);
404+
391405
let schemas = schemas.into_iter().map(|(_, schema)| schema);
392406

393407
let schema = if self.skip_metadata() {

‎datafusion/datasource/src/file_format.rs‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
//! See write.rs for write related helper methods
2020
2121
use std::any::Any;
22-
use std::collections::HashMap;
22+
use std::collections::{HashMap, HashSet};
2323
use std::fmt;
2424
use std::sync::Arc;
2525

@@ -28,9 +28,11 @@ use crate::file_compression_type::FileCompressionType;
2828
use crate::file_scan_config::FileScanConfig;
2929
use crate::file_sink_config::FileSinkConfig;
3030

31-
use arrow::datatypes::SchemaRef;
31+
use arrow::datatypes::{Schema, SchemaRef};
3232
use datafusion_common::file_options::file_type::FileType;
33-
use datafusion_common::{GetExt, Result, Statistics, internal_err, not_impl_err};
33+
use datafusion_common::{
34+
GetExt, Result, SchemaError, Statistics, internal_err, not_impl_err, schema_err,
35+
};
3436
use datafusion_physical_expr::LexRequirement;
3537
use datafusion_physical_expr_common::sort_expr::LexOrdering;
3638
use datafusion_physical_plan::ExecutionPlan;
@@ -42,6 +44,27 @@ use object_store::{ObjectMeta, ObjectStore};
4244
/// Default max records to scan to infer the schema
4345
pub const DEFAULT_SCHEMA_INFER_MAX_RECORD: usize = 1000;
4446

47+
/// Rejects an inferred schema that names the same field more than once.
48+
///
49+
/// [`Schema::try_merge`] coalesces fields by name, so callers validate each
50+
/// inferred file schema before merging. `seen` is cleared before use so callers
51+
/// can reuse its allocation across schemas.
52+
pub fn ensure_unique_field_names<'a>(
53+
schema: &'a Schema,
54+
seen: &mut HashSet<&'a str>,
55+
) -> Result<()> {
56+
seen.clear();
57+
seen.reserve(schema.fields().len());
58+
for field in schema.fields() {
59+
if !seen.insert(field.name()) {
60+
return schema_err!(SchemaError::DuplicateUnqualifiedField {
61+
name: field.name().clone(),
62+
});
63+
}
64+
}
65+
Ok(())
66+
}
67+
4568
/// Metadata fetched from a file, including statistics and ordering.
4669
///
4770
/// This struct is returned by [`FileFormat::infer_stats_and_ordering`] to

‎datafusion/sqllogictest/test_files/csv_files.slt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,13 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18+
# duplicate_header_is_rejected
19+
statement error Schema contains duplicate unqualified field name value
20+
CREATE EXTERNAL TABLE csv_with_duplicate_header
21+
STORED AS CSV
22+
LOCATION '../core/tests/data/duplicate_header.csv'
23+
OPTIONS ('format.has_header' 'true');
24+
1825
# create_external_table_with_quote_escape
1926
statement ok
2027
CREATE EXTERNAL TABLE csv_with_quote (

0 commit comments

Comments
 (0)