Skip to content

Commit 6185dee

Browse files
Developer1010xPrajwal Narayanaclaude
authored andcommitted
fix: return an error for an invalid CSV null_regex instead of panicking (apache#25261)
## Which issue does this PR close? - Closes apache#25260. ## Rationale for this change A malformed `null_regex` panics the query task instead of returning an error. `CsvFormat::infer_schema_from_stream` compiled the pattern with ```rust let regex = Regex::new(null_regex.as_str()) .expect("Unable to parse CSV null regex."); ``` so any pattern the `regex` crate rejects aborted the task. It is reachable straight from SQL, through a `CREATE EXTERNAL TABLE` that leaves its columns to schema inference: ```sql CREATE EXTERNAL TABLE t STORED AS CSV LOCATION 'data.csv' OPTIONS ('format.has_header' 'true', 'format.null_regex' '('); ``` ``` task 9 panicked with message "Unable to parse CSV null regex.: Syntax( regex parse error: ( ^ error: unclosed group )" ``` An invalid regex is a bad option value, not an internal invariant, so it should come back as an error naming the pattern. ## What changes are included in this PR? The regex is compiled once, before the per-chunk loop, and a failure is propagated with `exec_datafusion_err!` rather than panicking. Hoisting it also stops the pattern being recompiled for every chunk of the inference stream. The error text matches the one used on the read side in apache#25254, so the same bad option reads the same whichever path hits it first. ## What is the testing strategy for this PR? A `statement error Unable to parse CSV null regex` case in `datafusion/sqllogictest/test_files/csv_files.slt`. I checked it is not vacuous: with this change reverted, that case fails with the panic quoted above rather than an error, so the test reproduces the bug. `cargo fmt --check`, `cargo clippy --all-targets` and the crate's unit tests are clean. ## Are there any user-facing changes? An invalid `null_regex` now produces an error and leaves the session usable, where it previously panicked the task. No API changes. Independent of apache#25254 — that one is the read path in `source.rs`, this is the inference path in `file_format.rs` — but they touch the same crate, so whichever lands second may want a trivial rebase. Co-authored-by: Prajwal Narayana <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 4dc1c6b commit 6185dee

2 files changed

Lines changed: 27 additions & 6 deletions

File tree

‎datafusion/datasource-csv/src/file_format.rs‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,8 @@ use arrow::error::ArrowError;
3030
use datafusion_common::config::{ConfigField, ConfigFileType, CsvOptions};
3131
use datafusion_common::file_options::csv_writer::CsvWriterOptions;
3232
use datafusion_common::{
33-
DEFAULT_CSV_EXTENSION, DataFusionError, GetExt, Result, Statistics, exec_err,
34-
not_impl_err,
33+
DEFAULT_CSV_EXTENSION, DataFusionError, GetExt, Result, Statistics,
34+
exec_datafusion_err, exec_err, not_impl_err,
3535
};
3636
use datafusion_common_runtime::SpawnedTask;
3737
use datafusion_datasource::TableSchema;
@@ -540,6 +540,21 @@ impl CsvFormat {
540540
let mut record_number = -1;
541541
let initial_records_to_read = records_to_read;
542542

543+
// Compile once rather than per chunk, and report a malformed pattern
544+
// instead of panicking on it.
545+
let null_regex = self
546+
.options
547+
.null_regex
548+
.as_ref()
549+
.map(|null_regex| {
550+
Regex::new(null_regex).map_err(|e| {
551+
exec_datafusion_err!(
552+
"Unable to parse CSV null regex '{null_regex}': {e}"
553+
)
554+
})
555+
})
556+
.transpose()?;
557+
543558
pin_mut!(stream);
544559

545560
while let Some(chunk) = stream.next().await.transpose()? {
@@ -557,10 +572,8 @@ impl CsvFormat {
557572
.with_quote(self.options.quote)
558573
.with_truncated_rows(self.options.truncated_rows.unwrap_or(false));
559574

560-
if let Some(null_regex) = &self.options.null_regex {
561-
let regex = Regex::new(null_regex.as_str())
562-
.expect("Unable to parse CSV null regex.");
563-
format = format.with_null_regex(regex);
575+
if let Some(regex) = &null_regex {
576+
format = format.with_null_regex(regex.clone());
564577
}
565578

566579
if let Some(escape) = self.options.escape {

‎datafusion/sqllogictest/test_files/csv_files.slt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@ STORED AS CSV
2222
LOCATION '../core/tests/data/duplicate_header.csv'
2323
OPTIONS ('format.has_header' 'true');
2424

25+
# A malformed null_regex is reported as an error rather than panicking the
26+
# query. Schema inference compiles the pattern, so CREATE is where it surfaces.
27+
statement error Unable to parse CSV null regex
28+
CREATE EXTERNAL TABLE csv_with_invalid_null_regex
29+
STORED AS CSV
30+
LOCATION '../core/tests/data/quote.csv'
31+
OPTIONS ('format.has_header' 'true', 'format.null_regex' '(');
32+
2533
# create_external_table_with_quote_escape
2634
statement ok
2735
CREATE EXTERNAL TABLE csv_with_quote (

0 commit comments

Comments
 (0)