Skip to content

Commit b22574a

Browse files
alambclaude
andcommitted
Enable parquet pushdown_filters and progressive_io by default
* `datafusion.execution.parquet.pushdown_filters` and `datafusion.execution.parquet.progressive_io` now default to `true` * Regenerate config docs, `information_schema.slt` and the sqllogictest plans that lose a `FilterExec` above the parquet scan * Update comments, the format options table and the upgrade guide * Two core unit tests assert the predicate on the scan instead of a `FilterExec` Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent a955737 commit b22574a

30 files changed

Lines changed: 229 additions & 409 deletions

‎datafusion/common/src/config.rs‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1370,7 +1370,7 @@ config_namespace! {
13701370

13711371
/// (reading) If true, filter expressions are be applied during the parquet decoding operation to
13721372
/// reduce the number of rows decoded. This optimization is sometimes called "late materialization".
1373-
pub pushdown_filters: bool, default = false
1373+
pub pushdown_filters: bool, default = true
13741374

13751375
/// (reading) If true, filter expressions evaluated during the parquet decoding operation
13761376
/// will be reordered heuristically to minimize the cost of evaluation. If false,
@@ -1384,17 +1384,17 @@ config_namespace! {
13841384
pub force_filter_selections: bool, default = false
13851385

13861386
/// (reading) Controls the I/O pattern used when `pushdown_filters` is
1387-
/// enabled. If false (the default), all data pages needed to read a row
1388-
/// group (for both filter evaluation and the output projection) are
1389-
/// fetched in a single request, the same I/O pattern used when
1390-
/// `pushdown_filters` is disabled. If true, data is fetched
1391-
/// progressively: first the columns needed by each filter, then, after
1392-
/// the filters are evaluated, the remaining projected columns for the
1393-
/// rows that passed. Progressive fetching can reduce the total bytes
1394-
/// read when the file has a Parquet offset index, at the cost of
1395-
/// additional I/O requests per row group; files without an offset index
1396-
/// are always read with a single request per row group.
1397-
pub progressive_io: bool, default = false
1387+
/// enabled. If false, all data pages needed to read a row group (for
1388+
/// both filter evaluation and the output projection) are fetched in a
1389+
/// single request, the same I/O pattern used when `pushdown_filters` is
1390+
/// disabled. If true (the default), data is fetched progressively:
1391+
/// first the columns needed by each filter, then, after the filters are
1392+
/// evaluated, the remaining projected columns for the rows that passed.
1393+
/// Progressive fetching can reduce the total bytes read when the file
1394+
/// has a Parquet offset index, at the cost of additional I/O requests
1395+
/// per row group; files without an offset index are always read with a
1396+
/// single request per row group.
1397+
pub progressive_io: bool, default = true
13981398

13991399
/// (reading) If true, parquet reader will read columns of `Utf8/Utf8Large` with `Utf8View`,
14001400
/// and `Binary/BinaryLarge` with `BinaryView`.

‎datafusion/core/benches/parquet_struct_shared_prefix_pushdown.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@
2626
//! Two properties of the planner shape these queries, and both are easy to get
2727
//! wrong:
2828
//!
29-
//! * `execution.parquet.pushdown_filters` must be enabled. It defaults to
30-
//! `false`, in which case no row filter is built and every case below
29+
//! * `execution.parquet.pushdown_filters` must be enabled (it is by
30+
//! default). If it is disabled no row filter is built and every case below
3131
//! degenerates to a plain scan.
3232
//! * The predicate must be a *single* conjunct. `build_row_filter` calls
3333
//! `split_conjunction` before building filter candidates, and each candidate

‎datafusion/core/src/dataframe/parquet.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,9 +162,11 @@ mod tests {
162162
.select_columns(&["bool_col", "int_col"])?;
163163

164164
let plan = df.explain(false, false)?.collect().await?;
165-
// Filters all the way to Parquet
165+
// Filters all the way to Parquet: with `pushdown_filters` enabled by
166+
// default the predicate is evaluated by the scan itself
166167
let formatted = pretty::pretty_format_batches(&plan)?.to_string();
167-
assert!(formatted.contains("FilterExec: id@0 = 1"), "{formatted}");
168+
assert!(formatted.contains("predicate=id@0 = 1"), "{formatted}");
169+
assert!(!formatted.contains("FilterExec"), "{formatted}");
168170

169171
Ok(())
170172
}

‎datafusion/core/src/datasource/view_test.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -322,11 +322,13 @@ mod tests {
322322

323323
let plan = df.explain(false, false)?.collect().await?;
324324

325-
// Filters all the way to Parquet
325+
// Filters all the way to Parquet: with `pushdown_filters` enabled by
326+
// default the predicate is evaluated by the scan itself
326327
let formatted = arrow::util::pretty::pretty_format_batches(&plan)
327328
.unwrap()
328329
.to_string();
329-
assert!(formatted.contains("FilterExec: id@0 = 1"));
330+
assert!(formatted.contains("predicate=id@0 = 1"), "{formatted}");
331+
assert!(!formatted.contains("FilterExec"), "{formatted}");
330332
Ok(())
331333
}
332334

‎datafusion/optimizer/src/extract_leaf_expressions.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,8 @@
4646
//! The reason is that the extraction projection is the node a source absorbs.
4747
//! A Parquet scan merges it into the file projection and reads only the struct
4848
//! leaf. With the opposite precedence the scan reads the whole struct whenever
49-
//! `datafusion.execution.parquet.pushdown_filters` is `false`, which is the
50-
//! default. The filter loses nothing by staying one node higher, because
49+
//! `datafusion.execution.parquet.pushdown_filters` is `false`. The filter
50+
//! loses nothing by staying one node higher, because
5151
//! `PushDownFilter` records the predicate in
5252
//! [`TableScan::filters`](datafusion_expr::logical_plan::TableScan) in the pass
5353
//! that runs before the extraction projection exists, so row group pruning and

0 commit comments

Comments
 (0)