-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Implement MultiStage PREWHERE for ParquetReaderV3 #89101
Copy link
Copy link
Closed
Labels
comp-datalakeData lake table formats (Iceberg/Delta/Hudi) integration.Data lake table formats (Iceberg/Delta/Hudi) integration.comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).performance
Description
Right now PREWHERE in ParquetReaderV3 is implemented in a single step (actually two steps, one of them is row-level security): link. This simplifies implementation, but not that effective, because during query analysis ClickHouse push ALL conditions to PREWHERE: setting + usage.
The idea is to implement Multistage PREWHERE similar to what MergeTree has.
Workaround solution is to disable move_all_conditions_to_prewhere for parquet and move only best condition to PREWHERE.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
comp-datalakeData lake table formats (Iceberg/Delta/Hudi) integration.Data lake table formats (Iceberg/Delta/Hudi) integration.comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).performance