-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Make KeyCondition usable outside *MergeTree when analyzer is enabled #53536
Copy link
Copy link
Closed
Labels
developmentDevelopement process & source code & implementation detailsDevelopement process & source code & implementation details
Description
If I'm reading the code correctly:
- when analyzer is enabled,
SelectQueryInfo::filter_astsdoesn't have sets, and is probably missing some casts, - so, KeyCondition shouldn't use it,
- instead it should use
SelectQueryInfo::filter_actions_dag, - which is assigned only in ReadFromMergeTree::applyFilters().
So, any code using KeyCondition without a merge tree won't work. Currently that's just StorageHive, which uses KeyCondition to skip files based on their min/max statistics. (I didn't test if it's actually broken with analyzer, so maybe I'm misreading the code and it actually works.) And #52951 added KeyCondition to Parquet reader (to skip row groups), and I'm not sure how to make it work with analyzer.
What's the intended way for this to work? Should filter_actions_dag be built by some non-merge-tree code, before IStorage::read() calls? Or should it be built lazily by whoever needs it first? Should everything work completely differently (e.g. non-merge-tree filter pushdowns shouldn't use KeyCondition?)?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
developmentDevelopement process & source code & implementation detailsDevelopement process & source code & implementation details