roapi icon indicating copy to clipboard operation
roapi copied to clipboard

Does ROAPI do row group pruning on parquet files stored on disk?

Open jpaye opened this issue 3 years ago • 0 comments

Hi,

We are trying to use ROAPI to efficiently run queries against Parquet files. These files are large, so storing them in-memory is not an option. We were hoping that ROAPI/Datafusion would use predicate pushdown/"pruning" on the row groups to speed up our queries, but creating more row groups and sorting our queried columns only seemed to slow queries down.

Just wondering if ROAPI is expected to prune queries on parquets when stored on disk? And if so, do we need to do something special to achieve this?

thank you!

jpaye avatar Jul 21 '22 21:07 jpaye