QuestDB currently stores files in its proprietary format.

With this feature, QuestDB will store its data in Parquet format in an object store, which will allow 3rd-party tools to access, query, and analyze it.

The hot data (most recent, likely to be still updated) will still be in our proprietary format, and we'll move it over to Parquet when it's ready to be archived.

Parquet format support assumes changes in the query engine. QuestDB's native storage format stores column values densely, i.e. each column file contains values for all rows stored in the corresponding partition. Thanks to this, the query engine is free to remember the position of a row and cheaply jump back to that row later on (see the below pic). Of course, not all queries allow this, e.g. if you do a UNION or query a pseudo-table random access is impossible. But if the base query supports random access, a more efficient execution plan is chosen, e.g. ORDER BY may store row IDs instead of the materialized result set in its R/B tree.

But with Parquet random access becomes more expensive. So, how are we going to deal with that?

Parquet (or any other similar storage format) support will take multiple steps. First, we're going to keep the same overall API. This means that queries that support random access will keep the support. When we need to jump to a row group in a Parquet partition, we'll do a lookup in the cache of recently deserialized row groups and, if we don't have anything in the cache, deserialize the required columns into a set of buffers to be used for query processing. For native partitions, this will be a no-op operation. The overall idea is to avoid groundbreaking changes in the first iteration.

Once we have that, the next steps will assume optimizations for the new format. Say, ORDER BY with materialization may be a better choice for queries involving Parquet partitions, full deserialization is not required in case of certain aggregations, filters (think, WHERE clause) or their part can be pushed down to the code responsible for the deserialization, and so on. These and other things will be based on the conducted benchmarks and user feedback.

Long story short, our Parquet support journey assumes iterative, evolutionary changes in the query engine.