Skip to content

Compression for Memory tables #20093

@alexey-milovidov

Description

@alexey-milovidov

Add very simple in-memory compression interface that works directly for columns.

Add method IColumn::compress. It returns either:

  • CompressedColumnPtr that is shared_ptr of ICompressedColumn, that has the only method virtual ColumnPtr decompress();
  • std::function<ColumnPtr()> - a function object to get decompressed column back;

This method will take no parameters for compression but it can be extended later.

It should compress the whole buffers of columns as a single block with lz4. The usual compression interface with Read/Write buffers, support for checksums, etc. is not used.

Add support for SETTINGS clause for Memory table with the setting compress = 0|1.

The hypothesis is that compressed Memory tables will work faster on SELECTs by using less memory bandwidth.
Actually we already confirmed this hypothesis with MergeTree tables: they are compressed by default and it works faster than with disabled compression even if data is in memory and disk is not used.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions