Tandem PR: https://github.com/questdb/questdb-enterprise/pull/932, questdb/documentation#384

Summary

• Adds per-column Parquet encoding and compression configuration via CREATE TABLE and ALTER TABLE SQL syntax
• Introduces a global min_compression_ratio parameter that discards compressed pages when the compression ratio (uncompressed/compressed) falls below a threshold, storing them uncompressed instead
• Add RLE dictionary encoding support for all column types except Boolean and Array
• Improve varchar dict encoding performance by switching from HashMap to RapidHashMap and storing indices directly in a Vec<u32> instead of fetching them from the hashmap
• Enable lto and strip debuginfo from qdbr library (~50% file size reduction).

Per-column Parquet encoding and compression

Users can now specify Parquet encoding and compression on a per-column basis. This only applies to Parquet partitions and is ignored for native partitions.

SQL syntax

CREATE TABLE

CREATE TABLE sensors (
    ts TIMESTAMP,
    temperature DOUBLE PARQUET(rle_dictionary, zstd(3)),
    humidity FLOAT PARQUET(rle_dictionary),
    device_id VARCHAR PARQUET(default, lz4_raw),
    status INT
) TIMESTAMP(ts) PARTITION BY DAY;

The syntax is PARQUET(encoding [, compression[(level)]]). Both encoding and compression are optional — use default for the encoding when specifying compression only. When omitted entirely, the column uses the global defaults (type-based encoding and the server-wide compression codec).

ALTER TABLE

Setting per-column config on existing tables:

-- Set encoding only
ALTER TABLE sensors ALTER COLUMN temperature SET PARQUET(rle_dictionary);

-- Set compression only (with optional level)
ALTER TABLE sensors ALTER COLUMN temperature SET PARQUET(default, zstd(3));

-- Set both
ALTER TABLE sensors ALTER COLUMN temperature SET PARQUET(rle_dictionary, zstd(3));

-- Reset to defaults
ALTER TABLE sensors ALTER COLUMN temperature DROP PARQUET;

SHOW CREATE TABLE

Per-column Parquet config appears in SHOW CREATE TABLE output:

CREATE TABLE 'sensors' (
  ts TIMESTAMP,
  temperature DOUBLE PARQUET(rle_dictionary, zstd(3)),
  humidity FLOAT PARQUET(rle_dictionary),
  device_id VARCHAR PARQUET(default, lz4_raw),
  status INT
) timestamp(ts) PARTITION BY DAY BYPASS WAL;

Supported encodings

The SQL parser also accepts byte_stream_split, but the Rust encoder does not implement it yet — columns configured with this encoding silently fall back to the type's default encoding.

When no encoding is specified, QuestDB picks a type-appropriate default: rle_dictionary for SYMBOL and VARCHAR, delta_length_byte_array for STRING and BINARY, and plain for everything else.

Supported compression codecs

When no per-column compression is specified, the column uses the global compression setting (cairo.partition.encoder.parquet.compression.codec).

Storage format

The per-column encoding config is stored as a packed 32-bit integer in the column metadata:

• Bits 0-7: encoding ID
• Bits 8-15: compression codec ID
• Bits 16-23: compression level
• Bit 24: explicit flag (1 = user-specified override, 0 = use defaults)

Minimum compression ratio

A new server configuration property controls whether compressed pages are worth keeping:

cairo.partition.encoder.parquet.min.compression.ratio=1.2

Semantics: the ratio is uncompressed_size / compressed_size. A threshold of 1.2 means "only keep compressed output if it achieves at least ~17% size reduction." When a compressed column chunk fails to meet this threshold, the encoder discards the compressed output and stores the column chunk as uncompressed instead.

• Default: 1.2
• A value of 0.0 (or any value <= 1.0) disables the check entirely, always keeping compressed output (backward-compatible behavior for the CairoConfiguration interface default).

The ratio check applies to both data pages and dictionary pages, and works with all compression codecs. It runs after compression, so the CPU cost of compression is still incurred -- this setting only avoids the I/O and storage penalty of keeping pages that barely compress.

The value flows from server.conf -> CairoConfiguration -> PartitionEncoder/PartitionUpdater -> JNI -> Rust WriteOptions -> the compress() function in parquet2.

Files changed

Java -- SQL parsing and metadata

• SqlParser.java -- Parses PARQUET ENCODING ... COMPRESSION ... in column definitions
• SqlCompilerImpl.java -- Handles ALTER TABLE ... ALTER COLUMN ... SET/DROP PARQUET ENCODING/COMPRESSION
• SqlKeywords.java -- Adds parquet, encoding, compression keywords
• CreateTableColumnModel.java -- Stores per-column parquet encoding/compression in the column model
• CreateTableOperationBuilderImpl.java / CreateTableOperationImpl.java -- Threads encoding config through table creation
• AlterOperation.java / AlterOperationBuilder.java -- New alter operation type for parquet encoding changes
• ShowCreateTableRecordCursorFactory.java -- Emits PARQUET ENCODING ... COMPRESSION ... in SHOW CREATE TABLE
• ParquetEncoding.java -- New file: encoding constants and validation (type-compatibility checks)
• ParquetCompression.java -- Compression constants, level validation, and codec name resolution

Java -- metadata storage

• TableColumnMetadata.java -- New parquetEncodingConfig field
• TableUtils.java -- Pack/unpack helpers for the 32-bit encoding config, read/write from metadata memory
• TableWriter.java / TableWriterMetadata.java -- Store and propagate encoding config
• TableReaderMetadata.java -- Read encoding config from metadata
• MetadataService.java / MetadataServiceStub.java -- Interface for setting column encoding config
• CairoColumn.java -- Encoding config in column descriptor

Java -- encoder plumbing

• PartitionEncoder.java -- New minCompressionRatio parameter on encodeWithOptions(), encodePartition() native, and createStreamingParquetWriter() native
• PartitionUpdater.java -- New minCompressionRatio parameter on of() and create() native
• PartitionDescriptor.java -- Passes per-column encoding config to Rust
• TableWriter.java -- Reads minCompressionRatio from config and passes to encoder
• O3PartitionJob.java -- Same for partition updater path

Java -- config

• PropertyKey.java -- New CAIRO_PARTITION_ENCODER_PARQUET_MIN_COMPRESSION_RATIO property
• PropServerConfiguration.java -- Loads the property (default: 1.2)
• CairoConfiguration.java -- New getPartitionEncoderParquetMinCompressionRatio() method (interface default: 0.0)

Rust -- compression ratio check

• parquet2/src/write/compression.rs -- Adds min_compression_ratio parameter to compress(), compress_data(), compress_dict(), and the Compressor struct. After compressing a page, checks the ratio and falls back to uncompressed if the threshold is not met.

Rust -- per-column encoding/compression

• src/parquet_write/schema.rs -- to_encodings() and to_compressions() extract per-column overrides from the packed config integer. encoding_from_config() and compression_from_config() decode the packed format. validate_encoding() allows RleDictionary for all column types except Boolean and Array.
• src/parquet_write/file.rs -- column_compression() selects per-column compression when available, falling back to the global setting. WriteOptions gains min_compression_ratio. ParquetWriter gains with_min_compression_ratio(). All compress()/Compressor::new() call sites thread the ratio through. column_chunk_to_dict_pages() dispatches dict encoding for all supported types. Multi-partition writes fall back to the type's default encoding for non-Symbol dict columns (to avoid invalid multi-DictPage column chunks).
• src/parquet_write/update.rs -- ParquetUpdater gains min_compression_ratio field.
• src/parquet_write/jni.rs -- All three JNI entry points (encodePartition, createStreamingParquetWriter, PartitionUpdater_create) accept the new min_compression_ratio parameter.

Rust -- RLE dictionary encoding for all types

The writer now supports RLE dictionary encoding for all column types except Boolean and Array. Previously only Symbol and Varchar had dict encoding. The new encoders build a RapidHashMap for value deduplication, emit a DictPage with unique values, and a DataPage with definition levels and RLE-encoded dictionary keys.

• src/parquet_write/primitive.rs -- slice_to_dict_pages_simd() for i32/i64/f32/f64 SIMD types (Int, Long, Date, Timestamp, Float, Double). int_slice_to_dict_pages_nullable() for narrower nullable types (GeoByte/Short/Int/Long, IPv4). int_slice_to_dict_pages_notnull() for non-nullable types (Byte, Short, Char). decimal_slice_to_dict_pages() for Decimal types with Int32/Int64 physical representation.
• src/parquet_write/string.rs -- string_to_dict_pages() converts QuestDB's UTF-16 string format to UTF-8, deduplicates, and emits length-prefixed ByteArray dict entries.
• src/parquet_write/binary.rs -- binary_to_dict_pages() deduplicates binary blobs and emits length-prefixed ByteArray dict entries.
• src/parquet_write/fixed_len_bytes.rs -- bytes_to_dict_pages() for fixed-length byte arrays (UUID, Long128, Long256, Decimal FLBA types). Handles byte reversal for UUID.
• src/parquet_write/util.rs -- dict_pages_iter() shared helper that assembles DictPage + DataPage into a DynIter<Page>.
• src/parquet_write/mod.rs -- Bench re-exports for all new dict functions. Writer-side roundtrip tests for dict encoding (Int, Long, Double, Byte notnull, all-nulls).

Tests

• SqlParserTest.java -- Syntax tests for CREATE TABLE with per-column encoding/compression
• AlterTableAlterColumnTest.java -- Tests for ALTER TABLE SET/DROP PARQUET ENCODING/COMPRESSION, including error cases (invalid encoding for column type, invalid codec, invalid level)
• ShowCreateTableTest.java -- Tests that SHOW CREATE TABLE correctly round-trips encoding config
• PartitionEncoderTest.java, PartitionUpdaterTest.java, ReadParquetFunctionTest.java, ParallelFilterTest.java -- Updated call sites
• O3ParquetPartitionFuzzTest.java, WalWriterFuzzTest.java -- Fuzz tests include per-column encoding operations
• Rust parquet_write/schema.rs tests -- Unit tests for packing/unpacking and per-column override logic. Updated RleDictionary validation tests for all supported types.
• Rust parquet_write/mod.rs tests -- Writer-side roundtrip tests for dict encoding: Int, Long, Double, Byte (notnull), and all-nulls columns. Verifies DictPage + DataPage structure and correct encoding metadata.
• Rust benches/encode_page.rs -- Dict encoding benchmarks for all supported types across 4 cardinalities (10, 100, 256, 1000) and null percentages (0%, 20%). Covers SIMD types, non-nullable int types, nullable int types, String, Binary, fixed-length byte arrays (Long128, UUID, Long256), and Decimal variants.

Varchar dict encoding performance improvements

Changing hasher to RapidHashMap (for varchar dict encoding)

Default

time:   [7.4973 ms 7.5934 ms 7.7161 ms]
thrpt:  [12.960 Melem/s 13.169 Melem/s 13.338 Melem/s]

FxHashMap

time:   [2.2165 ms 2.2247 ms 2.2345 ms]
thrpt:  [44.753 Melem/s 44.950 Melem/s 45.115 Melem/s]

RapidHashMap

time:   [2.0327 ms 2.0368 ms 2.0411 ms]
thrpt:  [48.992 Melem/s 49.097 Melem/s 49.195 Melem/s]

Storing indices directly in a Vec<u32> instead of fetching them from the hashmap

Before

time:   [2.0327 ms 2.0368 ms 2.0411 ms]
thrpt:  [48.992 Melem/s 49.097 Melem/s 49.195 Melem/s]

After

time:   [1.1375 ms 1.1406 ms 1.1439 ms]
thrpt:  [87.420 Melem/s 87.671 Melem/s 87.912 Melem/s]