Skip to content

[C++][Parquet] Honor configured column encoding on dictionary fallback #55571

Open

Description

@ryanvickr

Describe the bug, including details regarding any error messages, version, and platform.

Three related issues occur when configuring per-column or default encodings (WriterProperties::Builder::encoding(...)) alongside dictionary encoding in the C++ Parquet writer:

  1. Dictionary fallback ignores configured column encoding (related to [Parquet] Make default fallback encoding choice smarter #40636):
    WriterProperties::Builder::encoding(...) rejects PLAIN_DICTIONARY and RLE_DICTIONARY with "Can't use dictionary encoding as fallback encoding", implying that the configured encoding serves as the fallback encoding when dictionary encoding is enabled. However, ColumnWriter::Make overwrites encoding with properties->dictionary_index_encoding() when use_dictionary is true without preserving the resolved non-dictionary encoding, and TypedColumnWriterImpl::FallbackToPlainEncoding() unconditionally instantiates MakeEncoder(..., Encoding::PLAIN, ...) and sets encoding_ = Encoding::PLAIN. As a result, setting e.g. enable_dictionary("col")->encoding("col", Encoding::DELTA_LENGTH_BYTE_ARRAY), Encoding::BYTE_STREAM_SPLIT, or Encoding::DELTA_BINARY_PACKED silently falls back to Encoding::PLAIN when dictionary_pagesize_limit is reached (or when writing arrow::DictionaryArray chunks with varying/duplicate dictionaries).

  2. Incompatible fallback encodings on dictionary-enabled columns are not validated upfront:
    Because MakeEncoder only receives dictionary_index_encoding() when use_dictionary is true, configuring an incompatible (physical_type, encoding) pair (such as Encoding::DELTA_LENGTH_BYTE_ARRAY on an INT64 column or Encoding::DELTA_BINARY_PACKED on a BYTE_ARRAY column) with dictionary encoding enabled does not throw on ColumnWriter::Make / NextColumn() unless the column happens to overflow its dictionary at runtime.

  3. CheckDictionarySizeLimit() can flush buffered dictionary pages mid-record on repeated columns:
    In WriteBatchInternal, WriteBatchSpacedInternal, and TypedColumnWriterImpl<ByteArrayType>::WriteArrowDense, DoInBatches sets check_page = false whenever a batch ends in the middle of a repeated record (rep_levels[offset + batch_size] != 0) and pages_change_on_record_boundaries() is true (DataPageV2 or page_index_enabled). While CommitWriteAndCheckPageLimit respects check_page, CheckDictionarySizeLimit() is called unconditionally after every batch. If the dictionary size limit is exceeded mid-record, FallbackToPlainEncoding() flushes any buffered data pages and switches encoders before the record finishes, which can split a repeated record across a page boundary when page_index_enabled() or DataPageV2 is active.

Component(s)

C++, Parquet

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions