Repository navigation
Description
Describe the bug, including details regarding any error messages, version, and platform.
Three related issues occur when configuring per-column or default encodings (WriterProperties::Builder::encoding(...)) alongside dictionary encoding in the C++ Parquet writer:
-
Dictionary fallback ignores configured column encoding (related to [Parquet] Make default fallback encoding choice smarter #40636):
WriterProperties::Builder::encoding(...)rejectsPLAIN_DICTIONARYandRLE_DICTIONARYwith"Can't use dictionary encoding as fallback encoding", implying that the configured encoding serves as the fallback encoding when dictionary encoding is enabled. However,ColumnWriter::Makeoverwritesencodingwithproperties->dictionary_index_encoding()whenuse_dictionaryis true without preserving the resolved non-dictionary encoding, andTypedColumnWriterImpl::FallbackToPlainEncoding()unconditionally instantiatesMakeEncoder(..., Encoding::PLAIN, ...)and setsencoding_ = Encoding::PLAIN. As a result, setting e.g.enable_dictionary("col")->encoding("col", Encoding::DELTA_LENGTH_BYTE_ARRAY),Encoding::BYTE_STREAM_SPLIT, orEncoding::DELTA_BINARY_PACKEDsilently falls back toEncoding::PLAINwhendictionary_pagesize_limitis reached (or when writingarrow::DictionaryArraychunks with varying/duplicate dictionaries). -
Incompatible fallback encodings on dictionary-enabled columns are not validated upfront:
BecauseMakeEncoderonly receivesdictionary_index_encoding()whenuse_dictionaryis true, configuring an incompatible(physical_type, encoding)pair (such asEncoding::DELTA_LENGTH_BYTE_ARRAYon anINT64column orEncoding::DELTA_BINARY_PACKEDon aBYTE_ARRAYcolumn) with dictionary encoding enabled does not throw onColumnWriter::Make/NextColumn()unless the column happens to overflow its dictionary at runtime. -
CheckDictionarySizeLimit()can flush buffered dictionary pages mid-record on repeated columns:
InWriteBatchInternal,WriteBatchSpacedInternal, andTypedColumnWriterImpl<ByteArrayType>::WriteArrowDense,DoInBatchessetscheck_page = falsewhenever a batch ends in the middle of a repeated record (rep_levels[offset + batch_size] != 0) andpages_change_on_record_boundaries()is true (DataPageV2orpage_index_enabled). WhileCommitWriteAndCheckPageLimitrespectscheck_page,CheckDictionarySizeLimit()is called unconditionally after every batch. If the dictionary size limit is exceeded mid-record,FallbackToPlainEncoding()flushes any buffered data pages and switches encoders before the record finishes, which can split a repeated record across a page boundary whenpage_index_enabled()orDataPageV2is active.
Component(s)
C++, Parquet
Activity