[Parquet] Make default fallback encoding choice smarter #40636

Describe the enhancement requested

Currently, the choice of default encoding for a non-dictionary data page is trivial.
It happens in two places:

1. in the FallbackToPlainEncoding function for columns for which dictionary encoding is attempted:
   

   arrow/cpp/src/parquet/column_writer.cc

   Lines 1567 to 1580 in 5718a28

   	void FallbackToPlainEncoding() {
   	if (IsDictionaryEncoding(current_encoder_->encoding())) {
   	WriteDictionaryPage();
   	// Serialize the buffered Dictionary Indices
   	FlushBufferedDataPages();
   	fallback_ = true;
   	// Only PLAIN encoding is supported for fallback in V1
   	current_encoder_ = MakeEncoder(DType::type_num, Encoding::PLAIN, false, descr_,
   	properties_->memory_pool());
   	current_value_encoder_ = dynamic_cast<ValueEncoderType*>(current_encoder_.get());
   	current_dict_encoder_ = nullptr; // not using dict
   	encoding_ = Encoding::PLAIN;
   	}
   	}

2. in the ColumnWriter::Make factory function for columns for which dictionary encoding is not attempted:
   

   arrow/cpp/src/parquet/column_writer.cc

   Lines 2375 to 2382 in 5718a28

   	Encoding::type encoding = properties->encoding(descr->path());
   	if (encoding == Encoding::UNKNOWN) {
   	encoding = (descr->physical_type() == Type::BOOLEAN &&
   	properties->version() != ParquetVersion::PARQUET_1_0 &&
   	properties->data_page_version() == ParquetDataPageVersion::V2)
   	? Encoding::RLE
   	: Encoding::PLAIN;
   	}

I'll note that parquet-mr does not limit dictionary encoding fallback to PLAIN, even for "v1" Parquet files:
https://github.com/apache/parquet-mr/blob/95b004c3df473e3ab0963dc5136934ce5235d5df/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java#L124-L139

We should probably consolidate the logic from the two functions above and make it more sophisticated, allowing the best encoding for the selected Parquet version.

Also related: #38441

Component(s)

C++, Parquet