Earlier this morning I managed to compare the top 50% of the original file with the one processed by ClickHouse using Spark & pyspark. They match! Didn't compare the other half because the pyspark API interface available to grab last N elements is tail and it was taking way more memory than limit.

I can't use the test file I have to implement a test because it's confidential data. I just managed to generate a file with the schema presented in the issue, but it doesn't raise the exception. The test file that was provided contains many more columns than what was in the select statement, maybe it has something to do with that. Will investigate further.

I have validated the implementation by doing the following:

1. Export the file processed by ClickHouse into two Parquet files, command: clickhouse-client --query "select id, _acc_fields_map from s3(<file_path>, 'Parquet', 'id Int64, _acc_fields_map Map(String, String)') order by id limit <lower_limit>, <upper_limit> FORMAT Parquet" > parquet validation/processed<part_number>.parquet. lower_limit, upper_limit and part_number being 0/80692, 80692/161384 and 1/2 for part1 and part2 respectively.
2. Using PySpark, concat both parts processed by ClickHouse into a single DataFrame. Then, assert the equality of the original dataframe (read from original file) with the processed data frame. Python script:

original_df = spark.read.parquet("original-strippedd.parquet").orderBy("id")

processed_df1 = spark.read.parquet("processed1.parquet")

processed_df2 = spark.read.parquet("processed2.parquet")

processed_concatened = processed_df1.union(processed_df2).withColumn("fields_map", F.col("fields_map").cast("map<string,string>"))

assert original_df_filtered.collect() == processed_concatened.collect()

To be sure the assertion was doing its job, I dropped the first row in the original DataFrame and the assertion failed. Dropped it with original_df_filtered = original_df.where(original_df.id > <first_id>). The assertion failed.

I have spent the last week trying to write a test, but failed to do so. Mainly because I could not generate a file that raises the exception. I have tried several combinations using VERY large strings with LOW and HIGH cardinality. None of them caused the issue. AFAIK, the data gets internally chunked when the chunk memory limit is reached within a rowgroup. As of now, it's set to 2^32 - 1.

Since I validated it works in that case, I'll set this PR as ready for review. I am open for discussions.

Marked it as draft again. While the initial case is working properly, the below isn't:

import pyarrow as pa
import pyarrow.parquet as pq

arr = pa.array([[("a" * 2**30, 1)]], type = pa.map_(pa.string(), pa.int32()))
arr = pa.chunked_array([arr, arr])
tab = pa.table({ "arr": arr })

pq.write_table(tab, "test.parquet")

pq.read_table("test.parquet")

Marked it as draft again. While the initial case is working properly, the below isn't:

import pyarrow as pa
import pyarrow.parquet as pq

arr = pa.array([[("a" * 2**30, 1)]], type = pa.map_(pa.string(), pa.int32()))
arr = pa.chunked_array([arr, arr])
tab = pa.table({ "arr": arr })

pq.write_table(tab, "test.parquet")

pq.read_table("test.parquet")

Arrow lib contains two variants of String: String and LargeString. After some hackish changes on arrow lib, I managed to force the use of LargeStrings and the data doesn't get chunked anymore. Using LargeString by default certainly has its implications and does use more resources. Suggestion from arrow upstream folks is to add a setting to the library API that allows conversion from non-large types to large types. Since it's a 3rd party library and a rather complicated change, it could take a long time to get this right & merged. More info on ARROW-17459.

The changes in this PR seems to fix the original case by avoiding that code path. It doesn't solve the latter, tho. It was a suggestion from arrow upstream folks in ARROW-17459-comment.

Based on that, I am re-opening this PR as ready-for-review.

@Avogar kind ping :)

arrow::Status read_status = rbr->ReadAll(&table);

WTF?
Revert?

Actually it's alright.