Repository navigation
[Python] Support row_group_size/chunk_size keyword in pq.write_to_dataset with use_legacy_dataset=False #31636
Description
The pq.write_to_dataset (legacy implementation) supports the row_group_size/chunk_size keyword to specify the row group size of the written parquet files.
Now that we made use_legacy_dataset=False the default, this keyword doesn't work anymore.
This is because dataset.write_dataset(..) doesn't support the parquet row_group_size keyword. The ParquetFileWriteOptions class doesn't support this keyword.
On the parquet side, this is also the only keyword that is not passed to the ParquetWriter init (and thus to parquet's WriterProperties or ArrowWriterProperties), but to the actual write_table call. In C++ this can be seen at
arrow/cpp/src/parquet/arrow/writer.h
Lines 62 to 71 in 76d064c
See discussion: #12811 (comment)
Reporter: Alenka Frim / @AlenkaF
Assignee: Alenka Frim / @AlenkaF
PRs and other links:
Note: This issue was originally created as ARROW-16240. Please see the migration documentation for further details.
Activity
asfimport commentedon Apr 21, 2022
Joris Van den Bossche / @jorisvandenbossche:
cc @westonpace do you remember if this has been discussed before how the row_group_size/chunk_size setting from Parquet fits into the dataset API?
The dataset API now has a
max_rows_per_group, I see, but that doesn't necessarily directly relate to Parquet row groups?It's more generic about how many rows are written in one go, but so effectively is therefore also a max parquet row group size? (since those need to be written in one go) In that sense the parquet
row_group_sizekeyword could be translated into that keyword to preserve the intended usecase?asfimport commentedon Apr 21, 2022
Weston Pace / @westonpace:
Your understanding is correct. I think
max_rows_per_groupis the correct choice here. Each call toWrite(e.g. one go) results inso it will create a new parquet row group.
It might also be useful to also set
min_rows_per_grouptorow_group_sizebut that would be a change in behavior so maybe we shouldn't do this too (the legacy behavior would just write tiny groups in this case).asfimport commentedon Apr 22, 2022
Krisztian Szucs / @kszucs:
Issue resolved by pull request 12955
#12955