Repository navigation
[Python] Start with deprecating ParquetDataset custom attributes #18710
Description
As a first step for ARROW-9720, we should start with deprecating attributes/methods of pq.ParquetDataset that we would definitely not keep / are conflicting with the "dataset API".
I am thinking of the pieces attribute (and the ParquetDatasetPiece class), the partitions attribute (and the ParquetPartitions class).
In addition, some of the keywords are also exposed as properties (memory_map, read_dictionary, buffer_size, fs), and could be deprecated.
Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Joris Van den Bossche / @jorisvandenbossche
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-13074. Please see the migration documentation for further details.
Activity
asfimport commentedon Jul 6, 2021
Lance Dacey / @ldacey:
Any idea if this includes the partition_filename_cb function? I am still using that pretty extensively to write my "final" datasets that Power BI connects to for visualization since it allows me to overwrite each partition.
asfimport commentedon Jul 6, 2021
Weston Pace / @westonpace:
@ldacey Does "basename_template" provide the functionality you seek? It's currently documented as "A template string used to generate basenames of written data files. The token '{i}' will be replaced with an automatically incremented integer. If not specified, it defaults to "part-{i}." + format.default_extname" I seem to recall there was some earlier discussion of this on a related Jira but I don't remember which one.
asfimport commentedon Jul 6, 2021
Joris Van den Bossche / @jorisvandenbossche:
@ldacey the current PR is only about the
ParquetDatasetclass, so for reading parquet, not for writing parquetLet's keep discussion about
partition_filename_cbfor ARROW-12365 (the issue @ldacey created specifically for that topic; there was also discussion at ARROW-10695 on this topic)asfimport commentedon Jul 6, 2021
Joris Van den Bossche / @jorisvandenbossche:
@ldacey can you post the comment (that you deleted again) about the issues you ran into with
basename_templatein ARROW-12365 ?asfimport commentedon Jul 7, 2021
Lance Dacey / @ldacey:
Sure Joris, I posted it and then I read that you said to keep the discussion separate so I tried to be sneaky and delete it before you noticed
asfimport commentedon Jul 15, 2021
David Li / @lidavidm:
Issue resolved by pull request 10549
#10549
asfimport commentedon Jul 31, 2021
Richard Shadrach:
Is my understanding correct that
ParquetDataset.partitionshas been deprecated but with no current alternative (as mentioned in ARROW-9720)?asfimport commentedon Aug 2, 2021
Joris Van den Bossche / @jorisvandenbossche:
There is actually some replacement (but not 1:1) if you use the Datasets API, see ARROW-8655 / #10661 (comment). That added a
partitioningattribute with similar information.Can you check if that gives sufficient information for your use case?
Now, we should certainly better document this / mention it in the deprecation message (I worked on both changes simultaneously, but should afterwards have updated the deprecation message that an alternative exists). We should maybe also expose the
partitioningattribute on ParquetDataset (if constructed withuse_legacy_dataset=False), as I did for thefilesystem/files/fragmentsattributes.asfimport commentedon Aug 2, 2021
Joris Van den Bossche / @jorisvandenbossche:
I opened ARROW-13525 to track this follow-up.
asfimport commentedon Aug 3, 2021
Richard Shadrach:
Thanks @jorisvandenbossche - it does. Warnings showed up on pandas CI , here is my proposed fix: pandas-dev/pandas#42873.