Repository navigation
[Python] Deprecate the legacy ParquetDataset custom python-based implementation #31529
Description
To be able to remove the custom python implementation (ARROW-15868), we first need to deprecate the various aspects.
This issue is meant as a parent issue to keep an overview of the different tasks.
Reporter: Joris Van den Bossche / @jorisvandenbossche
Subtasks:
- [Python] Start with deprecating ParquetDataset custom attributes
- [Python] Start to raise deprecation warnings when using use_legacy_dataset=True in parquet.py
- [Python] Start raising deprecation warnings for ParquetDataset keywords that won't be supported with the new API
- [Python] ParquetDataset deprecation: change Deprecation to FutureWarnings
- [Python] Deprecate the (common_)metadata(_path) attributes of ParquetDataset
- [Python] Change use_legacy_dataset default and deprecate no-longer supported keywords in parquet.write_to_dataset
- [Python] Support row_group_size/chunk_size keyword in pq.write_to_dataset with use_legacy_dataset=False
- [Python] Switch default and deprecate use_legacy_dataset=True in ParquetDataset
Related issues:
- [Python] Remove the legacy ParquetDataset custom python-based implementation #31303 (blocks)
- [Python] Long-term fate of pyarrow.parquet.ParquetDataset #25775 (is a child of)
- [Python] Remove the test usage of legacy dataset #20233 (is required by)
Note: This issue was originally created as ARROW-16119. Please see the migration documentation for further details.
Activity
asfimport commentedon Apr 25, 2022
Joris Van den Bossche / @jorisvandenbossche:
We included many additional deprecations in pyarrow 8.0.0, but we still need to deprecate / switch the
use_legacy_datasetkeyword inParquetDataset(..)constructor itself (we are first deprecating the parameters and properties that are enabled by the legacy dataset constructor).asfimport commentedon Apr 25, 2022
Alenka Frim / @AlenkaF:
Sorry for closing it, I misunderstood!
asfimport commentedon Jun 2, 2022
Joris Van den Bossche / @jorisvandenbossche:
The ParquetDataset() constructor itself still defaults to
use_legacy_dataset=True(although using specific attributes or keywords related to that will raise a warning). So a next step will be to actually deprecate passing that and switching the default, and then only afterwards we can remove the code (-> opened ARROW-16728).AlenkaF commentedon Dec 4, 2023
As the
ParquetDatasetalready defaults touse_legacy_dataset=Falsearrow/python/pyarrow/parquet/core.py
Lines 1807 to 1809 in f7947cc
jorisvandenbossche commentedon Dec 5, 2023
Indeed, and we also warn when specifically choosing the legacy one.
GH-31303: [Python] Remove the legacy ParquetDataset custom python-bas…
apacheGH-31303: [Python] Remove the legacy ParquetDataset custom pyth…