Repository navigation
[Python] Switch default and deprecate use_legacy_dataset=True in ParquetDataset #32067
Description
The ParquetDataset() constructor itself still defaults to use_legacy_dataset=True (although using specific attributes or keywords related to that will raise a warning). So a next step will be to actually deprecate passing that and switching the default, and then only afterwards we can remove the code.
Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Joris Van den Bossche / @jorisvandenbossche
PRs and other links:
Note: This issue was originally created as ARROW-16728. Please see the migration documentation for further details.
Activity
asfimport commentedon Aug 31, 2022
Raúl Cumplido / @raulcd:
@jorisvandenbossche was your idea for this one to be done on two different releases?
First - DeprecationWarning and switch default to use_legacy_dataset=False
Second - Remove possibility of using use_legacy_dataset=True
asfimport commentedon Sep 6, 2022
Joris Van den Bossche / @jorisvandenbossche:
Yes, indeed (but so the two first items together in one release (switching default + deprecating the option, and then only in a later release actually remove the option).
We should certainly do that first step in 10.0
asfimport commentedon Dec 23, 2022
Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 14052
#14052
0x26res commentedon Feb 3, 2023
Now that this change is effective in 11.0, we get this warning when loading data with
use_legacy_dataset=True.I'm in the process of migrating to
use_legacy_dataset=False, but was wondering what differences to expect between the 2 implementations. Is this documented somewhere?I have noticed one significant difference in behaviour. The legacy implementation would complain if the parquet schema are heterogeneous. The new implementation will try to convert all files to the schema of the first file it found (or the
schemaargument when provided).Are there other differences to expect?