Skip to content

Commit b70ad0b

Browse files
authored
GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation (#39112)
### Rationale for this change Legacy ParquetDataset has been deprecated for a while now, see #31529. This PR is removing the legacy implementation from the code. ### What changes are included in this PR? The PR is removing: - `ParquetDatasetPiece ` - `ParquetManifest` - `_ParquetDatasetMetadata ` - `ParquetDataset` The PR is renaming `_ParquetDatasetV2` to `ParquetDataset` which was removed. It is also updating the docstrings. The PR is updating: - `read_table` - `write_to_dataset` The PR is updating all the tests to not use `use_legacy_dataset` keyword or legacy parametrisation. ### Are these changes tested? Yes. ### Are there any user-facing changes? Deprecated code is removed. * Closes: #31303
1 parent e5145bf commit b70ad0b

15 files changed

Lines changed: 630 additions & 3029 deletions

‎docs/source/python/parquet.rst‎

Lines changed: 11 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -511,36 +511,20 @@ from a remote filesystem into a pandas dataframe you may need to run
511511
``sort_index`` to maintain row ordering (as long as the ``preserve_index``
512512
option was enabled on write).
513513

514-
.. note::
515-
516-
The ParquetDataset is being reimplemented based on the new generic Dataset
517-
API (see the :ref:`dataset` docs for an overview). This is not yet the
518-
default, but can already be enabled by passing the ``use_legacy_dataset=False``
519-
keyword to :class:`ParquetDataset` or :func:`read_table`::
520-
521-
pq.ParquetDataset('dataset_name/', use_legacy_dataset=False)
522-
523-
Enabling this gives the following new features:
524-
525-
- Filtering on all columns (using row group statistics) instead of only on
526-
the partition keys.
527-
- More fine-grained partitioning: support for a directory partitioning scheme
528-
in addition to the Hive-like partitioning (e.g. "/2019/11/15/" instead of
529-
"/year=2019/month=11/day=15/"), and the ability to specify a schema for
530-
the partition keys.
531-
- General performance improvement and bug fixes.
514+
Other features:
532515

533-
It also has the following changes in behaviour:
516+
- Filtering on all columns (using row group statistics) instead of only on
517+
the partition keys.
518+
- Fine-grained partitioning: support for a directory partitioning scheme
519+
in addition to the Hive-like partitioning (e.g. "/2019/11/15/" instead of
520+
"/year=2019/month=11/day=15/"), and the ability to specify a schema for
521+
the partition keys.
534522

535-
- The partition keys need to be explicitly included in the ``columns``
536-
keyword when you want to include them in the result while reading a
537-
subset of the columns
523+
Note:
538524

539-
This new implementation is already enabled in ``read_table``, and in the
540-
future, this will be turned on by default for ``ParquetDataset``. The new
541-
implementation does not yet cover all existing ParquetDataset features (e.g.
542-
specifying the ``metadata``, or the ``pieces`` property API). Feedback is
543-
very welcome.
525+
- The partition keys need to be explicitly included in the ``columns``
526+
keyword when you want to include them in the result while reading a
527+
subset of the columns
544528

545529

546530
Using with Spark

‎python/benchmarks/parquet.py‎

Lines changed: 0 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -29,35 +29,6 @@
2929
pq = None
3030

3131

32-
class ParquetManifestCreation(object):
33-
"""Benchmark creating a parquet manifest."""
34-
35-
size = 10 ** 6
36-
tmpdir = None
37-
38-
param_names = ('num_partitions', 'num_threads')
39-
params = [(10, 100, 1000), (1, 8)]
40-
41-
def setup(self, num_partitions, num_threads):
42-
if pq is None:
43-
raise NotImplementedError("Parquet support not enabled")
44-
45-
self.tmpdir = tempfile.mkdtemp('benchmark_parquet')
46-
rnd = np.random.RandomState(42)
47-
num1 = rnd.randint(0, num_partitions, size=self.size)
48-
num2 = rnd.randint(0, 1000, size=self.size)
49-
output_df = pd.DataFrame({'num1': num1, 'num2': num2})
50-
output_table = pa.Table.from_pandas(output_df)
51-
pq.write_to_dataset(output_table, self.tmpdir, ['num1'])
52-
53-
def teardown(self, num_partitions, num_threads):
54-
if self.tmpdir is not None:
55-
shutil.rmtree(self.tmpdir)
56-
57-
def time_manifest_creation(self, num_partitions, num_threads):
58-
pq.ParquetManifest(self.tmpdir, metadata_nthreads=num_threads)
59-
60-
6132
class ParquetWriteBinary(object):
6233

6334
def setup(self):

0 commit comments

Comments
 (0)