ARROW-8655: [C++][Python] Preserve partitioning information for a discovered Dataset - #10661

https://issues.apache.org/jira/browse/ARROW-8655

See the JIRA for more details and motivating use cases, but a small demo of what this exposes:

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

df = pd.DataFrame({"year": [2020, 2020, 2021, 2021], "month":[1, 2, 1, 2], "values": [1, 2, 3, 4]})
df.to_parquet("test_partitioned", partition_cols=["year", "month"], engine="pyarrow")

A discovered dataset now stores the Partitioning object that was created/discovered in the FileSystemDatasetFactory, so you can still access this information on the dataset:

>>> dataset = ds.dataset("test_partitioned/", partitioning="hive")
>>> dataset
<pyarrow._dataset.FileSystemDataset at 0x7fe481b6e270>
>>> dataset.partitioning
<pyarrow._dataset.HivePartitioning at 0x7fe48162c7f0>

# the schema (field names and types) of the partitioning
>>> dataset.partitioning.schema 
year: int32
month: int32
>>> dataset.partitioning.schema.names
['year', 'month']
# and all partition field values discovered during the factory
>>> dataset.partitioning.dictionaries
[<pyarrow.lib.Int32Array object at 0x7fe480fd9fa0>
 [
   2020,
   2021
 ],
 <pyarrow.lib.Int32Array object at 0x7fe480fd9b80>
 [
   1,
   2
 ]]

With inferring a dictionary type for the partitioning, the schema of the partitioning object indeed has dictionary type:

>>> dataset2 = ds.dataset("test_partitioned/", partitioning=ds.HivePartitioning.discover(infer_dictionary=True))
>>> datatset2.partitioning.schema
year: dictionary<values=int32, indices=int32, ordered=0>
month: dictionary<values=int32, indices=int32, ordered=0>
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [{"name": null, "field_n' + 352

When the FileSystemDataset is not discovered with a partitioning or not created through a discovery, the partitioning attribute is None:

>>> dataset3 = ds.dataset("test_partitioned/")
# TODO this is still something to fix, right now this is a "default" partitioning (which isn't exposed in Python)
>>> dataset3.partitioning
...
~/scipy/repos/arrow/python/pyarrow/_dataset.pyx in pyarrow._dataset.Partitioning.wrap()
TypeError: default

>>> dataset4 = ds.FileSystemDataset(list(dataset.get_fragments()), dataset.schema, dataset.format, dataset.filesystem)
>>> dataset4.partitioning is None
True

I'll add my comment from the JIRA here as you suggested. Another approach could be to expose the dictionaries as part of the partitioning factory output. So the partitioning factory could have a dictionaries() property that is set after Finish has been called. This puts a bit more complexity on python since it hides the partitioning factory from the user but it could track the dictionaries and expose them where needed.

Updated the PR according to the comments + added tests and docstrings.

Another approach could be to expose the dictionaries as part of the partitioning factory output. So the partitioning factory could have a dictionaries() property that is set after Finish has been called. This puts a bit more complexity on python since it hides the partitioning factory from the user but it could track the dictionaries and expose them where needed.

In the end, I am fine either way how we achieve it, and I suppose this could also work (in the python factory functions, we pass the PartitioningFactory before calling Finish on the dataset factory, so we have access to this object, could extract dictionaries, and attach this to the Dataset class only at the cython level).
If this would be preferred, I can take a look at it. But, I think the changes on the C++ side in the current implementation are rather limited and not invasive, and this makes this easier at the Python side. I also don't know to what extent there would be interest in this on the R side (it could make a read CSV -> write parquet flow easier for the case where you want to preserve the exact partitioning).