Repository navigation
[Python] Expose decryption properties in ParquetFragmentScanOptions #31410
Description
Parquet modular encryption is now available in the Python Parquet APIs. However, it's not available through datasets, as ParquetFragmentScanOptions currently doesn't expose the FileDecryptionProperties (it does in C++).
Reporter: Antoine Pitrou / @pitrou
Related issues:
Note: This issue was originally created as ARROW-15986. Please see the migration documentation for further details.
Activity
asfimport commentedon Mar 21, 2022
Antoine Pitrou / @pitrou:
cc @jorisvandenbossche @andersonm-ibm
asfimport commentedon Mar 23, 2022
Joris Van den Bossche / @jorisvandenbossche:
@andersonm-ibm this might be a duplicate of ARROW-13593 ?
asfimport commentedon Mar 23, 2022
Antoine Pitrou / @pitrou:
ARROW-13593 seems to be about dataset writing, so I'm not sure.
asfimport commentedon Mar 23, 2022
Maya Anderson / @andersonm-ibm:
@jorisvandenbossche , @pitrou , There is a problem with this suggestion of exposing the
FileDecryptionPropertiesas they are currently exposed in C++, in that it doesn't enable external key material, which is WIP. See https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit# .Therefore, before we can expose PME in the new dataset API in Python, there is work needed in the C++ layer, which is being addressed by ARROW-13593 .
jorisvandenbossche commentedon Aug 15, 2023
Closing as duplicate of #29238 (the ARROW-13593 mentioned above)