GH-31869: [Python][Parquet] Implement external key material features in Python by pfparsons · Pull Request #48009 · apache/arrow

pfparsons · 2025-10-30T23:51:16Z

Rationale for this change

Enables external key material and rotation for individual parquet files in PyArrow. This change does not address any parquet dataset encryption functionality.

What changes are included in this PR?

This PR enables external key material for parquet encryption from PyArrow:
Optional parquet_file_path and FileSystem paramters to CryptoFactory - mirroring the interface for CryptoFactory in C++

Exposes the rotate_master_keys method of CryptoFactory
Adds Cython classes for FileKeyMaterialStore, FileSystemKeyMaterialStore, and KeyMaterial - but does not expose these from PyArrow encryption. I included these changes only so that a unit test may verify an external store without leaking the implementation details for the store into the test.

Are these changes tested?

Yes - I've modified an existing test (previously marked pytest.xfail) to do a basic read write test and verify creation of the external key material store and added a test for CryptoFactory.rotate_master_keys.

Are there any user-facing changes?

Users may optionally supply a parquet file path and FileSystem to CryptoFactory methods that provide en/decryption_properties. Doing so in conjunction with setting EncryptionConfiguration.internal_key_material=False enables external key material from pyarrow.
PyArrow CryptoFactory now has a rotate_master_keys method exposing key rotation functionality from C++ CryptoFactory.

GitHub Issue: [Python][Parquet] Enable usage of external key material and rotation for encryption keys in PyArrow #31869

github-actions · 2025-10-30T23:51:41Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2025-10-30T23:55:46Z

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

github-actions · 2025-10-30T23:57:21Z

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

github-actions · 2025-11-04T17:50:33Z

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

pfparsons · 2025-11-04T17:54:30Z

Hi @raulcd would you mind taking a look at this?

raulcd

Thanks for the PR @pfparsons !

I am not too familiar with the Parquet encryption side of things but I'll try to take a look. I am running CI at the moment. Maybe @EnricoMi @rok @adamreeve who have worked on Parquet Encryption in the past can take a look

adamreeve

Hi @pfparsons, thanks for the contribution! There are some minor formatting issues that the pre-commit script is reporting in https://github.com/apache/arrow/actions/runs/19077444221/job/54553464743?pr=48009, but the other CI failures are unrelated and are failing on main at the moment.

You can install pre-commit and run it locally to reproduce these errors, see https://pre-commit.com/. You can use something like pre-commit run --from-ref main --to-ref feature/31869 to run it on changes already committed.

I've left some comments with suggested changes, but this mostly looks really good to me. I'm not super familiar with cython though.

Co-authored-by: Adam Reeve <[email protected]>

adamreeve

This looks good to me thanks @pfparsons. I've just left a couple of minor comments.

EnricoMi

LGTM, just a minor comment on the default value of a string parameter.

Changed defaults for get_{en,de}cryption_properties() parquet_file_path args from empty string to None Removed unsed custom_kms_conf arg from test_external_key_material_rotation

github-actions · 2025-11-08T19:09:07Z

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

github-actions · 2025-11-11T07:26:25Z

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

github-actions · 2025-11-11T20:36:58Z

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

conbench-apache-arrow · 2025-11-12T10:23:18Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 5112de2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 137 possible false positives for unstable benchmarks that are known to sometimes produce them.

…tures in Python (apache#48009) ### Rationale for this change Enables external key material and rotation for individual parquet files in PyArrow. This change does not address any parquet dataset encryption functionality. ### What changes are included in this PR? This PR enables external key material for parquet encryption from PyArrow: Optional parquet_file_path and FileSystem paramters to CryptoFactory - mirroring the interface for CryptoFactory in C++ 1. Exposes the rotate_master_keys method of CryptoFactory 2. Adds Cython classes for FileKeyMaterialStore, FileSystemKeyMaterialStore, and KeyMaterial - but does not expose these from PyArrow encryption. I included these changes only so that a unit test may verify an external store without leaking the implementation details for the store into the test. ### Are these changes tested? Yes - I've modified an existing test (previously marked pytest.xfail) to do a basic read write test and verify creation of the external key material store and added a test for CryptoFactory.rotate_master_keys. ### Are there any user-facing changes? 1. Users may optionally supply a parquet file path and FileSystem to CryptoFactory methods that provide en/decryption_properties. Doing so in conjunction with setting EncryptionConfiguration.internal_key_material=False enables external key material from pyarrow. 2. PyArrow CryptoFactory now has a rotate_master_keys method exposing key rotation functionality from C++ CryptoFactory. * GitHub Issue: apache#31869 Lead-authored-by: Patrick Parsons <[email protected]> Co-authored-by: Adam Reeve <[email protected]> Signed-off-by: Adam Reeve <[email protected]>

First pass at implementing external key material features in python

3c3e85e

pfparsons requested review from AlenkaF, raulcd, rok and wgtmac as code owners October 30, 2025 23:51

github-actions Bot added Component: Parquet Component: C++ Component: Python awaiting review labels Oct 30, 2025

pfparsons changed the title ~~First pass at implementing external key material features in python~~ GH-31869: [Python][Parquet] First pass at implementing external key material features in python Oct 30, 2025

pfparsons added 2 commits October 30, 2025 20:03

removed unused json import from _parquet_encryption.pyx

92c510f

Removed bare except in test_encrypted_parquet_write_read_external

58fcae3

raulcd reviewed Nov 5, 2025

View reviewed changes

github-actions Bot added awaiting changes and removed awaiting review labels Nov 5, 2025

adamreeve requested changes Nov 5, 2025

View reviewed changes

Apply suggestions from code review

09f0b3a

Co-authored-by: Adam Reeve <[email protected]>

github-actions Bot added awaiting change review and removed awaiting changes labels Nov 6, 2025

pfparsons added 2 commits November 7, 2025 20:47

Unit test for CryptoFactory.rotate_master_keys

cf6608c

removed unused test helper function

c336a93

adamreeve approved these changes Nov 8, 2025

View reviewed changes

Comment thread python/pyarrow/tests/parquet/conftest.py Resolved

Comment thread python/pyarrow/tests/parquet/test_encryption.py Resolved

EnricoMi reviewed Nov 8, 2025

View reviewed changes

Comment thread python/pyarrow/_parquet_encryption.pyx Resolved

Addressed PR feedback on CryptoFactory method args

f5199f0

Changed defaults for get_{en,de}cryption_properties() parquet_file_path args from empty string to None Removed unsed custom_kms_conf arg from test_external_key_material_rotation

pfparsons commented Nov 8, 2025

View reviewed changes

Comment thread python/pyarrow/tests/parquet/test_encryption.py Resolved

Consolidated pyarrow parquet key rotation tests

6477931

github-actions Bot added awaiting changes and removed awaiting change review labels Nov 10, 2025

Changed encryption key rotation test to parameterized

fe0a715

pfparsons force-pushed the feature/31869 branch from c7cb646 to fe0a715 Compare November 11, 2025 07:14

rok mentioned this pull request Nov 11, 2025

[Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

Closed

adamreeve changed the title ~~GH-31869: [Python][Parquet] First pass at implementing external key material features in python~~ GH-31869: [Python][Parquet] Implement external key material features in Python Nov 11, 2025

adamreeve merged commit 5112de2 into apache:main Nov 12, 2025

adamreeve removed the awaiting change review label Nov 12, 2025

adamreeve mentioned this pull request Nov 12, 2025

[Python][Parquet] Enable usage of external key material and rotation for encryption keys in PyArrow #31869

Closed

Conversation

pfparsons commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

github-actions Bot commented Nov 4, 2025

Uh oh!

pfparsons commented Nov 4, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

adamreeve left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamreeve left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

EnricoMi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 8, 2025

Uh oh!

github-actions Bot commented Nov 11, 2025

Uh oh!

github-actions Bot commented Nov 11, 2025

Uh oh!

conbench-apache-arrow Bot commented Nov 12, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pfparsons commented Oct 30, 2025 •

edited

Loading

adamreeve left a comment •

edited

Loading