Skip to content

Commit 64b5b51

Browse files
author
Laurens Kuiper
authored
Enable Parquet reader support for uniformly encrypted Parquet files written by Arrow C++ API (#20046)
This PR enables to read _uniformly encrypted parquet files_ written by the Arrow C++ API as addressed in apache/arrow#47254. Fixes duckdb/duckdb#18524 and https://github.com/duckdblabs/duckdb-internal/issues/5511. Previously, we were not able to read uniformly encrypted parquet files written with Arrow since we did not use the additional authenticated data (aad). The aad consist of a unique prefix and a suffix. The suffix consists of the module type, row group ordinal (present in metadata), column ordinal (column index of the _original_ scheme of a parquet file) and a page ordinal (page index). More details can be found in the [Parquet Encryption Spec](https://parquet.apache.org/docs/file-format/data-pages/encryption/). For testing I've added the uniformly encrypted file from the [parquet-testing](https://github.com/apache/parquet-testing/tree/master/data) repo and used the [gist](https://gist.github.com/rok/8a68066c51801458a3746772a2c736c5) of @rok. On top of this, I've added an Arrow file containing multiple row groups based on the previous gist ([code](https://gist.github.com/ccfelius/3def798ade5de006e6d0073c1c457990)). As also mentioned in apache/arrow#47254, PyArrow uses a KMS which creates randomly generated keys. This means that DuckDB is not able to read files written with PyArrow, since keys are used directly in DuckDB. The fix ensures backwards compatibility with our own encrypted parquet files. In a follow up PR I'll address the parquet writer.
2 parents 3a4b865 + 75e1eb0 commit 64b5b51

3 files changed

File tree

5.57 KB
Binary file not shown.
833 Bytes
Binary file not shown.
Binary file not shown.

0 commit comments

Comments
 (0)