Repository navigation
Commit 64b5b51
Laurens Kuiper
Enable Parquet reader support for uniformly encrypted Parquet files written by Arrow C++ API (#20046)
This PR enables to read _uniformly encrypted parquet files_ written by
the Arrow C++ API as addressed in
apache/arrow#47254. Fixes
duckdb/duckdb#18524 and
https://github.com/duckdblabs/duckdb-internal/issues/5511.
Previously, we were not able to read uniformly encrypted parquet files
written with Arrow since we did not use the additional authenticated
data (aad). The aad consist of a unique prefix and a suffix. The suffix
consists of the module type, row group ordinal (present in metadata),
column ordinal (column index of the _original_ scheme of a parquet file)
and a page ordinal (page index). More details can be found in the
[Parquet Encryption
Spec](https://parquet.apache.org/docs/file-format/data-pages/encryption/).
For testing I've added the uniformly encrypted file from the
[parquet-testing](https://github.com/apache/parquet-testing/tree/master/data)
repo and used the
[gist](https://gist.github.com/rok/8a68066c51801458a3746772a2c736c5) of
@rok. On top of this, I've added an Arrow file containing multiple row
groups based on the previous gist
([code](https://gist.github.com/ccfelius/3def798ade5de006e6d0073c1c457990)).
As also mentioned in apache/arrow#47254,
PyArrow uses a KMS which creates randomly generated keys. This means
that DuckDB is not able to read files written with PyArrow, since keys
are used directly in DuckDB.
The fix ensures backwards compatibility with our own encrypted parquet
files. In a follow up PR I'll address the parquet writer.3 files changed
File tree
- parquet-testing/encryption
Binary file not shown.
Binary file not shown.
Binary file not shown.
0 commit comments