Repository navigation
[C++][Parquet] Encrypted parquet written by C++ parquet writer not readable with DucKDB #47254
Description
Describe the usage question you have. Please include as many useful details as possible.
I am trying to write encrypted parquet file with teh above code in c++ with arrow and tryign to decrypt the parquet in duckdb using Dbeaver with the below query. But i am getting the error "SQL Error: Java.sql.SQLException: Invalid Input Error: COmputed AES tag differs from read AES tag, are you using the right key?"
Plesae find the sample c++ code and SQL used in Dbeaver in the attached file.
sample_code.txt
Duckdb supports only AES_GCM_V1 and I have used the same. Is there issue in encryption in my code?
Component(s)
C++
Activity
rok commentedon Aug 4, 2025
Thanks for reporting this @ssvenkatesh. Could you provide a minimal example of how you write the parquet file in C++? The
WriterPropertiescode looks good.ssvenkatesh commentedon Aug 4, 2025
Please find the entire sample code attached
[Sample_Test_Encryption.hpp.txt](https://github.com/user-attachments/files/21577000/Sample_Test_Encryption.hpp.txt
Without Encryption by calling Save(false) i am able to open the file in dbeaver and view the data. With encryption i am getting the above error.
rok commentedon Aug 4, 2025
Does is help if you replace
file_encryption_builder.footer_key_id(KEY_LABEL)with
file_encryption_builder.footer_key_metadata(KEY_LABEL)ssvenkatesh commentedon Aug 5, 2025
Tried with footer_key_metadata(KEY_LABEL) and disable_aad_prefix_storage()
got the same error.
also tried to read the AES tag from the parquet file with a python file (readparFooter.txt). So that i can compare parquet generated from ducdkb and arrow. Will that work?
readParFooter..txt
rok commentedon Aug 5, 2025
@ssvenkatesh as per docs DuckDBs parquet encryption is not compatible with PyArrow's parquet encryption (PyArrow is just a wrapper around Arrow C++). I hoped that DuckDB only reads uniformly encrypted files, however even that appears to not be true as per this gist.
Original DuckDBs encryption PR states "We mostly follow the Parquet Modular Encryption specification, save for some details.". PyArrow aims for supporting the encryption spec as do other parquet readers (e.g. arrow-rs) and would probably not support implementation specific encryption.
Edit: My pyarrow gist uses KMS and hence randomly generated keys so DuckDB shouldn't be able to read the generated file (as keys are used without KMS). Uniformly encrypted test file can be generated with this c++ gist.
rok commentedon Aug 5, 2025
@lnkuiper any input here? Should I open an issue?
adamreeve commentedon Aug 5, 2025
Looking at duckdb/duckdb#9392, I don't see anywhere where DuckDB computes the AAD suffix (https://github.com/apache/parquet-format/blob/master/Encryption.md#442-aad-suffix). I might be missing something, but this would explain why the tags differ.
adamreeve commentedon Aug 5, 2025
Looks like DuckDB have made an initial step towards fixing this, although it might be awkward for them to fix this and keep backwards compatibility with files they've previously written: duckdb/duckdb#17508
lnkuiper commentedon Aug 6, 2025
My initial PR tried to implement the Parquet spec, but I found too vague to understand what AAD goes where, so I bailed and stuck to just implementing round-trippable encryption for DuckDB instead.
Lotte's PR indeed made a step towards fixing this.
Do you know whether any other Parquet encryption libraries are compatible with one another? Can PyArrow read encrypted Parquet files written by Spark and vice-versa?
rok commentedon Aug 6, 2025
Hey @lnkuiper, thank you for the insight!
Parquet encryption implementations could do more integration testing, but there is some. E.g. encrypted tables from parquet-testing repo are read by Arrow c++ and arrow-rs tests. I imagine other implementations do something similar.
As @adamreeve points out, fixing compatibility later might be awkward if your users expect that you have by the spec implementation now. In part because you might need to keep supporting current implementation and the spec specified one. But in larger part because by-the-spec implementation implies you can move encrypted files between implementations without any additional work.
Aside - uniformly encrypted test file can be generated with this c++ gist.
lnkuiper commentedon Aug 6, 2025
Thanks! We'll most likely look into this again at some point in the future. It has become more important now that we support encrypted DuckLake
adamreeve commentedon Aug 6, 2025
In addition to what Rok has said, I just want to add that integration with PyArrow is a bit more complicated than C++ Arrow and other implementations, as PyArrow doesn't expose the low-level encryption API where you can set AES keys directly. PyArrow only provides the "key management tools" API (see this design doc), which requires users to implement a client for a Key Management Service. Data encryption keys are randomly generated and encrypted with the KMS and these encrypted keys are stored in the Parquet metadata. This isn't a part of the official Parquet spec, but is implemented by parquet-java / Spark too.
[-]Invalid Input Error: Computed AES tag differs from read AES tag while trying to read encrypted parquet file in Dbeaver/Duckdb[/-][+][C++][Parquet] Encrypted parquet written by C++ parquet writer not readable with DucKDB[/+]rok commentedon Aug 6, 2025
Since this is a DuckDB issue I opened duckdb/duckdb#18524 and will close this. Feel free to reopen if it turns out otherwise.
Enable Parquet reader support for uniformly encrypted Parquet files w…
Enable Parquet reader support for uniformly encrypted Parquet files w…