Repository navigation
Add "read_kms_url" setting to control reading the URL from file metadata - #46
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The shared KMS client cache can reuse a metadata-derived URL even when URL reading is disabled.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Adds an opt-in decryption setting that permits reading KMS URLs from Parquet key material, mirroring Apache Arrow’s security behavior.
Changes:
- Adds
read_kms_url, defaulting tofalse. - Exposes the setting through DataFusion options.
- Updates decryption tests for enabled and disabled behavior.
| File | Description |
|---|---|
key_unwrapper.rs |
Controls whether metadata supplies the KMS URL. |
crypto_factory.rs |
Adds configuration, builder support, and tests. |
datafusion.rs |
Adds DataFusion configuration conversion and tests. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -103,13 +105,17 @@ impl KeyUnwrapper { | |||
| } | |||
|
|
|||
| if mut_config.kms_instance_url().is_empty() { | |||
There was a problem hiding this comment.
Is there a use-case where no URL is configured, read_kms_url is true, no URL is given in the metadata, and user wants to fallback to DEFAULT?
There was a problem hiding this comment.
When writing, the URL is set to "DEFAULT" if a URL wasn't provided. So it's expected that there should always be something in the metadata. And this matches the existing behaviour. But maybe it would make sense to just fall back to "DEFAULT" on read too if one wasn't set.
There was a problem hiding this comment.
If it i expected to be an URL in the metadata and there isn't, it should fail. If user doesn't require an URL in the metadata but want to give it precedence, it should fallback to "DEFAULT" with read_kms_url being true. We should check with Arrow folks first and add this feature there if they think this is useful.
There was a problem hiding this comment.
Yeah I'm not sure how much this feature is used, if at all. I expect most KMS client implementations would just hard code the URL, or always provide it in the KMS configuration rather than expect to read it from files.
| } | ||
| mut_config.set_kms_instance_url(kms_instance_url.to_owned()); | ||
| } else { | ||
| mut_config.set_kms_instance_url("DEFAULT".to_owned()); |
There was a problem hiding this comment.
Yeah I guess it's expected that the client checks for this and knows how to handle it. I will add some documentation so it's more obvious that this value might be used.
This behaviour has been ported from the C++ implementation, which originally came from the Java implementation. Maybe just using an empty string would have been a lot simpler but I think it makes sense to keep the same behaviour as those other implementations.
There was a problem hiding this comment.
I've pushed a change to document this (66ddfb9)

This adds a new decryption option to control whether KMS URLs are read from file key material, which is disabled by default.
Any users previously relying on this behaviour will now need to opt-in to it, asserting that the KMS implementation validates the URLs it receives.
This change mirrors a similar change in C++ Parquet: apache/arrow#51350