Skip to content

Add "read_kms_url" setting to control reading the URL from file metadata - #46

Merged
adamreeve merged 2 commits into
G-Research:mainfrom
adamreeve:read-kms-url-flag
Sep 30, 2026
Merged

adamreeve merged 2 commits into
G-Research:mainfrom
adamreeve:read-kms-url-flag

Conversation

@adamreeve

Copy link
Copy Markdown
Contributor

This adds a new decryption option to control whether KMS URLs are read from file key material, which is disabled by default.

Any users previously relying on this behaviour will now need to opt-in to it, asserting that the KMS implementation validates the URLs it receives.

This change mirrors a similar change in C++ Parquet: apache/arrow#51350

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The shared KMS client cache can reuse a metadata-derived URL even when URL reading is disabled.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Adds an opt-in decryption setting that permits reading KMS URLs from Parquet key material, mirroring Apache Arrow’s security behavior.

Changes:

  • Adds read_kms_url, defaulting to false.
  • Exposes the setting through DataFusion options.
  • Updates decryption tests for enabled and disabled behavior.
File Description
key_unwrapper.rs Controls whether metadata supplies the KMS URL.
crypto_factory.rs Adds configuration, builder support, and tests.
datafusion.rs Adds DataFusion configuration conversion and tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@adamreeve
adamreeve requested a review from EnricoMi September 27, 2026 10:32
@@ -103,13 +105,17 @@ impl KeyUnwrapper {
}

if mut_config.kms_instance_url().is_empty() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a use-case where no URL is configured, read_kms_url is true, no URL is given in the metadata, and user wants to fallback to DEFAULT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When writing, the URL is set to "DEFAULT" if a URL wasn't provided. So it's expected that there should always be something in the metadata. And this matches the existing behaviour. But maybe it would make sense to just fall back to "DEFAULT" on read too if one wasn't set.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it i expected to be an URL in the metadata and there isn't, it should fail. If user doesn't require an URL in the metadata but want to give it precedence, it should fallback to "DEFAULT" with read_kms_url being true. We should check with Arrow folks first and add this feature there if they think this is useful.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm not sure how much this feature is used, if at all. I expect most KMS client implementations would just hard code the URL, or always provide it in the KMS configuration rather than expect to read it from files.

}
mut_config.set_kms_instance_url(kms_instance_url.to_owned());
} else {
mut_config.set_kms_instance_url("DEFAULT".to_owned());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "DEFAULT" mean as an URL?

@adamreeve adamreeve Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I guess it's expected that the client checks for this and knows how to handle it. I will add some documentation so it's more obvious that this value might be used.

This behaviour has been ported from the C++ implementation, which originally came from the Java implementation. Maybe just using an empty string would have been a lot simpler but I think it makes sense to keep the same behaviour as those other implementations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pushed a change to document this (66ddfb9)

@EnricoMi EnricoMi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@adamreeve
adamreeve merged commit 6dec9e4 into G-Research:main Sep 30, 2026
@adamreeve
adamreeve deleted the read-kms-url-flag branch September 30, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants