Skip to content

Commit 33ed1db

Browse files
adamreeverevit13andersonm-ibm
authored
GH-25986: [C++] Enable external material and rotation for encryption keys (#34181)
This PR is a replacement for #10491 which appears to be abandoned. I've fixed the issues pointed out in review comments as well as a few more things I noticed. I've also made one significant change to the API; rather than `CryptoFactory::RotateMasterKeys` operating on a whole directory, it rotates keys for a single file, as this is much more flexible and allows users to decide what files to rotate keys for, and there didn't appear to be a good reason for this to work at the whole directory level. This does mean the API diverges slightly from the parquet-mr API though. ### Rationale for this change Use of external key material allows rotating master encryption keys without having to rewrite Parquet file data. See https://docs.google.com/document/d/1bEu903840yb95k9q2X-BlsYKuXoygE4VnMDl9xz_zhk/edit?usp=sharing for more details. ### What changes are included in this PR? Adds support for writing and reading external key material for Parquet files from C++, as well as a new `CryptoFactory::RotateMasterKeys` function that allows re-encrypting key encryption keys or data encryption keys with latest versions of master keys. ### Are these changes tested? Yes, unit tests are included. I've added an additional test that reads a file generated with parquet-mr (used by Spark) from the parquet-testing repository. This requires merging the PR at apache/parquet-testing#36 and updating the parquet-testing submodule before the new test will pass. ### Are there any user-facing changes? Yes, the existing `internal_key_material` option in `parquet::encryption::EncryptionConfiguration` will now work and use external key material. This requires using two new parameters (`file_path` and `file_system`) in `CryptoFactory::GetFileEncryptionProperties` and `CryptoFactory::GetFileDecryptionProperties`, which are needed so that we know where to write/read the external key material. Note that this means external key material won't work from Python until the new parameters are exposed in Python too. This changes the `CryptoFactory` ABI but the API is still source compatible. Is this okay or should new overloads be added that take the new parameters? The `CryptoFactory::RotateMasterKeys` function is also a new public facing API. * Closes: #25986 Lead-authored-by: Adam Reeve <[email protected]> Co-authored-by: Revital Sur <[email protected]> Co-authored-by: Maya Anderson <[email protected]> Signed-off-by: Will Jones <[email protected]>
1 parent c2e8f25 commit 33ed1db

23 files changed

Lines changed: 855 additions & 153 deletions

‎cpp/cmake_modules/DefineOptions.cmake‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -586,7 +586,10 @@ Always OFF if building binaries" OFF)
586586
"Build the Parquet examples. Requires static libraries to be built." OFF)
587587

588588
define_option(PARQUET_REQUIRE_ENCRYPTION
589-
"Build support for encryption. Fail if OpenSSL is not found" OFF)
589+
"Build support for encryption. Fail if OpenSSL is not found"
590+
OFF
591+
DEPENDS
592+
ARROW_FILESYSTEM)
590593

591594
#----------------------------------------------------------------------
592595
set_option_category("Gandiva")

‎cpp/src/arrow/json/object_parser.cc‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,24 @@ class ObjectParser::Impl {
5252
return document_[key].GetString();
5353
}
5454

55+
Result<std::unordered_map<std::string, std::string>> GetStringMap() const {
56+
std::unordered_map<std::string, std::string> map;
57+
for (auto itr = document_.MemberBegin(); itr != document_.MemberEnd(); ++itr) {
58+
const auto& json_name = itr->name;
59+
const auto& json_value = itr->value;
60+
if (!json_name.IsString()) {
61+
return Status::TypeError("Key is not a string");
62+
}
63+
std::string name = json_name.GetString();
64+
if (!json_value.IsString()) {
65+
return Status::TypeError("Key '", name, "' does not have a string value");
66+
}
67+
std::string value = json_value.GetString();
68+
map.insert({std::move(name), std::move(value)});
69+
}
70+
return map;
71+
}
72+
5573
Result<bool> GetBool(const char* key) const {
5674
if (!document_.HasMember(key)) {
5775
return Status::KeyError("Key '", key, "' does not exist");
@@ -78,6 +96,10 @@ Result<std::string> ObjectParser::GetString(const char* key) const {
7896

7997
Result<bool> ObjectParser::GetBool(const char* key) const { return impl_->GetBool(key); }
8098

99+
Result<std::unordered_map<std::string, std::string>> ObjectParser::GetStringMap() const {
100+
return impl_->GetStringMap();
101+
}
102+
81103
} // namespace internal
82104
} // namespace json
83105
} // namespace arrow

‎cpp/src/arrow/json/object_parser.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
#include <memory>
2121
#include <string_view>
22+
#include <unordered_map>
2223

2324
#include "arrow/result.h"
2425
#include "arrow/util/visibility.h"
@@ -37,8 +38,12 @@ class ARROW_EXPORT ObjectParser {
3738
Status Parse(std::string_view json);
3839

3940
Result<std::string> GetString(const char* key) const;
41+
4042
Result<bool> GetBool(const char* key) const;
4143

44+
// Get all members of the object as a map from string keys to string values
45+
Result<std::unordered_map<std::string, std::string>> GetStringMap() const;
46+
4247
private:
4348
class Impl;
4449
std::unique_ptr<Impl> impl_;

‎cpp/src/parquet/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,7 @@ if(PARQUET_REQUIRE_ENCRYPTION)
224224
encryption/crypto_factory.cc
225225
encryption/file_key_unwrapper.cc
226226
encryption/file_key_wrapper.cc
227+
encryption/file_system_key_material_store.cc
227228
encryption/kms_client.cc
228229
encryption/key_material.cc
229230
encryption/key_metadata.cc

‎cpp/src/parquet/encryption/crypto_factory.cc‎

Lines changed: 29 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,13 @@
1717

1818
#include <string_view>
1919

20-
#include "arrow/result.h"
2120
#include "arrow/util/logging.h"
2221
#include "arrow/util/string.h"
2322

2423
#include "parquet/encryption/crypto_factory.h"
2524
#include "parquet/encryption/encryption_internal.h"
26-
#include "parquet/encryption/file_key_material_store.h"
2725
#include "parquet/encryption/file_key_unwrapper.h"
26+
#include "parquet/encryption/file_system_key_material_store.h"
2827
#include "parquet/encryption/key_toolkit_internal.h"
2928

3029
namespace parquet {
@@ -37,7 +36,8 @@ void CryptoFactory::RegisterKmsClientFactory(
3736

3837
std::shared_ptr<FileEncryptionProperties> CryptoFactory::GetFileEncryptionProperties(
3938
const KmsConnectionConfig& kms_connection_config,
40-
const EncryptionConfiguration& encryption_config) {
39+
const EncryptionConfiguration& encryption_config, const std::string& file_path,
40+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system) {
4141
if (!encryption_config.uniform_encryption && encryption_config.column_keys.empty()) {
4242
throw ParquetException("Either column_keys or uniform_encryption must be set");
4343
} else if (encryption_config.uniform_encryption &&
@@ -47,10 +47,16 @@ std::shared_ptr<FileEncryptionProperties> CryptoFactory::GetFileEncryptionProper
4747
const std::string& footer_key_id = encryption_config.footer_key;
4848
const std::string& column_key_str = encryption_config.column_keys;
4949

50-
std::shared_ptr<FileKeyMaterialStore> key_material_store = NULL;
50+
std::shared_ptr<FileKeyMaterialStore> key_material_store = nullptr;
5151
if (!encryption_config.internal_key_material) {
52-
// TODO: using external key material store with Hadoop file system
53-
throw ParquetException("External key material store is not supported yet.");
52+
try {
53+
key_material_store =
54+
FileSystemKeyMaterialStore::Make(file_path, file_system, false);
55+
} catch (ParquetException& e) {
56+
std::stringstream ss;
57+
ss << "Failed to get key material store.\n" << e.what() << "\n";
58+
throw ParquetException(ss.str());
59+
}
5460
}
5561

5662
FileKeyWrapper key_wrapper(&key_toolkit_, kms_connection_config, key_material_store,
@@ -88,6 +94,9 @@ std::shared_ptr<FileEncryptionProperties> CryptoFactory::GetFileEncryptionProper
8894
}
8995
}
9096

97+
if (key_material_store != nullptr) {
98+
key_material_store->SaveMaterial();
99+
}
91100
return properties_builder.build();
92101
}
93102

@@ -162,15 +171,26 @@ ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties
162171

163172
std::shared_ptr<FileDecryptionProperties> CryptoFactory::GetFileDecryptionProperties(
164173
const KmsConnectionConfig& kms_connection_config,
165-
const DecryptionConfiguration& decryption_config) {
166-
std::shared_ptr<DecryptionKeyRetriever> key_retriever(new FileKeyUnwrapper(
167-
&key_toolkit_, kms_connection_config, decryption_config.cache_lifetime_seconds));
174+
const DecryptionConfiguration& decryption_config, const std::string& file_path,
175+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system) {
176+
auto key_retriever = std::make_shared<FileKeyUnwrapper>(
177+
&key_toolkit_, kms_connection_config, decryption_config.cache_lifetime_seconds,
178+
file_path, file_system);
168179

169180
return FileDecryptionProperties::Builder()
170181
.key_retriever(key_retriever)
171182
->plaintext_files_allowed()
172183
->build();
173184
}
174185

186+
void CryptoFactory::RotateMasterKeys(
187+
const KmsConnectionConfig& kms_connection_config,
188+
const std::string& parquet_file_path,
189+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system, bool double_wrapping,
190+
double cache_lifetime_seconds) {
191+
key_toolkit_.RotateMasterKeys(kms_connection_config, parquet_file_path, file_system,
192+
double_wrapping, cache_lifetime_seconds);
193+
}
194+
175195
} // namespace encryption
176196
} // namespace parquet

‎cpp/src/parquet/encryption/crypto_factory.h‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,24 +105,45 @@ class PARQUET_EXPORT CryptoFactory {
105105
/// GetFileEncryptionProperties()/GetFileDecryptionProperties() methods.
106106
void RegisterKmsClientFactory(std::shared_ptr<KmsClientFactory> kms_client_factory);
107107

108+
/// Get the encryption properties for a Parquet file.
109+
/// If external key material is used then a file system and path to the
110+
/// parquet file must be provided.
108111
std::shared_ptr<FileEncryptionProperties> GetFileEncryptionProperties(
109112
const KmsConnectionConfig& kms_connection_config,
110-
const EncryptionConfiguration& encryption_config);
113+
const EncryptionConfiguration& encryption_config, const std::string& file_path = "",
114+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system = NULLPTR);
111115

116+
/// Get decryption properties for a Parquet file.
112117
/// The returned FileDecryptionProperties object will use the cache inside this
113118
/// CryptoFactory object, so please keep this
114119
/// CryptoFactory object alive along with the returned
115120
/// FileDecryptionProperties object.
121+
/// If external key material is used then a file system and path to the
122+
/// parquet file must be provided.
116123
std::shared_ptr<FileDecryptionProperties> GetFileDecryptionProperties(
117124
const KmsConnectionConfig& kms_connection_config,
118-
const DecryptionConfiguration& decryption_config);
125+
const DecryptionConfiguration& decryption_config, const std::string& file_path = "",
126+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system = NULLPTR);
119127

120128
void RemoveCacheEntriesForToken(const std::string& access_token) {
121129
key_toolkit_.RemoveCacheEntriesForToken(access_token);
122130
}
123131

124132
void RemoveCacheEntriesForAllTokens() { key_toolkit_.RemoveCacheEntriesForAllTokens(); }
125133

134+
/// Rotates master encryption keys for a Parquet file that uses external key material.
135+
/// In single wrapping mode, data encryption keys are decrypted with the old master keys
136+
/// and then re-encrypted with new master keys.
137+
/// In double wrapping mode, key encryption keys are decrypted with the old master keys
138+
/// and then re-encrypted with new master keys.
139+
/// This relies on the KMS supporting versioning, such that the old master key is
140+
/// used when unwrapping a key, and the latest version is used when wrapping a key.
141+
void RotateMasterKeys(const KmsConnectionConfig& kms_connection_config,
142+
const std::string& parquet_file_path,
143+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system,
144+
bool double_wrapping = kDefaultDoubleWrapping,
145+
double cache_lifetime_seconds = kDefaultCacheLifetimeSeconds);
146+
126147
private:
127148
ColumnPathToEncryptionPropertiesMap GetColumnEncryptionProperties(
128149
int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper);

‎cpp/src/parquet/encryption/file_key_material_store.h‎

Lines changed: 35 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
// distributed with this work for additional information
44
// regarding copyright ownership. The ASF licenses this file
55
// to you under the Apache License, Version 2.0 (the
6-
// "License") = 0; you may not use this file except in compliance
6+
// "License"); you may not use this file except in compliance
77
// with the License. You may obtain a copy of the License at
88
//
99
// http://www.apache.org/licenses/LICENSE-2.0
@@ -17,15 +17,43 @@
1717

1818
#pragma once
1919

20+
#include <set>
21+
#include <string>
22+
#include <unordered_map>
23+
24+
#include "arrow/filesystem/filesystem.h"
25+
#include "parquet/platform.h"
26+
2027
namespace parquet {
2128
namespace encryption {
2229

23-
// Key material can be stored outside the Parquet file, for example in a separate small
24-
// file in the same folder. This is important for “key rotation”, when MEKs have to be
25-
// changed (if compromised; or periodically, just in case) - without modifying the Parquet
26-
// files (often immutable).
27-
// TODO: details will be implemented later
28-
class FileKeyMaterialStore {};
30+
/// Stores encryption key material outside the Parquet file, for example in a separate
31+
/// small file in the same folder. This is important for “key rotation”, when MEKs have to
32+
/// be changed (if compromised; or periodically, just in case) - without modifying the
33+
/// Parquet files (often immutable).
34+
class PARQUET_EXPORT FileKeyMaterialStore {
35+
public:
36+
/// Add key material for one encryption key.
37+
virtual void AddKeyMaterial(std::string key_id_in_file, std::string key_material) = 0;
38+
39+
/// Get key material
40+
virtual std::string GetKeyMaterial(std::string key_id_in_file) = 0;
41+
42+
/// After key material was added for all keys in the given Parquet file,
43+
/// save material in persistent store.
44+
virtual void SaveMaterial() = 0;
45+
46+
/// Remove key material from persistent store. Used in key rotation.
47+
virtual void RemoveMaterial() = 0;
48+
49+
/// Move key material to another store. Used in key rotation.
50+
virtual void MoveMaterialTo(std::shared_ptr<FileKeyMaterialStore> target_key_store) = 0;
51+
52+
/// Returns the Set of all key IDs in this store (for the given Parquet file)
53+
virtual std::vector<std::string> GetKeyIDSet() = 0;
54+
55+
virtual ~FileKeyMaterialStore() {}
56+
};
2957

3058
} // namespace encryption
3159
} // namespace parquet

‎cpp/src/parquet/encryption/file_key_unwrapper.cc‎

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,14 +25,27 @@
2525
namespace parquet {
2626
namespace encryption {
2727

28-
using internal::KeyWithMasterId;
28+
FileKeyUnwrapper::FileKeyUnwrapper(
29+
KeyToolkit* key_toolkit, const KmsConnectionConfig& kms_connection_config,
30+
double cache_lifetime_seconds, const std::string& file_path,
31+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system)
32+
: key_toolkit_(key_toolkit),
33+
kms_connection_config_(kms_connection_config),
34+
cache_entry_lifetime_seconds_(cache_lifetime_seconds),
35+
file_path_(file_path),
36+
file_system_(file_system) {
37+
kek_per_kek_id_ = key_toolkit_->kek_read_cache_per_token().GetOrCreateInternalCache(
38+
kms_connection_config.key_access_token(), cache_entry_lifetime_seconds_);
39+
}
2940

30-
FileKeyUnwrapper::FileKeyUnwrapper(KeyToolkit* key_toolkit,
31-
const KmsConnectionConfig& kms_connection_config,
32-
double cache_lifetime_seconds)
41+
FileKeyUnwrapper::FileKeyUnwrapper(
42+
KeyToolkit* key_toolkit, const KmsConnectionConfig& kms_connection_config,
43+
double cache_lifetime_seconds,
44+
std::shared_ptr<FileKeyMaterialStore> key_material_store)
3345
: key_toolkit_(key_toolkit),
3446
kms_connection_config_(kms_connection_config),
35-
cache_entry_lifetime_seconds_(cache_lifetime_seconds) {
47+
cache_entry_lifetime_seconds_(cache_lifetime_seconds),
48+
key_material_store_(std::move(key_material_store)) {
3649
kek_per_kek_id_ = key_toolkit_->kek_read_cache_per_token().GetOrCreateInternalCache(
3750
kms_connection_config.key_access_token(), cache_entry_lifetime_seconds_);
3851
}
@@ -47,12 +60,25 @@ std::string FileKeyUnwrapper::GetKey(const std::string& key_metadata_bytes) {
4760
}
4861
KeyMetadata key_metadata = KeyMetadata::Parse(key_metadata_bytes);
4962

50-
if (!key_metadata.key_material_stored_internally()) {
51-
throw ParquetException("External key material store is not supported yet.");
63+
KeyMaterial key_material;
64+
if (key_metadata.key_material_stored_internally()) {
65+
key_material = key_metadata.key_material();
66+
} else {
67+
if (key_material_store_ == nullptr) {
68+
key_material_store_ =
69+
FileSystemKeyMaterialStore::Make(file_path_, file_system_, false);
70+
}
71+
// External key material storage: key metadata contains a reference
72+
// to a key in the material store
73+
std::string key_id_in_file = key_metadata.key_reference();
74+
std::string key_material_string = key_material_store_->GetKeyMaterial(key_id_in_file);
75+
if (key_material_string.empty()) {
76+
throw ParquetException("Could not find key material with ID '" + key_id_in_file +
77+
"' in external key material file");
78+
}
79+
key_material = KeyMaterial::Parse(key_material_string);
5280
}
5381

54-
const KeyMaterial& key_material = key_metadata.key_material();
55-
5682
return GetDataEncryptionKey(key_material).data_key();
5783
}
5884

‎cpp/src/parquet/encryption/file_key_unwrapper.h‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include "arrow/util/concurrent_map.h"
2121

2222
#include "parquet/encryption/encryption.h"
23+
#include "parquet/encryption/file_system_key_material_store.h"
2324
#include "parquet/encryption/key_material.h"
2425
#include "parquet/encryption/key_toolkit.h"
2526
#include "parquet/encryption/key_toolkit_internal.h"
@@ -31,9 +32,7 @@ namespace encryption {
3132

3233
// This class will retrieve the key from "key metadata", following these steps:
3334
// 1. Parse "key metadata" (see structure in KeyMetadata class).
34-
// 2. Retrieve "key material" which can be stored inside or outside "key metadata"
35-
// Currently we don't support the case "key material" stores outside "key metadata"
36-
// yet.
35+
// 2. Retrieve "key material" which can be stored inside or outside "key metadata".
3736
// 3. Unwrap the "data encryption key" from "key material". There are 2 modes:
3837
// 3.1. single wrapping: decrypt the wrapped "data encryption key" directly with "master
3938
// encryption key" 3.2. double wrapping: 2 steps: 3.2.1. "key encryption key" is decrypted
@@ -44,14 +43,28 @@ class PARQUET_EXPORT FileKeyUnwrapper : public DecryptionKeyRetriever {
4443
/// key_toolkit and kms_connection_config is to get KmsClient from cache or create
4544
/// KmsClient if it's not in the cache yet. cache_entry_lifetime_seconds is life time of
4645
/// KmsClient in the cache.
46+
/// If the file uses external key material then the Parquet file path and file
47+
/// system must be specified.
4748
FileKeyUnwrapper(KeyToolkit* key_toolkit,
4849
const KmsConnectionConfig& kms_connection_config,
49-
double cache_lifetime_seconds);
50+
double cache_lifetime_seconds, const std::string& file_path = "",
51+
const std::shared_ptr<::arrow::fs::FileSystem>& file_system = NULLPTR);
5052

53+
/// Constructor overload that accepts an existing key_material_store rather than using
54+
/// the file path and file system to create one when needed. This is useful for key
55+
/// rotation to allow accessing the key material store after it is used.
56+
FileKeyUnwrapper(KeyToolkit* key_toolkit,
57+
const KmsConnectionConfig& kms_connection_config,
58+
double cache_lifetime_seconds,
59+
std::shared_ptr<FileKeyMaterialStore> key_material_store);
60+
61+
/// Get the data key from key metadata
5162
std::string GetKey(const std::string& key_metadata) override;
5263

64+
/// Get the data key along with the master key id from key material
65+
KeyWithMasterId GetDataEncryptionKey(const KeyMaterial& key_material);
66+
5367
private:
54-
internal::KeyWithMasterId GetDataEncryptionKey(const KeyMaterial& key_material);
5568
std::shared_ptr<KmsClient> GetKmsClientFromConfigOrKeyMaterial(
5669
const KeyMaterial& key_material);
5770

@@ -60,6 +73,9 @@ class PARQUET_EXPORT FileKeyUnwrapper : public DecryptionKeyRetriever {
6073
KeyToolkit* key_toolkit_;
6174
KmsConnectionConfig kms_connection_config_;
6275
const double cache_entry_lifetime_seconds_;
76+
std::shared_ptr<FileKeyMaterialStore> key_material_store_;
77+
const std::string file_path_;
78+
std::shared_ptr<::arrow::fs::FileSystem> file_system_;
6379
};
6480

6581
} // namespace encryption

0 commit comments

Comments
 (0)