Skip to content

About

Implements the Parquet Key Management Tools API in Rust to enable integration with a Key Management Server when using Parquet modular encryption.

Resources

Stars

4 stars

Watchers

12 watching

Forks

Repository files navigation

Parquet Key Management for Rust

crates.io docs.rs

This library implements the Parquet Key Management Tools API in Rust to enable using a Key Management Server (KMS) to write and read encrypted Parquet files. It is used in conjunction with the Parquet modular encryption support in the Apache parquet crate.

This library can be used to write and read encrypted Parquet files that are compatible with other Parquet implementations, for example PyArrow and Apache Spark.

Concrete KMS client implementations are not included. To integrate with your KMS client you need to implement the KmsClient or AsyncKmsClient traits.

Please see the API documentation for examples of how to use this library and the full API reference.

Dependency versions

The parquet, arrow and DataFusion requirements each span several major versions. Cargo does not unify a dependency across major versions, so a project that uses an older version within the supported range can end up with a second, newer copy for this crate and incompatible types. In that case pin this crate's copies to your versions, for example:

cargo update -p [email protected] --precise 58.4.1

(and likewise for arrow, datafusion-common and datafusion-execution).

Feature Flags

The parquet_key_management crate provides the following features which may be enabled in your Cargo.toml:

  • parquet - integration with the parquet crate (enabled by default)
  • ring - use ring for cryptographic operations (enabled by default)
  • aws-lc-rs - use aws-lc-rs for cryptographic operations
  • async - enables the asynchronous KMS client API
  • async-std - enables support for the async-std runtime in the asynchronous KMS client API
  • smol - enables support for the smol runtime in the asynchronous KMS client API
  • tokio - enables support for the Tokio runtime in the asynchronous KMS client API
  • datafusion - enables the datafusion module, which implements integration with Apache DataFusion

One of the ring or aws-lc-rs features must be enabled. To use aws-lc-rs, disable default features and enable the aws-lc-rs feature, for example:

parquet-key-management = { version = "0.8", default-features = false, features = ["parquet", "aws-lc-rs"] }

If both features are enabled, aws-lc-rs is used.

About

Implements the Parquet Key Management Tools API in Rust to enable integration with a Key Management Server when using Parquet modular encryption.

Resources

Stars

4 stars

Watchers

12 watching

Forks

Releases

Packages

Used by

Contributors

Languages