Background

Secrets refer to any sensitive information required during the running process of APISIX, which may be part of the core configuration (such as the etcd's password) or some sensitive information in the plugin. Common types of Secrets in APISIX include:

• username, the password for some components (etcd, Redis, Kafka, etc.)
• the private key of the certificate
• API key
• Sensitive plugin configuration fields, typically used for authentication, hashing, signing, or encryption

Secrets Management refers to allowing users to store Secrets in APISIX through some secrets management services (vault, etc.), and read them according to the key when using them to ensure that Secrets do not exist in plaintext in the entire platform.

Currently, APISIX only supports the use of vault to manage secrets in the jwt-auth plugin. We plan to use more secrets management services in more plugins.

Scheme

Overall Design

In order to conveniently use the Secrets Management capability in various plugins of APISIX to protect secrets or other sensitive data, we will implement the following steps:

1. Add the environment variable module to support referencing environment variables in plugins and other APISIX resources in a specific way
2. Design a general KMS component, compatible with different secrets management services (vault, aws, etc.), and support referring to the KMS component in a specific way in the plugin
3. Add a new KMS resource in APISIX to configure and manage KMS components
4. Use a "non-intrusive" way to reference KMS components in plugins
5. We will first use secrets Management in the authentication class plugin

It works as follows:

Detailed Design

Environment variable

APISIX loads all environment variables at startup, so there is no need to add them through custom resources. Environment variables can be referenced in the following ways:

$ENV://$env_name/$sub_key

• env_name: environment variable name
• sub_key: get the value of a property when the value of the environment variable is a JSON string

If the value of the environment variable is of type string, such as:

export JACK_JWT_KEY=abc

It can be referenced as follows:

$ENV://JACK_JWT_KEY

If the value of the environment variable is a JSON string like:

export JACK={"jwt-key":"abc","openid-key": "def"}

It can be referenced as follows:

# Get the jwt-key of the environment variable JACK
$ENV://JACK/jwt-key
# Get the openid-key of the environment variable JACK
$ENV://JACK/openid-key

Example: use in jwt-auth plugin

First, create environment variables before the APISIX instance starts

export JACK_JWT_KEY=abc

Reference environment variables in the jwt-auth plugin

curl http://127.0.0.1:9180/apisix/admin/consumers \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "jack",
    "plugins": {
        "jwt-auth": {
            "key": "user-key",
            "secret": "$ENV://JACK_JWT_KEY"
        }
    }
}'

Through the steps shown above, you can save the secret configuration in the jwt-auth plugin in an environment variable instead of displaying it in plaintext when configuring the plugin.

KMS Component Design

Core Points

1. In order to facilitate the management of KMS components, combined with the characteristics of APISIX itself, KMS resources are introduced to configure KMS components, and configuration files are also supported to configure KMS components.
2. The KMS component is the encapsulation of the specific secrets management components (env, vault, etc.), and the specific secrets management components are not exposed to the outside world
3. The KMS component exposes the get operation, which must meet the following core functions:

• Parse a string similar to $KMS://vault/jwt/secret_id/jwt to get the secrets management service object and key
• Call the secrets management service to obtain the secret corresponding to the key
• The secret is loaded lazily and supports caching for a certain period of time. This can not only avoid stress on secrets management services but also be compatible with the "secret rotation" function of some secrets management services

4. The KMS component does not provide write operations to the secrets management service, as this may expose the user's secret on APISIX.
5. The secret support required by each secrets management service in the KMS component is configured through environment variables.
6. The communication method between the KMS component and the secrets management service is determined by the final selected Lua library, and we do not care about the specific implementation protocol (HTTP or gRPC).

Reference Method

In the plugin, KMS components can be referenced through special variables in the following format:

$KMS://$secretmanager/$id/$secret_id/$key

• secretmanager: secrets management service, could be the vault, aws, etc.
• id: KMS resource id, which needs to be consistent with the one specified when adding the KMS resource
• secret_id: the secret id in the secrets management service
• key: the key corresponding to the secret in the secrets management service

Example: use in jwt-auth plugin

First, create the corresponding secret in the vault. You can use the following command:

vault kv put apisix/jack jwt-key=value

Next, you can configure APISIX through the following steps:
Step 1: Add KMS resources through the Admin API, and configure the connection information such as the address of the vault:

curl http://127.0.0.1:9180/apisix/admin/kms/vault/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "https://127.0.0.1:8200"，
    "prefix": "apisix",
    "token": "root"
}'

Step 2: Refer to the KMS resource in the jwt-auth plugin and fill in the secret information

curl http://127.0.0.1:9180/apisix/admin/consumers \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "jack",
    "plugins": {
        "jwt-auth": {
            "key": "user-key",
            "secret": "$KMS://vault/1/jack/jwt-key"
        }
    }
}'

Through the above two steps, when the user request hits the jwt-auth plugin, the user-configured secrets management service will be called through the interface provided by the KMS component to obtain the real value of the secret in the vault.
In addition, the token required for APISIX to interact with the vault can also be stored in the environment variable. Before APISIX starts, the environment variable can be set by the following command:

export VAULT_TOKEN="root"

Reference the environment variable when adding the KMS resource:

curl http://127.0.0.1:9180/apisix/admin/kms/vault/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "https://127.0.0.1:8200"，
    "prefix": "apisix",
    "token": "$ENV://VAULT_TOKEN"
}'

Through the above steps, you can configure the secret in the jwt-auth plugin to the vault instead of displaying it in plaintext when configuring the plugin. Among them, the token used by APISIX to connect to the vault can also be saved in the environment variable.

KMS Resource Admin API Design

When adding KMS resources through the Admin API, the request is designed as follows:

Request URI

http://127.0.0.1:9180/apisix/admin/kms/${secretmanager}/${id}

• secretmanager: secrets management service, currently only vault, more secrets management services can be expanded in the future
• id: resource id

• APISIX can obtain all environment variables at startup, so users can directly refer to it without adding additional environment variable resources.

Request Method

• Note: According to the current design of APISIX, the POST method will automatically generate an id. When referring to a KMS component, an id needs to be used. The automatically generated id is not easy to manage and use, so the POST method is not provided.

Request Body

The secrets management service currently only supports vaults, so the request URI can currently only be:

http://127.0.0.1:9180/apisix/admin/kms/vault/${id}

The definitions of the fields in the corresponding request body are as follows: