Skip to content

Commit 79d805e

Browse files
committed
docs(openbao): key rotation needs an explicit pod replacement and a re-wrap check
The upstream chart updates the StatefulSet with OnDelete and the server copies its HCL to /tmp/storageconfig.hcl at startup, so changing seal.secretName never reaches a running pod. Document the replacement order for standalone and HA (standbys first, active last), the log line the new active node writes when it re-wraps the stored keys, and why the previous key has to stay configured until then. Signed-off-by: txmazing <[email protected]>
1 parent 3c7753b commit 79d805e

5 files changed

Lines changed: 25 additions & 6 deletions

File tree

‎api/apps/v1alpha1/openbao/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/openbao/README.md‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,26 @@ By default OpenBAO uses Shamir key shares: after every pod restart an operator h
2727
2828
`keyId` is a permanent identifier for that key material. OpenBAO refuses to start with a key whose identifier it has already seen with different material, so change `keyId` whenever you change the key.
2929

30-
To rotate the key, create a second Secret, move the current pair to `previousSecretName`/`previousKeyId` and put the new pair in `secretName`/`keyId`. This is the n-1 rotation the static seal documents: `current_key` is used for new seal operations while `previous_key` still decrypts what was written before. Keep both Secrets in place until OpenBAO has re-wrapped storage with the new key, then drop the `previous*` values and delete the old Secret.
30+
To rotate the key:
31+
32+
1. Create a second Secret with the new key material and pick a new `keyId`.
33+
2. Move the current pair to `previousSecretName`/`previousKeyId`, put the new pair in `secretName`/`keyId` and upgrade the release. This only changes the StatefulSet template. The upstream chart uses the `OnDelete` update strategy and the server copies its HCL to `/tmp/storageconfig.hcl` at startup, so running pods keep the old key and the old config until they are replaced.
34+
3. Replace the pods. Standalone: delete the single pod. HA: delete the standby pods one at a time, waiting for each to rejoin and unseal, then delete the active pod so that leadership moves to a pod that already runs the new config. Do not delete the active pod first: once a node with the new config has taken over, a standby still on the old template cannot unseal after a restart.
35+
36+
```bash
37+
kubectl -n <namespace> delete pod <release>-2 # standby
38+
kubectl -n <namespace> delete pod <release>-1 # standby
39+
kubectl -n <namespace> delete pod <release>-0 # active, last
40+
```
41+
42+
4. Confirm the rotation before touching the old key. The node that becomes active with the new config re-wraps the stored barrier and recovery keys with `current_key` and logs it:
43+
44+
```bash
45+
kubectl -n <namespace> logs <active pod> | grep -E 'upgrading (stored keys|recovery key)'
46+
```
47+
48+
Until those lines have appeared, storage is still wrapped with the previous key. The static seal decrypts by the key identifier stored with the data and refuses an identifier it does not know (`unknown key id for data`), which is why both keys have to stay configured until the re-wrap is done.
49+
5. Drop the `previous*` values, upgrade, and replace the pods once more in the same order. A pod that unseals with the new key alone proves the re-wrap. Delete the old Secret last.
3150

3251
> Switching an existing Shamir-sealed instance to `static` is a seal migration and needs `bao operator unseal -migrate`; see the OpenBAO [seal migration](https://openbao.org/docs/concepts/seal/#seal-migration) documentation. The chart does not run the migration for you. It does refuse to change `seal.type` on an instance that already runs with another seal, because a plain config change would leave a barrier nobody can decrypt. Set `seal.allowMigration: true` only for the upgrade in which you run the migration, then remove it. Migrating away from `static` needs the old stanza kept with `disabled = "true"` and its key still mounted, which this chart does not render; that direction is refused regardless of `allowMigration`.
3352

@@ -64,7 +83,7 @@ Keep a copy of the key outside the cluster: a Raft snapshot or a data PVC is unr
6483
| `seal.type` | Seal type. `shamir` keeps the current behaviour; `static` enables auto-unseal from `secretName`. | `string` | `shamir` |
6584
| `seal.secretName` | Existing Secret in the release namespace whose key `key` holds the base64 text of 32 random bytes (for example `openssl rand -base64 32`). Required when `type` is `static`. | `string` | `""` |
6685
| `seal.keyId` | Permanent identifier of the current key. Change it whenever the key material changes; OpenBAO refuses a key whose identifier it has already seen with different material. Required when `type` is `static`. | `string` | `""` |
67-
| `seal.previousSecretName` | Secret holding the previous key during an n-1 key rotation. | `string` | `""` |
86+
| `seal.previousSecretName` | Secret holding the previous key during an n-1 key rotation. Keep it until the pods have been replaced and the active node has logged `upgrading stored keys`; see the README. | `string` | `""` |
6887
| `seal.previousKeyId` | Identifier of the previous key. Required when `previousSecretName` is set. | `string` | `""` |
6988
| `seal.allowMigration` | Acknowledge a seal migration. The chart refuses to change `type` on an existing instance unless this is `true`; set it only while running `bao operator unseal -migrate`, then remove it. | `bool` | `false` |
7089

‎packages/apps/openbao/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,7 @@
158158
"default": ""
159159
},
160160
"previousSecretName": {
161-
"description": "Secret holding the previous key during an n-1 key rotation.",
161+
"description": "Secret holding the previous key during an n-1 key rotation. Keep it until the pods have been replaced and the active node has logged `upgrading stored keys`; see the README.",
162162
"type": "string",
163163
"default": ""
164164
},

‎packages/apps/openbao/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ ui: true
9494
## @field {SealType} type - Seal type. `shamir` keeps the current behaviour; `static` enables auto-unseal from `secretName`.
9595
## @field {string} secretName - Existing Secret in the release namespace whose key `key` holds the base64 text of 32 random bytes (for example `openssl rand -base64 32`). Required when `type` is `static`.
9696
## @field {string} keyId - Permanent identifier of the current key. Change it whenever the key material changes; OpenBAO refuses a key whose identifier it has already seen with different material. Required when `type` is `static`.
97-
## @field {string} [previousSecretName] - Secret holding the previous key during an n-1 key rotation.
97+
## @field {string} [previousSecretName] - Secret holding the previous key during an n-1 key rotation. Keep it until the pods have been replaced and the active node has logged `upgrading stored keys`; see the README.
9898
## @field {string} [previousKeyId] - Identifier of the previous key. Required when `previousSecretName` is set.
9999
## @field {bool} [allowMigration] - Acknowledge a seal migration. The chart refuses to change `type` on an existing instance unless this is `true`; set it only while running `bao operator unseal -migrate`, then remove it.
100100

‎packages/system/openbao-rd/cozyrds/openbao.yaml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)