Skip to content

Commit 6f49e5f

Browse files
authored
feat(openbao): optional static-key auto-unseal and apiserver egress label (#4766)
## What this PR does This continues #4168 by @txmazing, which has been idle since 2026-09-10. The first commit is the work from #4168 squashed into one, with @txmazing as author. The other two commits constrain the new values and complete the documentation. Five changes to the managed OpenBAO app (`packages/apps/openbao`). 1. **Optional static-key auto-unseal** (#2787, #2483). `seal.type` is `shamir`, the unchanged default, or `static`. With `static` the chart mounts the Secret named in `seal.secretName` through the upstream chart's `server.extraVolumes` and adds OpenBAO's `seal "static"` stanza to the standalone and the Raft config, so the pods unseal themselves after a restart. `seal.previousSecretName` and `seal.previousKeyId` carry the n-1 key during a rotation. The chart never generates the key. A cluster administrator creates the Secret in the tenant namespace, because the `cozy:tenant:*` roles grant nothing on Secrets. A way for tenants to supply secrets by reference is proposed separately in cozystack/community#82. 2. **No silent seal change.** The template reads the seal in use from the `<release>-config` ConfigMap and refuses a different `seal.type` unless `seal.allowMigration: true` acknowledges a migration. Going from `static` back to `shamir` is refused, because the chart cannot render the disabled stanza that direction needs. 3. **Apiserver egress label** (#2793). The server pods carry `policy.cozystack.io/allow-to-apiserver: "true"`, so `service_registration "kubernetes"` reaches the API server and HA pods no longer hang at start. 4. **Constrained values.** The upstream chart passes the server config through `tpl`, and its start script replaces upper case placeholders such as `HOSTNAME` in it. A key identifier with template delimiters would run as a template inside helm-controller, which runs as cluster-admin, so `seal.keyId` and `seal.previousKeyId` take lowercase alphanumerics, hyphens, dots and underscores. The Secret names become pod volume names `userconfig-<name>`, so they take DNS labels of at most 52 characters. A previous Secret or key identifier equal to the current one fails at render. The apps API only defaults against the schema, and Helm enforces these patterns at install and upgrade. 5. **README.** The trade-off of `static`, the key Secret, the rotation with the pod replacement that the `OnDelete` strategy requires, the migration from Shamir step by step after the OpenBAO guide, and the one-time pod replacement after an upgrade so that existing pods get the label. ### Validation - `helm unittest .` in `packages/apps/openbao` passes 28 of 28 with helm-unittest 1.1.2, the version CI pins, and with 1.2.1. - Dropping the key identifier patterns, moving the Secret name limit to 51 or 53, or removing either equality guard turns the suite red each time. - Rendering this chart and then the vendored `openbao` chart with a template in `seal.keyId` now stops at the schema. Without the pattern the same two-stage render executed it, `lookup` included. - With default values, standalone and with `replicas: 3`, `helm template` renders the same as `main` apart from the six-line `extraLabels` block. - `make generate` in the package with cozyvalues-gen v1.7.0 and at the repository root with Go 1.27.1 leaves the tree unchanged. `make unit-tests` passes apart from seven BATS tests that fail the same way on a clean `main` under macOS, most of them because macOS ships no `timeout`. In `make test-controllers` only `internal/backupcontroller` fails, because its tests do not build on current `main` either, which #4768 fixes. `hack/check-commit-trailers.sh` passes and `cmd/api-gate` reports no sizeable API change. - Nothing here was run on a cluster. #4168 reports a Cozystack v1.6.3 cluster where a `replicas: 1` instance with `seal.type: static` came back `Sealed false` after `kubectl delete pod`. TLS between pods and an init or bootstrap job are not covered. Both are on the #2787 list and independent of this change. ### Screenshots No dashboard code change. The application form gains the `seal` group from the regenerated `openAPISchema` and `keysOrder`. ### Downstream repositories Walked the trigger map against the diff. The new `seal` field in `packages/apps/openbao/values.schema.json` reaches the hand-written `cozystack_openbao` resource of the Terraform provider. That is tracked in cozystack/terraform-provider-cozystack#44, which also needs `allowMigration` and the value constraints. The website reference page is generated from this `README.md` for the apps listed in its `Makefile`, and `openbao` is in `APPS`, so the page follows the next stable tag without a follow-up. The website pages that mention the apiserver label describe it in general terms and carry no OpenBAO workaround. There is no `hack/` change, no platform values change, no variant and no change to ApplicationDefinition semantics. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#44 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(openbao): the OpenBAO app can auto-unseal with OpenBAO's static-key seal (`seal.type: static`) from a Secret that a cluster administrator creates in the tenant namespace, with n-1 rotation through `seal.previousSecretName` and `seal.previousKeyId`. Shamir stays the default, and changing the seal of an existing instance needs `seal.allowMigration`. Server pods now carry `policy.cozystack.io/allow-to-apiserver: "true"`, so HA instances no longer hang at start behind the apiserver egress policy. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * OpenBAO supports static sealing with keys stored in an existing Secret; Shamir sealing remains the default. * Configure current and previous keys to support key rotation in standalone and high-availability deployments. Keep the previous key until stored keys have been re-wrapped. * Seal-type changes are validated. Shamir-to-static migration requires explicit acknowledgement; static-to-Shamir migration is not supported. * Static-seal settings validate required key details and Secret names. Server pods receive a label needed by the configured API-server egress policy. * **Documentation** * Added guidance for configuring static sealing, managing Secrets, rotating keys, and migrating seal types. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 3dc0d04 + 3ea7a95 commit 6f49e5f

8 files changed

Lines changed: 756 additions & 2 deletions

File tree

‎api/apps/v1alpha1/openbao/types.go‎

Lines changed: 33 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/openbao/zz_generated.deepcopy.go‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/openbao/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,70 @@ It provides identity-based secrets and encryption management for cloud infrastru
55

66
> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it.
77
8+
## Auto-unseal
9+
10+
By default OpenBAO uses Shamir key shares, so after every pod restart an operator has to unseal each replica by hand. Setting `seal.type: static` switches the instance to OpenBAO's [`seal "static"`](https://openbao.org/docs/configuration/seal/static/), which unseals the barrier from a 32-byte key read from a Secret in the tenant namespace. The chart never generates, stores or rotates the key. It only mounts the Secret you name.
11+
12+
The choice is a trade-off. With `static` the key sits in a Secret in the same cluster as the data it unwraps, so the barrier is only as strong as access to Secrets in that namespace, to etcd and to backups. Shamir keeps the key material out of the cluster at the price of a manual unseal after every restart. Pick `static` when availability after a restart matters more than that boundary.
13+
14+
1. A cluster administrator creates the key Secret in the tenant namespace. Tenant roles grant no access to Secrets, so tenant users can neither create the key nor read it back. The pods mount the Secret and do not start until it exists, so have it created before you set `seal.type: static`. The value is the base64 text of 32 random bytes, stored under the key `key`.
15+
16+
```bash
17+
kubectl -n tenant-<name> create secret generic openbao-unseal \
18+
--from-literal=key="$(head -c 32 /dev/urandom | base64 | tr -d '\n')"
19+
```
20+
21+
2. Point the instance at it:
22+
23+
```yaml
24+
seal:
25+
type: static
26+
secretName: openbao-unseal
27+
keyId: key-1
28+
```
29+
30+
`keyId` is a permanent identifier for that key material. OpenBAO stores it with the data the key wraps and picks the key by it, so change `keyId` whenever you change the key.
31+
32+
To rotate the key:
33+
34+
1. Have a cluster administrator create a second Secret with the new key material, and pick a new `keyId`.
35+
2. Move the current pair to `previousSecretName`/`previousKeyId`, put the new pair in `secretName`/`keyId` and upgrade the release. The upgrade changes the config and the StatefulSet template, not the running pods. The upstream chart uses the `OnDelete` update strategy and the server copies its HCL to `/tmp/storageconfig.hcl` at startup, so running pods keep the old key and the old config until they are replaced.
36+
3. Replace the pods. In standalone mode delete the single pod. In HA mode delete the standby pods one at a time, waiting for each to rejoin and unseal, then delete the active pod so that leadership moves to a pod that already runs the new config. Do not delete the active pod first. Once a node with the new config has taken over, a standby still on the old template cannot unseal after a restart.
37+
38+
```bash
39+
kubectl -n <namespace> get pods -l openbao-active=true # the active pod
40+
kubectl -n <namespace> delete pod <standby pod> # each standby, one at a time
41+
kubectl -n <namespace> delete pod <active pod> # last
42+
```
43+
44+
4. Confirm the rotation before touching the old key. The node that becomes active with the new config re-wraps the stored barrier and recovery keys with `current_key` and logs it. Tenant roles do not grant `pods/log`, so a cluster administrator reads that log.
45+
46+
```bash
47+
kubectl -n <namespace> logs <active pod> | grep -E 'upgrading (stored keys|recovery key)'
48+
```
49+
50+
Until those lines have appeared, storage is still wrapped with the previous key. The static seal decrypts by the key identifier stored with the data and refuses an identifier it does not know (`unknown key id for data`), which is why both keys have to stay configured until the re-wrap is done.
51+
5. Drop the `previous*` values, upgrade, and replace the pods once more in the same order. A pod that unseals with the new key alone proves the re-wrap. The cluster administrator deletes the old Secret last.
52+
53+
### Upgrading an existing instance
54+
55+
The chart labels the server pods with `policy.cozystack.io/allow-to-apiserver: "true"`, which the tenant network policy needs before `service_registration "kubernetes"` can reach the API server (#2793). The StatefulSet uses the `OnDelete` strategy, so pods created before the label was added keep running without it after an upgrade. Delete them once, in HA mode one at a time, so that they come back labelled. With the Shamir seal every replaced pod has to be unsealed again. An HA instance that hangs at start because of #2793 stays hung until its pods are replaced.
56+
57+
### Migrating an existing instance from Shamir
58+
59+
Switching a running Shamir-sealed instance to `static` is a seal migration, see the OpenBAO [seal migration](https://openbao.org/docs/concepts/seal/#seal-migration) documentation. The chart does not run it for you. It refuses a changed `seal.type` on an instance that already runs with another seal, because a plain config change leaves an instance that cannot unseal. OpenBAO documents the migration as a procedure with downtime, so plan a window.
60+
61+
1. Take a restorable backup first, as OpenBAO recommends before any seal migration. In HA mode `bao operator raft snapshot save <file>` writes it. In standalone mode the file storage backend is not transactional, so a copy of the data volume taken while the server runs can be inconsistent. Copy the volume while OpenBAO is stopped, or take a volume snapshot with a storage class that snapshots atomically.
62+
2. A cluster administrator creates the key Secret as described above.
63+
3. Set `seal.type: static` with `secretName` and `keyId`, add `seal.allowMigration: true` and upgrade the release. The StatefulSet uses the `OnDelete` strategy, so the running pods keep the Shamir configuration until they are replaced. This upgrade is the point of no return for the chart. The server config now names the static seal while every pod still runs Shamir, and from here on the chart refuses `seal.type: shamir`. If you abort before any pod is replaced, a cluster administrator has to remove the `seal "static"` stanza from the `openbao-<name>-config` ConfigMap before the release accepts `shamir` again.
64+
4. Standalone. Delete the pod. When it is back, run `bao operator unseal -migrate` with each Shamir unseal key. The server performs the migration and the Shamir keys become the recovery keys.
65+
5. HA. Delete one standby pod, wait for it to come back and run `bao operator unseal -migrate` on it with each Shamir unseal key. Repeat for the other standby pods one at a time. Then run `bao operator step-down` on the active node and wait until another node has become active. The new active node performs the migration and logs its completion. Give that a little time to replicate to the other Raft nodes, then delete the old active pod, so that it restarts with the static configuration and unseals on its own.
66+
6. Remove `seal.allowMigration` and upgrade the release once more.
67+
68+
Migrating away from `static` needs the old stanza kept with `disabled = "true"` and its key still mounted, which this chart does not render. That direction is refused regardless of `allowMigration`.
69+
70+
Keep a copy of the key outside the cluster. A Raft snapshot or a data PVC is unreadable without it.
71+
872
## Parameters
973

1074
### Common parameters
@@ -27,3 +91,16 @@ It provides identity-based secrets and encryption management for cloud infrastru
2791
| ---- | -------------------------- | ------ | ------ |
2892
| `ui` | Enable the OpenBAO web UI. | `bool` | `true` |
2993

94+
95+
### Seal
96+
97+
| Name | Description | Type | Value |
98+
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------- |
99+
| `seal` | Seal configuration. The default is Shamir key shares. Set `type: static` for auto-unseal. | `object` | `{}` |
100+
| `seal.type` | Seal type. `shamir` keeps the current behaviour, `static` enables auto-unseal from `secretName`. | `string` | `shamir` |
101+
| `seal.secretName` | Existing Secret in the tenant namespace whose key `key` holds the base64 text of 32 random bytes (for example `openssl rand -base64 32`). A cluster administrator creates it, because tenant roles cannot create Secrets. Lowercase alphanumerics and hyphens, at most 52 characters, because the pod volume is named `userconfig-<name>` and a volume name is a DNS label. Required when `type` is `static`. | `string` | `""` |
102+
| `seal.keyId` | Permanent identifier of the current key. OpenBAO stores it with the data the key wraps and picks the key by it, so change it whenever the key material changes. Lowercase alphanumerics, `-`, `.` and `_`, starting and ending with an alphanumeric, because the upstream chart evaluates the server config as a template and rewrites upper case placeholders such as `HOSTNAME` in it. Required when `type` is `static`. | `string` | `""` |
103+
| `seal.previousSecretName` | Secret holding the previous key during an n-1 key rotation, with the same name constraints as `secretName`. Keep it until the pods have been replaced and the active node has logged `upgrading stored keys`, see the README. | `string` | `""` |
104+
| `seal.previousKeyId` | Identifier of the previous key, with the same character constraints as `keyId`. Required when `previousSecretName` is set. | `string` | `""` |
105+
| `seal.allowMigration` | Acknowledge a seal migration. The chart refuses to change `type` on an existing instance unless this is `true`. Set it only while running `bao operator unseal -migrate`, then remove it. | `bool` | `false` |
106+

0 commit comments

Comments
 (0)