Skip to content

Commit 8a18549

Browse files
committed
fix(api,postgres,mariadb): surface a leftover password and pending restore creds
A users[].password removed from the chart render is still accepted and stored -- the user object keeps additionalProperties open -- and through the lookup-preserve path it stays the LIVE credential until a passwordRotation bump, so an operator editing it to rotate gets a silent 200 and no effect. Emit an admission warning for a still-present users[].password on Postgres and MariaDB, wired into Create and Update with per-callsite tests, and say in the field docs that a leftover value stays live until a bump retires it. A Postgres restore into a copy renders freshly generated passwords into <release>-credentials while the role-applying init-job is skipped under bootstrap, so the advertised passwords do not authenticate until the app leaves bootstrap. Flag that Secret with postgres.cozystack.io/credentials-pending and print the same warning in NOTES so a reader does not treat the value as usable mid-restore. Document the equivalent MariaDB restore-into-a-copy behaviour -- application users realign only on the operator's requeue tick and root stays desynced because rootPasswordSecretKeyRef lands only at datadir bootstrap -- and narrow the Postgres in-place-restore caveat to hold only until a rotation. Drop the claim that the watch label re-applies root, and note that a helm rollback rewinds the rotation marker but not the database. Assisted-by: LLM Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 435ba2f commit 8a18549

17 files changed

Lines changed: 299 additions & 21 deletions

File tree

‎api/apps/v1alpha1/mariadb/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/postgresql/types.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/operations/backup-classes.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,11 @@ The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound
5555

5656
Managed Postgres no longer accepts a plaintext `users[].password`; every application-user password is chart-generated into `<release>-credentials` and read only from there. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext.
5757

58-
The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `<release>-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. Until the target leaves bootstrap — clear `bootstrap.enabled` on the Postgres app, or run a later user-driven `helm upgrade` — the passwords advertised in `<target>-credentials` do not match the recovered roles, so a client reading that Secret cannot log in as an application user. The data is intact and reachable as the CNPG superuser; only the chart-managed application logins lag, and they converge (the init-job runs `ALTER ROLE … WITH PASSWORD` onto the target Secret's values) as soon as bootstrap is cleared. After a to-copy restore, converge deliberately rather than assuming the advertised application password works immediately; an in-place restore is unaffected because the release keeps its existing `<release>-credentials`. Automating this convergence (clearing `bootstrap.enabled` once recovery is healthy) is tracked as follow-up.
58+
The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `<release>-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. Until the target leaves bootstrap — clear `bootstrap.enabled` on the Postgres app, or run a later user-driven `helm upgrade` — the passwords advertised in `<target>-credentials` do not match the recovered roles, so a client reading that Secret cannot log in as an application user. The data is intact and reachable as the CNPG superuser; only the chart-managed application logins lag, and they converge (the init-job runs `ALTER ROLE … WITH PASSWORD` onto the target Secret's values) as soon as bootstrap is cleared. After a to-copy restore, converge deliberately rather than assuming the advertised application password works immediately. The credentials Secret is flagged `postgres.cozystack.io/credentials-pending: "true"` and the chart NOTES print the same warning while bootstrap is on. An in-place restore is unaffected only as long as `passwordRotation` has not been bumped since the backup was taken: bump the counter, take no new backup, restore in place, and the Secret now carries the post-rotation passwords while the recovered roles carry the pre-rotation ones — the same mismatch, converging the same way once bootstrap is cleared. Automating this convergence (clearing `bootstrap.enabled` once recovery is healthy) is tracked as follow-up.
59+
60+
## MariaDB: a restored copy re-applies application passwords late, and root stays desynced
61+
62+
MariaDB has the mirror of the Postgres restore caveat, plus a sharper edge on `root`. The `strategy.backups.cozystack.io/MariaDB` driver takes a logical dump that includes `mysql.global_priv` (the grant table, password hashes and all) — mariadb-operator only excludes it when `ignoreGlobalPriv` is set, and that field defaults to `false` for a replication (non-Galera) instance, which is what this chart runs. So a restore into a copy replays the source's grants onto the target, while the target chart has generated fresh random passwords into `<target>-credentials`. Application users recover, but only on the operator's `--requeue-sql` tick (up to ~11h) rather than promptly, because the Secret content itself did not change so the `k8s.mariadb.com/watch` label does not fire. `root` never recovers on its own: `rootPasswordSecretKeyRef` is applied only at datadir bootstrap and the rotation counter deliberately skips `root`, so the password a tenant reads for `root` from `<target>-credentials` stays wrong from the restore onward. Rotate the application users (bump `passwordRotation`) after a to-copy restore to realign them promptly, and treat the advertised `root` password as unreliable on a restored copy. Making the target's own grants authoritative by setting `ignoreGlobalPriv: true` on the shipped `cozy-default-mariadb` strategy is tracked as follow-up.
5963

6064
## ClickHouse: opt-in to the system bucket
6165

‎packages/apps/mariadb/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,7 +205,7 @@ Uninstalling the instance does this for you; only the turn-it-off-and-keep-runni
205205

206206
| Name | Description | Type | Value |
207207
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | ----- |
208-
| `users` | Users configuration map. Passwords (including the `root` account) are always auto-generated and stored in the `<release>-credentials` Secret; they cannot be set from values. Read a user's password from that Secret, or rotate every managed password by bumping `passwordRotation`. | `map[string]object` | `{}` |
208+
| `users` | Users configuration map. Passwords (including the `root` account) are always auto-generated and stored in the `<release>-credentials` Secret; they cannot be set from values. Read a user's password from that Secret, or rotate every managed password by bumping `passwordRotation`. A `password` left over in values from before the field was removed is ignored by the chart but stays the live credential until a `passwordRotation` bump retires it. | `map[string]object` | `{}` |
209209
| `users[name].maxUserConnections` | Maximum number of connections. | `int` | `0` |
210210
| `passwordRotation` | Rotation counter for auto-generated application-user passwords. Change it to any new value (0 -> 1 -> 2 ...; any change, not only an increment, triggers a rotation) to regenerate every managed user password on the next reconcile; leaving it unchanged keeps the passwords already stored in the `<release>-credentials` Secret. Existing releases adopt their current passwords as the baseline on first upgrade: the counter value is recorded without rotating, so a bump requested during that same first upgrade does not rotate — bump it once more afterwards to rotate a legacy release. The `root` password is auto-generated but not rotated by this counter, since mariadb-operator only applies it at datadir bootstrap. | `int` | `0` |
211211
| `databases` | Databases configuration map. | `map[string]object` | `{}` |

‎packages/apps/mariadb/templates/secret.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,10 +48,12 @@ kind: Secret
4848
metadata:
4949
name: {{ .Release.Name }}-credentials
5050
labels:
51-
# mariadb-operator only re-applies a User/root password from
51+
# mariadb-operator only re-applies a User password from its
5252
# passwordSecretKeyRef when the referenced Secret carries this label;
5353
# without it a rotation lands only on the operator's --requeue-sql tick
54-
# (10h + up to 1h of jitter) instead of promptly.
54+
# (10h + up to 1h of jitter) instead of promptly. (root is not indexed on
55+
# the Secret at all — MariaDB.spec.rootPasswordSecretKeyRef lands only at
56+
# datadir bootstrap — so the label is load-bearing for application users.)
5557
k8s.mariadb.com/watch: ""
5658
annotations:
5759
{{ $rotationAnnotation }}: {{ $wantRotation | quote }}

‎packages/apps/mariadb/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@
159159
]
160160
},
161161
"users": {
162-
"description": "Users configuration map. Passwords (including the `root` account) are always auto-generated and stored in the `\u003crelease\u003e-credentials` Secret; they cannot be set from values. Read a user's password from that Secret, or rotate every managed password by bumping `passwordRotation`.",
162+
"description": "Users configuration map. Passwords (including the `root` account) are always auto-generated and stored in the `\u003crelease\u003e-credentials` Secret; they cannot be set from values. Read a user's password from that Secret, or rotate every managed password by bumping `passwordRotation`. A `password` left over in values from before the field was removed is ignored by the chart but stays the live credential until a `passwordRotation` bump retires it.",
163163
"type": "object",
164164
"default": {},
165165
"additionalProperties": {

‎packages/apps/mariadb/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ version: v11.8
112112
## @typedef {struct} User - User configuration.
113113
## @field {int} maxUserConnections - Maximum number of connections.
114114

115-
## @param {map[string]User} users - Users configuration map. Passwords (including the `root` account) are always auto-generated and stored in the `<release>-credentials` Secret; they cannot be set from values. Read a user's password from that Secret, or rotate every managed password by bumping `passwordRotation`.
115+
## @param {map[string]User} users - Users configuration map. Passwords (including the `root` account) are always auto-generated and stored in the `<release>-credentials` Secret; they cannot be set from values. Read a user's password from that Secret, or rotate every managed password by bumping `passwordRotation`. A `password` left over in values from before the field was removed is ignored by the chart but stays the live credential until a `passwordRotation` bump retires it.
116116
users: {}
117117
## Example:
118118
## users:

0 commit comments

Comments
 (0)