Skip to content

Commit 66a20d9

Browse files
committed
docs(postgres): describe minted passwords
The README told users to read a password from <release>-credentials, which the chart no longer renders, the createRole text named the init job as the place a planted role is refused, and the restore notes described the init job applying chart-generated passwords. They now describe the mint, the pre-upgrade check, how a restored copy gets its passwords, and why a downgrade past the conversion is unsupported. Assisted-by: LLM Signed-off-by: Timofei Larkin <[email protected]>
1 parent 72b81a5 commit 66a20d9

6 files changed

Lines changed: 19 additions & 10 deletions

File tree

‎api/apps/v1alpha1/postgresql/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/operations/backup-classes.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -60,13 +60,13 @@ The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound
6060

6161
## Postgres and MariaDB: a restored copy converges its application passwords shortly after recovery
6262

63-
Managed Postgres no longer accepts a plaintext `users[].password`; every application-user password is chart-generated into `<release>-credentials` and read only from there. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext.
63+
Managed Postgres no longer accepts a plaintext `users[].password`; every application-user password is issued by a mint on the account's `Credential` (`core.cozystack.io`), and only its SCRAM verifier is kept, in an account Secret named `<release>.<user>.account` that no tenant role can read. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext.
6464

65-
**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `<release>-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retained it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` held that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `<release>-init-script` rendered `ALTER ROLE … WITH PASSWORD '<literal>'` into its manifest, retained for `MaxHistory` revisions. There is no in-band rotation to invalidate the exposed value (chart-based rotation was intentionally not added — a chart-rendered Secret cannot revoke a leaked credential; rotation is being reworked as a controller, cozystack/community#72). The backup strategy controller removes `users.*.password` from every CNPG `Backup` snapshot each time its leader starts, so the first sink is cleaned in-cluster after the upgrade, including a `Backup` written by a pre-upgrade replica during the rollout; the start-up log line `legacy snapshot password scrub finished` reports how many it changed. That does not reach copies made outside the cluster — apiserver audit logs, etcd backups, or object-level exports of the `Backup` resources — and the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`. So after upgrading, treat any password that was ever set through `users[].password` as exposed until it is rotated.
65+
**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `<release>-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retained it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` held that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `<release>-init-script` rendered `ALTER ROLE … WITH PASSWORD '<literal>'` into its manifest, retained for `MaxHistory` revisions. The conversion to minted passwords carries each existing password over as the account's verifier, with origin `Migrated` on its `Credential`, so that password keeps working until a mint on the account replaces it. The backup strategy controller removes `users.*.password` from every CNPG `Backup` snapshot each time its leader starts, so the first sink is cleaned in-cluster after the upgrade, including a `Backup` written by a pre-upgrade replica during the rollout; the start-up log line `legacy snapshot password scrub finished` reports how many it changed. That does not reach copies made outside the cluster — apiserver audit logs, etcd backups, or object-level exports of the `Backup` resources — and the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`. So after upgrading, treat any password that was ever set through `users[].password` as exposed until a mint replaces it.
6666

67-
The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `<release>-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. The restore driver clears `bootstrap.enabled` automatically once recovery is healthy, so the next HelmRelease reconcile runs the init-job, which applies `ALTER ROLE … WITH PASSWORD` for every `spec.users` entry and converges the recovered roles onto the target Secret. Convergence is therefore automatic but not instant: between the RestoreJob reaching `Succeeded` and that init-job completing, the passwords advertised in `<target>-credentials` do not yet match the recovered roles, so a client reading that Secret cannot log in as an application user for that brief window. The chart flags the Secret `postgres.cozystack.io/credentials-pending: "true"` and the NOTES print the same caution while bootstrap is still on, so a reader does not treat the advertised password as usable mid-restore; both clear on their own once the driver clears bootstrap. The data is intact and reachable as the CNPG superuser throughout. After a to-copy restore, wait for the application login to succeed rather than assuming the advertised password works the instant the RestoreJob succeeds. The driver clears `bootstrap.enabled` by writing it onto the Postgres app's own spec, so a GitOps source (Argo CD, Flux, Terraform) that pins `bootstrap.enabled: true` for that app will revert the clear on its next apply and the copy's credentials never converge — exclude the restored app from such a source until convergence completes, or clear `bootstrap.enabled` in the source itself. If the driver cannot clear the flag at all (the app was deleted or renamed mid-restore), the RestoreJob fails with reason `BootstrapDisableFailed` once its deadline elapses rather than sitting `Running` forever with no signal. An in-place restore is unaffected: the release keeps its existing `<release>-credentials`, whose passwords already match the roles the recovery brings back. One-time migration: an app restored to a copy under an earlier release — before this auto-clear existed — may still be sitting with `bootstrap.enabled` set and its init-job permanently skipped; the driver clears the flag only for a restore it drives to completion, never retroactively for one that already finished, so such an app keeps advertising unconverged credentials until you clear `bootstrap.enabled` on it once by hand.
67+
The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes until CloudNativePG applies the target's account Secrets, which it does once the recovered primary is up. Those Secrets were seeded when the target application was created, with the verifier of a random password nobody received, so no application user can log in to the copy until someone mints a password on the target's `Credential`; the source's passwords stop working there as soon as the verifiers are applied. Mint on the target, then wait for a login to succeed rather than assuming it works the instant the RestoreJob succeeds. The driver clears `bootstrap.enabled` by writing it onto the Postgres app's own spec, so a GitOps source (Argo CD, Flux, Terraform) that pins `bootstrap.enabled: true` for that app will revert the clear on its next apply and the copy never leaves recovery — exclude the restored app from such a source until the restore completes, or clear `bootstrap.enabled` in the source itself. If the driver cannot clear the flag at all (the app was deleted or renamed mid-restore), the RestoreJob fails with reason `BootstrapDisableFailed` once its deadline elapses rather than sitting `Running` forever with no signal. An in-place restore keeps the account Secrets and their records, and CloudNativePG applies the current verifiers over the restored catalog, so the passwords issued before the restore keep working. One-time migration: an app restored to a copy under an earlier release — before the driver cleared the flag itself — may still be sitting with `bootstrap.enabled` set; the driver clears the flag only for a restore it drives to completion, never retroactively for one that already finished, so clear `bootstrap.enabled` on such an app once by hand.
6868

69-
Databases and users after a Postgres restore, in place or into a copy: the driver sets the app's `spec.databases` and `spec.users` to the snapshot taken when the Backup was created, while recovery replays WAL past that point (to the end of the archive when no `recoveryTime` is set), so the recovered cluster can hold databases and users the snapshot does not list. On the first init-job run after the restore, a database the chart manages but the restored spec does not declare is released instead of dropped: its data stays, and it and its `<db>_admin` and `<db>_readonly` roles lose the `managed by helm` comment, so the chart no longer grants on it or drops it. Add it to `databases` to manage it again. Users are not released: a recovered user the restored spec does not declare, including a source user the target of a copy does not declare, is dropped on that run, so a client still logging in as that user loses the login. Every object it owned is reassigned to `cozystack:orphaned`, a role that cannot log in and that the chart grants nothing. Functions it owns are switched to `SECURITY INVOKER`, and on PostgreSQL 15 and later its views to `security_invoker`. A materialized view, a rule, and a view on PostgreSQL 13 or 14 have no such switch and keep reading with the rights of `cozystack:orphaned`. The user's grants and memberships do not carry over, so a view or function of that user can lose access it had through them. Declare the user in `users` to keep it; the chart then generates a new password for it into `<release>-credentials`.
69+
Databases and users after a Postgres restore, in place or into a copy: the driver sets the app's `spec.databases` and `spec.users` to the snapshot taken when the Backup was created, while recovery replays WAL past that point (to the end of the archive when no `recoveryTime` is set), so the recovered cluster can hold databases and users the snapshot does not list. On the first init-job run after the restore, a database the chart manages but the restored spec does not declare is released instead of dropped: its data stays, and it and its `<db>_admin` and `<db>_readonly` roles lose the `managed by helm` comment, so the chart no longer grants on it or drops it. Add it to `databases` to manage it again. Users are not released: a recovered user the restored spec does not declare, including a source user the target of a copy does not declare, is dropped on that run, so a client still logging in as that user loses the login. Every object it owned is reassigned to `cozystack:orphaned`, a role that cannot log in and that the chart grants nothing. Functions it owns are switched to `SECURITY INVOKER`, and on PostgreSQL 15 and later its views to `security_invoker`. A materialized view, a rule, and a view on PostgreSQL 13 or 14 have no such switch and keep reading with the rights of `cozystack:orphaned`. The user's grants and memberships do not carry over, so a view or function of that user can lose access it had through them. Declare the user in `users` to keep it; the API then seeds its account, and a mint issues its password.
7070

7171
**Objects reassigned by earlier releases stay as they were.** Before this change the chart reassigned a dropped user's or role's objects to the `postgres` superuser, and upgrading does not move them. After upgrading, run the query below in each database: it lists the relations and routines `postgres` owns outside the system catalog and extensions, leaving out a sequence that belongs to a column, which moves with its table. Of those, the chart itself owns only its `auto_grant_schema_privileges()` event-trigger function, so anything else came from a tenant role or was created by hand. Hand what a tenant role created to `cozystack:orphaned` with the matching `ALTER TABLE`, `ALTER VIEW`, `ALTER ROUTINE` or other `ALTER … OWNER TO "cozystack:orphaned"`; the next init-job run switches its functions and, on PostgreSQL 15 and later, its views as above. Handle first the rows with `security_definer` set, and views, materialized views and tables with rules, since those run with their owner's rights:
7272

‎packages/apps/postgres/README.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -273,7 +273,7 @@ For `sslmode=verify-full` to work, the CA bundle retrieved above must be saved t
273273
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------- |
274274
| `users` | Users configuration map. Each user is a PostgreSQL role the operator manages from an account Secret the platform keeps. Passwords are issued through the Cozystack credentials API: a mint on the account's `Credential` (`core.cozystack.io`) returns a new password once and stores only its SCRAM verifier, and a revoke retires the password without issuing one. A `password` key here is ignored and draws an admission warning. A password set inside PostgreSQL is reverted by the operator. | `map[string]object` | `{}` |
275275
| `users[name].replication` | Whether the user has replication privileges. | `bool` | `false` |
276-
| `users[name].createRole` | Whether the user can create and manage roles (CREATEROLE). Requires PostgreSQL 16 or newer. Setting it back to `false` withdraws the attribute only: the roles the user created, and its ADMIN and SET memberships in them, stay. Once any user has held it, the chart no longer adopts an existing role it did not create, so a user or database declared later under the name of such a role fails the init job until a platform administrator drops that role. | `bool` | `false` |
276+
| `users[name].createRole` | Whether the user can create and manage roles (CREATEROLE). Requires PostgreSQL 16 or newer. Setting it back to `false` withdraws the attribute only: the roles the user created, and its ADMIN and SET memberships in them, stay. Once any user has held it, the chart no longer adopts an existing role it did not create, so a user or database declared later under the name of such a role fails the upgrade, in a pre-upgrade check, until a platform administrator drops that role. | `bool` | `false` |
277277
| `users[name].bypassRls` | Whether the user can bypass row-level security policies (BYPASSRLS). A role can only create other roles carrying BYPASSRLS if it holds the attribute itself, so this is required alongside `createRole` to provision a privileged service role. | `bool` | `false` |
278278

279279

@@ -354,7 +354,16 @@ users:
354354
replication: true
355355
```
356356

357-
Passwords cannot be set here — they are auto-generated and stored in the `<release>-credentials` Secret. Read a user's password with `kubectl get secret <release>-credentials -o json | jq -r '.data["user1"]' | base64 -d` (a username may contain a `.`, which `jsonpath` would treat as a path step and return nothing).
357+
Passwords cannot be set here. Each user is an account of the Cozystack credentials API: a `Credential` named `<release>.<account>` in `core.cozystack.io/v1alpha1`, where `<account>` is the user name lowercased with every character outside `[a-z0-9-]` replaced by `-`. A mint returns a new password once and replaces the previous one; only its SCRAM verifier is stored, so the password cannot be read back:
358+
359+
```bash
360+
kubectl create --raw "/apis/core.cozystack.io/v1alpha1/namespaces/<namespace>/credentials/postgres-<name>.user1/mint" \
361+
-f <(echo '{"apiVersion":"core.cozystack.io/v1alpha1","kind":"CredentialRequest"}') | jq -r .status.password
362+
```
363+
364+
A user added to an existing application has no password until the first mint. Names PostgreSQL or CloudNativePG reserve (`postgres`, `streaming_replica`, `pg_*`, `cnpg_*`), the name of a database role (`<db>_admin`, `<db>_readonly`), and two users mapping to one account name are refused.
365+
366+
Downgrading the platform past the conversion to minted passwords is not supported: the previous chart generates a new password for every account. A manual `helm rollback` of a release to a revision from before the conversion is undone at its next reconcile, and meanwhile the old revision's `<release>-credentials` shows a password that no longer works; mint again once the release is back.
358367

359368
### databases
360369

0 commit comments

Comments
 (0)