Skip to content

Commit 19764c4

Browse files
committed
fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots
A CNPG Backup taken while the Postgres app still accepted users[].password copied that password into status.underlyingResources, where anyone with `get backups` in the namespace can read it. Dropping the field stopped new snapshots from carrying it but left every existing Backup exposed. The backup strategy controller now removes users.*.password from CNPG snapshots when its leader starts. It runs on every start rather than once, so a Backup written by a pre-upgrade replica during the rollout, or brought back from an object-level export, is still cleaned; on a clean cluster it is a single cached List. The patch is a diff-computed merge patch that only nulls the password keys, and a Backup that cannot be decoded or patched is logged and skipped instead of stopping the manager. Fixes #4179 Assisted-by: LLM Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 5e32ef3 commit 19764c4

4 files changed

Lines changed: 294 additions & 1 deletion

File tree

‎cmd/backupstrategy-controller/main.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,11 @@ func main() {
212212
}
213213
}
214214

215+
if err := mgr.Add(&backupcontroller.LegacyPasswordScrubber{Client: mgr.GetClient()}); err != nil {
216+
setupLog.Error(err, "unable to add LegacyPasswordScrubber runnable")
217+
os.Exit(1)
218+
}
219+
215220
// The default Strategy CRs and the Velero BSL are Helm-templated behind
216221
// a lookup of the BucketClaim the same chart creates, so a fresh install
217222
// renders them empty — permanently, because helm-controller does not

‎docs/operations/backup-classes.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound
5959

6060
Managed Postgres no longer accepts a plaintext `users[].password`; every application-user password is chart-generated into `<release>-credentials` and read only from there. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext.
6161

62-
**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `<release>-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retain it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` holds that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `<release>-init-script` rendered `ALTER ROLE … WITH PASSWORD '<literal>'` into its manifest, retained for `MaxHistory` revisions. There is no in-band rotation to invalidate the exposed value (chart-based rotation was intentionally not added — a chart-rendered Secret cannot revoke a leaked credential; rotation is being reworked as a controller, cozystack/community#72). So after upgrading, treat any pre-upgrade Postgres backup as credential-bearing: delete or re-take the pre-upgrade `Backup` objects, and be aware that the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`.
62+
**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `<release>-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retained it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` held that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `<release>-init-script` rendered `ALTER ROLE … WITH PASSWORD '<literal>'` into its manifest, retained for `MaxHistory` revisions. There is no in-band rotation to invalidate the exposed value (chart-based rotation was intentionally not added — a chart-rendered Secret cannot revoke a leaked credential; rotation is being reworked as a controller, cozystack/community#72). The backup strategy controller removes `users.*.password` from every CNPG `Backup` snapshot each time its leader starts, so the first sink is cleaned in-cluster after the upgrade, including a `Backup` written by a pre-upgrade replica during the rollout; the start-up log line `legacy snapshot password scrub finished` reports how many it changed. That does not reach copies made outside the cluster — apiserver audit logs, etcd backups, or object-level exports of the `Backup` resources — and the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`. So after upgrading, treat any password that was ever set through `users[].password` as exposed until it is rotated.
6363

6464
The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `<release>-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. The restore driver clears `bootstrap.enabled` automatically once recovery is healthy, so the next HelmRelease reconcile runs the init-job, which applies `ALTER ROLE … WITH PASSWORD` for every `spec.users` entry and converges the recovered roles onto the target Secret. Convergence is therefore automatic but not instant: between the RestoreJob reaching `Succeeded` and that init-job completing, the passwords advertised in `<target>-credentials` do not yet match the recovered roles, so a client reading that Secret cannot log in as an application user for that brief window. The chart flags the Secret `postgres.cozystack.io/credentials-pending: "true"` and the NOTES print the same caution while bootstrap is still on, so a reader does not treat the advertised password as usable mid-restore; both clear on their own once the driver clears bootstrap. The data is intact and reachable as the CNPG superuser throughout. After a to-copy restore, wait for the application login to succeed rather than assuming the advertised password works the instant the RestoreJob succeeds. The driver clears `bootstrap.enabled` by writing it onto the Postgres app's own spec, so a GitOps source (Argo CD, Flux, Terraform) that pins `bootstrap.enabled: true` for that app will revert the clear on its next apply and the copy's credentials never converge — exclude the restored app from such a source until convergence completes, or clear `bootstrap.enabled` in the source itself. If the driver cannot clear the flag at all (the app was deleted or renamed mid-restore), the RestoreJob fails with reason `BootstrapDisableFailed` once its deadline elapses rather than sitting `Running` forever with no signal. An in-place restore is unaffected: the release keeps its existing `<release>-credentials`, whose passwords already match the roles the recovery brings back. One-time migration: an app restored to a copy under an earlier release — before this auto-clear existed — may still be sitting with `bootstrap.enabled` set and its init-job permanently skipped; the driver clears the flag only for a restore it drives to completion, never retroactively for one that already finished, so such an app keeps advertising unconverged credentials until you clear `bootstrap.enabled` on it once by hand.
6565

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
package backupcontroller
2+
3+
import (
4+
"bytes"
5+
"context"
6+
"encoding/json"
7+
8+
"k8s.io/apimachinery/pkg/runtime"
9+
"sigs.k8s.io/controller-runtime/pkg/client"
10+
"sigs.k8s.io/controller-runtime/pkg/log"
11+
"sigs.k8s.io/controller-runtime/pkg/manager"
12+
13+
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
14+
)
15+
16+
// LegacyPasswordScrubber is a one-shot Runnable that removes plaintext
17+
// users[].password values from CNPG Backup snapshots in
18+
// status.underlyingResources.
19+
//
20+
// Snapshots taken while the Postgres app still accepted users[].password
21+
// copied it verbatim, where anyone with `get backups` in the namespace can
22+
// read it. New snapshots no longer carry the field, and restore ignores it,
23+
// so dropping it changes nothing but the exposure.
24+
//
25+
// It runs on every leader start rather than once per cluster, so a Backup
26+
// written by a pre-upgrade replica during the rollout, or brought back from
27+
// an object-level backup, is still cleaned. On a clean cluster it is a
28+
// single cached List.
29+
//
30+
// TODO(#4179): remove once upgrading from a release that accepted
31+
// users[].password is no longer supported.
32+
type LegacyPasswordScrubber struct {
33+
Client client.Client
34+
}
35+
36+
var (
37+
_ manager.Runnable = (*LegacyPasswordScrubber)(nil)
38+
_ manager.LeaderElectionRunnable = (*LegacyPasswordScrubber)(nil)
39+
)
40+
41+
func (s *LegacyPasswordScrubber) NeedLeaderElection() bool { return true }
42+
43+
// Start never returns an error: a returned error stops the manager, and a
44+
// Backup that cannot be cleaned must not take backups down with it.
45+
func (s *LegacyPasswordScrubber) Start(ctx context.Context) error {
46+
logger := log.FromContext(ctx).WithName("legacy-password-scrubber")
47+
48+
var backups backupsv1alpha1.BackupList
49+
if err := s.Client.List(ctx, &backups); err != nil {
50+
logger.Error(err, "listing Backups; legacy snapshot passwords were not scrubbed")
51+
return nil
52+
}
53+
54+
scrubbed := 0
55+
for i := range backups.Items {
56+
b := &backups.Items[i]
57+
if b.Status.UnderlyingResources == nil {
58+
continue
59+
}
60+
raw, changed, err := scrubCNPGSnapshotPasswords(b.Status.UnderlyingResources.Raw)
61+
if err != nil {
62+
logger.Error(err, "decoding status.underlyingResources", "namespace", b.Namespace, "backup", b.Name)
63+
continue
64+
}
65+
if !changed {
66+
continue
67+
}
68+
// A merge patch computed from the diff carries only
69+
// `password: null` per user, so it cannot overwrite status fields
70+
// another writer changed since the List.
71+
orig := b.DeepCopy()
72+
b.Status.UnderlyingResources = &runtime.RawExtension{Raw: raw}
73+
if err := s.Client.Patch(ctx, b, client.MergeFrom(orig)); client.IgnoreNotFound(err) != nil {
74+
logger.Error(err, "patching Backup", "namespace", b.Namespace, "backup", b.Name)
75+
continue
76+
}
77+
scrubbed++
78+
logger.Info("removed legacy plaintext passwords from Backup snapshot", "namespace", b.Namespace, "backup", b.Name)
79+
}
80+
logger.Info("legacy snapshot password scrub finished", "scanned", len(backups.Items), "scrubbed", scrubbed)
81+
return nil
82+
}
83+
84+
// scrubCNPGSnapshotPasswords returns raw with users.*.password removed when
85+
// raw is a CNPG snapshot carrying one, and changed=false otherwise. It
86+
// decodes into generic maps so fields the current snapshot type does not
87+
// know survive the round trip.
88+
func scrubCNPGSnapshotPasswords(raw []byte) ([]byte, bool, error) {
89+
if len(raw) == 0 {
90+
return raw, false, nil
91+
}
92+
dec := json.NewDecoder(bytes.NewReader(raw))
93+
dec.UseNumber()
94+
var snap map[string]any
95+
if err := dec.Decode(&snap); err != nil {
96+
return nil, false, err
97+
}
98+
if snap["kind"] != cnpgBackupSnapshotKind || snap["apiVersion"] != cnpgBackupSnapshotAPIVersion {
99+
return raw, false, nil
100+
}
101+
users, _ := snap["users"].(map[string]any)
102+
changed := false
103+
for _, u := range users {
104+
user, ok := u.(map[string]any)
105+
if !ok {
106+
continue
107+
}
108+
if _, ok := user["password"]; ok {
109+
delete(user, "password")
110+
changed = true
111+
}
112+
}
113+
if !changed {
114+
return raw, false, nil
115+
}
116+
out, err := json.Marshal(snap)
117+
if err != nil {
118+
return nil, false, err
119+
}
120+
return out, true, nil
121+
}
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
package backupcontroller
2+
3+
import (
4+
"bytes"
5+
"context"
6+
"encoding/json"
7+
"errors"
8+
"testing"
9+
10+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
11+
"k8s.io/apimachinery/pkg/runtime"
12+
"k8s.io/apimachinery/pkg/types"
13+
"k8s.io/client-go/kubernetes/scheme"
14+
"sigs.k8s.io/controller-runtime/pkg/client"
15+
clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake"
16+
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
17+
18+
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
19+
)
20+
21+
func backupWithSnapshot(name, raw string) *backupsv1alpha1.Backup {
22+
b := &backupsv1alpha1.Backup{
23+
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "tenant-a"},
24+
}
25+
if raw != "" {
26+
b.Status.UnderlyingResources = &runtime.RawExtension{Raw: []byte(raw)}
27+
}
28+
return b
29+
}
30+
31+
func legacyCNPGSnapshot() string {
32+
return `{"kind":"CNPGBackupSnapshot","apiVersion":"` + cnpgBackupSnapshotAPIVersion + `",` +
33+
`"databases":{"app":{"roles":{"admin":["alice"]}}},` +
34+
`"users":{"alice":{"password":"hunter2","replication":true},"bob":{"password":"s3cret"},"carol":{}},` +
35+
`"parameters":{"retention":"30d"},"futureField":{"n":12345678901234567890}}`
36+
}
37+
38+
func newScrubberClient(objs ...client.Object) client.WithWatch {
39+
s := runtime.NewScheme()
40+
_ = scheme.AddToScheme(s)
41+
_ = backupsv1alpha1.AddToScheme(s)
42+
return clientfake.NewClientBuilder().WithScheme(s).WithObjects(objs...).Build()
43+
}
44+
45+
func getSnapshot(t *testing.T, c client.Client, name string) map[string]any {
46+
t.Helper()
47+
var b backupsv1alpha1.Backup
48+
if err := c.Get(context.Background(), types.NamespacedName{Namespace: "tenant-a", Name: name}, &b); err != nil {
49+
t.Fatalf("get Backup %s: %v", name, err)
50+
}
51+
if b.Status.UnderlyingResources == nil {
52+
return nil
53+
}
54+
var snap map[string]any
55+
dec := json.NewDecoder(bytes.NewReader(b.Status.UnderlyingResources.Raw))
56+
dec.UseNumber()
57+
if err := dec.Decode(&snap); err != nil {
58+
t.Fatalf("decode snapshot of %s: %v", name, err)
59+
}
60+
return snap
61+
}
62+
63+
func TestLegacyPasswordScrubber_RemovesPasswordsAndKeepsTheRest(t *testing.T) {
64+
c := newScrubberClient(backupWithSnapshot("legacy", legacyCNPGSnapshot()))
65+
66+
if err := (&LegacyPasswordScrubber{Client: c}).Start(context.Background()); err != nil {
67+
t.Fatalf("Start: %v", err)
68+
}
69+
70+
snap := getSnapshot(t, c, "legacy")
71+
users := snap["users"].(map[string]any)
72+
for name, u := range users {
73+
if _, ok := u.(map[string]any)["password"]; ok {
74+
t.Errorf("user %s still carries a password", name)
75+
}
76+
}
77+
if len(users) != 3 {
78+
t.Errorf("users = %v, want alice, bob and carol kept", users)
79+
}
80+
if got := users["alice"].(map[string]any)["replication"]; got != true {
81+
t.Errorf("alice.replication = %v, want true", got)
82+
}
83+
if got := snap["parameters"].(map[string]any)["retention"]; got != "30d" {
84+
t.Errorf("parameters.retention = %v, want 30d", got)
85+
}
86+
if _, ok := snap["databases"].(map[string]any)["app"]; !ok {
87+
t.Errorf("databases.app was dropped: %v", snap["databases"])
88+
}
89+
if got := snap["futureField"].(map[string]any)["n"]; got != json.Number("12345678901234567890") {
90+
t.Errorf("unknown field lost precision or was dropped: %v", got)
91+
}
92+
93+
// The restore path must still accept the scrubbed snapshot.
94+
var b backupsv1alpha1.Backup
95+
_ = c.Get(context.Background(), types.NamespacedName{Namespace: "tenant-a", Name: "legacy"}, &b)
96+
if _, _, _, err := unmarshalCNPGBackupSnapshot(&b); err != nil {
97+
t.Errorf("scrubbed snapshot no longer decodes for restore: %v", err)
98+
}
99+
}
100+
101+
func TestLegacyPasswordScrubber_LeavesOtherBackupsUntouched(t *testing.T) {
102+
cases := map[string]string{
103+
"clean-cnpg": `{"kind":"CNPGBackupSnapshot","apiVersion":"` + cnpgBackupSnapshotAPIVersion + `","users":{"alice":{}}}`,
104+
"other-kind": `{"kind":"SomeOtherSnapshot","apiVersion":"` + cnpgBackupSnapshotAPIVersion + `","users":{"alice":{"password":"keep"}}}`,
105+
"other-version": `{"kind":"CNPGBackupSnapshot","apiVersion":"example.com/v9","users":{"alice":{"password":"keep"}}}`,
106+
"no-snapshot": ``,
107+
}
108+
var objs []client.Object
109+
for name, raw := range cases {
110+
objs = append(objs, backupWithSnapshot(name, raw))
111+
}
112+
c := newScrubberClient(objs...)
113+
before := map[string]string{}
114+
for name := range cases {
115+
var b backupsv1alpha1.Backup
116+
_ = c.Get(context.Background(), types.NamespacedName{Namespace: "tenant-a", Name: name}, &b)
117+
before[name] = b.ResourceVersion
118+
}
119+
120+
if err := (&LegacyPasswordScrubber{Client: c}).Start(context.Background()); err != nil {
121+
t.Fatalf("Start: %v", err)
122+
}
123+
124+
for name := range cases {
125+
var b backupsv1alpha1.Backup
126+
_ = c.Get(context.Background(), types.NamespacedName{Namespace: "tenant-a", Name: name}, &b)
127+
if b.ResourceVersion != before[name] {
128+
t.Errorf("%s was patched but carries nothing to scrub", name)
129+
}
130+
}
131+
}
132+
133+
func TestLegacyPasswordScrubber_OneBadBackupDoesNotStopTheRest(t *testing.T) {
134+
c := newScrubberClient(
135+
backupWithSnapshot("not-an-object", `["not","an","object"]`),
136+
backupWithSnapshot("a-patch-fails", legacyCNPGSnapshot()),
137+
backupWithSnapshot("z-legacy", legacyCNPGSnapshot()),
138+
)
139+
failing := interceptor.NewClient(c, interceptor.Funcs{
140+
Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error {
141+
if obj.GetName() == "a-patch-fails" {
142+
return errors.New("apiserver unavailable")
143+
}
144+
return cl.Patch(ctx, obj, patch, opts...)
145+
},
146+
})
147+
148+
if err := (&LegacyPasswordScrubber{Client: failing}).Start(context.Background()); err != nil {
149+
t.Fatalf("Start must not fail the manager, got %v", err)
150+
}
151+
152+
users := getSnapshot(t, c, "z-legacy")["users"].(map[string]any)
153+
if _, ok := users["alice"].(map[string]any)["password"]; ok {
154+
t.Error("legacy Backup was not scrubbed after an earlier Backup failed")
155+
}
156+
}
157+
158+
func TestLegacyPasswordScrubber_ListFailureDoesNotStopTheManager(t *testing.T) {
159+
failing := interceptor.NewClient(newScrubberClient(), interceptor.Funcs{
160+
List: func(context.Context, client.WithWatch, client.ObjectList, ...client.ListOption) error {
161+
return errors.New("forbidden")
162+
},
163+
})
164+
if err := (&LegacyPasswordScrubber{Client: failing}).Start(context.Background()); err != nil {
165+
t.Fatalf("Start must not fail the manager, got %v", err)
166+
}
167+
}

0 commit comments

Comments
 (0)