Skip to content

Commit 3a369fe

Browse files
authored
feat(redis): add opt-in TLS for Redis and Sentinel (#2729)
## What this PR does `tls.enabled` puts Redis and Sentinel on a TLS-only listener with a per-release cert-manager chain, and hands the tenant the CA certificate as `redis-<name>.tenant-ca`. Off unless set. redis-operator v3.3.5 has no TLS support. The work is ported upstream in freshworks-oss/redis-operator#85, which is open and not in a release, so this carries it as three patches over the v3.3.5 tarball. `images/redis-operator/patches/tls.diff` is the source half, cut from that PR's commit `2ad2ed8d1`, which the patch header pins; it is applied by the glob the image Dockerfile already runs, after the Dockerfile strips the tarball's `manifests`, `charts`, `example` and `.github`, so the image builds from source alone and the upstream tests that read the CRD YAML skip instead of failing against the stale v3.3.5 copy. `patches/crd-tls.diff` adds `spec.tls` to the vendored CRD, and with it the CEL rule that fixes `tls.enabled` at creation; without the schema block the apiserver prunes the field on admission and the patched binary waits for something it never sees, and the rule cannot exist without the block it references. `patches/rbac-tls.diff` grants Create, Get and Update on Secrets, so the operator can publish the CA-only Secret it addresses by name. That is the whole delta, and it is narrower than the ported operator's own chart asks for: those three are the only Secret calls in the patched source, and cert-manager Certificates get no grant at all, because the operator only touches one under `spec.tls.certManager` and this chart pins `certificateSecret` mode instead. One change in `tls.diff` is not TLS and takes effect with TLS off, named in the patch header: `MakeSlaveOfWithPort` connects to the replica on its own port rather than on the master's. It changes a dial, not a pod template, so shipping the rebuilt image rolls nothing; the plaintext pod templates stay byte-identical to v3.3.5. The image build registration this branch used to carry (`make -C packages/system/redis-operator image` in the root `build:` target) landed on main with the Valkey service (#3380), so the rebuilt operator image is produced by CI from main as well; the digest in `values.yaml` is still refreshed by release preparation, not here. The chart patches re-apply at the end of `update:`, the same way reloader, grafana-operator, ingress-nginx, seaweedfs and kamaji carry theirs. Otherwise `rm -rf charts` takes TLS back out on the next `make update` and nothing fails. `make update` reproduces the committed tree byte for byte. TLS is chosen when the instance is created. The config gets `port 0` and the listener moves to `tls-port`, so TLS replaces plaintext rather than running beside it, and a running instance cannot change mode: a replica restarted onto the TLS listener cannot replicate from a master still on plaintext, and Sentinels already on TLS would fail over to a stale replica. The RedisFailover schema in the carried CRD patch rejects a change of `tls.enabled`, which on an existing instance surfaces as a failed HelmRelease upgrade while the instance keeps serving what it did; the README points at creating a new instance and migrating. That is also why TLS is not inferred from `external`: an externally reachable instance is the one most likely to have clients an upgrade must not drop. Existing releases keep their plaintext listener and are not restarted by the rebuilt operator image. The tenant gets `ca.crt` and nothing else, through `tenantsecrets`. The chart's own Secrets take a dot between the release and the suffix (`<release>.ca-tls`, `<release>.tls`, `<release>.ca-cert`): the suffixes are dot-free, so the split is unique and release `foo`'s CA Secret can no longer be release `foo-ca`'s leaf, the collision the hyphenated form had. The chart renders a `TenantProjection` pointing at `<release>.ca-tls`, and the CA-extraction controller re-encodes the one named key from the parsed certificate, so nothing that is not a certificate gets through. Same path as postgres, mongodb, nats and qdrant. No Secret of this release is granted to the tenant by name beyond the generated password, and that includes `<release>.ca-cert`, the CA-only copy the operator publishes for itself. It holds no key today, but the operator has Create and Update on Secrets cluster-wide, so what sits at that name is the operator's call rather than the platform's, and RBAC cannot filter by key. `hack/check-redis-rd-secrets.bats` pins the whole rendered Secret surface instead of a list of known-bad names, because an include with no `resourceNames` and no `matchLabels` restricts nothing. The chainsaw suite for redis gains two cases. A TLS fixture asserts the cert-manager chain is issued, the TLS-probed Redis replicas and Sentinel come up, the operator publishes its CA-only Secret under the widened ClusterRole, the tenant receives exactly `ca.crt` through `tenantsecrets` while the key-bearing Secrets stay off that path, and a `redis-cli --tls` PING through the service name completes with the same CA. A second case applies `tls.enabled` to a running plaintext release and asserts the refusal: the HelmRelease upgrade fails with the schema's message, the StatefulSet keeps its revision with both replicas Ready, plaintext still answers and no TLS listener appears. Certificate renewal rolls the pods from inside the operator. Redis and Sentinel read the certificate once at startup, so the patched operator stamps a hash of the TLS Secret's `tls.crt` and `ca.crt` into the pod template of both the Redis StatefulSet and the Sentinel Deployment on every reconcile, as `redis-failover.freshworks.com/tls-secret-hash`. A renewal changes the hash; the operator's own roller replaces the Redis pods one at a time and the Deployment controller rolls Sentinel. README names the fallback: if the roll cannot run, the deadline is the old certificate's expiry, past which the operator's own TLS connections fail along with the clients. `tls.authClients` maps to `tls-auth-clients` and defaults to `no`. The platform mints no client certificate for the tenant and the CA key is out of reach, so `yes` is only usable when the client certificate comes from whoever also controls the CA. Changing the value on a running release restarts the Redis and Sentinel pods: Redis reads the directive from its config file only at startup, so the operator folds the effective value into the certificate hash that drives the rollout, and the change lands when it is made rather than at the next unrelated restart. Follow-ups in the carried operator patch go upstream first, to freshworks-oss/redis-operator#85 or a successor there, and are picked up here afterwards; the carried patch does not grow Go that upstream has not seen. `tlsConfigFor` fetches the Secret on every call, tens of GETs per reconcile. The renewal roll is static analysis so far: the live check, `cmctl renew` on a running release while watching the `rfr-*` and `rfs-*` pods roll, is still owed and belongs with the e2e follow-up. Two facts about the shipped image rather than the patch. `packages/system/redis-operator/values.yaml` keeps the digest of the image built before the patch until release preparation rebuilds it, so on `main` between this merge and the next release-prep, `tls.enabled` renders the chain and `spec.tls` while the running operator ignores them and serves plaintext. And the pinned v3.3.5 tarball carries GO-2026-5026 in `golang.org/x/net` v0.53.0, reached through `PodService.UpdatePodLabels`; the port neither introduces nor fixes it, upstream `master` is already on v0.55.0, so it lands with the next tag bump rather than here. Outside the patch: `cozyvalues-gen` renders `{}` in the README Value column for `tls.authClients`: a child property without a default inherits the parent object's `{}`, and `make generate` reproduces it byte for byte, so it is a generator bug rather than something to hand-edit here. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#37 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: I walked the trigger map against the diff file by file. terraform-provider-cozystack is reached, tracked as cozystack/terraform-provider-cozystack#37 rather than a speculative PR: `values.schema.json` gains a `tls` object, and the provider hand-writes a schema, model and expand/flatten pair per Kind. It already models `tls` for postgres, kafka, nats and qdrant, not for redis. Redis needs different wording from those four, whose descriptions say to omit the block to follow `external`, which is the one thing redis does not do. Nothing else is reached. The website regenerates this package's reference page from `README.md` on a stable tag, so the README changes need no PR there. No package added, renamed or removed. `packages/core/platform/values.yaml`, variant and bundle composition and release asset names are untouched, and so are `hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh`, which is what ccp and external-apps-example read. The two new files under `hack/` are bats unit tests. No node prerequisite, installer value, telemetry metric or cozy-proxy label changes. ### Release note ```release-note feat(redis): add opt-in TLS for Redis and Sentinel. Set `tls.enabled: true` to serve both on a TLS-only listener backed by a per-release cert-manager chain; the tenant reads the CA certificate as the `<release>.tenant-ca` tenant secret. TLS is off unless set and is not inferred from `external`, so existing releases keep serving plaintext and are not restarted by the rebuilt operator image. TLS is chosen at creation: a running instance cannot switch modes, the change is rejected and surfaces as a failed HelmRelease upgrade, so add TLS by creating a new instance and migrating to it. Requires cert-manager, which the platform's `system` bundle installs on every variant. A renewed certificate rolls the Redis and Sentinel pods through the operator. TLS support is carried as a downstream patch over redis-operator v3.3.5 until freshworks-oss/redis-operator#85 lands in a release. ```
2 parents d800526 + 3cf9431 commit 3a369fe

29 files changed

Lines changed: 9170 additions & 10 deletions

‎api/apps/v1alpha1/redis/types.go‎

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/redis/zz_generated.deepcopy.go‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/agents/image-refs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ If a package stamps a reference somewhere the two globs do not cover, add that f
4444

4545
- **Every first-party reference is tag+digest.** A bare tag with no digest is not reproducible and must not be committed. `securitygroup-controller:v0.0.0` is a live violation.
4646
- **Exactly one producer stamps each reference.** A reference no `Makefile` writes will never be refreshed by anything, including release preparation — `chBackupClientImage` in `packages/system/backupstrategy-controller/values.yaml` is a live violation, still pinned to `platform-migrations:v1.4.0-rc.2`.
47-
- **A package that stamps a reference is in the root `Makefile` `build:` list.** `build-matrix.sh`, the main build and release preparation all iterate that one list, so a package outside it is never rebuilt by any path and its pin freezes permanently. `flux-plunger`, `keycloak-operator`, `kilo` and `redis-operator` are live violations (tracked in #3416).
47+
- **A package that stamps a reference is in the root `Makefile` `build:` list.** `build-matrix.sh`, the main build and release preparation all iterate that one list, so a package outside it is never rebuilt by any path and its pin freezes permanently. `flux-plunger`, `keycloak-operator` and `kilo` are live violations (tracked in #3416).
4848
- **Every consumer of references reads all three storage shapes**, via `hack/lib/image-refs.sh`.
4949
- **Scan wider than you rewrite.** A rewrite is deliberately narrow to avoid corrupting vendored defaults, so the check that follows it must not share that blind spot. `hack/promote-rewrite-tags.sh` rewrites the enumerated files, then greps the whole tree and fails if any reference to the release candidate survives.
5050

‎hack/check-redis-rd-secrets.bats‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
#!/usr/bin/env bats
2+
# Unit test: the redis ResourceDefinition hands tenants the key-free trust
3+
# anchor and nothing else.
4+
#
5+
# The chart issues redis-<name>.ca-tls (CA certificate AND CA private key, the
6+
# latter under tls.key) and redis-<name>.tls (server certificate AND server
7+
# private key). Neither may reach a tenant: read access to the CA key would
8+
# let the holder issue certificates for anything rather than merely verify the
9+
# server. What a client actually needs is ca.crt alone, delivered by the
10+
# CA-extraction controller as the key-free <release>.tenant-ca projection and
11+
# selected by LABEL rather than by name, because a name grant conveys whatever
12+
# happens to occupy that name.
13+
#
14+
# Redis has a third Secret the other engines do not: redis-<name>.ca-cert, the
15+
# CA-only copy the patched operator publishes for itself. It is key-free today,
16+
# which is exactly why it is tempting to grant by name — and why it is not.
17+
# The operator holds create and update on Secrets cluster-wide, so what lands
18+
# under that name is the operator's choice rather than the platform's, and RBAC
19+
# has no key-level filter to fall back on.
20+
#
21+
# Asserted against the RENDERED chart, not against cozyrds/redis.yaml on
22+
# disk. templates/cozyrd.yaml globs cozyrds/* and emits each match through
23+
# .Files.Get, so what ships is the render: reading one source file would miss
24+
# a second file in that directory, and a dotfile the glob picks up. Helm
25+
# ignores dotfiles only under templates/, so one here does ship.
26+
#
27+
# This lives in bats rather than helm-unittest because packages/system/
28+
# redis-rd has no `test:` target, and hack/helm-unit-tests.sh skips any
29+
# package without one. If that package ever grows one, this belongs there.
30+
#
31+
# spec.secrets is compared WHOLE rather than probed for known-bad names.
32+
# Three ways this breaks are invisible to a name-by-name check:
33+
#
34+
# - a name added under a different spelling, or via a helper;
35+
# - an include with NO resourceNames and NO matchLabels, which restricts
36+
# nothing;
37+
# - a SECOND matchLabels include selecting a label the key-bearing Secrets
38+
# also carry. cert-manager stamps controller.cert-manager.io/fao on the
39+
# Secrets it creates, both of these among them, so one such entry grants
40+
# the CA private key while every name in the file stays correct.
41+
42+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
43+
CHART="$REPO_ROOT/packages/system/redis-rd"
44+
45+
# The complete tenant-visible Secret surface. The generated password by name;
46+
# the trust anchor by label. Nothing else. The two key-bearing Secrets are
47+
# named in exclude, which is evaluated before include and so backstops the
48+
# label-only entry against a mis-stamp.
49+
EXPECTED_SECRETS='{"exclude":[{"resourceNames":["redis-{{ .name }}.ca-tls","redis-{{ .name }}.tls","redis-{{ .name }}.ca-cert"]}],"include":[{"resourceNames":["redis-{{ .name }}-auth"]},{"matchLabels":{"internal.cozystack.io/tenant-ca":"true"}}]}'
50+
51+
# Order within include is meaningless -- the matcher ORs across entries -- so
52+
# both arrays are sorted before comparing, and a missing exclude is normalized
53+
# to the empty list it means. Neither can hide a grant: sort preserves every
54+
# entry (it is not unique), and exclude only ever narrows.
55+
#
56+
# Kept as a filter string rather than a shell function because hack/cozytest.sh
57+
# inserts `return 0` before any line that is exactly `}`. A helper whose own
58+
# last command decides the outcome returns 0 regardless -- and whether a
59+
# wrapper propagates a failure at all then depends on the shell running it.
60+
# Repeating the helm invocation inline in each test costs two lines and owes
61+
# nothing to that.
62+
NORMALIZE='{exclude: ((.exclude // []) | sort), include: (.include | sort)}'
63+
64+
# Every document count below uses `yq eval-all`, never `yq eval`. `eval` runs
65+
# the expression once PER DOCUMENT, so `[.] | length` reports 1 per document
66+
# rather than the number of documents -- it prints "1" for a single document,
67+
# "1\n1" for two, and "1" for empty input, which would make a count check here
68+
# pass on exactly the cases it exists to catch. `eval-all` folds the stream
69+
# into one expression. Empty input still yields 1, since yq synthesizes a null
70+
# document, so emptiness is checked separately before counting.
71+
72+
@test "redis-rd renders exactly one ApplicationDefinition" {
73+
# stderr is deliberately not discarded: when the render breaks, helm's own
74+
# message is the only thing that says why.
75+
rendered=$(helm template redis-rd "$CHART") || return 1
76+
77+
# Fail closed: an empty render makes every assertion below vacuous, and yq
78+
# cannot distinguish it from one document.
79+
if [ -z "$rendered" ]; then
80+
echo "helm template produced no output for $CHART -- guard is blind." >&2
81+
return 1
82+
fi
83+
84+
count=$(printf '%s\n' "$rendered" | yq eval-all '[.] | length' -) || return 1
85+
86+
if [ "$count" != "1" ]; then
87+
echo "Expected exactly 1 rendered document from $CHART, got: $count" >&2
88+
echo "templates/cozyrd.yaml globs cozyrds/*, so every file there -- including" >&2
89+
echo "a dotfile -- ships its own tenant grants. Extend this suite before" >&2
90+
echo "adding one." >&2
91+
return 1
92+
fi
93+
}
94+
95+
@test "redis-rd selects the key-free tenant-CA projection by label" {
96+
rendered=$(helm template redis-rd "$CHART") || return 1
97+
98+
if [ -z "$rendered" ]; then
99+
echo "helm template produced no output for $CHART -- guard is blind." >&2
100+
return 1
101+
fi
102+
103+
count=$(printf '%s\n' "$rendered" | yq eval-all \
104+
'[.. | select(has("secrets")) | .secrets.include[] | select(.matchLabels."internal.cozystack.io/tenant-ca" == "true")] | length' \
105+
-) || return 1
106+
107+
if [ "$count" -lt 1 ]; then
108+
echo 'redis-rd does not select internal.cozystack.io/tenant-ca: "true"' >&2
109+
echo "Without it the tenant cannot read ca.crt and TLS verification is impossible." >&2
110+
return 1
111+
fi
112+
}
113+
114+
# Whole-structure comparison. Anything added to the tenant's Secret surface --
115+
# by name, by label, or by an unrestricted entry -- changes this string.
116+
@test "redis-rd exposes no Secret beyond the password and the trust anchor" {
117+
rendered=$(helm template redis-rd "$CHART") || return 1
118+
119+
if [ -z "$rendered" ]; then
120+
echo "helm template produced no output for $CHART -- guard is blind." >&2
121+
return 1
122+
fi
123+
124+
# eval-all again: with `eval`, a second rendered document would emit a
125+
# second JSON object and the comparison below would run on a concatenation.
126+
actual=$(printf '%s\n' "$rendered" \
127+
| yq eval-all --output-format=json '[.. | select(has("secrets")) | .secrets]' - \
128+
| jq --sort-keys --compact-output "if length == 1 then .[0] else . end | $NORMALIZE") || return 1
129+
130+
expected=$(printf '%s' "$EXPECTED_SECRETS" \
131+
| jq --sort-keys --compact-output "$NORMALIZE") || return 1
132+
133+
# Fail closed: an empty or null result means spec.secrets stopped being
134+
# readable at this path, so the guard is no longer observing what it claims.
135+
if [ -z "$actual" ] || [ "$actual" = "null" ]; then
136+
echo "spec.secrets is missing or unreadable in the rendered ResourceDefinition -- guard is blind." >&2
137+
return 1
138+
fi
139+
140+
if [ "$actual" != "$expected" ]; then
141+
echo "Unexpected tenant Secret surface in the rendered redis-rd spec.secrets:" >&2
142+
echo " actual: $actual" >&2
143+
echo " expected: $expected" >&2
144+
echo "redis-<name>.ca-tls holds the CA PRIVATE KEY and redis-<name>.tls the" >&2
145+
echo "server private key. Neither may be granted to a tenant, by name or by" >&2
146+
echo "a label they also carry -- cert-manager stamps its own labels on both." >&2
147+
echo "redis-<name>.ca-cert is key-free but operator-written, so it is not" >&2
148+
echo "granted either; the trust anchor travels as the tenant-ca projection." >&2
149+
return 1
150+
fi
151+
}
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#!/usr/bin/env bats
2+
# Unit test: no generated values.schema.json may declare a non-string default
3+
# on a property typed "string".
4+
#
5+
# cozyvalues-gen quotes a default only when the declared type is the literal
6+
# `string` or `quantity` (formatDefault in internal/openapi/openapi.go). A
7+
# named alias — including one declared `@enum {string}` — falls through to a
8+
# YAML re-parse of the bare value, where the 1.1 boolean words resolve. So a
9+
# chart that materialises `authClients: "no"` under an `@enum` type emits
10+
# `"default": false` into a property whose type is "string" and whose enum is
11+
# ["no","optional","yes"] — a default the field's own schema rejects. The same
12+
# coercion reaches api/ as `+kubebuilder:default:=false` on a string-typed Go
13+
# field, so the CRD defaults an enum-constrained string to a boolean and every
14+
# CR relying on the default is rejected. "optional" survives; "no" and "yes"
15+
# do not. The quotes in values.yaml do not help: the value reaches
16+
# formatDefault as a flat string with no record of having been quoted.
17+
#
18+
# The damage is invisible to the chart test suites: templates read the value
19+
# from values.yaml, not from the schema, so rendering stays correct and only
20+
# the shipped schema and CRD are wrong. This test is the guard, and it is why
21+
# packages/apps/redis/values.yaml leaves `tls: {}` empty and resolves the
22+
# authClients default in templates/_tls.tpl instead.
23+
#
24+
# Runs under hack/cozytest.sh, which is POSIX /bin/sh — no process
25+
# substitution, no arrays, no bats `run`.
26+
27+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
28+
29+
@test "no values.schema.json declares a non-string default on a string-typed property" {
30+
# Vendored charts under packages/*/*/charts/ are pulled by `make update`, not
31+
# generated, so a violation there has no fix inside this tree.
32+
# An empty result means "no violations" only if the scan actually reached
33+
# some schemas. A renamed directory, a wrong root or a missing jq all yield
34+
# the same empty string, and the test would pass having read nothing.
35+
scanned="$(find "$REPO_ROOT/packages" -name values.schema.json -not -path '*/charts/*' | wc -l | tr -d ' ')"
36+
if [ "$scanned" -eq 0 ]; then
37+
echo "no values.schema.json found under $REPO_ROOT/packages -- guard is blind." >&2
38+
return 1
39+
fi
40+
41+
violations="$(find "$REPO_ROOT/packages" -name values.schema.json -not -path '*/charts/*' -exec jq -r '
42+
input_filename as $f
43+
| [ .. | objects
44+
| select(.type == "string" and has("default") and (.default | type) != "string")
45+
| "\($f): default=\(.default | tojson) enum=\(.enum // "none" | tojson)"
46+
] | .[]' {} +)"
47+
48+
if [ -n "$violations" ]; then
49+
echo "String-typed schema properties carrying a non-string default:" >&2
50+
echo "$violations" >&2
51+
echo "This is cozyvalues-gen coercing a YAML 1.1 boolean word (no/yes/on/off)" >&2
52+
echo "out of values.yaml. Leave the field absent from values.yaml and resolve" >&2
53+
echo "its default in the template instead." >&2
54+
return 1
55+
fi
56+
}

0 commit comments

Comments
 (0)