Skip to content

Commit d800526

Browse files
authored
feat(mariadb): add TLS support via cert-manager (#2680)
## What this PR does Gives MariaDB a chart-managed certificate chain, makes TLS enforcement explicit, and hands the tenant a key-free trust anchor instead of a private key. - `tls.issuer` selects who issues the server certificate: `operator` (the CA mariadb-operator manages itself, the default and today's behaviour) or `cert-manager` (the instance gets its own CA, and this is what covers the external hostname). The field is named for the issuer rather than for on/off because TLS is served under both — unlike the similarly spelled field in some other charts, which really is a switch. - `tls.required` refuses plaintext connections. It is opt-in and defaults to false: enforcing TLS before the tenant can obtain the CA would be backwards, and a platform upgrade must not sever connections that work today. - Neither issuer is inferred from `external`. Existing instances keep the CA they have and change nothing on upgrade. - The tenant no longer receives `<release>-ca-tls` or `<release>-tls` — the CA private key and the server private key. It receives the operator's `ca.crt`-only bundle, which works today, and the ResourceDefinition additionally declares that bundle as the source for the CA-extraction controller. ### Fixes along the way - The TLS backup path could never have worked: it passed `--ssl-mode=VERIFY_IDENTITY`, a MySQL-8 flag the MariaDB client rejects during argument parsing. Replaced with `--ssl-ca` + `--ssl-verify-server-cert`, and the test now asserts the dialect rather than the presence of a substring. - The leaf certificate carried `clientAuth`. Because `REQUIRE X509` accepts any certificate signed by a trusted CA, the server's own key was usable as a client credential. Removed. - The CA and server private keys survived deletion of the instance: cert-manager runs with `enableCertificateOwnerRef: false`, so removing the `Certificate` left the Secrets behind. Both are now cleaned up by the existing post-delete hook. - The Secrets are labelled for the operator's watch predicate, without which a renewed leaf would not reach the pods until a full resync. ### Why the include carries an exclude list The trust-anchor selector matches by label with no name constraint, so the enumerable dangerous set is excluded explicitly — both issuers' private keys, the internal credentials, the backup keys, and the metrics configuration, which the operator writes unconditionally with credentials in plain text for a user holding `SELECT` on everything. The backstop is bounded and the comment says so: matching is by exact name with no globs, so a Secret introduced by a future operator version, or a transient object created mid-renewal, stays in scope of the label-only include. The primary defence remains that the controller stamps only its own projection. ## Known limitations 1. **Merge is blocked on #3299 — and the blocking is mechanical, not just procedural.** The CRD on `main` does not carry `spec.caCert`, and under Flux's Server-Side Apply the API server rejects a field the schema does not declare (`field not declared in schema`); under the older client-side apply the same field was silently dropped, so installing this branch without #3299 fails outright. An earlier revision of this description claimed the field would be silently pruned; that was wrong. The tenant still gets a usable trust anchor once merged in order, through the operator's bundle. 2. **The deprecated backup CronJob cannot start at all**, for reasons that predate this branch: its `secretKeyRef` names a Secret and key the chart never renders. That is filed separately as #3347 and deliberately not fixed here, since repairing it would start scheduled dumps on production instances from a TLS change. The tests that pin the backup TLS flags therefore pin a path that does not currently execute — the suite header says so. 3. **The operator's `clusterName` is hardcoded**, so on a cluster with a non-default domain the SANs and the name the operator dials diverge. Pre-existing, raised separately. 4. **No e2e coverage for the managed path.** Unit coverage pins the rendered chain, the SAN set, the exclusion set and the guards; the end-to-end handshake is not asserted in CI. ### On review coverage Two independent review passes were run repeatedly against this branch — seven rounds, converging from a security finding, through comment accuracy, to guard strength, and finally to nothing new. A third, model-based pass reviewed the branch through `6cb1bd236` and its findings were fixed; it could not be run against the final revision because of an external account quota lasting until 25 July. The delta since that revision is substantial and includes runtime code, not only tests and docs. Recording this so the coverage is not overstated: if the quota returns before this is ready to merge, that pass is worth running again. ### Downstream repositories `packages/apps/mariadb/values.schema.json` gains the `tls` object, so `cozystack/terraform-provider-cozystack` needs the field modelled for `cozystack_mariadb`; tracked in cozystack/terraform-provider-cozystack#40. ### Release note ```release-note feat(mariadb): chart-managed TLS via cert-manager, opt-in enforcement, and the CA published to tenants as a key-free trust anchor ```
2 parents 18d02d8 + b24ed36 commit d800526

20 files changed

Lines changed: 2707 additions & 14 deletions

‎api/apps/v1alpha1/mariadb/types.go‎

Lines changed: 14 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/mariadb/zz_generated.deepcopy.go‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎hack/check-mariadb-rd-secrets.bats‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
#!/usr/bin/env bats
2+
# Unit test: mariadb-rd's tenant grant surface. The chart's helm-unittest suite
3+
# cannot reach this package, so its Secret-exposure invariants are pinned here.
4+
#
5+
# The tenant reaches the TLS trust anchor as a key-free projection, never by
6+
# name. The chart renders a TenantProjection sentinel naming the operator's
7+
# key-free <release>-ca-bundle as the extraction source
8+
# (packages/apps/mariadb/templates/tenant-projection.yaml); the CA-extraction
9+
# controller publishes the result as "<release>.tenant-ca" carrying
10+
# internal.cozystack.io/tenant-ca; and this ApplicationDefinition selects that
11+
# label. The sentinel's own fields belong to the chart and are pinned by
12+
# packages/apps/mariadb/tests/tenant_projection_test.yaml, which compares
13+
# sourceSecretName against an exact key-free name — a stronger guard than any
14+
# grep here could be, and one this file cannot make, since it never sees the
15+
# chart.
16+
#
17+
# Asserted against the RENDERED chart, not against cozyrds/mariadb.yaml on
18+
# disk, following hack/check-qdrant-rd-secrets.bats. templates/cozyrd.yaml
19+
# globs cozyrds/* and emits each match through .Files.Get, so what ships is the
20+
# render: reading one source file would miss a second file in that directory,
21+
# and a dotfile the glob picks up. Helm ignores dotfiles only under templates/,
22+
# so one here does ship.
23+
#
24+
# spec.secrets and spec.services are each compared WHOLE rather than probed for
25+
# known-bad shapes. Probing is what a name-shaped check cannot see:
26+
#
27+
# - an exclude entry that selects by LABEL instead of by name. matchName
28+
# returns true for every name when resourceNames is nil
29+
# (internal/lineagecontrollerwebhook/matcher.go), and exclude wins over
30+
# include, so one such entry withholds everything — the tenant's
31+
# connection string and its trust anchor included. A pass reading
32+
# .exclude[].resourceNames[]? cannot see an entry that has no
33+
# resourceNames at all.
34+
# - an entry that restricts nothing, spelled as an empty matchLabels, an
35+
# empty matchExpressions, or a bare {}. Only the first is a literal anyone
36+
# greps for; ApplicationDefinitionResourceSelector inlines
37+
# metav1.LabelSelector, so all three are legal.
38+
# - a SECOND matchLabels include selecting a label the key-bearing Secrets
39+
# also carry. cert-manager stamps controller.cert-manager.io/fao on both
40+
# .ca-tls and .tls, so one such entry grants the CA private key while every
41+
# name in the file stays correct.
42+
#
43+
# Order within each list is meaningless — the matcher ORs across entries — so
44+
# both are sorted before comparing, and an absent list is normalized to the
45+
# empty list it means. The sort is jq's, which orders objects; yq's sort would
46+
# leave a list of maps as it found it.
47+
#
48+
# Every document count uses `yq eval-all`, never `yq eval`: `eval` runs the
49+
# expression once PER DOCUMENT, so `[.] | length` reports 1 per document rather
50+
# than the number of them. Emptiness is checked separately, since yq
51+
# synthesizes a null document for empty input.
52+
#
53+
# Everything is spelled inline rather than through a shell helper:
54+
# hack/cozytest.sh inserts `return 0` before any line that is exactly `}`, so a
55+
# helper whose own last command decides the outcome returns 0 regardless.
56+
57+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
58+
CHART="$REPO_ROOT/packages/system/mariadb-rd"
59+
60+
NORMALIZE='{exclude: (.exclude // [] | sort), include: (.include // [] | sort)}'
61+
62+
# The complete tenant-visible Secret surface: the credentials Secret by name,
63+
# the trust anchor by label, every key-bearing and credential Secret withheld
64+
# by name. mariadb-<name>.ca-tls holds the CA private key and mariadb-<name>.tls
65+
# the server key; the operator's own -ca, -server-cert and -client-cert hold
66+
# keys too. None of them may reach a tenant, by name or by a label they carry.
67+
EXPECTED_SECRETS='{"exclude":[{"resourceNames":["mariadb-{{ .name }}.ca-tls","mariadb-{{ .name }}.tls","mariadb-{{ .name }}-ca","mariadb-{{ .name }}-server-cert","mariadb-{{ .name }}-client-cert","mariadb-{{ .name }}-root","mariadb-{{ .name }}-password","mariadb-{{ .name }}-repl-password","mariadb-{{ .name }}-metrics-password","mariadb-{{ .name }}-metrics-config","mariadb-{{ .name }}-backup","mariadb-{{ .name }}-regsecret"]}],"include":[{"matchLabels":{"internal.cozystack.io/tenant-ca":"true"}},{"resourceNames":["mariadb-{{ .name }}-credentials"]}]}'
68+
69+
# The Services an instance exposes, pinned for the same reason: a catch-all
70+
# selector reaches this list too, and nothing else in the tree holds it.
71+
EXPECTED_SERVICES='{"exclude":[],"include":[{"resourceNames":["mariadb-{{ .name }}","mariadb-{{ .name }}-primary","mariadb-{{ .name }}-secondary"]}]}'
72+
73+
@test "mariadb-rd renders exactly one ApplicationDefinition" {
74+
# stderr is deliberately not discarded: when the render breaks, helm's own
75+
# message is the only thing that says why.
76+
rendered=$(helm template mariadb-rd "$CHART") || return 1
77+
78+
# Fail closed: an empty render makes every assertion below vacuous, and yq
79+
# cannot distinguish it from one document.
80+
if [ -z "$rendered" ]; then
81+
echo "helm template produced no output for $CHART -- guard is blind." >&2
82+
return 1
83+
fi
84+
85+
count=$(printf '%s\n' "$rendered" | yq eval-all '[.] | length' -) || return 1
86+
87+
if [ "$count" != "1" ]; then
88+
echo "Expected exactly 1 rendered document from $CHART, got: $count" >&2
89+
echo "templates/cozyrd.yaml globs cozyrds/*, so every file there -- including" >&2
90+
echo "a dotfile -- ships its own tenant grants. Extend this suite before" >&2
91+
echo "adding one." >&2
92+
return 1
93+
fi
94+
}
95+
96+
# Subsumed by the whole-value comparison below, which cannot match without this
97+
# entry. Kept because the runner stops at the first failing test, so this one
98+
# fires first and says what the loss costs rather than printing two JSON blobs
99+
# to diff.
100+
@test "mariadb-rd selects the key-free tenant-CA projection by label" {
101+
rendered=$(helm template mariadb-rd "$CHART") || return 1
102+
103+
if [ -z "$rendered" ]; then
104+
echo "helm template produced no output for $CHART -- guard is blind." >&2
105+
return 1
106+
fi
107+
108+
count=$(printf '%s\n' "$rendered" | yq eval-all \
109+
'[.. | select(has("secrets")) | .secrets.include[] | select(.matchLabels."internal.cozystack.io/tenant-ca" == "true")] | length' \
110+
-) || return 1
111+
112+
if [ "$count" -lt 1 ]; then
113+
echo 'mariadb-rd does not select internal.cozystack.io/tenant-ca: "true"' >&2
114+
echo "Without it the tenant cannot read ca.crt and TLS verification is impossible." >&2
115+
return 1
116+
fi
117+
}
118+
119+
# Also subsumed, and kept for the same reason: a dropped name is the likeliest
120+
# single edit here, and this names which one went rather than leaving it to a
121+
# diff of two twelve-element lists.
122+
@test "mariadb-rd withholds every key-bearing and credential Secret by name" {
123+
rendered=$(helm template mariadb-rd "$CHART") || return 1
124+
125+
if [ -z "$rendered" ]; then
126+
echo "helm template produced no output for $CHART -- guard is blind." >&2
127+
return 1
128+
fi
129+
130+
# The suffix carries its own separator: the two Secrets this chart asks
131+
# cert-manager for are joined with a dot, everything else with a hyphen, so
132+
# a loop that supplied the hyphen itself would look for names nothing renders.
133+
for n in .ca-tls .tls -ca -server-cert -client-cert \
134+
-root -password -repl-password -metrics-password -metrics-config \
135+
-backup -regsecret; do
136+
hits=$(printf '%s\n' "$rendered" | yq eval-all \
137+
"[.. | select(has(\"secrets\")) | .secrets.exclude[].resourceNames[]? | select(. == \"mariadb-{{ .name }}$n\")] | length" \
138+
-) || return 1
139+
if [ "$hits" -lt 1 ]; then
140+
echo "mariadb-<name>$n is missing from secrets.exclude" >&2
141+
echo "The include selector matches by label with no name constraint, so this" >&2
142+
echo "list is the enumerable part of that gap; exclude wins over include." >&2
143+
return 1
144+
fi
145+
done
146+
}
147+
148+
# Whole-structure comparison. Anything added to the tenant's Secret surface --
149+
# by name, by label, or by an unrestricted entry -- changes this string, and so
150+
# does anything removed from the backstop.
151+
@test "mariadb-rd grants exactly the credentials Secret and the trust anchor" {
152+
rendered=$(helm template mariadb-rd "$CHART") || return 1
153+
154+
if [ -z "$rendered" ]; then
155+
echo "helm template produced no output for $CHART -- guard is blind." >&2
156+
return 1
157+
fi
158+
159+
# eval-all again: with `eval`, a second rendered document would emit a second
160+
# JSON object and the comparison below would run on a concatenation.
161+
actual=$(printf '%s\n' "$rendered" \
162+
| yq eval-all --output-format=json '[.. | select(has("secrets")) | .secrets]' - \
163+
| jq --sort-keys --compact-output "if length == 1 then .[0] else . end | $NORMALIZE") || return 1
164+
165+
expected=$(printf '%s' "$EXPECTED_SECRETS" \
166+
| jq --sort-keys --compact-output "$NORMALIZE") || return 1
167+
168+
# Fail closed: an empty or null result means spec.secrets stopped being
169+
# readable at this path, so the guard is no longer observing what it claims.
170+
if [ -z "$actual" ] || [ "$actual" = "null" ]; then
171+
echo "spec.secrets is missing or unreadable in the rendered ResourceDefinition -- guard is blind." >&2
172+
return 1
173+
fi
174+
175+
if [ "$actual" != "$expected" ]; then
176+
echo "Unexpected tenant Secret surface in the rendered mariadb-rd spec.secrets:" >&2
177+
echo " actual: $actual" >&2
178+
echo " expected: $expected" >&2
179+
echo "Both lists are pinned: include decides what a tenant receives, and" >&2
180+
echo "exclude is evaluated first and wins outright, so an entry added there" >&2
181+
echo "revokes the connection string or the trust anchor in silence." >&2
182+
return 1
183+
fi
184+
}
185+
186+
@test "mariadb-rd exposes exactly the instance Services" {
187+
rendered=$(helm template mariadb-rd "$CHART") || return 1
188+
189+
if [ -z "$rendered" ]; then
190+
echo "helm template produced no output for $CHART -- guard is blind." >&2
191+
return 1
192+
fi
193+
194+
actual=$(printf '%s\n' "$rendered" \
195+
| yq eval-all --output-format=json '[.. | select(has("services")) | .services]' - \
196+
| jq --sort-keys --compact-output "if length == 1 then .[0] else . end | $NORMALIZE") || return 1
197+
198+
expected=$(printf '%s' "$EXPECTED_SERVICES" \
199+
| jq --sort-keys --compact-output "$NORMALIZE") || return 1
200+
201+
if [ -z "$actual" ] || [ "$actual" = "null" ]; then
202+
echo "spec.services is missing or unreadable in the rendered ResourceDefinition -- guard is blind." >&2
203+
return 1
204+
fi
205+
206+
if [ "$actual" != "$expected" ]; then
207+
echo "Unexpected tenant Service surface in the rendered mariadb-rd spec.services:" >&2
208+
echo " actual: $actual" >&2
209+
echo " expected: $expected" >&2
210+
return 1
211+
fi
212+
}

0 commit comments

Comments
 (0)