Skip to content

Commit 33cf697

Browse files
authored
test(qdrant): exercise the tenant trust anchor end to end (#4701)
## What this PR does The qdrant e2e suite now runs with TLS on and checks the tenant trust anchor end to end. Before this, the fixture left `tls` unset with `external: false`, so the chart rendered no CA and no `TenantProjection`, and nothing in e2e reached the controller or `tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the LoadBalancer that `external: true` would allocate. The first new step checks that the anchor is published. The `qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and `qdrant-test.tenant-ca` has to carry both the `tenant-ca` and `tenantresource=true` labels. The projection must hold only `ca.crt` with the same fingerprint as the chart's CA, and `tenantsecrets` must serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must answer NotFound there, and any other error fails the step. The second step checks that the anchor is what decides trust. It reads the anchor through `tenantsecrets`, port-forwards the qdrant Service and runs `openssl s_client` with that file as the only CA, hostname check included. A CA with the same subject and another key must fail, and so must the runner's default trust store. A handshake failure counts only when the output shows a verification error, so a refused connection does not pass as a rejection. Plaintext HTTP must get no response. Curl exits 7 and 28 fail the step, and a second verified handshake afterwards proves the forward was still alive. TLS does not break the existing steps, because none of them talk to qdrant. `create-qdrant`, `wait-helmreleases-ready`, `verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or exec, and the suite has no backup round trip. The release now also renders two Issuers, two Certificates and the sentinel. The chart switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1` now covers a pod serving TLS. Local checks: `chainsaw lint test` (v0.2.15) says the file is valid. `helm template` with the fixture's spec renders 2 Certificates, 2 Issuers and 1 TenantProjection, and none without TLS. I ran the handshake logic against a local `openssl s_server` with a CA and leaf shaped like the chart's. The anchor passes, the wrong CA and the default store fail on verification, a wrong hostname fails, plaintext gets `000`, and a closed port is not counted as a rejection. A sentinel with a missing source (`SourceNotFound`) is not tested here, the reconciler unit tests cover it. Review notes I left as they are: - Through `kubectl port-forward` the local socket is always accepted, so curl exits 7 and 28 only catch a dead forward process. A false pass would need qdrant to answer plaintext and a pod-side blip on that one request at the same time. - qdrant may reset the socket on the plaintext request, and some kubectl versions drop the whole port-forward on a pod-side error. Then the handshake after it turns the step red, not green. If CI shows `lost connection to pod` there, the plaintext check should get its own port-forward. - The suite's `catch` has no `describe` of the `TenantProjection`. The sentinel assert prints its conditions when it fails, but a red Secret assert or script after it would not. - The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`, so metrics scraping fails for any qdrant with TLS on. That was already the case before this PR, and the suite does not check metrics. Tracked in #4703. cert-manager does not set owner references here, so `qdrant-test-ca` and `qdrant-test-tls` stay in the namespace with their private keys after the app is deleted. That is tracked in #4704. The nats half of #3706 is tracked in #4702. Closes #3706 ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the trigger map covers. ### Release note ```release-note test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused. ```
2 parents 7ea7104 + efb4b1f commit 33cf697

2 files changed

Lines changed: 203 additions & 0 deletions

File tree

‎hack/e2e-chainsaw/qdrant/chainsaw-test.yaml‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,204 @@ spec:
128128
status:
129129
phase: Bound
130130

131+
# The trust-anchor chain from the sentinel to the tenantsecrets read path:
132+
# the chart renders a TenantProjection beside its cert-manager CA, the
133+
# cacert controller lifts ca.crt into the key-free <release>.tenant-ca, and
134+
# the lineage webhook marks it tenantresource=true because the
135+
# ApplicationDefinition selects the tenant-ca label. That last verdict is
136+
# computed from spec.secrets, and nothing short of a cluster proves it.
137+
- name: verify-tenant-anchor-published
138+
try:
139+
- assert:
140+
timeout: 2m
141+
resource:
142+
apiVersion: internal.cozystack.io/v1alpha1
143+
kind: TenantProjection
144+
metadata:
145+
name: qdrant-test-ca
146+
status:
147+
(conditions[?type == 'Ready']):
148+
- status: "True"
149+
reason: Projected
150+
- assert:
151+
timeout: 2m
152+
resource:
153+
apiVersion: v1
154+
kind: Secret
155+
metadata:
156+
name: qdrant-test.tenant-ca
157+
labels:
158+
internal.cozystack.io/tenant-ca: "true"
159+
internal.cozystack.io/tenantresource: "true"
160+
type: Opaque
161+
# Deterministic once the asserts above pass, so no retry.
162+
- script:
163+
timeout: 2m
164+
content: |
165+
set -eu
166+
167+
secret_keys() {
168+
kubectl -n "$NAMESPACE" get secret "$1" \
169+
-o go-template='{{if .data}}{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}{{end}}'
170+
}
171+
172+
# Exactly one key, and it is ca.crt. The source holds the CA private
173+
# key under tls.key, so a second key here is that key leaving.
174+
keys=$(secret_keys qdrant-test.tenant-ca | sort | tr '\n' ' ')
175+
if [ "$keys" != "ca.crt " ]; then
176+
echo "projection must hold exactly ca.crt, holds: $keys" >&2
177+
exit 1
178+
fi
179+
180+
# It lifted the right certificate. Compared by fingerprint, because
181+
# the controller re-encodes ca.crt from the DER it parsed.
182+
src=$(kubectl -n "$NAMESPACE" get secret qdrant-test-ca -o jsonpath='{.data.ca\.crt}' | base64 -d)
183+
proj=$(kubectl -n "$NAMESPACE" get secret qdrant-test.tenant-ca -o jsonpath='{.data.ca\.crt}' | base64 -d)
184+
if [ -z "$src" ]; then
185+
echo "qdrant-test-ca has no ca.crt — the declared source name or key is wrong" >&2
186+
exit 1
187+
fi
188+
src_fp=$(echo "$src" | openssl x509 -noout -fingerprint -sha256)
189+
proj_fp=$(echo "$proj" | openssl x509 -noout -fingerprint -sha256)
190+
if [ "$src_fp" != "$proj_fp" ]; then
191+
echo "projection does not carry the chart's CA certificate" >&2
192+
exit 1
193+
fi
194+
if echo "$proj" | grep -qi 'BEGIN .*PRIVATE KEY'; then
195+
echo "projection carries PEM private key material" >&2
196+
exit 1
197+
fi
198+
199+
# The delivery path a tenant uses. Everything above reads raw
200+
# Secrets, which only an admin can do.
201+
anchor=$(kubectl -n "$NAMESPACE" get tenantsecret qdrant-test.tenant-ca \
202+
-o jsonpath='{.data.ca\.crt}' | base64 -d)
203+
if ! echo "$anchor" | grep -q 'BEGIN CERTIFICATE'; then
204+
echo "the trust anchor is not served through core.cozystack.io/tenantsecrets" >&2
205+
exit 1
206+
fi
207+
208+
# The negative that gives the positive its meaning: the key-bearing
209+
# cert-manager Secrets must not be served the same way. Each must
210+
# hold tls.key first, so a renamed or emptied Secret cannot 404 for
211+
# the wrong reason and pass. NotFound is the only acceptable
212+
# failure: an RBAC denial or an unavailable aggregated apiserver also
213+
# fails the read, and that is not "safely absent".
214+
for s in qdrant-test-ca qdrant-test-tls; do
215+
if ! secret_keys "$s" | grep -qx 'tls.key'; then
216+
echo "$s no longer carries tls.key — this check has stopped proving anything" >&2
217+
exit 1
218+
fi
219+
if err=$(kubectl -n "$NAMESPACE" get tenantsecret "$s" 2>&1 >/dev/null); then
220+
echo "key-bearing $s is being served as a tenant secret" >&2
221+
exit 1
222+
fi
223+
case "$err" in
224+
*NotFound*) echo "ok $s: key-bearing and not served" ;;
225+
*)
226+
echo "tenantsecret $s did not answer NotFound, so this check proved nothing: $err" >&2
227+
exit 1
228+
;;
229+
esac
230+
done
231+
232+
# What the anchor is for: a client that trusts only what the tenant was
233+
# handed reaches qdrant over TLS, and the same client fails closed when the
234+
# anchor is wrong or missing, or when it falls back to plaintext.
235+
- name: verify-endpoint-trusts-only-the-anchor
236+
try:
237+
- script:
238+
# Inner budgets: the port-forward wait (30s) plus four handshakes and
239+
# one plaintext request at 20s each, about 130s at worst, so an inner
240+
# timeout fires before this ceiling does.
241+
timeout: 3m
242+
content: |
243+
set -eu
244+
workdir=$(mktemp -d)
245+
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
246+
247+
# Read through tenantsecrets, the tenant's own path, not the raw
248+
# Secret an admin would read.
249+
kubectl -n "$NAMESPACE" get tenantsecret qdrant-test.tenant-ca \
250+
-o jsonpath='{.data.ca\.crt}' | base64 -d > "$workdir/anchor.crt"
251+
252+
# A wrong anchor with the real CA's subject and a different key, so
253+
# a pass here cannot come from matching on the name.
254+
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
255+
-keyout "$workdir/wrong.key" -out "$workdir/wrong.crt" \
256+
-subj /CN=qdrant-test-ca -days 1 >/dev/null 2>&1
257+
258+
kubectl -n "$NAMESPACE" port-forward service/qdrant-test 6333:6333 >/dev/null 2>&1 &
259+
pf_pid=$!
260+
# TODO(e2e-replace-fixed-timeouts): nothing models a port-forward
261+
# becoming ready; the socket opening is the only signal.
262+
timeout 30 sh -ec 'until nc -z localhost 6333; do sleep 1; done'
263+
264+
# -verify_return_error turns a chain that does not build into a
265+
# non-zero exit, and -verify_hostname checks the SAN the chart
266+
# issues for the in-cluster Service name.
267+
handshake() {
268+
timeout 20 openssl s_client -connect 127.0.0.1:6333 \
269+
-servername qdrant-test -verify_hostname qdrant-test \
270+
-verify_return_error "$@" </dev/null >"$workdir/handshake.log" 2>&1
271+
}
272+
# A negative counts only when verification is what failed. A refused
273+
# connection or a hang also exits non-zero, and reading that as
274+
# "rejected" would pass while proving nothing.
275+
expect_verify_failure() {
276+
what=$1
277+
shift
278+
if handshake "$@"; then
279+
echo "qdrant verified against $what; the anchor is not what decides trust" >&2
280+
cat "$workdir/handshake.log" >&2
281+
exit 1
282+
fi
283+
if ! grep -q 'verify error' "$workdir/handshake.log"; then
284+
echo "the handshake against $what failed, but not on verification:" >&2
285+
cat "$workdir/handshake.log" >&2
286+
exit 1
287+
fi
288+
echo "ok: $what is rejected"
289+
}
290+
291+
if ! handshake -CAfile "$workdir/anchor.crt"; then
292+
echo "qdrant does not verify against the anchor published to the tenant" >&2
293+
cat "$workdir/handshake.log" >&2
294+
exit 1
295+
fi
296+
echo "ok: the published anchor verifies qdrant-test:6333"
297+
298+
expect_verify_failure "a CA with the same subject and another key" -CAfile "$workdir/wrong.crt"
299+
# No anchor at all: the runner's default trust store only.
300+
expect_verify_failure "the default trust store"
301+
302+
# Last, because it is the one request that is not TLS: the listener
303+
# must not answer plaintext HTTP, or a client without the anchor
304+
# could skip verification entirely. 000 means no HTTP response, but
305+
# curl also prints it when nothing is listening, so a refused
306+
# connection (7) or a timeout (28) proves nothing and fails here.
307+
rc=0
308+
code=$(curl --silent --output /dev/null --write-out '%{http_code}' --max-time 20 \
309+
http://127.0.0.1:6333/readyz) || rc=$?
310+
case "$rc" in
311+
7|28)
312+
echo "plaintext request never reached qdrant (curl exit $rc), so this check proved nothing" >&2
313+
exit 1
314+
;;
315+
esac
316+
if [ "$code" != "000" ]; then
317+
echo "qdrant answered plaintext HTTP with status $code" >&2
318+
exit 1
319+
fi
320+
# And the listener was still there to refuse it: the same forward
321+
# still carries a verified handshake afterwards.
322+
if ! handshake -CAfile "$workdir/anchor.crt"; then
323+
echo "the port-forward died around the plaintext request, so its rejection proves nothing" >&2
324+
cat "$workdir/handshake.log" >&2
325+
exit 1
326+
fi
327+
echo "ok: plaintext HTTP gets no response (curl exit $rc)"
328+
131329
- name: verify-pvc-reclaim-on-delete
132330
try:
133331
# Issue #3059: Kubernetes never GCs StatefulSet-templated PVCs; the

‎hack/e2e-chainsaw/qdrant/qdrant.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,8 @@ spec:
99
resourcesPreset: "nano"
1010
resources: {}
1111
external: false
12+
# TLS is set explicitly rather than through `external`, which would also
13+
# allocate a LoadBalancer. With TLS off the chart renders no CA and no
14+
# TenantProjection, so the trust-anchor steps would have nothing to check.
15+
tls:
16+
enabled: true

0 commit comments

Comments
 (0)