test(qdrant): exercise the tenant trust anchor end to end by lexfrei · Pull Request #4701 · cozystack/cozystack

Aleksei Sviridkin (lexfrei) · 2026-10-02T03:49:42Z

What this PR does

The qdrant e2e suite now runs with TLS on and checks the tenant trust anchor end to end. Before this, the fixture left tls unset with external: false, so the chart rendered no CA and no TenantProjection, and nothing in e2e reached the controller or tenantsecrets. Setting tls.enabled: true turns TLS on without the LoadBalancer that external: true would allocate.

The first new step checks that the anchor is published. The qdrant-test-ca sentinel has to report Ready=True/Projected, and qdrant-test.tenant-ca has to carry both the tenant-ca and tenantresource=true labels. The projection must hold only ca.crt with the same fingerprint as the chart's CA, and tenantsecrets must serve it. The key-bearing qdrant-test-ca and qdrant-test-tls must answer NotFound there, and any other error fails the step.

The second step checks that the anchor is what decides trust. It reads the anchor through tenantsecrets, port-forwards the qdrant Service and runs openssl s_client with that file as the only CA, hostname check included. A CA with the same subject and another key must fail, and so must the runner's default trust store. A handshake failure counts only when the output shows a verification error, so a refused connection does not pass as a rejection. Plaintext HTTP must get no response. Curl exits 7 and 28 fail the step, and a second verified handshake afterwards proves the forward was still alive.

TLS does not break the existing steps, because none of them talk to qdrant. create-qdrant, wait-helmreleases-ready, verify-statefulset-and-storage and verify-pvc-reclaim-on-delete only read Kubernetes objects: no http://, no port 6333 or 6334, no curl or exec, and the suite has no backup round trip. The release now also renders two Issuers, two Certificates and the sentinel. The chart switches its probes to HTTPS with enable_tls, so readyReplicas: 1 now covers a pod serving TLS.

Local checks: chainsaw lint test (v0.2.15) says the file is valid. helm template with the fixture's spec renders 2 Certificates, 2 Issuers and 1 TenantProjection, and none without TLS. I ran the handshake logic against a local openssl s_server with a CA and leaf shaped like the chart's. The anchor passes, the wrong CA and the default store fail on verification, a wrong hostname fails, plaintext gets 000, and a closed port is not counted as a rejection.

A sentinel with a missing source (SourceNotFound) is not tested here, the reconciler unit tests cover it.

Review notes I left as they are:

Through kubectl port-forward the local socket is always accepted, so curl exits 7 and 28 only catch a dead forward process. A false pass would need qdrant to answer plaintext and a pod-side blip on that one request at the same time.
qdrant may reset the socket on the plaintext request, and some kubectl versions drop the whole port-forward on a pod-side error. Then the handshake after it turns the step red, not green. If CI shows lost connection to pod there, the plaintext check should get its own port-forward.
The suite's catch has no describe of the TenantProjection. The sentinel assert prints its conditions when it fails, but a red Secret assert or script after it would not.
The vendored chart's servicemonitor.yaml hardcodes scheme: http, so metrics scraping fails for any qdrant with TLS on. That was already the case before this PR, and the suite does not check metrics. Tracked in qdrant: the ServiceMonitor scrapes http even when TLS is on #4703.

cert-manager does not set owner references here, so qdrant-test-ca and qdrant-test-tls stay in the namespace with their private keys after the app is deleted. That is tracked in #4704. The nats half of #3706 is tracked in #4702.

Closes #3706

Screenshots

No UI change.

Downstream repositories

The diff touches only hack/e2e-chainsaw/qdrant/, which no entry in the trigger map covers.

Release note

test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused.

The fixture left TLS unset with external off, so the chart rendered no CA and no TenantProjection, and no e2e run walked the chain from the sentinel through the CA-extraction controller to the tenantsecrets read path. Turn TLS on explicitly, without allocating a LoadBalancer, and check both ends: the projection is key-free, carries the chart's CA and is served through tenantsecrets while the key-bearing Secrets are not; and a client trusting only that anchor verifies the listener, while a wrong anchor, no anchor and plaintext HTTP all fail. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>

coderabbitai · 2026-10-02T03:50:12Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: cb6d0af3-ab7a-45d5-89f3-2b15732a5cfe

📥 Commits

Reviewing files that changed from the base of the PR and between bc400df and efb4b1f.

📒 Files selected for processing (2)

hack/e2e-chainsaw/qdrant/chainsaw-test.yaml
hack/e2e-chainsaw/qdrant/qdrant.yaml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 3 remain after this review.

📝 Walkthrough

Walkthrough

The Qdrant e2e fixture now enables TLS. New checks validate tenant anchor publication and test which trust sources can verify the Qdrant TLS endpoint.

Changes

Qdrant trust-anchor verification

Layer / File(s)	Summary
Enable TLS in the e2e fixture `hack/e2e-chainsaw/qdrant/qdrant.yaml`	The fixture enables TLS explicitly and documents its relationship to the trust-anchor checks.
Verify projected anchor publication `hack/e2e-chainsaw/qdrant/chainsaw-test.yaml`	The check validates the projected Secret’s readiness, labels, type, and certificate contents. It confirms that `tenantsecrets` serves the anchor but does not serve source Secrets containing private keys.
Verify endpoint trust behavior `hack/e2e-chainsaw/qdrant/chainsaw-test.yaml`	The check requires the endpoint to accept the published anchor and reject a different CA and the default trust store. It also checks plaintext behavior and confirms that a verified TLS handshake succeeds afterward.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to efb4b

This change adds end-to-end coverage for Qdrant's tenant trust anchor by enabling TLS in the test fixture. Production code is unchanged, and no defects were identified. It is ready to merge.

Security Architecture Review

Security architecture risk: 🔵 Low · up to efb4b

The changes strengthen trust checks without changing production templates or enabling external exposure. No introduced vulnerability was established. Tenant-identity enforcement and certificate cleanup after interrupted runs remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

inferred — The introduced behavior is bounded to the Qdrant test namespace and its runner: fixture TLS activation, certificate inspection, and local endpoint probes. The scoped comparison shows no production-template modification, and the fixture does not enable LoadBalancer exposure.

Trust Boundaries and Controls

observed — The checks explicitly distinguish the tenant-readable certificate from CA and service Secrets containing tls.key. They fail if either key-bearing Secret is served through tenantsecrets, and accept only NotFound rather than an authorization denial or unavailable API as evidence of exclusion.
observed — The scripts do not select or impersonate a tenant identity, and the TLS handshake supplies no client certificate. Consequently, these checks do not establish least-privilege tenant authorization or server-side client authentication.

Resilience and Maintainability Implications

inferred — Verification-specific failure checks and a final positive handshake reduce the chance of mistaking transport failure for a working security control. Nevertheless, readiness checks only a fixed local socket; it does not independently establish ownership by this invocation's port-forward under concurrent execution.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Issue `#3706` asks for Qdrant e2e coverage of the trust-anchor path. The fixture enables TLS while keeping `external: false`. The test checks that `qdrant-test-ca` reaches `Ready=True/Projected`, that `…
Out of Scope Changes check	✅ Passed	Both changed files support issue `#3706`. The TLS endpoint checks validate use of the published anchor, and the key-bearing Secret checks strengthen the same trust-anchor security boundary. The changes …
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding end-to-end Qdrant tests for the tenant trust anchor.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

IvanHunters

LGTM

Test-only, nothing under production code moves: both files live in hack/e2e-chainsaw/qdrant/. The new coverage actually walks the tenant trust-anchor path instead of asserting something trivial, and every name it pins lines up with the chart and the cacert controller.

What I checked against the source:

Fixture tls.enabled is a real field (values.schema.json and the generated qdrant-rd CRD); the release prefix makes the release name qdrant-test.
certmanager.yaml renders CA secret qdrant-test-ca and serving secret qdrant-test-tls, and the serving cert has CN qdrant-test with qdrant-test in its SAN list, so -servername/-verify_hostname qdrant-test matches. No client cert is required, so the server-auth-only handshake the test performs is right.
tenant-projection.yaml names the TenantProjection qdrant-test-ca, which is exactly what the first step asserts.
cacert/reconciler.go publishes .tenant-ca as an Opaque, key-free Secret labeled internal.cozystack.io/tenant-ca=true, with condition Ready/Projected.
The qdrant-rd cozyrd spec.secrets selects only the tenant-ca label plus the apikey Secret by name, so the key-bearing qdrant-test-ca and qdrant-test-tls are correctly not served through tenantsecrets, which is the negative the test leans on.
The chart sets service.enable_tls on 6333/6334, so the plaintext probe expecting no HTTP response (http_code 000) holds up.

The negated checks are the opposite of inert. Each one separates "the thing I wanted to fail failed" from "something else failed": verify error versus a refused connection or a hang, NotFound versus an RBAC denial or an unavailable apiserver, curl 000 versus exit 7/28. Every timeout is bounded, and the existing steps are untouched, so no coverage quietly goes away.

[NIT] hack/e2e-chainsaw/qdrant/chainsaw-test.yaml:256 the port-forward readiness gate (nc -z localhost 6333) only proves the local socket is open, not that kubectl finished dialing the backend, so the first positive handshake can fail on the connection rather than on verification and then report a trust failure that isn't real. You already called this out in the TODO(e2e-replace-fixed-timeouts) comment, and it matches what the bucket test does, so the flake risk is low and nothing needs to change.

Aleksei Sviridkin (lexfrei) requested review from IvanHunters, Andrei Kvapil (kvaps), Timofei Larkin (lllamnyp) and myasnikovdaniil as code owners October 2, 2026 03:49

github-actions Bot added area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) size/L This PR changes 100-499 lines, ignoring generated files labels Oct 2, 2026

IvanHunters approved these changes Oct 5, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) enabled auto-merge October 5, 2026 11:14

Aleksei Sviridkin (lexfrei) merged commit 33cf697 into main Oct 5, 2026
20 checks passed

Aleksei Sviridkin (lexfrei) deleted the test/e2e-qdrant-trust-anchor branch October 5, 2026 11:14

Navigation Menu

Repository navigation

test(qdrant): exercise the tenant trust anchor end to end - #4701

test(qdrant): exercise the tenant trust anchor end to end#4701
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
test/e2e-qdrant-trust-anchor

Aleksei Sviridkin (lexfrei) commented Oct 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 2, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

IvanHunters left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Aleksei Sviridkin (lexfrei) commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Screenshots

Downstream repositories

Release note

Uh oh!

coderabbitai Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Security Architecture Review

Uh oh!

IvanHunters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Aleksei Sviridkin (lexfrei) commented Oct 2, 2026 •

edited

Loading

coderabbitai Bot commented Oct 2, 2026 •

edited

Loading