Skip to content

Commit a198cad

Browse files
committed
feat(platform)!: publish the passthrough services by gateway.passthroughMode
With the port-443 passthrough services on a Gateway of their own, the tenant API, VM export and CDI upload need a second address. Not every installation has one, so the platform chooses: singleAddress needs no second address. The Kubernetes API is passed through on 6443 of the main Gateway, where client certificates still reach kube-apiserver. A native-port listener admits routes from the publishing tenant's namespace alone, and cozystack-api installs before cozystack-basics creates that namespace, so the TLSRoute ships with the tenant's gateway chart, next to the listener, and reaches the kubernetes Service through a ReferenceGrant. The VM export and CDI upload proxies authenticate tokens rather than client certificates, so the Gateway terminates TLS for them on 443 and re-encrypts under a BackendTLSPolicy whose CA the controller keeps in step. For the export proxy that is the main KubeVirt CA, which signs the proxy's own certificate; the export CA signs only the per-export servers behind it. Their routes disable the request timeout, which Gateway API leaves to the implementation: an export streams a whole disk image in one response. separateAddress keeps all three passed through on 443 of the second Gateway. Left unset, the mode is singleAddress on a new install and separateAddress on a cluster that already ran the Gateway, so an upgrade does not move a working API endpoint to another port. Migration 59 pins that in the platform Package. It runs as the hook of the render it would have to change, so the chart makes the same choice from the version stamp, which also covers the variants that run no migrations. Only the publishing tenant renders anything for these services, only while gateway.enabled is on and only for what the platform exposes: they are platform routes, rendered only then, so anything else rendering them reserved names and, with a second Gateway, would have cost an address for routes that never come. Every chart renders nothing for them while the mode is absent from its values, since that is a channel written before the mode existed. The controller renders the second Gateway only where the platform values channel names the tenant as publishing under separateAddress, because a TenantGateway still carrying the services from an older chart would otherwise get a second Gateway, and a load-balancer address, before its chart re-renders. BREAKING CHANGE: a new install defaults to singleAddress, which serves the Kubernetes API on api.<host>:6443; set publishing.apiServerEndpoint to match, except on isp-full-generic, where the CNI reaches kube-apiserver through that value and it has to stay on a direct address. The BackendTLSPolicy path requires a GatewayClass that supports it. An existing install with gateway.enabled is pinned to separateAddress: its URLs stay on 443, but the DNS records for the three names have to point at the second Gateway's address. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 0271996 commit a198cad

34 files changed

Lines changed: 1690 additions & 295 deletions

‎api/gateway/v1alpha1/tenantgateway_types.go‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -332,9 +332,10 @@ type TenantGatewaySpec struct {
332332
// TLSPassthroughServices names services exposed via TLS-passthrough
333333
// (mode: Passthrough listeners). Each service gets a dedicated
334334
// listener on port 443 of a second Gateway, <name>-passthrough,
335-
// which exists only while this list is non-empty; HTTPRoutes attach
336-
// to TLS-terminate listeners instead. Not rendered when
337-
// CertMode=edge.
335+
// rendered only while this list is non-empty and the platform values
336+
// name this tenant as publishing under passthroughMode
337+
// separateAddress; HTTPRoutes attach to TLS-terminate listeners
338+
// instead. Not rendered when CertMode=edge.
338339
//
339340
// An entry becomes both the listener name tls-<svc> and the listener
340341
// hostname <svc>.<apex>, and Gateway API bounds each at 253

‎cmd/cozystack-controller/main.go‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -203,17 +203,23 @@ func main() {
203203
// Scope the shared Secret informer so the WildcardSecret reconciler's
204204
// cluster-wide Secret watch does not cache every Secret (and its key
205205
// material) in memory. Only managed wildcard replicas and the values
206-
// channel are cached; no other reconciler reads Secrets through THIS
207-
// (the manager's) typed Secret cache. See wildcardsecret.SecretCacheByObject.
206+
// channel are cached. Two reconcilers read through THIS (the
207+
// manager's) typed Secret cache: wildcardsecret, and tenantgateway,
208+
// which reads the values channel to decide whether a TenantGateway
209+
// gets a passthrough Gateway and reads its absence as "no". Dropping
210+
// the channel from the selector would therefore remove every
211+
// passthrough Gateway without an error;
212+
// TestPlatformValuesChannelIsInTheSecretCache pins it. See
213+
// wildcardsecret.SecretCacheByObject.
208214
//
209215
// The CA-extraction reconciler does NOT use this cache at all — neither its
210216
// typed Secret informer nor a metadata one. A metadata informer for
211217
// v1/Secret would route to THIS same per-GVK cache and inherit its
212218
// wildcard-replica label selector, so its source watch would never fire for
213219
// an unlabelled CA source. It therefore owns a SEPARATE metadata-only cache
214220
// (caSecretCluster, below) and reads the one source it projects through the
215-
// uncached APIReader. So this manager-level Secret scoping is
216-
// wildcardsecret's alone.
221+
// uncached APIReader. So this manager-level Secret scoping serves
222+
// wildcardsecret and tenantgateway alone.
217223
Cache: cache.Options{
218224
ByObject: map[client.Object]cache.ByObject{
219225
&corev1.Secret{}: wildcardsecret.SecretCacheByObject(),

‎hack/e2e-chainsaw/gateway/chainsaw-test.yaml‎

Lines changed: 24 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1057,15 +1057,14 @@ spec:
10571057
check:
10581058
($error == null): true
10591059
---
1060-
# What a tlsPassthroughServices name costs an HTTPRoute that publishes it,
1061-
# measured on the objects rather than on the renderer. The chart ships api,
1062-
# vm-exportproxy and cdi-uploadproxy on every tenant Gateway while the platform
1063-
# TLSRoutes for them exist once cluster-wide and name one tenant's Gateway, so
1064-
# a tenant that is not that one renders all three listeners with a route on
1065-
# none. Both halves of the rule live in that gap: a declared listener nothing
1066-
# attaches to contributes no Envoy filter chain, so the terminate listener is
1067-
# what answers the name and keeps it; once a TLSRoute can be served on the
1068-
# listener the two chains carry one server name and the terminate listener goes.
1060+
# What a passthrough listener costs an HTTPRoute that publishes its name,
1061+
# measured on the objects rather than on the renderer. The listener is the
1062+
# shape the publishing tenant renders for the Kubernetes API under
1063+
# gateway.passthroughMode=singleAddress: api.<apex> passed through on 6443.
1064+
# Both halves of the rule are measured: a declared listener nothing attaches
1065+
# to contributes no Envoy filter chain, so the terminate listener is what
1066+
# answers the name and keeps it; once a TLSRoute can be served on the
1067+
# listener, the terminate listener goes.
10691068
#
10701069
# The two phases are one measurement on purpose. Asserting the terminate
10711070
# listener is present proves nothing on its own — a withdrawal that is merely
@@ -1165,8 +1164,7 @@ spec:
11651164
tgw_ready='jsonpath={range .status.conditions[?(@.type=="Ready")]}{.status}{"\n"}{end}'
11661165
tgw_listener='jsonpath={range .status.listeners[?(@.name=="tls-pgprobe")]}{.ready}{"\n"}{end}'
11671166
1168-
# A tenant owning its own Gateway, which is where the shipped
1169-
# tlsPassthroughServices entries land with nothing routed to them.
1167+
# A tenant owning its own Gateway.
11701168
kubectl apply -n "$ns" -f - <<EOF
11711169
apiVersion: apps.cozystack.io/v1alpha1
11721170
kind: Tenant
@@ -1182,6 +1180,12 @@ spec:
11821180
[ -n "$apex" ]
11831181
route_host="api.${apex}"
11841182
1183+
# Patched onto the TenantGateway because only the publishing
1184+
# tenant's chart renders this listener. It survives the patch
1185+
# because helm-controller runs here with drift detection off.
1186+
api_listener=$(printf '{"name":"api","port":6443,"hostname":"%s"}' "$route_host")
1187+
kubectl -n "$tenant_ns" patch tenantgateway cozystack --type=merge -p "{\"spec\":{\"tlsPassthroughListeners\":[${api_listener}]}}"
1188+
11851189
# Without the passthrough listener the rest measures an ordinary
11861190
# HTTPRoute and says nothing about the withdrawal.
11871191
poll present 60 tls-api kubectl -n "$tenant_ns" get gateway cozystack -o "$tls_names"
@@ -1278,20 +1282,15 @@ spec:
12781282
# ...and the HTTPRoute hears why its hostname went away.
12791283
poll present 60 NoMatchingListenerHostname kubectl -n "$tenant_ns" get httproute api-probe -o "$accepted_reason"
12801284
1281-
# Phase 3: the native-port form, which nothing above reaches. The
1282-
# entry renders a listener on its own port with no TLSRoute behind
1283-
# it, which is the shape the package README rests a claim on: the
1284-
# pinned Cilium judges a listener on its protocol, its allowed
1285-
# kinds and its certificate refs and never on its route set, so a
1286-
# routeless passthrough listener is Accepted and Programmed and
1287-
# tenant readiness is untouched. The unit tests run against a fake
1288-
# client and cannot measure that; only a cluster can.
1289-
#
1290-
# Patched onto the TenantGateway because no chart value reaches
1291-
# this field yet. It survives the patch because helm-controller
1292-
# runs here with drift detection off, the same fact the README
1293-
# documents.
1294-
patch=$(printf '{"spec":{"tlsPassthroughListeners":[{"name":"pgprobe","port":5432,"hostname":"pgprobe.%s"}]}}' "$apex")
1285+
# Phase 3: a native-port listener with no TLSRoute behind it. The
1286+
# package README rests a claim on that shape: Cilium judges a
1287+
# listener on its protocol, its allowed kinds and its certificate
1288+
# refs and never on its route set, so a routeless passthrough
1289+
# listener is Accepted and Programmed and tenant readiness is
1290+
# untouched. The unit tests run against a fake client and cannot
1291+
# measure that; only a cluster can. The api listener stays in the
1292+
# list, which the merge patch replaces whole.
1293+
patch=$(printf '{"spec":{"tlsPassthroughListeners":[%s,{"name":"pgprobe","port":5432,"hostname":"pgprobe.%s"}]}}' "$api_listener" "$apex")
12951294
kubectl -n "$tenant_ns" patch tenantgateway cozystack --type=merge -p "$patch"
12961295
12971296
# The per-listener entry is the measurement, and it is read before
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for platform migration 59 --> 60 (pin
4+
# gateway.passthroughMode=separateAddress on a cluster that already runs the
5+
# Gateway, so the upgrade does not move its tenant API to :6443).
6+
#
7+
# Four properties are pinned here:
8+
#
9+
# 1. ONLY A GATEWAY INSTALL WITH NO MODE. gateway.enabled true, as a bool or as
10+
# the string the Package's values JSON may carry, with no passthroughMode,
11+
# gets exactly the separateAddress patch. A set mode, either one, is left
12+
# alone, and so are a Gateway that is off and a Package that is not there.
13+
#
14+
# 2. EXACTLY THIS PATCH, on the platform Package, as a merge patch.
15+
#
16+
# 3. FAIL CLOSED. A failed read or a failed patch aborts before the version
17+
# stamp: migrations never re-run, and a swallowed error would stamp past a
18+
# cluster that then renders singleAddress.
19+
#
20+
# 4. EVERY PATH STAMPS 60, and only once.
21+
#
22+
# These drive the real migration script end-to-end against a fake kubectl
23+
# (hack/testdata/migration-59-passthrough-mode/), mocking only the cluster
24+
# boundary.
25+
#
26+
# SHELL. Production runs the migration under /bin/sh = busybox ash, by path, so
27+
# run_migration() runs it by path inside the migrations image's own pinned base
28+
# with jq added, --network none. See hack/migration-54-redis-adopt.bats for the
29+
# full reasoning; it applies unchanged.
30+
#
31+
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its own
32+
# line, rewriting the latter into `return 0` + `}`. A helper whose exit status
33+
# matters must capture it and `return` it by hand before its closing brace.
34+
#
35+
# Run with: hack/cozytest.sh hack/migration-59-passthrough-mode.bats
36+
# -----------------------------------------------------------------------------
37+
38+
FAKEBIN="$PWD/hack/testdata/migration-59-passthrough-mode"
39+
MIG_DIR="$PWD/packages/core/platform/images/migrations/migrations"
40+
PIN='{"spec":{"components":{"platform":{"values":{"gateway":{"passthroughMode":"separateAddress"}}}}}}'
41+
42+
ALPINE=$(sed -n 's/^FROM \(alpine:[^ ]*\).*$/\1/p' \
43+
"$PWD/packages/core/platform/images/migrations/Dockerfile" | head -1)
44+
45+
TESTIMG="cozystack-migration59-test:$(printf '%s' "$ALPINE" | sed 's/[^a-zA-Z0-9]/-/g')"
46+
47+
run_migration() {
48+
_run_migration_rc=0
49+
docker run --rm --network none \
50+
--user "$(id -u):$(id -g)" \
51+
-v "$MIG_DIR:/migrations:ro" \
52+
-v "$FAKEBIN:/fakebin:ro" \
53+
-v "$WORK:/work" \
54+
-e PATH=/fakebin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
55+
-e FAKE_CMDLOG=/work/cmdlog \
56+
-e NAMESPACE=cozy-system \
57+
-e FAKE_VALUES="${FAKE_VALUES-}" \
58+
-e FAKE_GET_FAIL="${FAKE_GET_FAIL-}" \
59+
-e FAKE_PATCH_FAIL="${FAKE_PATCH_FAIL-}" \
60+
"$TESTIMG" /migrations/59 || _run_migration_rc=$?
61+
return "$_run_migration_rc"
62+
}
63+
64+
prep() {
65+
docker info >/dev/null 2>&1 || {
66+
echo "docker is required: these tests run migration 59 inside a jq-enabled" >&2
67+
echo "build of $ALPINE, the migrations image's base." >&2
68+
return 1
69+
}
70+
docker build -q -t "$TESTIMG" - >/dev/null <<DOCKERFILE
71+
FROM $ALPINE
72+
RUN apk add --no-cache jq
73+
DOCKERFILE
74+
chmod +x "$FAKEBIN/kubectl"
75+
WORK=$(mktemp -d "${TMPDIR:-/tmp}/mig59.XXXXXX")
76+
export FAKE_CMDLOG="$WORK/cmdlog"
77+
: > "$FAKE_CMDLOG"
78+
export FAKE_VALUES="-"
79+
unset FAKE_GET_FAIL FAKE_PATCH_FAIL || true
80+
return 0
81+
}
82+
83+
# stamped_once -- the run stamped 60 exactly once and nothing else.
84+
stamped_once() {
85+
[ "$(grep -cE '^STAMP ' "$FAKE_CMDLOG")" -eq 1 ] && grep -qxF -- "STAMP 60" "$FAKE_CMDLOG"
86+
return $?
87+
}
88+
89+
# --- 1, 2 and 4. who is pinned, with what --------------------------------------
90+
91+
@test "a Gateway install with no mode is pinned to separateAddress" {
92+
prep
93+
for values in '{"gateway":{"enabled":true}}' '{"gateway":{"enabled":"true"}}' '{"gateway":{"enabled":true,"passthroughMode":null}}' '{"gateway":{"enabled":true,"passthroughMode":""}}'; do
94+
: > "$FAKE_CMDLOG"
95+
export FAKE_VALUES="$values"
96+
rc=0
97+
run_migration >"$WORK/out" 2>&1 || rc=$?
98+
cat "$WORK/out"; cat "$FAKE_CMDLOG"
99+
[ "$rc" -eq 0 ]
100+
[ "$(grep -cE '^PATCH ' "$FAKE_CMDLOG")" -eq 1 ]
101+
grep -qxF -- "PATCH $PIN" "$FAKE_CMDLOG"
102+
stamped_once
103+
done
104+
rm -rf "$WORK"
105+
}
106+
107+
@test "a mode already set, a Gateway that is off and a missing Package are left alone" {
108+
prep
109+
for values in '{"gateway":{"enabled":true,"passthroughMode":"singleAddress"}}' '{"gateway":{"enabled":true,"passthroughMode":"separateAddress"}}' '{"gateway":{"enabled":false}}' '{"gateway":{}}' '{}' '-'; do
110+
: > "$FAKE_CMDLOG"
111+
export FAKE_VALUES="$values"
112+
rc=0
113+
run_migration >"$WORK/out" 2>&1 || rc=$?
114+
cat "$WORK/out"; cat "$FAKE_CMDLOG"
115+
[ "$rc" -eq 0 ]
116+
[ "$(grep -cE '^PATCH ' "$FAKE_CMDLOG")" -eq 0 ]
117+
[ "$(grep -c '^UNHANDLED ' "$FAKE_CMDLOG")" -eq 0 ]
118+
stamped_once
119+
done
120+
rm -rf "$WORK"
121+
}
122+
123+
# --- 3. fail closed -------------------------------------------------------------
124+
125+
@test "a failed read of the Package aborts before stamping" {
126+
prep
127+
export FAKE_VALUES='{"gateway":{"enabled":true}}'
128+
export FAKE_GET_FAIL=1
129+
rc=0
130+
run_migration >"$WORK/out" 2>&1 || rc=$?
131+
cat "$WORK/out"; cat "$FAKE_CMDLOG"
132+
[ "$rc" -ne 0 ]
133+
[ "$(grep -cE '^(PATCH|STAMP) ' "$FAKE_CMDLOG")" -eq 0 ]
134+
rm -rf "$WORK"
135+
}
136+
137+
@test "a failed patch aborts before stamping" {
138+
prep
139+
export FAKE_VALUES='{"gateway":{"enabled":true}}'
140+
export FAKE_PATCH_FAIL=1
141+
rc=0
142+
run_migration >"$WORK/out" 2>&1 || rc=$?
143+
cat "$WORK/out"; cat "$FAKE_CMDLOG"
144+
[ "$rc" -ne 0 ]
145+
[ "$(grep -cE '^STAMP ' "$FAKE_CMDLOG")" -eq 0 ]
146+
rm -rf "$WORK"
147+
}
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/bin/sh
2+
# Fake kubectl for hack/migration-59-passthrough-mode.bats. It serves only the
3+
# calls migration 59 (and lib/cozystack-version.sh) makes; behaviour is driven by
4+
# FAKE_* env and every invocation is appended to $FAKE_CMDLOG.
5+
#
6+
# FAKE_CMDLOG file to append a one-line record of each call to
7+
# FAKE_VALUES JSON of spec.components.platform.values on the platform
8+
# Package, or "-" for no Package at all
9+
# FAKE_GET_FAIL non-empty => reading the Package exits 1, as an apiserver
10+
# error
11+
# FAKE_PATCH_FAIL non-empty => the patch exits 1
12+
set -u
13+
14+
log() { [ -n "${FAKE_CMDLOG:-}" ] && printf '%s\n' "$*" >> "$FAKE_CMDLOG"; }
15+
log "KUBECTL $*"
16+
17+
args="$*"
18+
19+
arg_after() { k="$1"; shift; p=""; for a in "$@"; do [ "$p" = "$k" ] && { printf '%s' "$a"; return; }; p="$a"; done; }
20+
21+
case "$args" in
22+
"get packages.cozystack.io cozystack.cozystack-platform -o json --ignore-not-found")
23+
if [ -n "${FAKE_GET_FAIL:-}" ]; then
24+
printf 'fake kubectl: the server is currently unable to handle the request\n' >&2
25+
exit 1
26+
fi
27+
[ "${FAKE_VALUES:--}" = "-" ] && exit 0
28+
printf '{"metadata":{"name":"cozystack.cozystack-platform"},"spec":{"components":{"platform":{"values":%s}}}}\n' "$FAKE_VALUES"
29+
exit 0
30+
;;
31+
32+
"patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p "*)
33+
if [ -n "${FAKE_PATCH_FAIL:-}" ]; then
34+
printf 'fake kubectl: admission webhook denied the patch\n' >&2
35+
exit 1
36+
fi
37+
log "PATCH $(arg_after -p "$@")"
38+
exit 0
39+
;;
40+
41+
*"apply --filename -"*)
42+
manifest=$(cat)
43+
v=$(printf '%s\n' "$manifest" | sed -n 's/^ version: "\{0,1\}\([0-9]\{1,\}\)"\{0,1\}$/\1/p' | head -1)
44+
log "STAMP ${v:-unknown}"
45+
exit 0
46+
;;
47+
esac
48+
49+
log "UNHANDLED $args"
50+
printf 'fake kubectl: unmodelled call: %s\n' "$args" >&2
51+
exit 1

‎internal/controller/tenantgateway/backends.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,8 @@ import (
2828
gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
2929
)
3030

31-
// tlsRouteForwards reports whether route names at least one backend the
32-
// pinned Cilium would forward to, which is what decides whether the
31+
// tlsRouteForwards reports whether route names at least one backend
32+
// Cilium would forward to, which is what decides whether the
3333
// route puts a filter chain on its hostname.
3434
//
3535
// The question is answered exactly as v1.19.5 answers it and no more

‎internal/controller/tenantgateway/conflict.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -783,9 +783,9 @@ func (r *Reconciler) updateRouteStatuses(
783783
// Edge renders no TLS-passthrough listener, so every attached TLSRoute
784784
// pins a section that is gone and is told so. NoMatchingParent is
785785
// Gateway API's reason for a parentRef whose port or sectionName
786-
// matches no listener on the Gateway, which is what the platform's own
787-
// api, vm-exportproxy and cdi-uploadproxy routes now have: all three
788-
// pin sectionName tls-<svc>.
786+
// matches no listener on the Gateway, which is what every platform
787+
// TLSRoute has here: each pins sectionName tls-<svc>, and edge renders
788+
// no such listener.
789789
//
790790
// The other whole-apex modes render those listeners and serve them, but
791791
// this controller collects no claims there and so has no basis to judge

‎internal/controller/tenantgateway/mapper.go‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,22 @@ func (r *Reconciler) mapGatewayClassToTenantGateways(ctx context.Context, obj cl
109109
return out
110110
}
111111

112+
// mapPlatformValuesToTenantGateways requeues every TenantGateway when the
113+
// platform values channel changes, since whether a TenantGateway gets a
114+
// passthrough Gateway is read from it (publishesPassthroughGateway).
115+
func (r *Reconciler) mapPlatformValuesToTenantGateways(ctx context.Context, _ client.Object) []reconcile.Request {
116+
list := &gatewayv1alpha1.TenantGatewayList{}
117+
if err := r.List(ctx, list); err != nil {
118+
log.FromContext(ctx).Error(err, "list TenantGateways for platform values mapper")
119+
return nil
120+
}
121+
out := make([]reconcile.Request, 0, len(list.Items))
122+
for i := range list.Items {
123+
out = append(out, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&list.Items[i])})
124+
}
125+
return out
126+
}
127+
112128
// backendToTenantGateways returns an EventHandler that maps a Service
113129
// or ReferenceGrant change back to the TenantGateways whose withdrawal
114130
// decision it can flip.

0 commit comments

Comments
 (0)