Skip to content

Commit d2e851f

Browse files
committed
feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native services with Cilium Gateway API, materialised per-tenant via a new TenantGateway CRD reconciled by cozystack-controller. The extra/gateway chart renders one gateway.cozystack.io/v1alpha1 TenantGateway per tenant; the controller owns the resulting Gateway, per-tenant Issuer, Certificate(s), and the http->https redirect HTTPRoute. This avoids the helm-vs-controller race on Gateway.spec.listeners when route-driven listeners are added. publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS listener + per-app Certificate) or DNS-01 (opt-in; single wildcard Certificate per apex; providers: cloudflare, route53, digitalocean, rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves Let's Encrypt budget on apex-shared clusters. Security model -- seven independent layers, all admission-time gates fail-closed: 1. Listener allowedRoutes namespace whitelist via the kube-apiserver-written kubernetes.io/metadata.name label; plain-HTTP listener is strictly narrower than HTTPS, and HTTPS / passthrough listeners restrict allowedRoutes.kinds to HTTPRoute / TLSRoute respectively. 2. cozystack-gateway-hostname-policy VAP on Gateway. 3. cozystack-gateway-attached-namespaces-policy VAP on Package -- blocks tenant-* in gateway.attachedNamespaces. 4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host writes from non-trusted callers. 5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks writes to namespace.cozystack.io/host from non-trusted callers. 6. Render-time fail in cozystack-basics for tenant-* in gateway.attachedNamespaces. 7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute, scoped to tenant-* namespaces. Within-apex cross-namespace hostname conflict is resolved at reconcile time by a HostnameConflict condition (cozy-* namespace wins). Five reconcile paths refuse to rewrite pre-existing objects with the controller-derived name but no OwnerReference back to the TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard Certificate, per-listener Certificate. Derived-apex tenants (tenants whose host is a subdomain of a parent tenant's) auto-enable Gateway when the platform-level gateway.enabled is on. Bonus fix: pkg/registry/apps/application/rest.go was ignoring the createValidation and deleteValidation callbacks that genericapiserver hands to every storage method, silently bypassing admission for Create and Delete on every apps.cozystack.io resource (Tenant, MariaDB, Postgres, Kubernetes, ...). Aligned with the existing updateValidation pattern. Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle), and the cozystack-api admission-chain fix. Everything else is gated behind gateway.enabled=false and tenant.spec.gateway=false; existing clusters see no behavioural change until an operator opts in. Tests: 60+ Go cases in internal/controller/tenantgateway/, admission- chain regression tests in pkg/registry/apps/application/, helm-unittest across 7 charts (extra/gateway, core/platform, apps/tenant, system/cozystack-basics, system/cert-manager-issuers, system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in hack/e2e-apps/gateway.bats. New make test-controllers target wired into pull-requests.yaml. Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 60357eb commit d2e851f

92 files changed

Lines changed: 27435 additions & 9860 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/pull-requests.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,9 @@ jobs:
5050
- name: Run unit tests
5151
run: make unit-tests
5252

53+
- name: Run controller Go tests
54+
run: make test-controllers
55+
5356
- name: Set up Docker config
5457
run: |
5558
if [ -d ~/.docker ]; then

‎Makefile‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: manifests assets unit-tests helm-unit-tests bats-unit-tests rd-presets-check preflight
1+
.PHONY: manifests assets unit-tests helm-unit-tests bats-unit-tests rd-presets-check test-controllers preflight
22

33
include hack/common-envs.mk
44

@@ -114,6 +114,15 @@ rd-presets-check:
114114
go-unit-tests:
115115
go test ./pkg/registry/... ./pkg/config/... ./pkg/cmd/server/...
116116

117+
# Go tests for the controllers and supporting packages under ./internal.
118+
# Excludes ./pkg/... and ./cmd/... — those are run separately by
119+
# go-unit-tests above (pkg subset) and skipped (cmd) until their tests
120+
# stabilise. Run as its own step in CI alongside helm/bats unit tests;
121+
# locally invoke directly (`make test-controllers`) or chain
122+
# (`make unit-tests test-controllers`).
123+
test-controllers:
124+
go test ./internal/... -count=1
125+
117126
# Discover every hack/*.bats file that is NOT an e2e test and run it
118127
# through cozytest.sh. Drop a new *.bats file in hack/ and it is picked
119128
# up automatically on the next `make unit-tests` run.

‎api/apps/v1alpha1/tenant/types.go‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// Package v1alpha1 contains API Schema definitions for the
18+
// gateway.cozystack.io v1alpha1 API group.
19+
// +kubebuilder:object:generate=true
20+
// +groupName=gateway.cozystack.io
21+
package v1alpha1
22+
23+
import (
24+
"k8s.io/apimachinery/pkg/runtime/schema"
25+
"sigs.k8s.io/controller-runtime/pkg/scheme"
26+
)
27+
28+
var (
29+
// GroupVersion is group version used to register these objects.
30+
GroupVersion = schema.GroupVersion{Group: "gateway.cozystack.io", Version: "v1alpha1"}
31+
32+
// SchemeBuilder is used to add go types to the GroupVersionKind scheme.
33+
SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
34+
35+
// AddToScheme adds the types in this group-version to the given scheme.
36+
AddToScheme = SchemeBuilder.AddToScheme
37+
)
Lines changed: 259 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,259 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package v1alpha1
18+
19+
import (
20+
corev1 "k8s.io/api/core/v1"
21+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
22+
)
23+
24+
// CertMode selects how the per-tenant Gateway sources TLS certificates.
25+
// +kubebuilder:validation:Enum=http01;dns01
26+
type CertMode string
27+
28+
const (
29+
// CertModeHTTP01 issues per-listener Certificates via HTTP-01 ACME
30+
// against the tenant Gateway's `http` listener. Default. Each
31+
// published app gets its own listener and its own Certificate.
32+
CertModeHTTP01 CertMode = "http01"
33+
34+
// CertModeDNS01 issues a single wildcard Certificate per tenant
35+
// (apex + *.apex) via DNS-01 ACME. Operator must configure a DNS
36+
// provider in Spec.DNS01.
37+
CertModeDNS01 CertMode = "dns01"
38+
)
39+
40+
// IssuerName selects the Let's Encrypt environment the per-tenant
41+
// Issuer points at. Operators running staging or non-production
42+
// clusters should pick "letsencrypt-stage" to avoid eating into the
43+
// production rate limits.
44+
// +kubebuilder:validation:Enum=letsencrypt-prod;letsencrypt-stage
45+
type IssuerName string
46+
47+
const (
48+
IssuerNameLetsEncryptProd IssuerName = "letsencrypt-prod"
49+
IssuerNameLetsEncryptStage IssuerName = "letsencrypt-stage"
50+
)
51+
52+
// DNS01Provider names a supported cert-manager DNS-01 solver.
53+
// +kubebuilder:validation:Enum=cloudflare;route53;digitalocean;rfc2136
54+
type DNS01Provider string
55+
56+
// DNS01Config configures the DNS-01 solver when CertMode=dns01. Only
57+
// the field corresponding to Provider is read; others are ignored.
58+
type DNS01Config struct {
59+
// Provider selects which DNS-01 solver block to render.
60+
// +kubebuilder:default=cloudflare
61+
Provider DNS01Provider `json:"provider,omitempty"`
62+
63+
// Cloudflare config. Required when Provider=cloudflare.
64+
// +optional
65+
Cloudflare *CloudflareDNS01 `json:"cloudflare,omitempty"`
66+
67+
// Route53 config. Required when Provider=route53.
68+
// +optional
69+
Route53 *Route53DNS01 `json:"route53,omitempty"`
70+
71+
// DigitalOcean config. Required when Provider=digitalocean.
72+
// +optional
73+
DigitalOcean *DigitalOceanDNS01 `json:"digitalocean,omitempty"`
74+
75+
// RFC2136 config. Required when Provider=rfc2136.
76+
// +optional
77+
RFC2136 *RFC2136DNS01 `json:"rfc2136,omitempty"`
78+
}
79+
80+
// CloudflareDNS01 configures the cloudflare solver.
81+
type CloudflareDNS01 struct {
82+
// APITokenSecretRef references a Secret holding a Cloudflare API
83+
// token with Zone:Read + Zone:DNS:Edit on the tenant's apex zone.
84+
// +required
85+
APITokenSecretRef corev1.SecretKeySelector `json:"apiTokenSecretRef"`
86+
}
87+
88+
// Route53DNS01 configures the AWS Route53 solver.
89+
type Route53DNS01 struct {
90+
// Region is the AWS region of the hosted zone.
91+
// +required
92+
Region string `json:"region"`
93+
94+
// AccessKeyID is the IAM access key ID. Optional when running with
95+
// IRSA / instance profile.
96+
// +optional
97+
AccessKeyID string `json:"accessKeyID,omitempty"`
98+
99+
// SecretAccessKeySecretRef references a Secret holding the IAM
100+
// secret access key. Optional when running with IRSA / instance
101+
// profile.
102+
// +optional
103+
SecretAccessKeySecretRef *corev1.SecretKeySelector `json:"secretAccessKeySecretRef,omitempty"`
104+
}
105+
106+
// DigitalOceanDNS01 configures the DigitalOcean solver.
107+
type DigitalOceanDNS01 struct {
108+
// TokenSecretRef references a Secret holding a DigitalOcean API
109+
// token with write access to the tenant's apex domain.
110+
// +required
111+
TokenSecretRef corev1.SecretKeySelector `json:"tokenSecretRef"`
112+
}
113+
114+
// RFC2136DNS01 configures the BIND-style RFC 2136 dynamic-update solver.
115+
type RFC2136DNS01 struct {
116+
// Nameserver is the host:port of the authoritative nameserver
117+
// accepting dynamic updates.
118+
// +required
119+
Nameserver string `json:"nameserver"`
120+
121+
// TSIGKeyName names the TSIG key authorising the update.
122+
// +required
123+
TSIGKeyName string `json:"tsigKeyName"`
124+
125+
// TSIGAlgorithm is the TSIG HMAC algorithm. Default HMACSHA256.
126+
// +kubebuilder:default=HMACSHA256
127+
// +optional
128+
TSIGAlgorithm string `json:"tsigAlgorithm,omitempty"`
129+
130+
// TSIGSecretSecretRef references the Secret holding the TSIG key
131+
// material.
132+
// +required
133+
TSIGSecretSecretRef corev1.SecretKeySelector `json:"tsigSecretSecretRef"`
134+
}
135+
136+
// TenantGatewaySpec describes the desired state of a per-tenant Gateway.
137+
type TenantGatewaySpec struct {
138+
// Apex is the tenant's apex hostname. The Gateway listeners are
139+
// constrained to this apex and its subdomains.
140+
// +required
141+
Apex string `json:"apex"`
142+
143+
// CertMode selects between HTTP-01 per-listener Certificates and
144+
// a wildcard cert via DNS-01.
145+
// +kubebuilder:default=http01
146+
CertMode CertMode `json:"certMode,omitempty"`
147+
148+
// IssuerName picks the Let's Encrypt environment the controller
149+
// configures the per-tenant Issuer with. Defaults to
150+
// letsencrypt-prod. Set to letsencrypt-stage on dev clusters to
151+
// avoid the LE production rate limits.
152+
// +kubebuilder:default=letsencrypt-prod
153+
IssuerName IssuerName `json:"issuerName,omitempty"`
154+
155+
// DNS01 configures the DNS-01 solver when CertMode=dns01. Ignored
156+
// otherwise. Required (provider + matching config block) when
157+
// CertMode=dns01.
158+
// +optional
159+
DNS01 *DNS01Config `json:"dns01,omitempty"`
160+
161+
// AttachedNamespaces lists namespace names that are allowed to
162+
// attach HTTPRoute or TLSRoute to this tenant's Gateway. The
163+
// publishing tenant namespace is implicit. Selector is by built-in
164+
// kubernetes.io/metadata.name (kube-apiserver-written, unspoofable).
165+
// +optional
166+
AttachedNamespaces []string `json:"attachedNamespaces,omitempty"`
167+
168+
// TLSPassthroughServices names services exposed via TLS-passthrough
169+
// (mode: Passthrough listeners). Each service gets a dedicated
170+
// listener; HTTPRoutes attach to TLS-terminate listeners instead.
171+
// +optional
172+
TLSPassthroughServices []string `json:"tlsPassthroughServices,omitempty"`
173+
174+
// GatewayClassName names the GatewayClass to attach the rendered
175+
// Gateway to. Default cilium.
176+
// +kubebuilder:default=cilium
177+
// +optional
178+
GatewayClassName string `json:"gatewayClassName,omitempty"`
179+
}
180+
181+
// TenantGatewayListenerStatus reports the observed state of a single
182+
// listener on the tenant's Gateway.
183+
type TenantGatewayListenerStatus struct {
184+
// Name is the listener's name (e.g. "https-harbor", "https-apex").
185+
Name string `json:"name"`
186+
187+
// Hostname is the hostname this listener serves.
188+
// +optional
189+
Hostname string `json:"hostname,omitempty"`
190+
191+
// Ready indicates the cert is issued and the Gateway has accepted
192+
// the listener.
193+
Ready bool `json:"ready"`
194+
195+
// CertificateName names the cert-manager Certificate backing this
196+
// listener.
197+
// +optional
198+
CertificateName string `json:"certificateName,omitempty"`
199+
200+
// Reason is a short machine-readable reason when Ready=false.
201+
// +optional
202+
Reason string `json:"reason,omitempty"`
203+
}
204+
205+
// TenantGatewayStatus reports the observed state of the tenant's Gateway.
206+
type TenantGatewayStatus struct {
207+
// ObservedGeneration mirrors the .metadata.generation reflected in
208+
// the latest reconciled state.
209+
// +optional
210+
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
211+
212+
// Conditions describes the current state of the TenantGateway.
213+
// Standard condition types: Ready, Programmed.
214+
// +optional
215+
// +patchMergeKey=type
216+
// +patchStrategy=merge
217+
// +listType=map
218+
// +listMapKey=type
219+
Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
220+
221+
// Listeners reports per-listener readiness and cert state.
222+
// +optional
223+
// +listType=map
224+
// +listMapKey=name
225+
Listeners []TenantGatewayListenerStatus `json:"listeners,omitempty"`
226+
}
227+
228+
// +kubebuilder:object:root=true
229+
// +kubebuilder:subresource:status
230+
// +kubebuilder:resource:scope=Namespaced,shortName=tgw
231+
// +kubebuilder:printcolumn:name="Apex",type="string",JSONPath=".spec.apex"
232+
// +kubebuilder:printcolumn:name="Mode",type="string",JSONPath=".spec.certMode"
233+
// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=`.status.conditions[?(@.type=="Ready")].status`
234+
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
235+
236+
// TenantGateway is the Schema for the tenantgateways API.
237+
// It expresses the operator-facing intent for a tenant's per-namespace
238+
// Gateway; the cozystack-controller reconciles the actual Gateway and
239+
// per-listener Certificate resources from this CR.
240+
type TenantGateway struct {
241+
metav1.TypeMeta `json:",inline"`
242+
metav1.ObjectMeta `json:"metadata,omitempty"`
243+
244+
Spec TenantGatewaySpec `json:"spec,omitempty"`
245+
Status TenantGatewayStatus `json:"status,omitempty"`
246+
}
247+
248+
// +kubebuilder:object:root=true
249+
250+
// TenantGatewayList contains a list of TenantGateway.
251+
type TenantGatewayList struct {
252+
metav1.TypeMeta `json:",inline"`
253+
metav1.ListMeta `json:"metadata,omitempty"`
254+
Items []TenantGateway `json:"items"`
255+
}
256+
257+
func init() {
258+
SchemeBuilder.Register(&TenantGateway{}, &TenantGatewayList{})
259+
}

0 commit comments

Comments
 (0)