Skip to content

Commit 1c136c3

Browse files
lexfreilexfrei
committed
fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex
The Gateway (cozystack-gateway-hostname-policy) and Route (cozystack-route-hostname-policy) admission policies constrain tenant hostnames on the Gateway API path. Gateway API is opt-in and off by default, so in the default configuration tenant applications publish through a legacy networking.k8s.io/Ingress on the shared ingress-nginx, which had no equivalent policy. Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[] is allowed when it is within the namespace's own apex (namespace.cozystack.io/host), or when it is a concrete (non-wildcard) name entirely outside the platform root apex (_cluster.root-host); a hostname under the platform root apex but outside the namespace's own apex, and a wildcard taken through the outside-root path (which could match the platform apex itself), are rejected. The outside-root branch keeps the kubernetes app's Proxied exposeMethod working, since it routes a user-supplied external domain to a nested cluster. A rule with no host (absent or empty, an unbounded catch-all) and spec.defaultBackend are also rejected. The policy templates the platform root apex into its CEL and renders only when _cluster.root-host is set, so it is never shipped in a state that would admit every hostname. A tenant-* namespace missing its own apex label is denied. Cover the policy with helm-unittest (render, including the root-host, empty-host, and wildcard guards) and behavioral admission tests against the live apiserver (in-apex allow, external-domain allow, own-apex-wildcard allow, external-tls allow, under-root-outside-apex deny, wildcard-matching-apex deny, hostless deny, empty-string-host deny, defaultBackend deny), and document it as an additional layer in the gateway security model. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 10f8e2d commit 1c136c3

4 files changed

Lines changed: 672 additions & 1 deletion

File tree

‎hack/e2e-apps/gateway.bats‎

Lines changed: 358 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -401,6 +401,364 @@ EOF
401401
kubectl -n tenant-test delete httproute route-hostname-allow-probe --ignore-not-found
402402
}
403403

404+
@test "cozystack-ingress-hostname-policy VAP and binding are installed" {
405+
# Legacy-Ingress counterpart of the gateway/route hostname VAPs. Gateway
406+
# API is off by default, so tenant apps publish through a legacy
407+
# networking.k8s.io/Ingress; if these resources are missing, admission
408+
# cannot constrain tenant Ingress hostnames on the shared ingress-nginx.
409+
# Make that loud instead of letting it fall through as a silent pass.
410+
kubectl get validatingadmissionpolicy cozystack-ingress-hostname-policy -o yaml
411+
kubectl get validatingadmissionpolicybinding cozystack-ingress-hostname-policy -o yaml
412+
actions=$(kubectl get validatingadmissionpolicybinding cozystack-ingress-hostname-policy -o jsonpath='{.spec.validationActions}')
413+
echo "binding.validationActions=$actions"
414+
case "$actions" in
415+
*Deny*) ;;
416+
*) echo "SETUP FAILURE: binding.validationActions lacks Deny (got '$actions')" >&2; return 1 ;;
417+
esac
418+
}
419+
420+
@test "cozystack-ingress-hostname-policy VAP rejects an Ingress with a foreign-apex host" {
421+
# tenant-test's own apex (namespace.cozystack.io/host) is
422+
# test.example.org; the platform root apex is example.org.
423+
# dashboard.example.org sits under the platform root apex but outside
424+
# the tenant's own apex, so it must be denied. As with the route tests,
425+
# use `if !` so the dash `set -e` runner captures the expected-rejection
426+
# exit status.
427+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
428+
kubectl -n tenant-test delete ingress ingress-out-of-apex-host-probe --ignore-not-found
429+
if ! output=$(kubectl apply -f - 2>&1 <<'EOF'
430+
apiVersion: networking.k8s.io/v1
431+
kind: Ingress
432+
metadata:
433+
name: ingress-out-of-apex-host-probe
434+
namespace: tenant-test
435+
spec:
436+
rules:
437+
- host: "dashboard.example.org"
438+
http:
439+
paths:
440+
- path: /
441+
pathType: Prefix
442+
backend:
443+
service:
444+
name: noop
445+
port:
446+
number: 80
447+
EOF
448+
); then
449+
echo "$output" | grep -qi "ValidatingAdmissionPolicy"
450+
echo "$output" | grep -qi "must be within"
451+
else
452+
kubectl -n tenant-test delete ingress ingress-out-of-apex-host-probe --ignore-not-found
453+
echo "BUG: admission accepted an out-of-apex Ingress host — ingress was created in tenant-test" >&2
454+
echo "$output" >&2
455+
return 1
456+
fi
457+
}
458+
459+
@test "cozystack-ingress-hostname-policy VAP rejects an Ingress with a foreign-apex tls host" {
460+
# The rule host is in-apex, but a spec.tls[].hosts[] entry
461+
# (harbor.example.org) sits under the platform root apex outside the
462+
# tenant's own apex. The tls branch must deny it on its own — an Ingress
463+
# carries a hostname in spec.tls[].hosts[] as well as spec.rules[].host,
464+
# so both are constrained the same way.
465+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
466+
kubectl -n tenant-test delete ingress ingress-tls-out-of-apex-probe --ignore-not-found
467+
if ! output=$(kubectl apply -f - 2>&1 <<'EOF'
468+
apiVersion: networking.k8s.io/v1
469+
kind: Ingress
470+
metadata:
471+
name: ingress-tls-out-of-apex-probe
472+
namespace: tenant-test
473+
spec:
474+
tls:
475+
- hosts:
476+
- "harbor.example.org"
477+
secretName: noop-tls
478+
rules:
479+
- host: "harbor.test.example.org"
480+
http:
481+
paths:
482+
- path: /
483+
pathType: Prefix
484+
backend:
485+
service:
486+
name: noop
487+
port:
488+
number: 80
489+
EOF
490+
); then
491+
echo "$output" | grep -qi "ValidatingAdmissionPolicy"
492+
echo "$output" | grep -qi "must be within"
493+
else
494+
kubectl -n tenant-test delete ingress ingress-tls-out-of-apex-probe --ignore-not-found
495+
echo "BUG: admission accepted an out-of-apex Ingress tls host in tenant-test" >&2
496+
echo "$output" >&2
497+
return 1
498+
fi
499+
}
500+
501+
@test "cozystack-ingress-hostname-policy VAP rejects a hostless catch-all Ingress rule" {
502+
# A rule with no .host matches all inbound HTTP on the shared
503+
# ingress-nginx — an unbounded catch-all. Every rule must carry an
504+
# in-apex host, so a hostless rule is denied.
505+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
506+
kubectl -n tenant-test delete ingress ingress-hostless-probe --ignore-not-found
507+
if ! output=$(kubectl apply -f - 2>&1 <<'EOF'
508+
apiVersion: networking.k8s.io/v1
509+
kind: Ingress
510+
metadata:
511+
name: ingress-hostless-probe
512+
namespace: tenant-test
513+
spec:
514+
rules:
515+
- http:
516+
paths:
517+
- path: /
518+
pathType: Prefix
519+
backend:
520+
service:
521+
name: noop
522+
port:
523+
number: 80
524+
EOF
525+
); then
526+
echo "$output" | grep -qi "ValidatingAdmissionPolicy"
527+
else
528+
kubectl -n tenant-test delete ingress ingress-hostless-probe --ignore-not-found
529+
echo "BUG: admission accepted a hostless catch-all Ingress rule in tenant-test" >&2
530+
echo "$output" >&2
531+
return 1
532+
fi
533+
}
534+
535+
@test "cozystack-ingress-hostname-policy VAP rejects an empty-string rule host" {
536+
# host: "" is a valid Ingress rule and a catch-all on the shared
537+
# ingress-nginx, the same as an absent host. Without an explicit empty
538+
# guard it would satisfy the "outside the root apex" branch and be
539+
# admitted, so the CEL denies it via r.host != "" (and via has(r.host)
540+
# if the apiserver drops the empty value). Either way, expect denial.
541+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
542+
kubectl -n tenant-test delete ingress ingress-empty-host-probe --ignore-not-found
543+
if ! output=$(kubectl apply -f - 2>&1 <<'EOF'
544+
apiVersion: networking.k8s.io/v1
545+
kind: Ingress
546+
metadata:
547+
name: ingress-empty-host-probe
548+
namespace: tenant-test
549+
spec:
550+
rules:
551+
- host: ""
552+
http:
553+
paths:
554+
- path: /
555+
pathType: Prefix
556+
backend:
557+
service:
558+
name: noop
559+
port:
560+
number: 80
561+
EOF
562+
); then
563+
echo "$output" | grep -qi "ValidatingAdmissionPolicy"
564+
else
565+
kubectl -n tenant-test delete ingress ingress-empty-host-probe --ignore-not-found
566+
echo "BUG: admission accepted an empty-string rule host in tenant-test" >&2
567+
echo "$output" >&2
568+
return 1
569+
fi
570+
}
571+
572+
@test "cozystack-ingress-hostname-policy VAP rejects an Ingress default backend" {
573+
# spec.defaultBackend is a catch-all for otherwise-unmatched traffic on
574+
# the shared ingress-nginx; a tenant may not declare one.
575+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
576+
kubectl -n tenant-test delete ingress ingress-defaultbackend-probe --ignore-not-found
577+
if ! output=$(kubectl apply -f - 2>&1 <<'EOF'
578+
apiVersion: networking.k8s.io/v1
579+
kind: Ingress
580+
metadata:
581+
name: ingress-defaultbackend-probe
582+
namespace: tenant-test
583+
spec:
584+
defaultBackend:
585+
service:
586+
name: noop
587+
port:
588+
number: 80
589+
EOF
590+
); then
591+
echo "$output" | grep -qi "ValidatingAdmissionPolicy"
592+
else
593+
kubectl -n tenant-test delete ingress ingress-defaultbackend-probe --ignore-not-found
594+
echo "BUG: admission accepted an Ingress with spec.defaultBackend in tenant-test" >&2
595+
echo "$output" >&2
596+
return 1
597+
fi
598+
}
599+
600+
@test "cozystack-ingress-hostname-policy VAP allows an in-apex Ingress" {
601+
# Inverse sanity check: an Ingress whose rule host and tls hosts are all
602+
# under test.example.org must NOT be rejected by the VAP.
603+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
604+
kubectl -n tenant-test delete ingress ingress-hostname-allow-probe --ignore-not-found
605+
kubectl apply -f - <<'EOF'
606+
apiVersion: networking.k8s.io/v1
607+
kind: Ingress
608+
metadata:
609+
name: ingress-hostname-allow-probe
610+
namespace: tenant-test
611+
spec:
612+
tls:
613+
- hosts:
614+
- "harbor.test.example.org"
615+
secretName: noop-tls
616+
rules:
617+
- host: "harbor.test.example.org"
618+
http:
619+
paths:
620+
- path: /
621+
pathType: Prefix
622+
backend:
623+
service:
624+
name: noop
625+
port:
626+
number: 80
627+
EOF
628+
629+
kubectl -n tenant-test delete ingress ingress-hostname-allow-probe --ignore-not-found
630+
}
631+
632+
@test "cozystack-ingress-hostname-policy VAP allows an Ingress host outside the platform root apex" {
633+
# Narrow rule: a tenant may route its own domain that lies entirely
634+
# outside the platform root apex (example.org). This is the path the
635+
# kubernetes app's Proxied exposeMethod uses — addons.ingressNginx.hosts
636+
# routes a user-supplied external domain to a nested cluster — and it
637+
# must be admitted. shop.example.net is not under example.org.
638+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
639+
kubectl -n tenant-test delete ingress ingress-external-host-probe --ignore-not-found
640+
kubectl apply -f - <<'EOF'
641+
apiVersion: networking.k8s.io/v1
642+
kind: Ingress
643+
metadata:
644+
name: ingress-external-host-probe
645+
namespace: tenant-test
646+
spec:
647+
rules:
648+
- host: "shop.example.net"
649+
http:
650+
paths:
651+
- path: /
652+
pathType: Prefix
653+
backend:
654+
service:
655+
name: noop
656+
port:
657+
number: 80
658+
EOF
659+
660+
kubectl -n tenant-test delete ingress ingress-external-host-probe --ignore-not-found
661+
}
662+
663+
@test "cozystack-ingress-hostname-policy VAP rejects a wildcard host that matches the platform apex" {
664+
# *.org is "outside" the platform root apex example.org by a plain string
665+
# test, but at ingress-nginx *.org also matches the bare root apex
666+
# example.org. The outside-root path is gated on the host not being a
667+
# wildcard, so this must be denied. (A wildcard under the tenant's own
668+
# apex, e.g. *.test.example.org, is still allowed via the own-apex path.)
669+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
670+
kubectl -n tenant-test delete ingress ingress-wildcard-host-probe --ignore-not-found
671+
if ! output=$(kubectl apply -f - 2>&1 <<'EOF'
672+
apiVersion: networking.k8s.io/v1
673+
kind: Ingress
674+
metadata:
675+
name: ingress-wildcard-host-probe
676+
namespace: tenant-test
677+
spec:
678+
rules:
679+
- host: "*.org"
680+
http:
681+
paths:
682+
- path: /
683+
pathType: Prefix
684+
backend:
685+
service:
686+
name: noop
687+
port:
688+
number: 80
689+
EOF
690+
); then
691+
echo "$output" | grep -qi "ValidatingAdmissionPolicy"
692+
else
693+
kubectl -n tenant-test delete ingress ingress-wildcard-host-probe --ignore-not-found
694+
echo "BUG: admission accepted a wildcard host matching the platform apex in tenant-test" >&2
695+
echo "$output" >&2
696+
return 1
697+
fi
698+
}
699+
700+
@test "cozystack-ingress-hostname-policy VAP allows a wildcard host under the own apex" {
701+
# A wildcard under the tenant's own apex (*.test.example.org) is admitted
702+
# via the own-apex path — the anti-wildcard gate applies only to the
703+
# outside-root path. Locks the behavior the CEL comment describes.
704+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
705+
kubectl -n tenant-test delete ingress ingress-ownapex-wildcard-probe --ignore-not-found
706+
kubectl apply -f - <<'EOF'
707+
apiVersion: networking.k8s.io/v1
708+
kind: Ingress
709+
metadata:
710+
name: ingress-ownapex-wildcard-probe
711+
namespace: tenant-test
712+
spec:
713+
rules:
714+
- host: "*.test.example.org"
715+
http:
716+
paths:
717+
- path: /
718+
pathType: Prefix
719+
backend:
720+
service:
721+
name: noop
722+
port:
723+
number: 80
724+
EOF
725+
726+
kubectl -n tenant-test delete ingress ingress-ownapex-wildcard-probe --ignore-not-found
727+
}
728+
729+
@test "cozystack-ingress-hostname-policy VAP allows an external domain on spec.tls[].hosts" {
730+
# The outside-root allowance covers spec.tls[].hosts[] as well as
731+
# spec.rules[].host: a concrete external domain not under the platform
732+
# root apex must be admitted on the tls block too.
733+
# Pre-clean a stale probe left by an interrupted prior run (e2e convention).
734+
kubectl -n tenant-test delete ingress ingress-external-tls-probe --ignore-not-found
735+
kubectl apply -f - <<'EOF'
736+
apiVersion: networking.k8s.io/v1
737+
kind: Ingress
738+
metadata:
739+
name: ingress-external-tls-probe
740+
namespace: tenant-test
741+
spec:
742+
tls:
743+
- hosts:
744+
- "shop.example.net"
745+
secretName: noop-tls
746+
rules:
747+
- host: "shop.example.net"
748+
http:
749+
paths:
750+
- path: /
751+
pathType: Prefix
752+
backend:
753+
service:
754+
name: noop
755+
port:
756+
number: 80
757+
EOF
758+
759+
kubectl -n tenant-test delete ingress ingress-external-tls-probe --ignore-not-found
760+
}
761+
404762
@test "HTTPRoute with a matching parentRef reaches Accepted status" {
405763
# Put a Gateway and a route in the same namespace so allowedRoutes: Same accepts them.
406764
kubectl apply -f - <<'EOF'

‎packages/extra/gateway/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ Tenants in Cozystack interact with the platform exclusively through `apps.cozyst
8181
The protections below split into three groups by who they defend against:
8282

8383
- **Tenant-user-input gates** — Layer 4 (`cozystack-tenant-host-policy`). `Tenant.spec.host` is the user-supplied field that surfaces as a security boundary at the hostname layer; it is gated on every Create / Update via `cozystack-api`'s admission chain (`pkg/registry/apps/application/rest.go`).
84-
- **Defense-in-depth** — Layers 1, 2, 5, 6, 7. These do not protect against tenant-user input (tenants don't hold the relevant RBAC). They guard against bugs in cozystack-controller / Flux, supply-chain compromise of an app chart that emits Gateway API resources, and confused-deputy mistakes by a cluster admin. Fail-closed via `failurePolicy: Fail` + `validationActions: [Deny]`.
84+
- **Defense-in-depth** — Layers 1, 2, 5, 6, 7, 8. These do not protect against tenant-user input (tenants don't hold the relevant RBAC). They guard against bugs in cozystack-controller / Flux, supply-chain compromise of an app chart that emits Gateway API or Ingress resources, and confused-deputy mistakes by a cluster admin. Fail-closed via `failurePolicy: Fail` + `validationActions: [Deny]`. Layers 1-7 cover the opt-in Gateway API dataplane; Layer 8 covers the legacy Ingress dataplane, which is the default (Gateway API is off by default).
8585
- **Admin-against-themselves** — Layer 3 (`cozystack-gateway-attached-namespaces-policy`). Rejects a `kubectl edit packages.cozystack.io` that would slip a `tenant-*` entry into the platform Package's `gateway.attachedNamespaces`. Layer 6 catches the same misconfiguration at helm render time.
8686

8787
1. **Namespace whitelist on listeners.** Every listener carries an `allowedRoutes.namespaces.from: Selector` matching the built-in `kubernetes.io/metadata.name` label (written by kube-apiserver, unspoofable). HTTPS / TLS-passthrough listeners accept routes from the publishing tenant's namespace plus `gateway.attachedNamespaces` in the platform chart (default includes the `cozy-*` namespaces for platform services and `default` for the Kubernetes API TLSRoute). A namespace outside the list literally cannot attach any `HTTPRoute` or `TLSRoute` to those listeners. The plain-HTTP listener (port 80) carries a strictly narrower selector — only the tenant namespace itself (where the controller-owned http→https redirect HTTPRoute lives) and `cozy-cert-manager` (HTTP-01 ACME challenge HTTPRoutes) — so app HTTPRoutes attaching by hostname cannot bind to port 80 and serve plaintext. HTTPS listeners additionally restrict `allowedRoutes.kinds` to `HTTPRoute` (and TLS-passthrough listeners to `TLSRoute`), preventing GRPCRoute / TCPRoute / UDPRoute from attaching outside the route-hostname VAP's coverage.
@@ -91,6 +91,7 @@ The protections below split into three groups by who they defend against:
9191
5. **`cozystack-namespace-host-label-policy`** — VAP on core `v1 Namespace` CREATE/UPDATE. Rejects any set or change of the `namespace.cozystack.io/host` label, except by the same trusted-caller whitelist as layer 4. This closes both first-time label writes on CREATE and first-time adds on UPDATE — only cozystack/Flux service accounts (which apply the tenant chart) can stamp the label.
9292
6. **Render-time `fail` in cozystack-basics.** The cozystack-basics chart fails the helm render if `_cluster.gateway-attached-namespaces` contains any `tenant-*` entry. Triggers on the helm-install path before the cluster ever sees the values — complements layer 3 which triggers at `kubectl apply` time.
9393
7. **`cozystack-route-hostname-policy`** — VAP on `gateway.networking.k8s.io/v1 HTTPRoute` and `v1alpha2 TLSRoute` CREATE/UPDATE. Scoped to `tenant-*` namespaces (cozy-* are cluster-admin-managed and trusted to publish under any apex). Rejects any `spec.hostnames` entry that is not equal to the namespace's `namespace.cozystack.io/host` label or a subdomain of it. Defense-in-depth against an app chart bug or supply-chain compromise that emits Gateway API resources outside the tenant's apex — tenants in Cozystack do not hold `gateway.networking.k8s.io/*` RBAC by design, so this is not a tenant-user defense. The within-apex cross-namespace case (a tenant chart claiming a hostname that is published by a `cozy-*` app) is handled by the controller at reconciliation time: when two routes from different namespaces claim the same hostname, the `cozy-*` namespace wins and the loser receives a `HostnameConflict` condition under the controller's name in `Status.Parents`.
94+
8. **`cozystack-ingress-hostname-policy`** — VAP on core `networking.k8s.io/v1 Ingress` CREATE/UPDATE. Gateway API is opt-in and **off by default**, so in the default configuration tenant applications publish through a legacy Ingress on the shared ingress-nginx; layers 1-7 constrain tenant hostnames on the opt-in Gateway path, and this VAP adds a hostname constraint on the default Ingress path. Scoped to `tenant-*` namespaces (cozy-* are cluster-admin-managed and trusted). A hostname on `spec.rules[].host` or `spec.tls[].hosts[]` is allowed when it is within the namespace's own `namespace.cozystack.io/host` apex, OR when it lies entirely outside the platform root apex (`_cluster.root-host`) — the second case lets a tenant route its own external custom domain (for example the `kubernetes` app's Proxied `addons.ingressNginx.hosts`, which routes a user-supplied domain to a nested cluster). A hostname that falls under the platform root apex but outside the namespace's own apex is rejected; a rule with no host (an unbounded catch-all) and `spec.defaultBackend` (a catch-all for unmatched traffic) are also rejected. Fail-closed: a `tenant-*` namespace missing its host label is denied, and the policy renders only when `_cluster.root-host` is set. This bounds which apexes a tenant Ingress may claim under the platform domain; it does not attempt to resolve every possible hostname collision.
9495

9596
For `tenant-root` the allowed host suffix is `publishing.host`; for any `tenant-<name>` that inherits from its parent the suffix is `<name>.<parent apex>`. A child tenant with an independent apex (`customer1.io` instead of a subdomain) is handled correctly because the VAP reads the per-namespace label rather than assuming a subdomain hierarchy.
9697

0 commit comments

Comments
 (0)