cozystack-tenant-host-policy VAP is always-on (not gated behind gateway.enabled) — silent breaking change for tenant-scoped tenant creation on v1.4→v1.5 #3252

Summary

cozystack-tenant-host-policy (the ValidatingAdmissionPolicy that restricts Tenant.spec.host to trusted callers, introduced in #2470 as layer 4 of the Gateway API security model) is rendered unconditionally. It ships in cozystack-basics, a core system package that is always installed, with no gateway.enabled guard. Combined with the rest.go admission-chain fix from the same PR (which makes apps.cozystack.io Create actually go through admission), a plain v1.4 → v1.5 upgrade silently takes away the ability of tenant-scoped service accounts (including tenant-root) to set spec.host on tenants or nested tenants.

This contradicts #2470's stated promise that "existing clusters see no behavioural change until an operator opts in," and it is not flagged as a breaking change in the v1.5.0 changelog.

To be clear: the security boundary itself is sound and I'm not asking to revert it. The ask is to make the rollout honest — either gate it like the rest of the feature, or document it as breaking with a migration path.

Environment

• Cozystack v1.5.2 (also present in the whole v1.5.x line; landed in v1.5.0)
• Upgraded from v1.4.x; Gateway API not opted into (publishing.gateway.enabled left at default)

Steps to reproduce

1. On a v1.5.x cluster, authenticate as a tenant-scoped SA — e.g. system:serviceaccount:tenant-root:tenant-root (the credential many operators use day-to-day).
2. Create a tenant (or nested tenant) with a custom spec.host.

Result:

Failed: tenants.apps.cozystack.io "cozytester" is forbidden: ValidatingAdmissionPolicy
'cozystack-tenant-host-policy' with binding 'cozystack-tenant-host-policy' denied request:
tenant.spec.host can only be set or changed by cluster-admins (system:masters) or by
cozystack/Flux service accounts; request user is system:serviceaccount:tenant-root:tenant-root.
A tenant with empty spec.host inherits the apex from its parent tenant.

Creating a tenant with an empty spec.host still works (it derives <name>.<parent-apex>); only setting a custom host is denied.

Root cause

Two always-on changes from #2470 (commit d2e851f6), neither gated behind the opt-in Gateway feature:

1. The VAP renders unconditionally. packages/system/cozystack-basics/templates/gateway-hostname-policy.yaml has no {{- if .Values.gateway.enabled }} guard around the cozystack-tenant-host-policy (or cozystack-namespace-host-label-policy) blocks. cozystack-basics has no gateway key in its values at all, and it is installed by default via packages/core/platform/templates/bundles/system.yaml.

2. The admission-chain fix makes it bite on Create. pkg/registry/apps/application/rest.go previously ignored the createValidation callback, silently bypassing admission on Create for every apps.cozystack.io resource. feat(networking): Gateway API support via Cilium (supersedes #2213) #2470 fixes this (aligning Create/Delete with the existing Update path). So the VAP now actually fires on kubectl create tenant, where before v1.5.0 a Create would have skipped admission entirely.

The policy's trustedCaller allows only these groups: system:masters, system:serviceaccounts:cozy-system, system:serviceaccounts:cozy-cert-manager, system:serviceaccounts:cozy-fluxcd, system:serviceaccounts:kube-system. Every tenant SA — including tenant-root — is excluded by design.

Why this is unexpected

• feat(networking): Gateway API support via Cilium (supersedes #2213) #2470's commit body: "Everything else is gated behind gateway.enabled=false and tenant.spec.gateway=false; existing clusters see no behavioural change until an operator opts in." — This does not hold for the spec.host admission gate, which is always-on.
• The v1.5.0 changelog mentions the admission-chain change generically ("cozystack-api now invokes admission on Create and Delete … so any custom VAPs/webhooks on those kinds will now fire on all three verbs") but does not call out that platform-shipped policy now blocks tenant-scoped host writes — a real behavioral break for anyone scripting tenant creation with tenant credentials.

Impact

Any workflow that creates tenants/nested tenants with a custom spec.host using tenant-scoped credentials (very common — tenant-root is the usual operator credential) breaks on upgrade to v1.5.x, with no advance notice and no documented workaround.

Note on "cluster-admin" — an easy trap

The VAP matches the literal group system:masters, not the cluster-admin ClusterRole. A user granted cluster-admin via RBAC, or a Keycloak/OIDC admin, is not in system:masters and is still denied. Only an identity whose credential carries the system:masters group (e.g. the raw admin kubeconfig) — or the management Flux SA (cozy-fluxcd) via GitOps — can set a custom host. Worth making explicit wherever the migration path is documented.

Proposed resolution (pick one)

1. Gate it like the rest of the feature — wrap the host-write VAPs (cozystack-tenant-host-policy, cozystack-namespace-host-label-policy) behind the same opt-in the rest of feat(networking): Gateway API support via Cilium (supersedes #2213) #2470 uses, so upgrading clusters that haven't adopted Gateway API keep prior behavior; or
2. Document it as a breaking change — add it to the v1.5.0 upgrade notes with the supported migration path:
   • leave spec.host empty → the tenant inherits the derived apex <name>.<parent-apex>; or
   • set a custom host via a system:masters credential or through the management Flux (cozy-fluxcd) GitOps repo (custom-apex tenants also need gateway: true for public exposure).

Happy to send a PR for whichever direction you prefer.

Refs: #2470, commit d2e851f6, packages/system/cozystack-basics/templates/gateway-hostname-policy.yaml, pkg/registry/apps/application/rest.go