Skip to content

Commit ded2354

Browse files
authored
fix(tenant): skip Cilium policies on isp-hosted without Cilium (cozystack#4710)
## What this PR does The tenant chart no longer fails to install on a hosted cluster without Cilium. The isp-hosted variants install no CNI of their own, but the chart rendered its Cilium policies unconditionally, so `tenant-root` failed with `no matches for kind "CiliumNetworkPolicy" in version "cilium.io/v2"`. Everything that depends on `cozystack-basics` then never converged. This also unblocks the isp-hosted e2e lane, which fails on exactly this. The guard is keyed on the platform variant. The platform already passes it to every tenant as `_cluster.bundle-name`. It covers every Cilium object the chart renders, including the shared-central tracing egress policies. - On variants where Cozystack installs Cilium itself, nothing changes and a missing CRD still fails the install. - On `isp-hosted` and `isp-hosted-slim`, the policies render only when the host serves both Cilium policy kinds, since a hosted cluster may run Cilium. The render on Cilium variants is byte-identical to `main`. I checked it with `helm template` over all four Cilium variants and an absent `_cluster`, with optional services on and off, with and without the Cilium API, for root, child and grandchild tenants. I didn't use the capability check alone because it fails open. helm-controller upgrades a deployed release only when the chart or the values digest changes, so a change in API discovery never triggers a re-render. That is `VerifyRelease` in `internal/action/verify.go` at v1.5.0, the version Cozystack ships. If the Cilium CRDs were not discoverable at install time on a Cilium cluster, a capability-only guard would leave the tenant without policies until the next chart or values change. An upgrade rendered in that window would delete existing ones. On isp-hosted without Cilium, tenants get no network isolation from Cozystack. Isolating them is up to the host. I didn't fall back to standard `NetworkPolicy`, because the host CNI may not enforce it. The tenant policies also use Cilium features that don't map one to one: the `world` and `kube-apiserver` entities, clusterwide policies, selectors on ancestor namespace labels. A partial translation would look like isolation without really being one. The tenant README now documents this, including that Cilium added to the host later takes effect only on the tenant's next chart or values change. The threat model and the self-assessment list it under non-goals. The new helm-unittest suite fails on `main` for the hosted cases without Cilium. Each guard mutation I tried turns at least one case red: no variant check, no capability check, `and` changed to `or`, `isp-hosted-slim` dropped from the list. I didn't add an explicit "require Cilium" value, since the Cilium variants already behave that way. One thing to keep in mind: the list of hosted variants now lives in the platform bundle and in the tenant helper, so a new hosted variant has to go into both. A platform `_cluster` flag saying whether Cozystack manages the CNI would remove that, but it's a platform change. Closes cozystack#3387 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The tenant README change reaches the website through the release bot that regenerates the managed-apps pages from each package README. ### Release note ```release-note fix(tenant): the tenant chart no longer fails to install on the isp-hosted variants when the host has no Cilium. There, the Cilium network policies are created only if the host serves the Cilium policy API; otherwise tenants get no network isolation from Cozystack. Other variants are unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Tenant network-isolation policies are rendered for `isp-hosted` and `isp-hosted-slim` only when the required Cilium APIs are available. If Cilium becomes available later, policies are applied after a subsequent chart or values change. Tracing policies also require the Cilium APIs. * **Documentation** * Clarified when Cozystack provides tenant network isolation and that isolation is the host’s responsibility when the required Cilium APIs are unavailable. * **Tests** * Added coverage for policy rendering across bundle types and Cilium API availability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 11910e0 + 3b5af9d commit ded2354

8 files changed

Lines changed: 125 additions & 2 deletions

File tree

‎docs/security/self-assessment.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ Actors are decomposed by **trust boundary**: two components are distinct actors
9393
- **Upstream-only vulnerabilities** not introduced or worsened by Cozystack packaging or defaults.
9494
- **Ingress-side isolation of tenant pods.** Tenant-pod ingress is open to `world` and `cluster`, so a tenant pod is reachable by any in-cluster peer whose own egress permits it (platform components) or by a workload not governed by Cozystack's tenant egress policies. Cross-tenant isolation is enforced on the *egress* side, not tenant-pod ingress.
9595
- **Blocking cross-tenant reachability through externally-exposed services.** If a tenant publishes a LoadBalancer external IP, another tenant can reach it via `world` egress.
96+
- **Network isolation between tenants on a hosted cluster without Cilium.** On the `isp-hosted` and `isp-hosted-slim` variants the CNI belongs to the host, and tenant Cilium policies are created only if the host serves the Cilium policy API. Without it, tenants get no network isolation from Cozystack, and isolating them is up to the host.
9697
- **Noisy-neighbour / side-channel** guarantees for co-located tenant pods beyond namespace scoping (and, for tenant Kubernetes/VMs, virtualization).
9798
- Acting as an **egress firewall**: tenant workloads may egress to `world` by default.
9899
- Supporting **pre-release builds and end-of-life versions**.

‎docs/security/threat-model.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ The following are explicitly out of scope for the platform's security guarantees
8383
- **Upstream-only vulnerabilities** that Cozystack packaging and defaults do not introduce or worsen.
8484
- **Ingress-side isolation of tenant pods.** Tenant-pod ingress is open to the `world` and `cluster` entities, so a tenant pod is reachable by any in-cluster peer whose own egress permits it (platform components) and by a workload not governed by Cozystack's tenant egress policies (e.g. a privileged or system pod). Cross-tenant isolation is enforced on the *egress* side, not on tenant-pod ingress.
8585
- **Blocking cross-tenant reachability through externally-exposed services.** If a tenant publishes a LoadBalancer with an external IP, another tenant can reach it via `world` egress.
86+
- **Network isolation between tenants on a hosted cluster without Cilium.** On the `isp-hosted` and `isp-hosted-slim` variants the CNI belongs to the host, and the tenant chart creates its Cilium policies only if the host serves the Cilium policy API (`packages/apps/tenant/templates/_helpers.tpl`, helper `tenant.ciliumPolicies`). Without it, tenants get no network isolation from Cozystack, and isolating them is up to the host.
8687
- **Noisy-neighbour and side-channel guarantees** for co-located tenant pods beyond namespace scoping (and, for tenant Kubernetes/VMs, virtualization). Co-located pods are not claimed to be side-channel isolated.
8788
- **Acting as an egress firewall.** Tenant workloads are allowed egress to `world` by default (`packages/apps/tenant/templates/networkpolicy.yaml`).
8889
- **Supporting pre-release builds and end-of-life versions** (`SECURITY.md`).

‎packages/apps/tenant/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,10 @@ tenant-u1
7070
└── postgres-db1
7171
```
7272

73+
### Network isolation
74+
75+
Tenant network isolation is enforced with Cilium network policies. On the `isp-hosted` and `isp-hosted-slim` platform variants the CNI belongs to the host, so these policies are created only if the host serves the Cilium policy API when the tenant is installed or updated. On a host without Cilium, tenants get no network isolation from Cozystack, and isolating them is up to the host. If Cilium appears on the host later, the policies are created on the tenant's next chart or values change, not automatically.
76+
7377
## Parameters
7478

7579
### Common parameters

‎packages/apps/tenant/templates/_helpers.tpl‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,3 +100,16 @@ false
100100
{{- define "tenant.tracingCentralEgress" -}}
101101
{{- if and (ne (include "tenant.name" .) "tenant-root") .Values.tracingCentral .Values.monitoring -}}true{{- end -}}
102102
{{- end -}}
103+
104+
{{- /* Every variant but the isp-hosted ones installs Cilium itself, so the
105+
policies render unconditionally there: a CRD missing at render time
106+
must fail the install, not leave the tenant unisolated for good, since
107+
helm-controller does not re-render a release when discovery changes.
108+
isp-hosted runs on the host's CNI, which may or may not be Cilium. */ -}}
109+
{{- define "tenant.ciliumPolicies" -}}
110+
{{- if not (has (include "cozy-lib.bundle-name" .) (list "isp-hosted" "isp-hosted-slim")) -}}
111+
true
112+
{{- else if and (.Capabilities.APIVersions.Has "cilium.io/v2/CiliumNetworkPolicy") (.Capabilities.APIVersions.Has "cilium.io/v2/CiliumClusterwideNetworkPolicy") -}}
113+
true
114+
{{- end -}}
115+
{{- end -}}

‎packages/apps/tenant/templates/networkpolicy-traces-central-read.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if include "tenant.tracingCentralEgress" . }}
1+
{{- if and (include "tenant.ciliumPolicies" .) (include "tenant.tracingCentralEgress" .) }}
22
# Read hop of shared-central tracing: Grafana reaches the tenant-root vmauth,
33
# never vtselect, which serves any account a client names.
44
apiVersion: cilium.io/v2

‎packages/apps/tenant/templates/networkpolicy-traces-central.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if include "tenant.tracingCentralEgress" . }}
1+
{{- if and (include "tenant.ciliumPolicies" .) (include "tenant.tracingCentralEgress" .) }}
22
# Write hop of shared-central tracing: the collector reaches the tenant-root
33
# vmauth, never vtinsert, which takes the account from the writer's headers.
44
# vmauth sets it from the tenant's VMUser login, so a pod that only carries

‎packages/apps/tenant/templates/networkpolicy.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
{{- if include "tenant.ciliumPolicies" . }}
12
---
23
apiVersion: cilium.io/v2
34
kind: CiliumNetworkPolicy
@@ -274,3 +275,4 @@ spec:
274275
- toEndpoints:
275276
- matchLabels:
276277
cozystack.io/service: ingress
278+
{{- end }}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
suite: tenant Cilium policies follow the platform variant's CNI
2+
templates:
3+
- templates/networkpolicy.yaml
4+
- templates/networkpolicy-traces-central.yaml
5+
- templates/networkpolicy-traces-central-read.yaml
6+
release:
7+
name: tenant-ktj
8+
namespace: tenant-root
9+
tests:
10+
- it: renders no Cilium objects on isp-hosted without the cilium.io API
11+
set:
12+
_cluster:
13+
bundle-name: isp-hosted
14+
asserts:
15+
- hasDocuments:
16+
count: 0
17+
18+
- it: renders no Cilium objects on isp-hosted-slim without the cilium.io API
19+
set:
20+
_cluster:
21+
bundle-name: isp-hosted-slim
22+
asserts:
23+
- hasDocuments:
24+
count: 0
25+
26+
- it: renders no Cilium objects on isp-hosted when only one Cilium kind is served
27+
set:
28+
_cluster:
29+
bundle-name: isp-hosted
30+
capabilities:
31+
apiVersions:
32+
- cilium.io/v2/CiliumNetworkPolicy
33+
asserts:
34+
- hasDocuments:
35+
count: 0
36+
37+
- it: renders the Cilium policies on isp-hosted when the host runs Cilium
38+
set:
39+
_cluster:
40+
bundle-name: isp-hosted
41+
capabilities:
42+
apiVersions:
43+
- cilium.io/v2/CiliumNetworkPolicy
44+
- cilium.io/v2/CiliumClusterwideNetworkPolicy
45+
asserts: &policiesRendered
46+
- template: templates/networkpolicy.yaml
47+
documentSelector:
48+
path: metadata.name
49+
value: allow-internal-communication
50+
isKind:
51+
of: CiliumNetworkPolicy
52+
- template: templates/networkpolicy.yaml
53+
documentSelector:
54+
path: metadata.name
55+
value: tenant-ktj-ingress
56+
isKind:
57+
of: CiliumClusterwideNetworkPolicy
58+
59+
- it: renders the Cilium policies on a Cilium variant even when the API is not discoverable
60+
set:
61+
_cluster:
62+
bundle-name: isp-full
63+
asserts: *policiesRendered
64+
65+
- it: renders the Cilium policies when the platform injected no _cluster
66+
asserts: *policiesRendered
67+
68+
- it: renders no shared-central tracing egress on isp-hosted without the cilium.io API
69+
set:
70+
_cluster:
71+
bundle-name: isp-hosted
72+
tracingCentral: true
73+
monitoring: true
74+
asserts:
75+
- hasDocuments:
76+
count: 0
77+
78+
- it: renders the shared-central tracing egress on isp-hosted when the host runs Cilium
79+
set:
80+
_cluster:
81+
bundle-name: isp-hosted
82+
tracingCentral: true
83+
monitoring: true
84+
capabilities:
85+
apiVersions:
86+
- cilium.io/v2/CiliumNetworkPolicy
87+
- cilium.io/v2/CiliumClusterwideNetworkPolicy
88+
asserts: &tracingRendered
89+
- template: templates/networkpolicy-traces-central.yaml
90+
isKind:
91+
of: CiliumClusterwideNetworkPolicy
92+
- template: templates/networkpolicy-traces-central-read.yaml
93+
isKind:
94+
of: CiliumClusterwideNetworkPolicy
95+
96+
- it: renders the shared-central tracing egress on a Cilium variant
97+
set:
98+
_cluster:
99+
bundle-name: isp-full
100+
tracingCentral: true
101+
monitoring: true
102+
asserts: *tracingRendered

0 commit comments

Comments
 (0)