Guest cluster monitoringAgents in a child tenant cannot reach the ancestor tenant's VictoriaLogs/VictoriaMetrics (tenant NetworkPolicy gap) #5256

What happened

In a tenant hierarchy, a child tenant that has its own monitoring disabled runs a managed Kubernetes cluster with the monitoringAgents addon enabled. The addon's Fluent Bit and VMAgent are rendered to ship to the nearest ancestor tenant that has monitoring (the parent). Delivery silently fails: no guest logs or metrics ever arrive in the parent's VictoriaLogs/VictoriaMetrics, while the monitoringAgents HelmRelease reports Ready.

Fluent Bit inside the guest cluster logs:

[error] [upstream] connection #160 to tcp://<clusterIP>:9481 timed out after 10 seconds (connection timeout)
[error] [output:http:http.0] no upstream connections available to vlinsert-generic.tenant-a.svc.cozy.local:9481

DNS resolves correctly; the TCP connection is dropped by the tenant CiliumNetworkPolicies.

Environment

• Cozystack v1.7.0
• Managed Kubernetes (apps.cozystack.io/Kubernetes) guest cluster inside a child tenant
• Child tenant: monitoring: false; ancestor tenant: monitoring: true

Reproduction

1. Tenant tenant-a with monitoring: true.
2. Child tenant tenant-a-b with monitoring: false.
3. A managed Kubernetes cluster in tenant-a-b with addons.monitoringAgents.enabled: true.
4. Expect guest logs/metrics to appear in tenant-a's VictoriaLogs/VictoriaMetrics (the agents are rendered to target vlinsert-generic.tenant-a.svc / vminsert-shortterm.tenant-a.svc).
5. Actual: nothing arrives. Fluent Bit times out against the parent's vlinsert; the HelmRelease stays Ready.

Root cause

packages/apps/tenant/templates/networkpolicy.yaml generates the per-tenant CiliumClusterwideNetworkPolicies, and the ancestor monitoring path is incomplete in both directions:

1. Child -egress is missing vlinsert. The ancestor loop allows egress to the ancestor's vminsert (metrics), etcd, and ingress, but never vlinsert (logs):

   - toEndpoints:
     - matchLabels:
         "k8s:app.kubernetes.io/name": "vminsert"
         "k8s:io.kubernetes.pod.namespace": <ancestor>

   There is no equivalent block for vlinsert, so guest log shipping to the ancestor is dropped at the child's egress.

2. Parent -ingress does not allow descendants. The ingress policy allows only kube-apiserver, cozystack.io/system namespaces, kube-system, and the tenant's own ancestors. Descendant tenants are not permitted as a source, so even the vminsert egress allow above has no matching ingress allow on the parent, and metrics from a descendant guest are dropped too.

Net effect: for a guest cluster in a child tenant without its own monitoring, neither logs nor metrics can reach the ancestor it is configured to use. The failure is silent (HelmRelease Ready), which makes it hard to diagnose.

Workaround

Two additive CiliumNetworkPolicies (they do not touch the generated tenant-*-egress/ingress):

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-to-parent-vlinsert
  namespace: tenant-a-b
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: tenant-a
            k8s:app.kubernetes.io/name: vlinsert
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-from-child-to-vlinsert
  namespace: tenant-a
spec:
  endpointSelector:
    matchLabels:
      k8s:app.kubernetes.io/name: vlinsert
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: tenant-a-b

After applying these, Fluent Bit immediately starts returning HTTP status=200 and the guest logs appear in the parent's VictoriaLogs (filter by the injected cluster field). Metrics need the same treatment for vminsert on the parent ingress side.

Suggested fix

When a tenant's monitoring target is an ancestor, the generated policies should cover the full monitoring-ingest flow both ways:

• Child -egress: add vlinsert alongside the existing vminsert ancestor rule.
• Parent -ingress: allow ingress to the monitoring ingest endpoints (vlinsert, vminsert) from descendant tenants (scoped to the ingest pods, not the whole namespace).

Separately, the monitoringAgents HelmRelease reporting Ready while delivery fails masks the problem; surfacing agent delivery health would make this class of issue visible.