kube-controller-manager CrashLoopBackOff: cozystack-api publishes additionalProperties:true on Tenant schema, panicking the VAP status type-checker #2863

Summary

During platform install, kube-controller-manager enters CrashLoopBackOff on every control-plane node, which stalls reconciliation and times out the install (kubectl wait hr --all never completes). The crash is a nil-pointer panic in KCM's validatingadmissionpolicystatus controller while it type-checks the cozystack-tenant-host-policy ValidatingAdmissionPolicy against the aggregated tenants.apps.cozystack.io schema.

Root cause: cozystack-api publishes additionalProperties: true (boolean form) on the Tenant/Application spec schema, and the Kubernetes VAP status type-checker nil-dereferences on that construct.

Observed on v1.5.0-rc.2 / Kubernetes v1.33.1 (e.g. CI run https://github.com/cozystack/cozystack/actions/runs/27255451373). This is not fixed by #2558 (see below).

Impact

• KCM crash-loops cluster-wide → control plane is only intermittently functional → installs time out and E2E fails. Now that ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown #2558 removed the 3× install retry, a single occurrence fails the job outright.
• The cozystack-tenant-host-policy VAP never gets a populated status (observedGeneration stays empty) because the type-check panics before it can complete.

Panic (captured from kube-controller-manager)

panic: runtime error: invalid memory address or nil pointer dereference
k8s.io/apiserver/pkg/cel/openapi.isExtension(...)                                  extensions.go:28
k8s.io/apiserver/pkg/cel/openapi.isXEmbeddedResource(...)                          extensions.go:39
k8s.io/apiserver/pkg/cel/openapi.(*Schema).IsXEmbeddedResource(...)                adaptor.go:192
k8s.io/apiserver/pkg/cel/common.SchemaDeclType(...)                                schemas.go:91
k8s.io/apiserver/pkg/cel/common.SchemaDeclType(...)                                schemas.go:120
k8s.io/apiserver/pkg/admission/plugin/policy/validating.(*TypeChecker).declType   typechecking.go:253
k8s.io/apiserver/pkg/admission/plugin/policy/validating.(*TypeChecker).CreateContext typechecking.go:149
k8s.io/apiserver/pkg/admission/plugin/policy/validating.(*TypeChecker).Check       typechecking.go:111
k8s.io/kubernetes/pkg/controller/validatingadmissionpolicystatus.(*Controller).reconcile  controller.go:147

Root cause

1. cozystack-tenant-host-policy (shipped by cozystack-basics) matches apps.cozystack.io/v1alpha1/tenants, an aggregated API served by cozystack-api.
2. KCM's validatingadmissionpolicystatus controller type-checks the policy against the published OpenAPI v3 schema of tenants. That schema contains:

   "spec": { "type": "object", "additionalProperties": true }

   (the free-form "Chart Values" spec — 28 such additionalProperties: true nodes exist across the apps.cozystack.io schemas).
3. The CEL type-checker recurses into the boolean additionalProperties node; IsXEmbeddedResource → isExtension dereferences a nil schema → KCM panics. It re-queues the policy and re-panics → CrashLoopBackOff.

The additionalProperties: true is emitted here:

• pkg/cmd/server/openapi.go, patchSpec():

  // empty (schemaless) spec:
  prop.AdditionalProperties = &spec.SchemaOrBool{Allows: true}   // ~line 73
  // custom spec without additionalProperties:
  custom.AdditionalProperties = &spec.SchemaOrBool{Allows: true} // ~line 83

  &spec.SchemaOrBool{Allows: true} (no Schema) serializes to additionalProperties: true.

Why #2558 does not fix this

#2558 added dependsOn: [cozystack.cozystack-engine, cozystack.gateway-api-crds] to cozystack-basics so its VAPs are created after the matched APIs are served. That ordering is correct and is present in the live cluster — verified the live cozystack.cozystack-basics PackageSource carries those dependsOn edges. But the panic happens when KCM type-checks the VAP against the schema, which is independent of install order: once both the tenants API and the VAP exist, KCM type-checks and panics. Ordering does not avoid it (arguably it makes it deterministic).

Minimal reproduction (stock Kubernetes, no cozystack)

kind cluster on v1.33.1:

kind create cluster --name kcm-vap --image kindest/node:v1.33.1

crd.yaml — a CRD whose served schema has additionalProperties: true:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.repro.example.com
spec:
  group: repro.example.com
  scope: Namespaced
  names: {plural: widgets, singular: widget, kind: Widget}
  versions:
  - name: v1alpha1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            additionalProperties: true

vap.yaml — a VAP matching it:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: widget-host-policy
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["repro.example.com"]
      apiVersions: ["v1alpha1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["widgets"]
  validations:
  - expression: "true"

kubectl apply -f crd.yaml
kubectl apply -f vap.yaml
# within seconds:
kubectl get pod -n kube-system -l component=kube-controller-manager -w   # -> CrashLoopBackOff
kubectl logs -n kube-system -l component=kube-controller-manager --previous | grep -A30 'nil pointer'  # the panic above
kubectl get validatingadmissionpolicy widget-host-policy -o jsonpath='{.status.observedGeneration}'  # empty — never type-checked

Verified fix

Emit x-kubernetes-preserve-unknown-fields: true instead of the boolean additionalProperties: true for the schemaless / open spec. Confirmed in the same kind cluster — swapping the CRD's spec to:

spec:
  type: object
  x-kubernetes-preserve-unknown-fields: true

→ served schema becomes {"type":"object","x-kubernetes-preserve-unknown-fields":true}, KCM stays healthy, and the VAP type-checks cleanly (observedGeneration=1, typeChecking={}). Semantically equivalent ("arbitrary fields allowed").

Concretely, in pkg/cmd/server/openapi.go::patchSpec replace the &spec.SchemaOrBool{Allows: true} assignments with setting the x-kubernetes-preserve-unknown-fields: true extension on the spec schema (and not emitting the boolean additionalProperties).

Upstream

The underlying nil-dereference is a Kubernetes KCM bug — the VAP status type-checker should treat additionalProperties: true as dynamic/unknown rather than panic. A separate upstream report is being prepared. cozystack can fix immediately via the schema change above.

Environment

• cozystack v1.5.0-rc.2
• Kubernetes v1.33.1 (Talos v1.13.0 in CI; reproduced on kindest/node:v1.33.1)
• VAP involved: cozystack-tenant-host-policy (matches apps.cozystack.io/v1alpha1/tenants)