Summary

On a fresh GKE Standard cluster, AICR bundle installation fails because the gpu-operator controller Deployment declares priorityClassName: system-node-critical and lands in the gpu-operator namespace, where GKE's ResourceQuota admission plugin rejects it. The ReplicaSet event makes the cause explicit:

FailedCreate: Error creating: insufficient quota to match these scopes:
[{PriorityClass In [system-node-critical system-cluster-critical]}]

The Deployment never gets to image-pull or readiness — admission blocks the pod before it's created — so the failure surfaces only as a helmfile apply --wait timeout 10 minutes later, which is high time-to-debug for a first-time user.

Repro

1. Provision the UAT GKE H100 cluster (e.g. gh workflow run uat-gcp.yaml -f skip_delete=true -f skip_tests=true).
2. Generate a helmfile bundle following demos/cuj1-gke-config.md (spec.bundle.deployment.deployer: helmfile).
3. cd ./bundle && helmfile apply --skip-diff-on-install (after installing the helm-diff plugin per [Bug]: helmfile bundle fails on fresh cluster — CRD/CR render order between nodewright-operator and nodewright-customizations #914).
4. After ~10 min, the gpu-operator release fails with Deployment/gpu-operator/gpu-operator not ready. status: InProgress, message: Replicas: 0/1 + context deadline exceeded.
5. kubectl -n gpu-operator describe deploy/gpu-operator shows the ReplicaSet stuck at 0/1 with ReplicaFailure: True (FailedCreate).
6. kubectl -n gpu-operator get events shows the quota rejection above.

Root cause

GKE Standard enables the ResourceQuota admission plugin and ships a ResourceQuota in kube-system with a scopeSelector admitting system-*-critical priority classes for that namespace only. Per the Kubernetes spec (ResourceQuota scope semantics): once any quota in the cluster scopes by PriorityClass for those values, pods that request a matching priority class can only be created in namespaces that have a matching quota. AICR emits gpu-operator (and likely a few other charts) into per-component namespaces with the upstream chart's default priorityClassName: system-node-critical, so admission blocks every pod.

EKS, AKS, and bare-metal clusters generally don't ship this default quota, which is why cuj1-eks.md and the existing EKS UAT path don't hit it.

Affected components (audit needed)

Any release emitted by AICR bundlers whose pod spec sets priorityClassName: system-node-critical (or system-cluster-critical) outside kube-system. Confirmed today: gpu-operator controller. Suspected, worth grepping templates for: network-operator, nvidia-dra-driver-gpu, nodewright-operator. NVIDIA's k8s-device-plugin, MIG manager, and the toolkit DaemonSets that gpu-operator manages all run in the gpu-operator namespace and inherit the same priority class — they would have hit the same wall on subsequent reconciliation.

Fix options (preferred → fallback)

• A — Bundler emits a matching ResourceQuota alongside any release that requests system-*-critical. Localized to the affected namespace, opaque to end users, no chart-value changes. The quota stays a permissive admission allowlist (not a real quota cap):

  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: <release>-critical-pods
    namespace: <release-namespace>
  spec:
    hard:
      pods: \"32\"   # generous; covers DaemonSets across N GPU nodes
    scopeSelector:
      matchExpressions:
        - operator: In
          scopeName: PriorityClass
          values:
            - system-node-critical
            - system-cluster-critical

  Wire it into pkg/bundler/component/ so any rendered release whose pod-spec walks turn up priorityClassName: system-*-critical triggers a sibling ResourceQuota. Idempotent re-apply is safe.

• B — Default the chart's operator.priorityClassName: \"\" (or a non-critical class) when the deployer target is GKE. Smaller blast radius than A in some respects (no extra resource), but globally suppresses the priority signal even where it does work (EKS/AKS/bare-metal), and we lose the operator's eviction-safety guarantee on resource-pressured GPU nodes. Not preferred.

• C — Document the manual ResourceQuota step in the GKE demo and the helmfile bundle README. Worst option — pushes Kubernetes admission trivia onto users for a problem AICR can fix once, centrally. Current state of demos/cuj1-gke-config.md is effectively at this level (the unblock is in chat, not in the file).

A is the right answer. Implementation pointers: pkg/bundler/component/ is the natural seam — render the chart, scan rendered templates for priorityClassName: system-*-critical, append a synthesized ResourceQuota for each affected namespace. Same fix benefits the helm, helmfile, flux, and argocd deployers because they all share the rendered-component plumbing.

Severity

User-visible: GKE demo blocking on a fresh cluster with no actionable error in the helmfile output — the user sees Replicas: 0/1 + context deadline exceeded, which sends them debugging the gpu-operator chart, not Kubernetes admission. Time-to-debug for a first-time user is high (10 min stall + opaque error). Risk is low: the fix is additive, no data loss path.

Workaround until fixed

kubectl apply -f - <<'EOF2'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-operator-critical-pods
  namespace: gpu-operator
spec:
  hard:
    pods: \"16\"
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values:
          - system-node-critical
          - system-cluster-critical
EOF2

cd ./bundle && helmfile apply --skip-diff-on-install

helmfile apply is idempotent — completed releases skip; only gpu-operator (and any downstream needs: releases) re-runs.

Repeat the same manifest, swapping metadata.namespace, for any other affected release the audit above turns up.

Related

• [Bug]: helmfile bundle fails on fresh cluster — CRD/CR render order between nodewright-operator and nodewright-customizations #914 (helmfile bundle fresh-cluster CRD/CR ordering) — different problem, same demo path. Both should be resolved before cuj1-gke-config.md runs cleanly end-to-end with no manual steps.

Are you willing to contribute?

Maybe, with guidance — happy to PR option A once a maintainer signs off on the seam (component bundler post-process vs. deployer-layer hook).