This is the multi-page printable view of this section. Click here to print.
Kubernetes Blog
- Scheduling the removal of the kube-proxy IPVS backend
- From Working Group to SIG Architecture: spotlight on AI Conformance
- Announcing the Workload Conformance Working Group
- Why Is My Kubernetes Job Still Pending? Tracing Kueue Admission with Headlamp
- The Shift to cgroup v2 in Kubernetes: What You Need to Know
- Scaling Kubernetes Workloads with Node Swap
- Spotlight on SIG Apps
- Kubernetes v1.37: Tracking When a PersistentVolumeClaim Was Last Used (Beta)
- Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions
- Kubernetes v1.37: Pod-Level Resource Managers graduated to Beta
- Kubernetes Changed Block Tracking API - Beta Differences
- Kubernetes v1.37: Memory QoS Graduates to Beta
- Kubernetes v1.37: Native Histograms Graduates to Beta
- Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize (Alpha)
- Kubernetes v1.37: Introducing Node Lifecycle Conditions
- Kubernetes v1.37: Advancing Workload-Aware Scheduling
- Kubernetes v1.37: KubeletInUserNamespace (aka Rootless mode) Graduates to Beta
- Kubernetes v1.37: DRA Updates
- Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler
- Kubernetes v1.37: etcd RangeStream Cuts Memory Use on Large List Reads
- Kubernetes v1.37: Storage Version Migration Enabled by Default
- Kubernetes v1.37: Pod Certificates and Cluster Trust Bundles
- Kubernetes v1.37: Metrics API graduates to stable
- Kubernetes v1.37: Garhwal
- How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To
- Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard
- Kubernetes v1.37 Sneak Peek
- How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server
- Building a Custom Metrics Exporter for Kubernetes
- Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow
- Kubernetes Dashboard to Headlamp: A Step-by-Step Guide
- Announcing etcd v3.7.0
- Open source maintainership in the age of AI
- Introducing the Cluster API plugin for Headlamp
- Inspect Volcano workloads faster with Headlamp
- See your serverless: introducing the Headlamp plugin for Knative
- Spotlight on WG Device Management
- Spotlight on SIG Storage
- From Kubernetes Dashboard to Headlamp: Understanding the Transition
- Reconciling the Past: Correcting Records for Unfixed Kubernetes CVEs
- Announcing etcd 3.7.0-beta.0
- Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager
- Kubernetes v1.36: Mixed Version Proxy Graduates to Beta
- Kubernetes v1.36: Deprecation and removal of Service ExternalIPs
- Kubernetes v1.36: Advancing Workload-Aware Scheduling
- Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA
- Kubernetes v1.36: Moving Volume Group Snapshots to GA
- Kubernetes v1.36: More Drivers, New Features, and the Next Era of DRA
- Kubernetes v1.36: Server-Side Sharded List and Watch
- Kubernetes v1.36: Declarative Validation Graduates to GA
- Kubernetes v1.36: Admission Policies That Can't Be Deleted
- Kubernetes v1.36: Pod-Level Resource Managers (Alpha)
- Kubernetes v1.36: In-Place Vertical Scaling for Pod-Level Resources Graduates to Beta
- Kubernetes v1.36: Tiered Memory Protection with Memory QoS
- Kubernetes v1.36: Staleness Mitigation and Observability for Controllers
- Kubernetes v1.36: Mutable Pod Resources for Suspended Jobs (beta)
- Kubernetes v1.36: Fine-Grained Kubelet API Authorization Graduates to GA
- Kubernetes v1.36: User Namespaces in Kubernetes are finally GA
- SELinux Volume Label Changes goes GA (and likely implications in v1.37)
- Kubernetes v1.36: ハル (Haru)
- Gateway API v1.5: Moving features to Stable
- Kubernetes v1.36 Sneak Peek
- Announcing Ingress2Gateway 1.0: Your Path to Gateway API
- Running Agents on Kubernetes with Agent Sandbox
- Securing Production Debugging in Kubernetes
- The Invisible Rewrite: Modernizing the Kubernetes Image Promoter
- Announcing the AI Gateway Working Group
- Before You Migrate: Five Surprising Ingress-NGINX Behaviors You Need to Know
- Spotlight on SIG Architecture: API Governance
- Introducing Node Readiness Controller
- New Conversion from cgroup v1 CPU Shares to v2 CPU Weight
- Ingress NGINX: Statement from the Kubernetes Steering and Security Response Committees
- Experimenting with Gateway API using kind
- Cluster API v1.12: Introducing In-place Updates and Chained Upgrades
- Headlamp in 2025: Project Highlights
- Announcing the Checkpoint/Restore Working Group
- Uniform API server access using clientcmd
- Kubernetes v1.35: Restricting executables invoked by kubeconfigs via exec plugin allowList added to kuberc
- Kubernetes v1.35: Mutable PersistentVolume Node Affinity (alpha)
- Kubernetes v1.35: A Better Way to Pass Service Account Tokens to CSI Drivers
- Kubernetes v1.35: Extended Toleration Operators to Support Numeric Comparisons (Alpha)
- Kubernetes v1.35: New level of efficiency with in-place Pod restart
- Kubernetes 1.35: Enhanced Debugging with Versioned z-pages APIs
- Kubernetes v1.35: Watch Based Route Reconciliation in the Cloud Controller Manager
- Kubernetes v1.35: Introducing Workload Aware Scheduling
- Kubernetes v1.35: Fine-grained Supplemental Groups Control Graduates to GA
- Kubernetes v1.35: Kubelet Configuration Drop-in Directory Graduates to GA
- Avoiding Zombie Cluster Members When Upgrading to etcd v3.6
- Kubernetes 1.35: In-Place Pod Resize Graduates to Stable
- Kubernetes v1.35: Job Managed By Goes GA
- Kubernetes v1.35: Timbernetes (The World Tree Release)
- Kubernetes v1.35 Sneak Peek
- Kubernetes Configuration Good Practices
- Ingress NGINX Retirement: What You Need to Know
- Announcing the 2025 Steering Committee Election Results
- Gateway API 1.4: New Features
- 7 Common Kubernetes Pitfalls (and How I Learned to Avoid Them)
- Spotlight on Policy Working Group
- Introducing Headlamp Plugin for Karpenter - Scaling and Visibility
- Announcing Changed Block Tracking API support (alpha)
- Kubernetes v1.34: Pod Level Resources Graduated to Beta
- Kubernetes v1.34: Recovery From Volume Expansion Failure (GA)
- Kubernetes v1.34: DRA Consumable Capacity
- Kubernetes v1.34: Pods Report DRA Resource Health
- Kubernetes v1.34: Moving Volume Group Snapshots to v1beta2
- Kubernetes v1.34: Decoupled Taint Manager Is Now Stable
- Kubernetes v1.34: Autoconfiguration for Node Cgroup Driver Goes GA
- Kubernetes v1.34: Mutable CSI Node Allocatable Graduates to Beta
- Kubernetes v1.34: Use An Init Container To Define App Environment Variables
- Kubernetes v1.34: Snapshottable API server cache
- Kubernetes v1.34: VolumeAttributesClass for Volume Modification GA
- Kubernetes v1.34: Pod Replacement Policy for Jobs Goes GA
- Kubernetes v1.34: PSI Metrics for Kubernetes Graduates to Beta
- Kubernetes v1.34: Service Account Token Integration for Image Pulls Graduates to Beta
- Kubernetes v1.34: Introducing CPU Manager Static Policy Option for Uncore Cache Alignment
- Kubernetes v1.34: DRA has graduated to GA
- Kubernetes v1.34: Finer-Grained Control Over Container Restarts
- Kubernetes v1.34: User preferences (kuberc) are available for testing in kubectl 1.34
- Kubernetes v1.34: Of Wind & Will (O' WaW)
- Tuning Linux Swap for Kubernetes: A Deep Dive
- Introducing Headlamp AI Assistant
- Kubernetes v1.34 Sneak Peek
- Post-Quantum Cryptography in Kubernetes
- Navigating Failures in Pods With Devices
- Image Compatibility In Cloud Native Environments
- Changes to Kubernetes Slack
- Enhancing Kubernetes Event Management with Custom Aggregation
- Introducing Gateway API Inference Extension
- Start Sidecar First: How To Avoid Snags
- Gateway API v1.3.0: Advancements in Request Mirroring, CORS, Gateway Merging, and Retry Budgets
- Kubernetes v1.33: In-Place Pod Resize Graduated to Beta
- Announcing etcd v3.6.0
- Kubernetes 1.33: Job's SuccessPolicy Goes GA
- Kubernetes v1.33: Updates to Container Lifecycle
- Kubernetes v1.33: Job's Backoff Limit Per Index Goes GA
- Kubernetes v1.33: Image Pull Policy the way you always thought it worked!
- Kubernetes v1.33: Streaming List responses
- Kubernetes 1.33: Volume Populators Graduate to GA
- Kubernetes v1.33: From Secrets to Service Accounts: Kubernetes Image Pulls Evolved
- Kubernetes v1.33: Fine-grained SupplementalGroups Control Graduates to Beta
- Kubernetes v1.33: Prevent PersistentVolume Leaks When Deleting out of Order graduates to GA
- Kubernetes v1.33: Mutable CSI Node Allocatable Count
- Kubernetes v1.33: New features in DRA
- Kubernetes v1.33: Storage Capacity Scoring of Nodes for Dynamic Provisioning (alpha)
- Kubernetes v1.33: Image Volumes graduate to beta!
- Kubernetes v1.33: HorizontalPodAutoscaler Configurable Tolerance
- Kubernetes v1.33: User Namespaces enabled by default!
- Kubernetes v1.33: Continuing the transition from Endpoints to EndpointSlices
- Kubernetes v1.33: Octarine
- Kubernetes Multicontainer Pods: An Overview
- Introducing kube-scheduler-simulator
- Kubernetes v1.33 sneak peek
- Fresh Swap Features for Linux Users in Kubernetes 1.32
- Ingress-nginx CVE-2025-1974: What You Need to Know
- Introducing JobSet
- Spotlight on SIG Apps
- Spotlight on SIG etcd
- NFTables mode for kube-proxy
- The Cloud Controller Manager Chicken and Egg Problem
- Spotlight on SIG Architecture: Enhancements
- Kubernetes 1.32: Moving Volume Group Snapshots to Beta
- Enhancing Kubernetes API Server Efficiency with API Streaming
- Kubernetes v1.32 Adds A New CPU Manager Static Policy Option For Strict CPU Reservation
- Kubernetes v1.32: Memory Manager Goes GA
- Kubernetes v1.32: QueueingHint Brings a New Possibility to Optimize Pod Scheduling
- Kubernetes v1.32: Penelope
- Gateway API v1.2: WebSockets, Timeouts, Retries, and More
- How we built a dynamic Kubernetes API Server for the API Aggregation Layer in Cozystack
- Kubernetes v1.32 sneak peek
- Spotlight on Kubernetes Upstream Training in Japan
- Announcing the 2024 Steering Committee Election Results
- Spotlight on CNCF Deaf and Hard-of-hearing Working Group (DHHWG)
- Spotlight on SIG Scheduling
- Kubernetes v1.31: kubeadm v1beta4
- Kubernetes 1.31: Custom Profiling in Kubectl Debug Graduates to Beta
- Kubernetes 1.31: Fine-grained SupplementalGroups control
- Kubernetes v1.31: New Kubernetes CPUManager Static Policy: Distribute CPUs Across Cores
- Kubernetes 1.31: Autoconfiguration For Node Cgroup Driver (beta)
- Kubernetes 1.31: Streaming Transitions from SPDY to WebSockets
- Kubernetes 1.31: Pod Failure Policy for Jobs Goes GA
- Kubernetes 1.31: MatchLabelKeys in PodAffinity graduates to beta
- Kubernetes 1.31: Prevent PersistentVolume Leaks When Deleting out of Order
- Kubernetes 1.31: Read Only Volumes Based On OCI Artifacts (alpha)
- Kubernetes 1.31: VolumeAttributesClass for Volume Modification Beta
- Kubernetes v1.31: Accelerating Cluster Performance with Consistent Reads from Cache
- Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode
- Kubernetes v1.31: PersistentVolume Last Phase Transition Time Moves to GA
- Kubernetes v1.31: Elli
- Introducing Feature Gates to Client-Go: Enhancing Flexibility and Control
- Spotlight on SIG API Machinery
- Kubernetes Removals and Major Changes In v1.31
- Spotlight on SIG Node
- 10 Years of Kubernetes
- Completing the largest migration in Kubernetes history
- Gateway API v1.1: Service mesh, GRPCRoute, and a whole lot more
- Container Runtime Interface streaming explained
- Kubernetes 1.30: Preventing unauthorized volume mode conversion moves to GA
- Kubernetes 1.30: Multi-Webhook and Modular Authorization Made Much Easier
- Kubernetes 1.30: Structured Authentication Configuration Moves to Beta
- Kubernetes 1.30: Validating Admission Policy Is Generally Available
- Kubernetes 1.30: Read-only volume mounts can be finally literally read-only
- Kubernetes 1.30: Beta Support For Pods With User Namespaces
- Kubernetes v1.30: Uwubernetes
- Spotlight on SIG Architecture: Code Organization
- DIY: Create Your Own Cloud with Kubernetes (Part 3)
- DIY: Create Your Own Cloud with Kubernetes (Part 2)
- DIY: Create Your Own Cloud with Kubernetes (Part 1)
- Introducing the Windows Operational Readiness Specification
- A Peek at Kubernetes v1.30
- CRI-O: Applying seccomp profiles from OCI registries
- Spotlight on SIG Cloud Provider
- A look into the Kubernetes Book Club
- Image Filesystem: Configuring Kubernetes to store containers on a separate filesystem
- Spotlight on SIG Release (Release Team Subproject)
- Contextual logging in Kubernetes 1.29: Better troubleshooting and enhanced logging
- Kubernetes 1.29: Decoupling taint manager from node lifecycle controller
- Kubernetes 1.29: PodReadyToStartContainers Condition Moves to Beta
- Kubernetes 1.29: New (alpha) Feature, Load Balancer IP Mode for Services
- Kubernetes 1.29: Single Pod Access Mode for PersistentVolumes Graduates to Stable
- Kubernetes 1.29: CSI Storage Resizing Authenticated and Generally Available in v1.29
- Kubernetes 1.29: VolumeAttributesClass for Volume Modification
- Kubernetes 1.29: Cloud Provider Integrations Are Now Separate Components
- Kubernetes v1.29: Mandala
- New Experimental Features in Gateway API v1.0
- Spotlight on SIG Testing
- Kubernetes Removals, Deprecations, and Major Changes in Kubernetes 1.29
- The Case for Kubernetes Resource Limits: Predictability vs. Efficiency
- Introducing SIG etcd
- Kubernetes Contributor Summit: Behind-the-scenes
- Spotlight on SIG Architecture: Production Readiness
- Gateway API v1.0: GA Release
- Introducing ingress2gateway; Simplifying Upgrades to Gateway API
- Plants, process and parties: the Kubernetes 1.28 release interview
- PersistentVolume Last Phase Transition Time in Kubernetes
- A Quick Recap of 2023 China Kubernetes Contributor Summit
- Bootstrap an Air Gapped Cluster With Kubeadm
- CRI-O is moving towards pkgs.k8s.io
- Spotlight on SIG Architecture: Conformance
- Announcing the 2023 Steering Committee Election Results
- Happy 7th Birthday kubeadm!
- kubeadm: Use etcd Learner to Join a Control Plane Node Safely
- User Namespaces: Now Supports Running Stateful Pods in Alpha!
- Comparing Local Kubernetes Development Tools: Telepresence, Gefyra, and mirrord
- Kubernetes Legacy Package Repositories Will Be Frozen On September 13, 2023
- Gateway API v0.8.0: Introducing Service Mesh Support
- Kubernetes 1.28: A New (alpha) Mechanism For Safer Cluster Upgrades
- Kubernetes v1.28: Introducing native sidecar containers
- Kubernetes 1.28: Beta support for using swap on Linux
- Kubernetes 1.28: Node podresources API Graduates to GA
- Kubernetes 1.28: Improved failure handling for Jobs
- Kubernetes v1.28: Retroactive Default StorageClass move to GA
- Kubernetes 1.28: Non-Graceful Node Shutdown Moves to GA
- pkgs.k8s.io: Introducing Kubernetes Community-Owned Package Repositories
- Kubernetes v1.28: Planternetes
- Spotlight on SIG ContribEx
- Spotlight on SIG CLI
- Confidential Kubernetes: Use Confidential Virtual Machines and Enclaves to improve your cluster security
- Verifying Container Image Signatures Within CRI Runtimes
- dl.k8s.io to adopt a Content Delivery Network
- Using OCI artifacts to distribute security profiles for seccomp, SELinux and AppArmor
- Having fun with seccomp profiles on the edge
- Kubernetes 1.27: KMS V2 Moves to Beta
- Kubernetes 1.27: updates on speeding up Pod startup
- Kubernetes 1.27: In-place Resource Resize for Kubernetes Pods (alpha)
- Kubernetes 1.27: Avoid Collisions Assigning Ports to NodePort Services
- Kubernetes 1.27: Safer, More Performant Pruning in kubectl apply
- Kubernetes 1.27: Introducing An API For Volume Group Snapshots
- Kubernetes 1.27: Quality-of-Service for Memory Resources (alpha)
- Kubernetes 1.27: StatefulSet PVC Auto-Deletion (beta)
- Kubernetes 1.27: HorizontalPodAutoscaler ContainerResource type metric moves to beta
- Kubernetes 1.27: StatefulSet Start Ordinal Simplifies Migration
- Updates to the Auto-refreshing Official CVE Feed
- Kubernetes 1.27: Server Side Field Validation and OpenAPI V3 move to GA
- Kubernetes 1.27: Query Node Logs Using The Kubelet API
- Kubernetes 1.27: Single Pod Access Mode for PersistentVolumes Graduates to Beta
- Kubernetes 1.27: Efficient SELinux volume relabeling (Beta)
- Kubernetes 1.27: More fine-grained pod topology spread policies reached beta
- Kubernetes v1.27: Chill Vibes
- Keeping Kubernetes Secure with Updated Go Versions
- Kubernetes Validating Admission Policies: A Practical Example
- Kubernetes Removals and Major Changes In v1.27
- k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know
- Forensic container analysis
- Introducing KWOK: Kubernetes WithOut Kubelet
- Free Katacoda Kubernetes Tutorials Are Shutting Down
- k8s.gcr.io Image Registry Will Be Frozen From the 3rd of April 2023
- Spotlight on SIG Instrumentation
- Consider All Microservices Vulnerable — And Monitor Their Behavior
- Protect Your Mission-Critical Pods From Eviction With PriorityClass
- Kubernetes 1.26: Eviction policy for unhealthy pods guarded by PodDisruptionBudgets
- Kubernetes v1.26: Retroactive Default StorageClass
- Kubernetes v1.26: Alpha support for cross-namespace storage data sources
- Kubernetes v1.26: Advancements in Kubernetes Traffic Engineering
- Kubernetes 1.26: Job Tracking, to Support Massively Parallel Batch Workloads, Is Generally Available
- Kubernetes v1.26: CPUManager goes GA
- Kubernetes 1.26: Pod Scheduling Readiness
- Kubernetes 1.26: Support for Passing Pod fsGroup to CSI Drivers At Mount Time
- Kubernetes v1.26: GA Support for Kubelet Credential Providers
- Kubernetes 1.26: Introducing Validating Admission Policies
- Kubernetes 1.26: Device Manager graduates to GA
- Kubernetes 1.26: Non-Graceful Node Shutdown Moves to Beta
- Kubernetes 1.26: Alpha API For Dynamic Resource Allocation
- Kubernetes 1.26: Windows HostProcess Containers Are Generally Available
- Kubernetes 1.26: We're now signing our binary release artifacts!
- Kubernetes v1.26: Electrifying
- Forensic container checkpointing in Kubernetes
- Finding suspicious syscalls with the seccomp notifier
- Boosting Kubernetes container runtime observability with OpenTelemetry
- registry.k8s.io: faster, cheaper and Generally Available (GA)
- Kubernetes Removals, Deprecations, and Major Changes in 1.26
- Live and let live with Kluctl and Server Side Apply
- Server Side Apply Is Great And You Should Be Using It
- Current State: 2019 Third Party Security Audit of Kubernetes
- Introducing Kueue
- Kubernetes 1.25: alpha support for running Pods with user namespaces
- Enforce CRD Immutability with CEL Transition Rules
- Kubernetes 1.25: Kubernetes In-Tree to CSI Volume Migration Status Update
- Kubernetes 1.25: CustomResourceDefinition Validation Rules Graduate to Beta
- Kubernetes 1.25: Use Secrets for Node-Driven Expansion of CSI Volumes
- Kubernetes 1.25: Local Storage Capacity Isolation Reaches GA
- Kubernetes 1.25: Two Features for Apps Rollouts Graduate to Stable
- Kubernetes 1.25: PodHasNetwork Condition for Pods
- Announcing the Auto-refreshing Official Kubernetes CVE Feed
- Kubernetes 1.25: KMS V2 Improvements
- Kubernetes’s IPTables Chains Are Not API
- Introducing COSI: Object Storage Management using Kubernetes APIs
- Kubernetes 1.25: cgroup v2 graduates to GA
- Kubernetes 1.25: CSI Inline Volumes have graduated to GA
- Kubernetes v1.25: Pod Security Admission Controller in Stable
- PodSecurityPolicy: The Historical Context
- Kubernetes v1.25: Combiner
- Spotlight on SIG Storage
- Meet Our Contributors - APAC (China region)
- Enhancing Kubernetes one KEP at a Time
- Kubernetes Removals and Major Changes In 1.25
- Spotlight on SIG Docs
- Kubernetes Gateway API Graduates to Beta
- Annual Report Summary 2021
- Kubernetes 1.24: Maximum Unavailable Replicas for StatefulSet
- Contextual Logging in Kubernetes 1.24
- Kubernetes 1.24: Avoid Collisions Assigning IP Addresses to Services
- Kubernetes 1.24: Introducing Non-Graceful Node Shutdown Alpha
- Kubernetes 1.24: Prevent unauthorised volume mode conversion
- Kubernetes 1.24: Volume Populators Graduate to Beta
- Kubernetes 1.24: gRPC container probes in beta
- Kubernetes 1.24: Storage Capacity Tracking Now Generally Available
- Kubernetes 1.24: Volume Expansion Now A Stable Feature
- Dockershim: The Historical Context
- Kubernetes 1.24: Stargazer
- Increasing the security bar in Ingress-NGINX v1.2.0
- Kubernetes Removals and Deprecations In 1.24
- Is Your Cluster Ready for v1.24?
- Meet Our Contributors - APAC (Aus-NZ region)
- Updated: Dockershim Removal FAQ
- SIG Node CI Subproject Celebrates Two Years of Test Improvements
- Spotlight on SIG Multicluster
- Securing Admission Controllers
- Meet Our Contributors - APAC (India region)
- Kubernetes is Moving on From Dockershim: Commitments and Next Steps
- Kubernetes-in-Kubernetes and the WEDOS PXE bootable server farm
- Using Admission Controllers to Detect Container Drift at Runtime
- What's new in Security Profiles Operator v0.4.0
- Kubernetes 1.23: StatefulSet PVC Auto-Deletion (alpha)
- Kubernetes 1.23: Prevent PersistentVolume leaks when deleting out of order
- Kubernetes 1.23: Kubernetes In-Tree to CSI Volume Migration Status Update
- Kubernetes 1.23: Pod Security Graduates to Beta
- Kubernetes 1.23: Dual-stack IPv4/IPv6 Networking Reaches GA
- Kubernetes 1.23: The Next Frontier
- Contribution, containers and cricket: the Kubernetes 1.22 release interview
- Quality-of-Service for Memory Resources
- Dockershim removal is coming. Are you ready?
- Non-root Containers And Devices
- Announcing the 2021 Steering Committee Election Results
- Use KPNG to Write Specialized kube-proxiers
- Introducing ClusterClass and Managed Topologies in Cluster API
- A Closer Look at NSA/CISA Kubernetes Hardening Guidance
- How to Handle Data Duplication in Data-Heavy Kubernetes Environments
- Spotlight on SIG Node
- Introducing Single Pod Access Mode for PersistentVolumes
- Alpha in Kubernetes v1.22: API Server Tracing
- Kubernetes 1.22: A New Design for Volume Populators
- Minimum Ready Seconds for StatefulSets
- Enable seccomp for all workloads with a new v1.22 alpha feature
- Alpha in v1.22: Windows HostProcess Containers
- Kubernetes Memory Manager moves to beta
- Kubernetes 1.22: CSI Windows Support (with CSI Proxy) reaches GA
- New in Kubernetes v1.22: alpha support for using swap memory
- Kubernetes 1.22: Server Side Apply moves to GA
- Kubernetes 1.22: Reaching New Peaks
- Roorkee robots, releases and racing: the Kubernetes 1.21 release interview
- Updating NGINX-Ingress to use the stable Ingress API
- Kubernetes Release Cadence Change: Here’s What You Need To Know
- Spotlight on SIG Usability
- Kubernetes API and Feature Removals In 1.22: Here’s What You Need To Know
- Announcing Kubernetes Community Group Annual Reports
- Writing a Controller for Pod Labels
- Using Finalizers to Control Deletion
- Kubernetes 1.21: Metrics Stability hits GA
- Evolving Kubernetes networking with the Gateway API
- Graceful Node Shutdown Goes Beta
- Annotating Kubernetes Services for Humans
- Defining Network Policy Conformance for Container Network Interface (CNI) providers
- Introducing Indexed Jobs
- Volume Health Monitoring Alpha Update
- Three Tenancy Models For Kubernetes
- Local Storage: Storage Capacity Tracking, Distributed Provisioning and Generic Ephemeral Volumes hit Beta
- kube-state-metrics goes v2.0
- Introducing Suspended Jobs
- Kubernetes 1.21: CronJob Reaches GA
- Kubernetes 1.21: Power to the Community
- PodSecurityPolicy Deprecation: Past, Present, and Future
- The Evolution of Kubernetes Dashboard
- A Custom Kubernetes Scheduler to Orchestrate Highly Available Applications
- Kubernetes 1.20: Pod Impersonation and Short-lived Volumes in CSI Drivers
- Third Party Device Metrics Reaches GA
- Kubernetes 1.20: Granular Control of Volume Permission Changes
- Kubernetes 1.20: Kubernetes Volume Snapshot Moves to GA
- Kubernetes 1.20: The Raddest Release
- GSoD 2020: Improving the API Reference Experience
- Dockershim Deprecation FAQ
- Don't Panic: Kubernetes and Docker
- Cloud native security for your clusters
- Remembering Dan Kohn
- Announcing the 2020 Steering Committee Election Results
- Contributing to the Development Guide
- GSoC 2020 - Building operators for cluster addons
- Introducing Structured Logs
- Warning: Helpful Warnings Ahead
- Scaling Kubernetes Networking With EndpointSlices
- Ephemeral volumes with storage capacity tracking: EmptyDir on steroids
- Increasing the Kubernetes Support Window to One Year
- Kubernetes 1.19: Accentuate the Paw-sitive
- Moving Forward From Beta
- Introducing Hierarchical Namespaces
- Physics, politics and Pull Requests: the Kubernetes 1.18 release interview
- Music and math: the Kubernetes 1.17 release interview
- SIG-Windows Spotlight
- Working with Terraform and Kubernetes
- A Better Docs UX With Docsy
- Supporting the Evolving Ingress Specification in Kubernetes 1.18
- K8s KPIs with Kuberhealthy
- My exciting journey into Kubernetes’ history
- An Introduction to the K8s-Infrastructure Working Group
- WSL+Docker: Kubernetes on the Windows Desktop
- How Docs Handle Third Party and Dual Sourced Content
- Introducing PodTopologySpread
- Two-phased Canary Rollout with Open Source Gloo
- Cluster API v1alpha3 Delivers New Features and an Improved User Experience
- How Kubernetes contributors are building a better communication process
- API Priority and Fairness Alpha
- Introducing Windows CSI support alpha for Kubernetes
- Improvements to the Ingress API in Kubernetes 1.18
- Kubernetes 1.18 Feature Server-side Apply Beta 2
- Kubernetes Topology Manager Moves to Beta - Align Up!
- Kubernetes 1.18: Fit & Finish
- Join SIG Scalability and Learn Kubernetes the Hard Way
- Kong Ingress Controller and Service Mesh: Setting up Ingress to Istio on Kubernetes
- Contributor Summit Amsterdam Postponed
- Bring your ideas to the world with kubectl plugins
- Contributor Summit Amsterdam Schedule Announced
- Deploying External OpenStack Cloud Provider with Kubeadm
- KubeInvaders - Gamified Chaos Engineering Tool for Kubernetes
- CSI Ephemeral Inline Volumes
- Reviewing 2019 in Docs
- Kubernetes on MIPS
- Announcing the Kubernetes bug bounty program
- Remembering Brad Childs
- Testing of CSI drivers
- Kubernetes 1.17: Stability
- Kubernetes 1.17 Feature: Kubernetes Volume Snapshot Moves to Beta
- Kubernetes 1.17 Feature: Kubernetes In-Tree to CSI Volume Migration Moves to Beta
- When you're in the release team, you're family: the Kubernetes 1.16 release interview
- Gardener Project Update
- Develop a Kubernetes controller in Java
- Running Kubernetes locally on Linux with Microk8s
- Grokkin' the Docs
- Kubernetes Documentation Survey
- Contributor Summit San Diego Schedule Announced!
- 2019 Steering Committee Election Results
- Contributor Summit San Diego Registration Open!
- Kubernetes 1.16: Custom Resources, Overhauled Metrics, and Volume Extensions
- Announcing etcd 3.4
- OPA Gatekeeper: Policy and Governance for Kubernetes
- Get started with Kubernetes (using Python)
- Deprecated APIs Removed In 1.16: Here’s What You Need To Know
- Recap of Kubernetes Contributor Summit Barcelona 2019
- Automated High Availability in kubeadm v1.15: Batteries Included But Swappable
- Introducing Volume Cloning Alpha for Kubernetes
- Future of CRDs: Structural Schemas
- Kubernetes 1.15: Extensibility and Continuous Improvement
- Join us at the Contributor Summit in Shanghai
- Kyma - extend and build on Kubernetes with ease
- Kubernetes, Cloud Native, and the Future of Software
- Expanding our Contributor Workshops
- Cat shirts and Groundhog Day: the Kubernetes 1.14 release interview
- Join us for the 2019 KubeCon Diversity Lunch & Hack
- How You Can Help Localize Kubernetes Docs
- Hardware Accelerated SSL/TLS Termination in Ingress Controllers using Kubernetes Device Plugins and RuntimeClass
- Introducing kube-iptables-tailer: Better Networking Visibility in Kubernetes Clusters
- The Future of Cloud Providers in Kubernetes
- Pod Priority and Preemption in Kubernetes
- Process ID Limiting for Stability Improvements in Kubernetes 1.14
- Kubernetes 1.14: Local Persistent Volumes GA
- Kubernetes v1.14 delivers production-level support for Windows nodes and Windows containers
- kube-proxy Subtleties: Debugging an Intermittent Connection Reset
- Running Kubernetes locally on Linux with Minikube - now with Kubernetes 1.14 support
- Kubernetes 1.14: Production-level support for Windows Nodes, Kubectl Updates, Persistent Local Volumes GA
- Kubernetes End-to-end Testing for Everyone
- A Guide to Kubernetes Admission Controllers
- A Look Back and What's in Store for Kubernetes Contributor Summits
- KubeEdge, a Kubernetes Native Edge Computing Framework
- Kubernetes Setup Using Ansible and Vagrant
- Raw Block Volume support to Beta
- Automate Operations on your Cluster with OperatorHub.io
- Building a Kubernetes Edge (Ingress) Control Plane for Envoy v2
- Runc and CVE-2019-5736
- Poseidon-Firmament Scheduler – Flow Network Graph Based Scheduler
- Update on Volume Snapshot Alpha for Kubernetes
- Container Storage Interface (CSI) for Kubernetes GA
- APIServer dry-run and kubectl diff
- Kubernetes Federation Evolution
- etcd: Current status and future roadmap
- New Contributor Workshop Shanghai
- Production-Ready Kubernetes Cluster Creation with kubeadm
- Kubernetes 1.13: Simplified Cluster Management with Kubeadm, Container Storage Interface (CSI), and CoreDNS as Default DNS are Now Generally Available
- Kubernetes Docs Updates, International Edition
- gRPC Load Balancing on Kubernetes without Tears
- Tips for Your First Kubecon Presentation - Part 2
- Tips for Your First Kubecon Presentation - Part 1
- Kubernetes 2018 North American Contributor Summit
- 2018 Steering Committee Election Results
- Topology-Aware Volume Provisioning in Kubernetes
- Kubernetes v1.12: Introducing RuntimeClass
- Introducing Volume Snapshot Alpha for Kubernetes
- Support for Azure VMSS, Cluster-Autoscaler and User Assigned Identity
- Introducing the Non-Code Contributor’s Guide
- KubeDirector: The easy way to run complex stateful applications on Kubernetes
- Building a Network Bootable Server Farm for Kubernetes with LTSP
- Health checking gRPC servers on Kubernetes
- Kubernetes 1.12: Kubelet TLS Bootstrap and Azure Virtual Machine Scale Sets (VMSS) Move to General Availability
- Hands On With Linkerd 2.0
- 2018 Steering Committee Election Cycle Kicks Off
- The Machines Can Do the Work, a Story of Kubernetes Testing, CI, and Automating the Contributor Experience
- Introducing Kubebuilder: an SDK for building Kubernetes APIs using CRDs
- Out of the Clouds onto the Ground: How to Make Kubernetes Production Grade Anywhere
- Dynamically Expand Volume with CSI and Kubernetes
- KubeVirt: Extending Kubernetes with CRDs for Virtualized Workloads
- Feature Highlight: CPU Manager
- The History of Kubernetes & the Community Behind It
- Kubernetes Wins the 2018 OSCON Most Impact Award
- 11 Ways (Not) to Get Hacked
- How the sausage is made: the Kubernetes 1.11 release interview, from the Kubernetes Podcast
- Resizing Persistent Volumes using Kubernetes
- Dynamic Kubelet Configuration
- CoreDNS GA for Kubernetes Cluster DNS
- Meet Our Contributors - Monthly Streaming YouTube Mentoring Series
- IPVS-Based In-Cluster Load Balancing Deep Dive
- Airflow on Kubernetes (Part 1): A Different Kind of Operator
- Kubernetes 1.11: In-Cluster Load Balancing and CoreDNS Plugin Graduate to General Availability
- Dynamic Ingress in Kubernetes
- 4 Years of K8s
- Say Hello to Discuss Kubernetes
- Introducing kustomize; Template-free Configuration Customization for Kubernetes
- Kubernetes Containerd Integration Goes GA
- Getting to Know Kubevirt
- Gardener - The Kubernetes Botanist
- Docs are Migrating from Jekyll to Hugo
- Announcing Kubeflow 0.1
- Current State of Policy in Kubernetes
- Developing on Kubernetes
- Zero-downtime Deployment in Kubernetes with Jenkins
- Kubernetes Community - Top of the Open Source Charts in 2017
- Kubernetes Application Survey 2018 Results
- Local Persistent Volumes for Kubernetes Goes Beta
- Migrating the Kubernetes Blog
- Container Storage Interface (CSI) for Kubernetes Goes Beta
- Fixing the Subpath Volume Vulnerability in Kubernetes
- Kubernetes 1.10: Stabilizing Storage, Security, and Networking
- Principles of Container-based Application Design
- Expanding User Support with Office Hours
- How to Integrate RollingUpdate Strategy for TPR in Kubernetes
- Apache Spark 2.3 with Native Kubernetes Support
- Kubernetes: First Beta Version of Kubernetes 1.10 is Here
- Reporting Errors from Control Plane to Applications Using Kubernetes Events
- Core Workloads API GA
- Introducing client-go version 6
- Extensible Admission is Beta
- Introducing Container Storage Interface (CSI) Alpha for Kubernetes
- Kubernetes v1.9 releases beta support for Windows Server Containers
- Five Days of Kubernetes 1.9
- Introducing Kubeflow - A Composable, Portable, Scalable ML Stack Built for Kubernetes
- Kubernetes 1.9: Apps Workloads GA and Expanded Ecosystem
- Using eBPF in Kubernetes
- PaddlePaddle Fluid: Elastic Deep Learning on Kubernetes
- Autoscaling in Kubernetes
- Certified Kubernetes Conformance Program: Launch Celebration Round Up
- Kubernetes is Still Hard (for Developers)
- Securing Software Supply Chain with Grafeas
- Containerd Brings More Container Runtime Options for Kubernetes
- Kubernetes the Easy Way
- Enforcing Network Policies in Kubernetes
- Using RBAC, Generally Available in Kubernetes v1.8
- It Takes a Village to Raise a Kubernetes
- kubeadm v1.8 Released: Introducing Easy Upgrades for Kubernetes Clusters
- Five Days of Kubernetes 1.8
- Introducing Software Certification for Kubernetes
- Request Routing and Policy Management with the Istio Service Mesh
- Kubernetes Community Steering Committee Election Results
- Kubernetes 1.8: Security, Workloads and Feature Depth
- Kubernetes StatefulSets & DaemonSets Updates
- Introducing the Resource Management Working Group
- Windows Networking at Parity with Linux for Kubernetes
- Kubernetes Meets High-Performance Computing
- High Performance Networking with EC2 Virtual Private Clouds
- Kompose Helps Developers Move Docker Compose Files to Kubernetes
- Happy Second Birthday: A Kubernetes Retrospective
- How Watson Health Cloud Deploys Applications with Kubernetes
- Kubernetes 1.7: Security Hardening, Stateful Application Updates and Extensibility
- Managing microservices with the Istio service mesh
- Draft: Kubernetes container development made easy
- Kubernetes: a monitoring guide
- Kubespray Ansible Playbooks foster Collaborative Kubernetes Ops
- Dancing at the Lip of a Volcano: The Kubernetes Security Process - Explained
- How Bitmovin is Doing Multi-Stage Canary Deployments with Kubernetes in the Cloud and On-Prem
- RBAC Support in Kubernetes
- Configuring Private DNS Zones and Upstream Nameservers in Kubernetes
- Advanced Scheduling in Kubernetes
- Scalability updates in Kubernetes 1.6: 5,000 node and 150,000 pod clusters
- Dynamic Provisioning and Storage Classes in Kubernetes
- Five Days of Kubernetes 1.6
- Kubernetes 1.6: Multi-user, Multi-workloads at Scale
- The K8sPort: Engaging Kubernetes Community One Activity at a Time
- Deploying PostgreSQL Clusters using StatefulSets
- Containers as a Service, the foundation for next generation PaaS
- Inside JD.com's Shift to Kubernetes from OpenStack
- Run Deep Learning with PaddlePaddle on Kubernetes
- Highly Available Kubernetes Clusters
- Fission: Serverless Functions as a Service for Kubernetes
- Running MongoDB on Kubernetes with StatefulSets
- How we run Kubernetes in Kubernetes aka Kubeception
- Scaling Kubernetes deployments with Policy-Based Networking
- A Stronger Foundation for Creating and Managing Kubernetes Clusters
- Kubernetes UX Survey Infographic
- Kubernetes supports OpenAPI
- Cluster Federation in Kubernetes 1.5
- Windows Server Support Comes to Kubernetes
- StatefulSet: Run and Scale Stateful Applications Easily in Kubernetes
- Five Days of Kubernetes 1.5
- Introducing Container Runtime Interface (CRI) in Kubernetes
- Kubernetes 1.5: Supporting Production Workloads
- From Network Policies to Security Policies
- Kompose: a tool to go from Docker-compose to Kubernetes
- Kubernetes Containers Logging and Monitoring with Sematext
- Visualize Kubelet Performance with Node Dashboard
- CNCF Partners With The Linux Foundation To Launch New Kubernetes Certification, Training and Managed Service Provider Program
- Bringing Kubernetes Support to Azure Container Service
- Modernizing the Skytap Cloud Micro-Service Architecture with Kubernetes
- Introducing Kubernetes Service Partners program and a redesigned Partners page
- Tail Kubernetes with Stern
- How We Architected and Run Kubernetes on OpenStack at Scale at Yahoo! JAPAN
- Building Globally Distributed Services using Kubernetes Cluster Federation
- Helm Charts: making it simple to package and deploy common applications on Kubernetes
- Dynamic Provisioning and Storage Classes in Kubernetes
- How we improved Kubernetes Dashboard UI in 1.4 for your production needs
- How we made Kubernetes insanely easy to install
- How Qbox Saved 50% per Month on AWS Bills Using Kubernetes and Supergiant
- Kubernetes 1.4: Making it easy to run on Kubernetes anywhere
- High performance network policies in Kubernetes clusters
- Creating a PostgreSQL Cluster using Helm
- Deploying to Multiple Kubernetes Clusters with kit
- Cloud Native Application Interfaces
- Security Best Practices for Kubernetes Deployment
- Scaling Stateful Applications using Kubernetes Pet Sets and FlexVolumes with Datera Elastic Data Fabric
- Kubernetes Namespaces: use cases and insights
- SIG Apps: build apps for and operate them in Kubernetes
- Create a Couchbase cluster using Kubernetes
- Challenges of a Remotely Managed, On-Premises, Bare-Metal Kubernetes Cluster
- Why OpenStack's embrace of Kubernetes is great for both communities
- A Very Happy Birthday Kubernetes
- Happy Birthday Kubernetes. Oh, the places you’ll go!
- The Bet on Kubernetes, a Red Hat Perspective
- Bringing End-to-End Kubernetes Testing to Azure (Part 2)
- Dashboard - Full Featured Web Interface for Kubernetes
- Steering an Automation Platform at Wercker with Kubernetes
- Citrix + Kubernetes = A Home Run
- Cross Cluster Services - Achieving Higher Availability for your Kubernetes Applications
- Stateful Applications in Containers!? Kubernetes 1.3 Says “Yes!”
- Thousand Instances of Cassandra using Kubernetes Pet Set
- Autoscaling in Kubernetes
- Kubernetes in Rancher: the further evolution
- Five Days of Kubernetes 1.3
- Minikube: easily run Kubernetes locally
- rktnetes brings rkt container engine to Kubernetes
- Updates to Performance and Scalability in Kubernetes 1.3 -- 2,000 node 60,000 pod clusters
- Kubernetes 1.3: Bridging Cloud Native and Enterprise Workloads
- Container Design Patterns
- The Illustrated Children's Guide to Kubernetes
- Bringing End-to-End Kubernetes Testing to Azure (Part 1)
- Hypernetes: Bringing Security and Multi-tenancy to Kubernetes
- CoreOS Fest 2016: CoreOS and Kubernetes Community meet in Berlin (& San Francisco)
- Introducing the Kubernetes OpenStack Special Interest Group
- SIG-UI: the place for building awesome user interfaces for Kubernetes
- SIG-ClusterOps: Promote operability and interoperability of Kubernetes clusters
- SIG-Networking: Kubernetes Network Policy APIs Coming in 1.3
- How to deploy secure, auditable, and reproducible Kubernetes clusters on AWS
- Adding Support for Kubernetes in Rancher
- Container survey results - March 2016
- Configuration management with Containers
- Using Deployment objects with Kubernetes 1.2
- Kubernetes 1.2 and simplifying advanced networking with Ingress
- Using Spark and Zeppelin to process big data on Kubernetes 1.2
- AppFormix: Helping Enterprises Operationalize Kubernetes
- Building highly available applications using Kubernetes new multi-zone clusters (a.k.a. 'Ubernetes Lite')
- 1000 nodes and beyond: updates to Kubernetes performance and scalability in 1.2
- Five Days of Kubernetes 1.2
- How container metadata changes your point of view
- Scaling neural network image classification using Kubernetes with TensorFlow Serving
- Kubernetes 1.2: Even more performance upgrades, plus easier application deployment and management
- ElasticBox introduces ElasticKube to help manage Kubernetes within the enterprise
- Kubernetes in the Enterprise with Fujitsu’s Cloud Load Control
- Kubernetes Community Meeting Notes - 20160225
- State of the Container World, February 2016
- KubeCon EU 2016: Kubernetes Community in London
- Kubernetes Community Meeting Notes - 20160218
- Kubernetes Community Meeting Notes - 20160211
- ShareThis: Kubernetes In Production
- Kubernetes Community Meeting Notes - 20160204
- Kubernetes Community Meeting Notes - 20160128
- State of the Container World, January 2016
- Kubernetes Community Meeting Notes - 20160114
- Kubernetes Community Meeting Notes - 20160121
- Why Kubernetes doesn’t use libnetwork
- Simple leader election with Kubernetes and Docker
- Creating a Raspberry Pi cluster running Kubernetes, the installation (Part 2)
- Managing Kubernetes Pods, Services and Replication Controllers with Puppet
- How Weave built a multi-deployment solution for Scope using Kubernetes
- Creating a Raspberry Pi cluster running Kubernetes, the shopping list (Part 1)
- Monitoring Kubernetes with Sysdig
- One million requests per second: Dependable and dynamic distributed systems at scale
- Kubernetes 1.1 Performance upgrades, improved tooling and a growing community
- Kubernetes as Foundation for Cloud Native PaaS
- Some things you didn’t know about kubectl
- Kubernetes Performance Measurements and Roadmap
- Using Kubernetes Namespaces to Manage Environments
- Weekly Kubernetes Community Hangout Notes - July 31 2015
- The Growing Kubernetes Ecosystem
- Weekly Kubernetes Community Hangout Notes - July 17 2015
- Strong, Simple SSL for Kubernetes Services
- Weekly Kubernetes Community Hangout Notes - July 10 2015
- Announcing the First Kubernetes Enterprise Training Course
- How did the Quake demo from DockerCon Work?
- Kubernetes 1.0 Launch Event at OSCON
- The Distributed System ToolKit: Patterns for Composite Containers
- Slides: Cluster Management with Kubernetes, talk given at the University of Edinburgh
- Cluster Level Logging with Kubernetes
- Weekly Kubernetes Community Hangout Notes - May 22 2015
- Kubernetes on OpenStack
- Docker and Kubernetes and AppC
- Weekly Kubernetes Community Hangout Notes - May 15 2015
- Kubernetes Release: 0.17.0
- Resource Usage Monitoring in Kubernetes
- Weekly Kubernetes Community Hangout Notes - May 1 2015
- Kubernetes Release: 0.16.0
- AppC Support for Kubernetes through RKT
- Weekly Kubernetes Community Hangout Notes - April 24 2015
- Borg: The Predecessor to Kubernetes
- Kubernetes and the Mesosphere DCOS
- Weekly Kubernetes Community Hangout Notes - April 17 2015
- Introducing Kubernetes API Version v1beta3
- Kubernetes Release: 0.15.0
- Weekly Kubernetes Community Hangout Notes - April 10 2015
- Faster than a speeding Latte
- Weekly Kubernetes Community Hangout Notes - April 3 2015
- Participate in a Kubernetes User Experience Study
- Weekly Kubernetes Community Hangout Notes - March 27 2015
- Kubernetes Gathering Videos
- Welcome to the Kubernetes Blog!
Scheduling the removal of the kube-proxy IPVS backend
The ipvs backend for kube-proxy was an experiment in providing a
service proxy implementation with better performance than the
iptables backend. While it succeeded in that goal, the kernel IPVS
API has turned out to be a bad match for the Kubernetes Services API,
and the ipvs backend was never able to implement all of the edge
cases of Kubernetes Service functionality correctly.
In Kubernetes 1.29, the project introduced the nftables backend for
kube-proxy, which became GA in 1.33. Part of the plan behind adding
the nftables backend was the idea that it would eventually replace
both the iptables and ipvs backends. Because the ipvs backend in
particular is complex, not 100% feature complete, and architecturally
quite different from the other two backends (and thus more work to
maintain), Kubernetes SIG Network decided to officially deprecate it
as of Kubernetes 1.35. Although it currently remains available in
kube-proxy (with a warning about its deprecation), we (Kubernetes SIG
Network) now plan to remove it in Kubernetes 1.40. All ipvs users
should therefore be planning to migrate to either the nftables or
iptables backends.
Why was the ipvs backend written?
The ipvs backend was originally designed to fix two different kinds
of performance problems in the iptables backend:
First,
iptableshad problems with data plane latency (the speed at which clients could establish new connections to services), because kube-proxy would write a separate iptables rule to match each service IP, so every new connection would need to be matched against a long series of rules that was O(n) in the number of services in the cluster.Second, it had an even larger problem with control plane latency (the speed at which kube-proxy could process changes to services and endpoints and update the data plane rules), because kube-proxy would rewrite the entire iptables ruleset every time any service or endpoint changed. While the list of rules itself was only O(n) in the total number of endpoints, the amount of time, RAM, and CPU that it took to process the updated ruleset grew faster than that. In very large clusters, kube-proxy would use 100% of one CPU all the time just to process iptables updates. (In fact, the control plane scaling problem was bad enough that for most users it was not even possible to grow the cluster to the size where the data plane scaling problem became noticeable.)
The ipvs backend solved both of these scaling problems. The data
plane problem was solved mostly automatically, because the kernel IPVS
subsystem uses hash tables rather than linear lookup. The
control-plane problem was solved by not rewriting the entire
ruleset on every update. (The kernel IPVS API doesn’t have a way of
doing batch updates anyway, so this was essentially forced.)
Why is the ipvs backend now being deprecated?
Almost from the start, the ipvs backend also had problems.
The first version didn’t have 100% compatibility with the iptables
backend, but it was assumed that this would be fixed over time.
Unfortunately, the original authors moved on to other things outside
the Kubernetes ecosystem (as inevitably sometimes happens in open
source), and it took a while to find new people interested in
maintaining the code.
Meanwhile, as people began using the ipvs backend in larger and
larger clusters, its own scaling problems started to emerge. In
particular, the lack of batch update operations in the IPVS API meant
that while the ipvs backend was faster than iptables when making a
few small changes, it was potentially slower when making a large
number of changes at once. And because the IPVS API is fairly limited
in what it can do, kube-proxy's ipvs backend needed to fall back to
using iptables to implement various features of Kubernetes Services
beyond basic "load balancing", making it run into some of the same
scaling problems the iptables backend has. While the ipvs backend
tried to mitigate this by using ipset for some of its rules, this
again ran into the problem of not having a batch update API.
Additionally, the fact that the ipvs backend uses iptables means
that it has the same long-term support problem as the iptables
backend: Red Hat's plan to remove the iptables API entirely from Red
Hat Enterprise Linux means the ipvs kube-proxy backend will also no
longer be available to Kubernetes users on RHEL.
What about IPVS schedulers?
Beyond performance, the other reason some users were excited about the
ipvs backend was the ability to pick from among various IPVS
"schedulers" for distributing traffic to endpoints, rather than
relying on the simple random selection used by iptables (and
nftables).
Unfortunately, the way IPVS is used by kube-proxy prevents it from
being able to use most of the IPVS schedulers in a useful way. The
problem is that kube-proxy is a distributed service that runs on all
nodes in the cluster, but the kube-proxy processes on different nodes
do not share their IPVS scheduler state, so each node picks endpoints
without knowing what is happening with clients on any other node. Even
the schedulers that are designed to work in a distributed way
generally don't work better than random selection in the context of
kube-proxy. For example, the mh scheduler uses Maglev Hashing to
select endpoints, which theoretically allows for extremely efficient
distributed load balancing. But that efficiency is only in the context
of a specific sort of two-layer load-balancer design, and kube-proxy
does not have the right architecture to be able to act as either of
the two layers in that design.
Switching away from ipvs
Switching to nftables
If you are running Kubernetes 1.36 or later, and your nodes are
running a Linux distribution with kernel 5.13 or later, and you use
the ipvs proxy mode, then Kubernetes recommends switching to the
nftables mode of kube-proxy. (Although the nftables backend is GA
—and fully usable—as of Kubernetes 1.33, there have been some
performance- and scaling-related fixes since then that that weren't
backported to releases earlier than 1.36.)
The nftables backend is fully supported, feature-complete, and even
has very slightly better performance than ipvs:
(Taller bars represent longer latency. The "p50" group is the average
connection establishment latency. Note that even the largest numbers
in this graph are quite tiny; the point is not to claim that
nftables is (noticeably) faster than ipvs, just to show that,
unlike iptables, it's not slower.)
Switching is as easy as changing the kube-proxy configuration and
restarting kube-proxy. For example, in clusters deployed via kubeadm,
change mode: ipvs to mode: nftables in the kube-proxy ConfigMap
and then restart the kube-proxy pods, as described in the
documentation on reconfiguring a kubeadm cluster. (If you are
overriding other ipvs-specific kube-proxy configuration, you will
also need to change those options to the corresponding nftables
ones.)
Switching to iptables: "It's Not As Slow As You Think!"
If you can't switch to nftables, either because you are on an older
version of Kubernetes, or because your cluster runs on an older Linux
distribution that isn't new enough to support the nftables backend,
you may be able to just switch to the iptables backend, which has
improved greatly since the original creation of the ipvs backend in
2017:
The generated ruleset has been optimized to remove redundant and irrelevant rules (#57461, #60306, #96959, #108251), reducing data plane latency.
The code that generates the ruleset has been optimized to do fewer allocations, and to avoid unnecessarily recomputing the same information multiple times (#65755, #65902, #67948, #85617, #90103, #97238), reducing CPU and memory usage when resyncing.
Rather than doing “check the current state of the rules, then update them to the new state”,
iptableskube-proxy now does “assume the current state is correct, update to the new state based on that assumption, and only explicitly check the current state if that update fails”. This greatly improves the resyncing speed, because it turns out that even reading the iptables ruleset is extremely slow in large clusters (#81517, #110334, #114181).To the extent possible within the constraints of the iptables APIs, kube-proxy’s iptables backend now only updates iptables rules for services/endpoints that have changed since the last sync, rather than rewriting the entire ruleset every time (#110268 / KEP-3453).
The net result of this is that the iptables backend of 2026, while
still not as good as the ipvs backend, is at least much better
than the iptables backend of 2017 was, and many people who found
iptables unusable in the past would not have problems with it now.
As with switching to nftables, switching to iptables is mostly
just a matter of updating your kube-proxy configuration, although for
technical reasons, in this case we recommend that you reboot each node
after changing the kube-proxy configuration, rather than merely
restarting kube-proxy.
Learn more
KEP-5495 gives the full timeline and plan for kube-proxy IPVS deprecation.
The blog post "NFTables mode for kube-proxy" has more information about
nftablesmode and its performance relative toipvsandiptables.
From Working Group to SIG Architecture: spotlight on AI Conformance
In this SIG Architecture spotlight we talked with Janet Kuo (Google) and Yuan Tang (TensorScale AI) about the journey of AI Conformance in Kubernetes, from the initial Working Group to the integration SIG Architecture as a subproject.
Introductions
Frederico Muñoz and Kirti Goyal (FK): Hi Janet, Yuan, and thank you for the opportunity to learn a bit more about AI Conformance. Can you tell us a bit about yourselves and how you got involved in Kubernetes?
Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. I started my journey writing the core workload controllers that standardized how applications are orchestrated on Kubernetes. Today, I lead SIG Apps as a Co-chair & TL, and also lead the AI Conformance program. My current focus is to ensure Kubernetes remains the best platform for the next generation of workloads, such as AI.
Yuan Tang: I'm a co-founder of TensorScale AI where we build AI infrastructure for world models. Prior to that, I was a Senior Principal Software Engineer at Red Hat, where I led the development of Red Hat AI platform and upstream open source strategy. I became involved in Kubernetes early on by fostering collaborations between the Kubernetes community and projects I was leading, including Kubeflow, Argo, and KServe. I later helped found and lead WG Serving and the AI Conformance program, advancing innovation and strengthening community collaboration around running AI workloads on Kubernetes.
Program, Working Group, Subproject: the many shapes of AI Conformance
FK: AI Conformance started as a CNCF Program, was it then you got involved?
Janet Kuo: I was actually involved from day one since I pitched the original idea. It became obvious then that everyone was starting to run AI on Kubernetes, but vendors were building their infrastructure stacks in different ways. I got worried about ecosystem fragmentation. So I wrote up the initial Kubernetes AI Conformance Proposal, took it to the CNCF leadership to get their buy-in, and started the AI Conformance Program. Once we had the green light, I reached out to my fellow co-chairs Yuan Tang, Rita Zhang, and Mario Fahlandt to get the community rallied around it.
Yuan Tang: Yes. I was a co-chair of WG Serving at that time and some of the initiatives such as the Gateway AI Inference Extension quickly gained traction. There was a strong interest among the community to include inference-related requirements in the Kubernetes AI Conformance program.
FK: How would you describe the motivation behind the creation of the AI Conformance Working Group?
Janet and Yuan: The motivation behind creating the Kubernetes AI Conformance Working Group (AI WG) was to address the rapid convergence of AI workloads onto Kubernetes while preventing ecosystem fragmentation.
As AI adoption accelerated, organizations began running increasingly complex workloads, including distributed training, large-scale inference, GPU-intensive pipelines, and emerging agentic AI systems across many different Kubernetes platforms. However, vendors were implementing AI infrastructure capabilities differently, especially around GPUs, networking, scheduling, and inference orchestration. That created portability challenges, operational inconsistency, and growing vendor lock-in concerns.
The AI WG was formed to bring the community together around:
- vendor-neutral AI infrastructure standards,
- interoperable Kubernetes-based AI platforms,
- consistent operational behavior for AI workloads,
- and common requirements for scalable AI deployment.
FK: WGs are temporary by nature, based on outcomes: would you say the main goals of the WG were achieved?
Janet and Yuan: Yes, the WG achieved its primary initial goals.
The WG has succeeded in moving the ecosystem from: fragmented, vendor-specific AI infrastructure approaches to a shared, community-defined baseline for AI workloads on Kubernetes.
Some concrete indicators of success include:
- Launching the Certified Kubernetes AI Conformance Program
- Defining formal Kubernetes AI Requirements (KARs)
- Getting broad participation from major ecosystem players
- Growing certified platforms from 18 to 31 within a few months
- Aligning around newer Kubernetes primitives needed for AI workloads
- Establishing a roadmap for automated conformance verification
Most importantly, the WG created consensus that AI workloads are now a first-class Kubernetes use case.

FK: AI Conformance was specifically highlighted at the latest KubeCon: is this a reflection of the growing importance of AI throughout the project?
Janet and Yuan: Yes, the prominence of AI Conformance at the latest KubeCon + CloudNativeCon is a strong reflection of how central AI has become to the Kubernetes ecosystem overall.
A few years ago, AI workloads on Kubernetes were still viewed as somewhat specialized, mostly tied to ML platforms or GPU-heavy research environments. What’s changed is that AI is now becoming a mainstream infrastructure concern for nearly every large-scale platform operator.
That shift naturally elevates AI from an application running on Kubernetes to a core workload category Kubernetes must optimize for.
The AI Conformance initiative reflects that evolution. It signals that the community now sees AI infrastructure requirements, things like accelerator scheduling, high-performance networking, workload-aware orchestration, and low-latency inference, as important enough to standardize at the ecosystem level.
Its visibility at KubeCon also shows a broader strategic realization: if Kubernetes is going to remain the universal control plane for modern infrastructure, it needs to provide a consistent foundation for AI workloads across clouds, on-prem, and edge environments.
So the emphasis on AI Conformance is not just about chasing AI hype; it's about Kubernetes adapting to a major shift in workload patterns and ensuring the ecosystem evolves cohesively rather than fragmenting into vendor-specific AI stacks.

From Working Group to SIG Architecture subproject
FK: How would you describe the change from WG to SIG? Is it a reflection of AI as something that requires ongoing architectural governance?
Janet and Yuan: In Kubernetes, Working Groups are meant to be temporary. Our WG had a specific goal to get the initial Kubernetes AI Conformance requirements published and get the certification process off the ground. Once we hit those milestones, we archived the WG and moved AI Conformance to be a permanent subproject under SIG Architecture.
Moving to SIG Architecture definitely proves that AI needs ongoing governance. AI isn’t just a passing trend or a one-off workload. It has become a core part of what Kubernetes runs every day. Because SIG Architecture already owns overall Kubernetes Conformance and API design principles, housing the AI Conformance subproject there makes perfect sense. It makes sure our AI Conformance requirements stay aligned with the rest of the Kubernetes ecosystem for the long haul.
FK: And with that change, how would you describe the main goals and areas of intervention of the subproject?
Janet and Yuan: Now that we’re a subproject, our main goal is to keep the AI Conformance standards current, vendor-neutral, and standard-setting. We want to make sure Kubernetes captures the latest advancements in AI infrastructure while keeping the ecosystem fair for everyone. When we find gaps where no clear community pattern exists, we help foster and form new standards to solve them.
AI Conformance and Kubernetes
FK: AI Infrastructure is evolving very quickly, while Kubernetes APIs are designed to remain stable for years. Is this something you consider a challenge?
Janet and Yuan: It’s definitely a tricky balance. AI moves very fast, while Kubernetes is designed to be stable and boring. But we actually structured the program around this exact tension.
First, we use a tiered graduation model. When new AI infrastructure concepts emerge, we introduce them as recommended SHOULD requirements. That gives the ecosystem room to experiment and agree on best practices without locking Kubernetes into rigid APIs too early. Once those APIs stabilize and with ecosystem adoption, we graduate them to mandatory MUST requirements.
Second, we intentionally keep our focus on the infrastructure layer. We test for core capabilities like accelerator support through Dynamic Resource Allocation (DRA), rather than getting bogged down in specific AI frameworks or model servers. That keeps the base platform stable while letting the AI tooling evolve as fast as it needs to.
FK: DRA has evolved, also due to the work of WG Device Management, as an important part of Kubernetes (and especially so for AI workloads), with SIG Architecture having an important role in how it’s integrated in Kubernetes: do you find any parallel in this? Specifically, do you think there will be a heavier focus in the integration with the evolving Kubernetes architecture as part of SIG Architecture?
Janet and Yuan: There is a huge parallel there. Whenever we face a massive, complicated challenge that touches multiple parts of the codebase and involves multiple SIGs, a Working Group is the best place to start. It lets experts from different SIGs get together to figure out the problem, and build consensus.
Being part of SIG Architecture means there will be a much heavier focus on how AI capabilities weave into the evolving Kubernetes architecture. Instead of treating AI as an add-on layer, we’re looking at how core Kubernetes primitives, like scheduling and resource allocation, need to adapt at a foundational level to support the next decade of AI workloads.
Final remarks
FK: For interested contributors, what’s the best way to get involved with the AI Conformance subproject?
Janet and Yuan: We're always looking for new contributors. The easiest way to get started is by checking out our
main repo at kubernetes-sigs/ai-conformance and
join our meetings. If
you're interested in defining new AI Conformance requirements, you can jump into PR reviews or
propose your own using our KAR template. If
you want to write code, we always need help expanding our automated Go test suites in the test/ directory.
For vendors looking to certify their platforms, all the details are at cncf/k8s-ai-conformance.
FK: Any final comments you would like to share?
Janet and Yuan: We’ve been blown away by how fast the community has rallied around this. When we first announced the AI Conformance program during the keynote at KubeCon NA 2025, we weren’t sure what the response would be. But by the time we gave our update keynote at KubeCon EU 2026, we had grown from 18 certified platforms to 31 in just a few months.
That kind of growth shows just how much the industry wanted a unified, vendor-neutral standard. It gives organizations the confidence that they can run their AI workloads on any certification Kubernetes platform without getting locked into a proprietary stack.
If you’re a platform vendor, we’d love to see you get certified. If you’re a developer interested in the intersection of AI and cloud-native infrastructure, come join us in SIG Architecture!
FK: Excellent, thank you both for guiding us in this journey on AI and AI Conformance in Kubernetes.
Announcing the Workload Conformance Working Group
We are excited to announce that the Kubernetes community now has a Workload Conformance Working Group. The working group is writing a shared standard for how workloads behave at runtime on Kubernetes. To get it right, we need input from the people who build those workloads and from the people who run the clusters they're deployed on.
Why workloads need a shared standard
Certified Kubernetes tests the cluster, and Kubernetes AI Conformance tests whether a cluster can run AI and ML workloads. Nothing comparable exists for the workloads themselves, so cluster operators still check each application by hand before routine maintenance or a Kubernetes upgrade.
The problems they find are familiar. A routine node drain is often how you discover that an application can't tolerate disruption. An upgrade can break an application that still depends on a removed API version. Others show up in day-to-day operation: workloads that ask for privileges they don't need, or reserve resources they never use.
Kubernetes already documents much of this, including security and handling disruptions. Even so, every cluster operator ends up repeating the same assessment, and application maintainers get the same questions from each environment they support. A common specification with repeatable tests would give both groups the same baseline to start from.
What the working group will deliver
The charter sets out three deliverables:
- A specification of workload behavior for each Kubernetes minor version.
- A method for testing each requirement against a running workload.
- Upstream conformance tooling that implements those tests.
The first step is deciding which workloads and behaviors the specification covers. The scope includes security, resilience during disruptions, resource use, networking, storage, API usage, and observability, but the group hasn't yet agreed on specific requirements or on how to test them. However, one rule is already set: every requirement needs an objective test that a test binary can run, so nothing subjective goes into the specification.
The specification covers how a workload runs, and importantly not what it does. Consequently, application logic is out of scope of workload conformance. The binary won't retest anything Certified Kubernetes already covers, and it won't require a particular workload packaging format, or vendor tool like policy engine, security scanner, etc.
How the work is organized
Two Special Interest Groups (SIGs) are sponsors for the working group.
- SIG Architecture will own the specification and tooling through a subproject, which will be created once the first artifacts are proposed by the Working Group.
- SIG Apps will bring the application perspective.
Whenever a requirement touches another Kubernetes SIG's area, such as node behavior or networking, the working group will initiate conversations with those SIGs as well.
The working group is a forum where SIGs can get involved and shape the direction of Workload Conformance while the specification is being written. We're aiming for a stable first specification and working tests within two to three Kubernetes release cycles.
Get involved
The specification will and needs to be built from real examples, so we want to hear your experiences. If you run clusters, tell us how you assess workloads today and what broke during your last upgrade or maintenance window. If you maintain an application, tell us what it needs from the cluster and where the existing guidance left you guessing.
- Beyond examples, the work needs reviewers for proposed requirements and contributors to write the tests and tooling.
- When early versions of the tests are ready, we also want early testers to run the binary against your own workloads and tell us where the results differ between runs or environments.
- If a requirement touches your SIG's area, help us check its scope and the test behind it.
- And because the subproject will outlive the working group, we're also looking for people who want to maintain it for the long term.
You can find us in #wg-workload-conformance on Kubernetes Slack (visit slack.k8s.io for an invitation) and on the mailing list.
The working group meets every other Wednesday at 15:00 UTC.
The Zoom link and the agenda are in the meeting notes.
If you want to raise a topic, add it to the agenda.
Joining the mailing list usually adds the meeting invite to your calendar.
Thanks to everyone who reviewed and supported the working group proposal.
Why Is My Kubernetes Job Still Pending? Tracing Kueue Admission with Headlamp
You submit a Kubernetes Job and wait for its Pods to appear. A minute passes. Then another. The Job is still suspended, there are no application logs, and there may not even be a Pod to inspect.
In a cluster managed by Kueue, this can be completely normal. Kueue may still be deciding whether the Job can use the requested resources. The useful question is not only “Why is the Pod pending?” It is first “Has Kueue admitted the Workload?”
The Kueue plugin for Headlamp brings that investigation into the Headlamp UI. Instead of reconstructing the admission path from several commands and large YAML objects, you can begin with the Workload, read its condition message, and follow links through the LocalQueue, ClusterQueue, ResourceFlavor, and Cohort.
This article follows that Headlamp-first path. The command line appears only in a short optional section for reproducing the example.
Before you begin
You need a Kubernetes cluster with Kueue installed,
a Headlamp installation, and the
0.1.0-alpha Kueue plugin.
Your Kubernetes identity must be allowed to read the Kueue resources used in this article.
The alpha release provides read-only views for Kueue v1beta2 Cohorts, ClusterQueues, LocalQueues,
ResourceFlavors, and Workloads. It does not add its own create, edit, stop, resume, or delete
workflows. Headlamp still provides its standard YAML and resource actions where your permissions
allow them.
First, separate admission from Pod scheduling
Kueue and the Kubernetes scheduler answer different questions.
Kueue admission: Should this Workload start now, considering queue policy, quota, ResourceFlavors, admission checks, and other waiting Workloads?
Pod scheduling: After the Workload is admitted and the Job resumes, which nodes can run its Pods?
That boundary changes the investigation. If the Workload is not admitted, stay in the Kueue section
of Headlamp. If it is admitted but its Pods are still Pending, move to Headlamp's normal Job and
Pod views and inspect scheduler events.

Kueue admission architecture
The Headlamp-first troubleshooting path
The quickest investigation starts with the Workload, not with a shell.

Tracing a pending Job through Headlamp
Step 1: Open the Workloads view
In Headlamp, open Kueue → Workloads. Select the namespace that contains the Job, then find the Workload created for it.
Kueue creates a Workload to represent the
resource request and admission state of a Job. In the alpha plugin, owner references are shown as
Kind/name text but are not yet clickable. The Workload name normally contains the Job name, so the
namespace filter and table search are the simplest ways to locate it.
Before opening the detail page, the list already gives you a useful first scan:
LocalQueue
priority and priority class
active state
admitted state
completion state
a readable current status such as
Admitted,Finished, orInadmissibleage
If there is no matching Workload, check three things before going further: the selected namespace, whether the Job was submitted to a LocalQueue, and whether your Kubernetes account can list Workloads. The plugin performs permission checks and displays an access message when the current identity cannot read a resource.
Step 2: Read status, conditions, and events
Open the Workload. Start with its current status and conditions, not with the full YAML.
For admission troubleshooting, the most important condition types are QuotaReserved and
Admitted.
Admitted=Truemeans Kueue has admitted the Workload. The Job can resume, so any remainingPendingPods belong to the scheduling branch of the investigation.QuotaReserved=Falsewith a reason such asPendingmeans Kueue has not reserved quota. The condition message explains what prevented the reservation.An
Inadmissiblestatus means the Workload currently cannot fit the available queue configuration. A missing covered resource, unsuitable ResourceFlavor, stopped queue, or admission check can all lead you here.
For example, a condition message may report that Kueue could not assign a ResourceFlavor because the ClusterQueue has insufficient CPU quota. That message is the first real answer to “Why is my Job still pending?” The rest of the Headlamp journey explains the configuration behind it.
Also check the Events section. Conditions describe the current state; events help show what recently changed or what Kueue attempted.
Step 3: Compare the Workload's request with its assignment
Stay on the Workload detail page and inspect the PodSets and resource requests. The plugin shows the requested Pod count, CPU, memory, and other resources that Kueue evaluates together.
This matters because a Job with parallelism: 3 and a one-CPU request per Pod asks Kueue for three
CPUs in total. Looking at only one Pod's template would underestimate the Workload.
If quota has already been reserved, the same page also shows the assigned ClusterQueue, ResourceFlavors, and admitted resource usage. If those admission assignments are absent, the Workload has not reached that point yet.
In our example, the Workload requests:
three Pods
three CPUs in total
600 MiB of memory in total
The next question is where that request entered Kueue.
Step 4: Follow the linked LocalQueue
Use the LocalQueue link on the Workload detail page. A LocalQueue is the namespace-level entry point used by the Job. It connects namespaced Workloads to a cluster-scoped ClusterQueue.
The LocalQueue page shows:
its namespace and connected ClusterQueue
stop policy and current status
pending, reserving, and admitted Workload counts
Kueue conditions and Kubernetes events
related Workloads in the same namespace
This page answers two practical questions: Did the Job enter the queue you expected, and is it the only Workload waiting there?
If the LocalQueue is inactive or stopped, that may already explain the wait. If it is active, follow the linked ClusterQueue to inspect the capacity behind it.
Step 5: Inspect ClusterQueue capacity and usage
A ClusterQueue represents the cluster-level quota and admission policy available to one or more LocalQueues.
On its Headlamp detail page, compare the Workload request with:
covered resources
nominal quota for each ResourceFlavor
current reservations and usage
borrowing and lending limits
pending, reserving, and admitted Workload counts
queueing strategy and stop policy
preemption, fair-sharing, flavor-fungibility, and admission-check settings
Kueue conditions and Kubernetes events
For the example in this article, the ClusterQueue provides a nominal quota of two CPUs and two Pod slots. The Workload asks for three of each. The condition message and the ClusterQueue page now tell the same story: the Workload does not fit, so Kueue leaves the Job suspended.
This is an admission-capacity problem. It is not evidence that kube-scheduler failed to find a node.
Step 6: Check ResourceFlavor and Cohort when needed
Quota is not always a single pool. A ResourceFlavor can represent a class of compute through node labels, taints, tolerations, or topology settings. Use the ResourceFlavor link from the ClusterQueue when the condition mentions flavor assignment or when the Workload requires a particular hardware class, such as GPU nodes.
If the ClusterQueue belongs to a Cohort, follow that link too. Cohorts let ClusterQueues share unused quota. The Headlamp page shows parent and child Cohorts, member ClusterQueues, referenced ResourceFlavors, resource groups, borrowing and lending limits, and fair-sharing information.
These pages help answer a more subtle question: Is the Workload blocked because the queue has no quota of its own, because it cannot borrow enough, or because no suitable flavor can be assigned?
What we learned without leaving Headlamp
For the sample pending Job, the UI gives us a complete explanation:
The Workload is not admitted.
Its
QuotaReservedcondition is false and the message points to insufficient quota.The Workload requests three CPUs and three Pod slots.
Its LocalQueue is active and points to the expected ClusterQueue.
The ClusterQueue provides only two CPUs and two Pod slots.
Kueue therefore keeps the Job suspended until quota becomes available or the queue configuration changes.
The important part is not that Headlamp invents a new status. It reads the same Kubernetes API objects as other clients, then places the related resources and human-readable values in one connected investigation.
If the Workload is admitted, switch views
Suppose the Workload detail page shows Admitted=True, but the application is still not running.
Kueue has completed its part of the decision. Continue in Headlamp's standard Kubernetes views:
Open Workloads → Jobs and inspect the Job state and events.
Open Workloads → Pods and filter by the Job name or namespace.
Select a
PendingPod and read its Events section.
At this stage, look for scheduler concerns such as insufficient node capacity, node affinity, taints and tolerations, topology constraints, persistent-volume binding, or image-pull failures.
This handoff is the most useful troubleshooting rule in the article:
Not admitted: investigate Kueue resources in Headlamp.
Admitted, but Pods arePending: investigate Job, Pod, node, and scheduler information in Headlamp.
A repeatable Headlamp checklist
When the next Job does not start, use this short path:
Open Kueue → Workloads and select the correct namespace.
Find the Workload associated with the Job.
Check the readable status,
Admitted, andQuotaReservedconditions.Read the condition message and recent events.
Compare PodSet resource requests with any admission assignments.
Follow the LocalQueue link and confirm its status and ClusterQueue.
Follow the ClusterQueue link and compare quota, reservations, usage, and policy.
Inspect ResourceFlavor or Cohort details when the message points to flavor selection or shared quota.
If admitted, move to Headlamp's Job and Pod views.
What Headlamp replaces during this investigation
The plugin does not replace kubectl or
kueuectl for scripts, exact object
output, or changes. It replaces the repetitive UI-less navigation needed during an interactive
investigation.
| Question | Headlamp path | Command-line equivalent |
|---|---|---|
| Is the Workload admitted? | Kueue → Workloads → status and conditions | kubectl describe workload <workload-name> -n <namespace> |
| Which LocalQueue did it enter? | Workload → linked LocalQueue | kubectl get localqueue <localqueue-name> -n <namespace> -o yaml |
| Which ClusterQueue provides quota? | LocalQueue → linked ClusterQueue | kubectl get clusterqueue <clusterqueue-name> -o yaml |
| Which flavor or shared quota matters? | ClusterQueue → ResourceFlavor or Cohort | kubectl get resourceflavor <flavor-name> -o yaml or kubectl get cohort <cohort-name> -o yaml |
| What happened after admission? | Workloads → Jobs and Pods → events | kubectl describe job <job-name> -n <namespace> or kubectl describe pod <pod-name> -n <namespace> |
For scripts and GitOps workflows, the command line remains the right tool. For a person asking “why is this waiting?”, the linked Headlamp path is usually faster to read.
Optional: reproduce the pending scenario
The plugin repository includes sample Cohort, ClusterQueue, LocalQueue, ResourceFlavor, and Job manifests for local testing. Apply those examples to a cluster where Kueue is installed, then open the Kueue section in Headlamp.
To reproduce the specific quota mismatch used in this article, configure a ClusterQueue with two CPUs and two Pod slots, then submit a three-Pod Job that requests one CPU per Pod. The Job enters its LocalQueue, Kueue creates a Workload, and the Workload remains pending because the combined request is larger than the available quota.
Keeping this setup separate from the main walkthrough is intentional: once the resources exist, the investigation itself should happen in Headlamp.
Current alpha limitations
The 0.1.0-alpha release is an initial read-only view of the Kueue admission path.
The plugin does not add its own create, edit, pause, resume, or delete flows.
Workload owner references are displayed as text and are not currently navigable.
The UI depends on the Kubernetes permissions of the identity used by Headlamp.
Metrics and a visual relationship map are possible follow-up areas rather than features of this release.
These limits are useful to state clearly: the plugin's current purpose is to make admission state and resource relationships easier to inspect.
Developed during LFX Mentorship
The plugin was developed during CNCF LFX Mentorship 2026 Term 2 using TypeScript, React, Kubernetes APIs, and Headlamp's plugin system.
Utkarsh Raj implemented the plugin with guidance and review from René Dudfield (@illume) and Kevin Hannon (@kannon92). Feedback from the Headlamp and Kueue communities shaped the resource views, status presentation, navigation, documentation, tests, and first alpha release.
Try it and share feedback
Explore the plugin source and README,
read the
0.1.0-alpha release documentation,
or share feedback in the release issue.
If a pending Workload is still difficult to explain after following the Headlamp path, that is exactly the kind of feedback that can improve the plugin.
The Shift to cgroup v2 in Kubernetes: What You Need to Know
In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications run smoothly without interfering with each other. With the release of Kubernetes v1.31, support for v1 cgroup management moved into maintenance mode. Support for v2 cgroup management has been stable since Kubernetes v1.25.
Compared with cgroup v1, cgroup v2 provides a single unified hierarchy, a more consistent interface, and a stronger foundation for resource isolation and modern resource-management features.
Deprecation of cgroup v1
Kubernetes has deprecated cgroup v1.
Starting with Kubernetes v1.35, failCgroupV1 defaults to true, so the kubelet does not start
on a cgroup v1 node by default. Administrators can temporarily set failCgroupV1: false in the
kubelet configuration file, but removal will
follow the Kubernetes deprecation policy.
Further removal work is tracked in KEP-5573: Remove cgroup v1 support.
If you are still on a release older than v1.35, migrate every Linux node to
cgroup v2 before upgrading, or plan to set the temporary failCgroupV1: false
override. If you are already on v1.35 or later, confirm that every Linux node
runs cgroup v2 (or that you intentionally keep the override). Under the default
configuration, a remaining cgroup v1 node fails during kubelet startup.
For kubeadm-managed clusters, Kubernetes v1.35 also makes this an earlier, stricter check. The
SystemVerification preflight check, provided by k8s.io/system-validators, returns an error during
kubeadm init, kubeadm join, and kubeadm upgrade when it detects cgroup v1 with kubelet v1.35 or
later; with an older kubelet, the check remains a warning. See
kubernetes/system-validators#1.12.1 release notes for details.
The top FAQs cover three main areas: why to migrate, the benefits and drawbacks, and key points to keep in mind when using cgroup v2.
Limitations of cgroup v1 and Improvements with cgroup v2
The Linux kernel documentation describes both interfaces:
Let's enumerate some known issues.
active_file memory is not considered available memory
The kubelet treats active_file memory as not reclaimable. For I/O-intensive workloads, a large
page cache can therefore make the kubelet report memory pressure and evict Pods. This is a
known kubelet issue
(kubernetes/kubernetes#43916); migrating to
cgroup v2 does not by itself change that calculation. The documented workaround is to set equal
memory requests and limits for containers that perform intensive I/O, after measuring an appropriate
value.
Memory QoS updates in Kubernetes v1.36
Memory QoS was introduced as an alpha feature in Kubernetes v1.22 and updated in v1.27. It remains alpha in v1.36, but now separates memory throttling from memory reservation and adds tiered memory protection:
Memory QoS is available only on Linux nodes that use cgroup v2. It relies on the cgroup v2 memory
controller: memory.high provides throttling, while memory.min and memory.low provide hard and
soft protection when tiered reservation is enabled. cgroup v1 cannot provide this protection model.
Enabling the
MemoryQoSfeature gate appliesmemory.highthrottling to Burstable containers. The threshold is derived from the request, limit, andmemoryThrottlingFactor(default0.9).memoryReservationPolicy: Noneis the default. It does not writememory.minormemory.low.memoryReservationPolicy: TieredReservationmaps Guaranteed Pod memory requests tomemory.min(hard protection) and Burstable Pod requests tomemory.low(soft protection). BestEffort Pods receive neither protection.The kubelet exposes Alpha metrics for the total
memory.minandmemory.lowreservations on a node.Kernel 5.9 or later is recommended. On older kernels,
memory.highreclaim can trigger a known livelock; from v1.36 the kubelet logs a warning when Memory QoS is enabled on an affected kernel.
For example, to opt in to tiered protection:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
MemoryQoS: true
memoryReservationPolicy: TieredReservation
memoryThrottlingFactor: 0.9
The overall Kubernetes recommendation is not to enable Alpha features in production; however, if you
judge that memory QoS with tiered reservations is useful for your platform in production, make sure
to test the configuration and account for hard-reserved memory before you enable
the TieredReservation feature gate.
See the
Pod QoS documentation
for the current mapping of Kubernetes QoS classes to cgroup v2 controls.
Container-aware OOM handling
On cgroup v2 nodes, the kubelet defaults singleProcessOOMKill to false. It therefore sets
memory.oom.group for each container cgroup so that an OOM event kills all processes in that
container together, rather than leaving a partially functioning multi-process container. Set
singleProcessOOMKill: true only if you need the cgroup v1-compatible behavior where the kernel
may kill one process at a time. See the
KubeletConfiguration reference
for this setting.
This behavior is scoped to a container cgroup, not the whole Pod. Also, cgroup.kill is a separate
administrative interface: writing 1 to it sends SIGKILL to every process in that cgroup and its
descendants; it does not configure OOM behavior. The cgroup v2 memory controller additionally
provides memory.events counters that monitoring systems and userspace OOM managers can observe.
Rootless support
In cgroup v1, delegating controllers to less privileged containers may be dangerous.
Unlike cgroup v1, cgroup v2 officially supports delegation. Most implementations of rootless containers rely on systemd for delegating v2 controllers to non-root users.
This delegation mechanism is separate from Kubernetes Pod user namespaces, which map container users to unprivileged host users. Pod user namespace support graduated to stable in Kubernetes v1.36; check its filesystem, kernel, CRI runtime, and OCI runtime prerequisites before enabling it.
What else?
- eBPF stories:
- In cgroup v1, device access controls are exposed through interface files.
- The cgroup v2 device controller has no interface files, and is implemented on top of cgroup BPF.
- Cilium attaches BPF cgroup programs for socket-based load balancing. Its
default cgroup root
is
/run/cilium/cgroupv2.
- Pressure Stall Information (PSI)
reports CPU, memory, and I/O contention at node, Pod, and container level. On supported
clusters, the kubelet exposes PSI by default (
KubeletPSIis stable and locked on). PSI requires cgroup v2, Linux 4.20 or later,CONFIG_PSI=y, and a kernel not booted withpsi=0. The kubelet surfaces the data through the Summary API and/metrics/cadvisor. - When migrating, update software that reads the cgroup filesystem directly. The
migration guidance recommends
cAdvisor v0.43.0 or later and lists compatible Java, Node.js, and
automaxprocsversions.
CPU weight conversion in newer OCI runtimes
cgroup v1 uses cpu.shares, whereas cgroup v2 uses cpu.weight. Newer OCI runtimes use an improved
non-linear conversion that preserves the default priority and gives small CPU requests more usable
granularity. The change is implemented in the OCI runtime rather than Kubernetes: it is available
in crun v1.23 and runc v1.3.2. After upgrading a runtime, monitoring or policy tools that predict
exact cpu.weight values may need updates. Read
New Conversion from cgroup v1 CPU Shares to v2 CPU Weight
for the formula, examples, and compatibility considerations.
In-place resource updates
In-place Pod vertical scaling graduated to stable in Kubernetes v1.35. Kubernetes v1.36 then [enabled] (/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/) in-place vertical scaling for Pod-level resources by default, as a Beta feature. The kubelet coordinates changes between the Pod-level and container cgroups so that increases create headroom before container limits grow, while decreases constrain containers before shrinking the Pod-level boundary. Accurate aggregate enforcement for this v1.36 feature requires cgroup v2.
Adopting cgroup version 2
Requirements
Here's what you need to use cgroup v2 with Kubernetes. First up, you need to be using a version of Kubernetes with support for v2 cgroup management; that's been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support.
- You need at least one Linux node; cgroup is a Linux-only concept
- Your OS install must run with cgroup v2 enabled
- The kernel version must be 5.8 or later (5.9 or later is recommended when using memory QoS)
- The container runtime must support cgroup v2. For example:
- containerd v1.4 or later supports cgroup v2; use containerd v2.0 or later for automatic cgroup-driver discovery
- CRI-O v1.20 or later
- The kubelet and the container runtime must both be configured to use the correct cgroup driver. See Configure the kubelet's cgroup driver to match the container runtime cgroup driver.
For now, you can opt back in to use cgroup v1; the Kubernetes project recommends using cgroup v2, but in Kubernetes 1.36 (the current release) the cgroup v1 option remains supported as a fallback. That fallback is scheduled for removal in Kubernetes v1.38. If you are running an older cluster, plan to migrate; if you are setting up a new cluster with Linux nodes, you should prefer cgroup v2. In either case, review both the Kubernetes runtime documentation and (if relevant) the containerd compatibility matrix.
kernel updates around cgroup v2
When Kubernetes was first announced, in 2014, only v1 cgroup existed. Version 2 cgroup management first appeared in Linux kernel 4.5, released in 2016.
- In Linux 4.5, the cgroup v2
io,memory, andpidscontrollers were supported. - Linux 4.15 added support for the cgroup v2
cpucontroller. - Pressure Stall Information (PSI) support began with Linux 4.20.
- The Kubernetes project does not recommend using cgroup v2 with a Linux kernel older than 5.2 due to lack of cgroup-level task freezer support.
- Kubernetes documents 5.8 as the minimum kernel version for cgroup v2; the root cgroup's
system-level
cpu.statfile was added in Linux 5.8. - The
memory.highlivelock fix used by Memory QoS is present in Linux 5.9 and later. memory.peakwas added in Linux 5.19.
cgroup driver configuration
Configure the kubelet's cgroup driver to match the container runtime cgroup driver.
If you use kubeadm to manage your cluster, Kubernetes recommends that you use the systemd cgroup driver,
because kubeadm manages the kubelet as a systemd service. For other management tooling,
check the documentation for the tool you're using to manage your cluster.
If you can pick either option, I recommend using the systemd driver.
Whatever tooling you've chosen, the kubelet automatically tries to detect the runtime's recommended cgroup driver.
This automatic detection relies on using a runtime
that implements the RuntimeConfig
CRI RPC (for example: containerd v2.0+ or CRI-O v1.28+).
If you're using a container runtime that supports cgroup v2 but doesn't support automatic cgroup driver detection, you can manually configure an override by editing the kubelet configuration file. For example:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
Automatic discovery of the runtime's cgroup driver through the CRI, tracked by
KEP-4033, graduated to stable in Kubernetes v1.34. It requires a runtime
that implements the RuntimeConfig CRI RPC (containerd v2.0+ or CRI-O v1.28+). When available,
the kubelet uses the value reported by the runtime instead of its configured cgroupDriver value.
Tools and commands for troubleshooting
Tools and commands that you should know about cgroups:
stat -fc %T /sys/fs/cgroup/: Check whether cgroup v2 is enabled; it returnscgroup2fs.systemctl list-units 'kube*' --type=sliceor--type=scope: List Kubernetes-related units that systemd currently has in memory.bpftool cgroup list /sys/fs/cgroup/*: List all programs attached to the cgroup CGROUP.systemd-cgls /sys/fs/cgroup/*: Recursively show control group contents.systemd-cgtop: Show top control groups by their resource usage.tree -L 2 -d /sys/fs/cgroup/kubepods.slice: Show Pods' related cgroups directories.
How to check if a Pod CPU or memory limit is successfully applied to the cgroup file?
Work from the API object down to the node. Identify the node, then compare desired resources in the spec with the enacted values in status (in-place resize can leave those out of sync):
kubectl get pod <pod-name> -n <namespace> \
-o jsonpath='{.spec.nodeName}{"\n"}'
kubectl get pod <pod-name> -n <namespace> \
-o jsonpath='{range .spec.containers[*]}{.name}{" spec\t"}{.resources}{"\n"}{end}'
kubectl get pod <pod-name> -n <namespace> \
-o jsonpath='{range .status.containerStatuses[*]}{.name}{" status\t"}{.resources}{"\n"}{end}'
On that node, as root, with the systemd cgroup driver and cgroup v2 (these
snippets also need jq). Pin the container with kubelet labels; a container
name alone is not unique on a node. Use crictl ps -a if the container is
not running:
CONTAINER_ID=$(crictl ps \
--label io.kubernetes.pod.namespace=<namespace> \
--label io.kubernetes.pod.name=<pod-name> \
--name <container-name> -q | head -n1)
# OCI view (not the CRI protobuf field names)
crictl inspect "$CONTAINER_ID" | jq '.info.runtimeSpec.linux.resources'
# Kernel cgroup path for this container
PID=$(crictl inspect "$CONTAINER_ID" | jq -r '.info.pid')
CGROUP="/sys/fs/cgroup$(awk -F: '$1=="0"{print $3}' /proc/$PID/cgroup)"
cat "$CGROUP/cpu.weight" # request.cpu → shares → weight; not a limit
cat "$CGROUP/cpu.max" # limit.cpu as "quota period"; unlimited is "max <period>"
cat "$CGROUP/memory.max" # limit.memory; unlimited is "max"
# Pod-level cgroup (parent slice), used by Pod-level resources
cat "$(dirname "$CGROUP")/cpu.max" "$(dirname "$CGROUP")/memory.max"
# Present when MemoryQoS is enabled
cat "$CGROUP/memory.high" "$CGROUP/memory.min" "$CGROUP/memory.low"
SCOPE=$(basename "$CGROUP")
systemctl show "$SCOPE" \
-p CPUWeight -p CPUQuotaPerSecUSec -p CPUQuotaPeriodUSec -p MemoryMax
Do not treat info.runtimeSpec.linux.cgroupsPath as a filesystem path when
the runtime uses the systemd driver; that value is a systemd unit path
(slice:runtime:id), not a directory under /sys/fs/cgroup.
Expected mapping at each layer:
- Kubernetes:
spec.containers[*].resources(desired) andstatus.containerStatuses[*].resources(enacted after in-place resize). Pod-level resources are onspec.resourcesand the parent pod slice. - CRI (kubelet → runtime; these protobuf names stay cgroup v1-style even on
v2):
cpu_shares,cpu_quota,cpu_period,memory_limit_in_bytes, plusunifiedfor Memory QoS.crictl inspectdoes not show these names; it shows the OCI spec the runtime derived from them. - OCI spec:
linux.resources.cpu.{shares,quota,period},linux.resources.memory.limit, andlinux.resources.unified. The runtime converts shares, quota, and period into cgroup v2 files. After upgrading crun or runc, the shares-to-weight formula may change; see CPU weight conversion. - systemd scope:
CPUWeight,CPUQuotaPerSecUSec,CPUQuotaPeriodUSec,MemoryMax. Unlimited or unset values often appear as[not set]or infinity. - cgroupfs:
cpu.weight(from CPU request; an unset request still yields the default of 2 shares),cpu.max, andmemory.maxon the container scope. Memory QoS addsmemory.high,memory.min, andmemory.low.
Further reading
- Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode
- Kubernetes v1.36: Tiered Memory Protection with Memory QoS
- Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA
- Kubernetes 1.25: cgroup v2 graduates to GA
- KubeCon NA 2022 cgroup v2: Before You Jump In by Tony Gosselin & Mike Tougeron, Adobe Systems
- KubeCon NA 2022 Cgroupv2 Is Coming Soon To a Cluster Near You - David Porter, Google & Mrunal Patel, RedHat
- KubeCon EU 2020 Kubernetes On cgroup v2 by Giuseppe Scrivano, Red Hat.
- This blog only covers the basic requirements and configuration of Kubernetes components. It will not include how to enable cgroup fs in OS distributions. For migration, you can refer to migrating cgroup v2
Scaling Kubernetes Workloads with Node Swap
Memory is often the first hard limit a Kubernetes cluster hits. Nodes run out of RAM long before they run out of CPU, and the new wave of agentic AI workloads makes this worse. These workloads demand large memory footprints to start up and run untrusted code, then sit idle waiting for the next prompt. That idle but resident memory is expensive, and it caps how many pods a node can hold. This is where swap helps. Kubernetes support for running nodes with swap enabled reached General Availability in v1.34, and by backing that swap with fast NVMe solid state drives (SSDs), a node can page out dormant memory and pack in far more pods. This post explains how we benchmarked that approach across three workloads, including CI/CD kernel builds, sandboxed headless browsers, and isolated Python runtimes; we found density gains of up to 3×, often with little or no latency cost.
The node density problem
The Kubernetes ecosystem has reached a fundamental physical resource constraint: the strict limits of hardware memory versus the growing demand for dynamic, bursty workloads in the new agentic era.
Historically, administrators provisioning memory-intensive workloads encountered a persistent dilemma: set memory limits too high and you waste expensive infrastructure on idle RAM; set them too low and you risk Out-Of-Memory (OOM) kills.
This conflict is amplified when deploying autonomous AI agents using secure execution environments like the agent-sandbox framework. These agentic pods require large memory footprints to initialize and execute untrusted code. However, after their burst of activity, they typically enter long-tail idle phases waiting for user prompts. Keeping this idle state in physical RAM caps cluster density and makes AI infrastructure expensive to run.
The solution: Kubernetes node swap
With the introduction of Kubernetes' support for running nodes with swap enabled (which reached General Availability in v1.34), this paradigm shifts. By enabling the Linux kernel to page out anonymous memory to disk, node swap acts as a shock absorber during traffic spikes or periods of heavy memory oversubscription.
Historically, swap was discouraged in Kubernetes for two reasons. The first was memory accounting. Under cgroup v1, the controls treated memory and swap as a single combined limit rather than letting operators set an independent limit for disk swap. Without independent tracking, a process could page large amounts of anonymous memory out to disk, which made a container's real memory usage unpredictable and hard to isolate. Kubernetes' swap support resolves this by relying on cgroup v2, whose separate swap accounting tracks disk swap on its own. The second reason was the latency penalty of paging to slow spinning disks, which fast NVMe Local SSDs largely eliminate. Together, these make it practical to increase pod density and buffer against memory spikes without sacrificing cluster stability.
The benchmark data
To quantify the performance boundaries and cost-saving potential of Local SSD-backed node swap, this analysis covers three distinct workload categories: a Traditional Build Workload for CI/CD pipelines, High-Density Browser Sandboxes, and Isolated Python Sandboxes.
| Workload Profile | Baseline Capacity (No Swap) | Local SSD Swap Capacity | Density Improvement |
|---|---|---|---|
| Linux CI/CD Kernel Build | 600 MB RAM Limit | 300 MB RAM Limit | -50% RAM Footprint |
| Headless Chrome (Kata) | 40 Concurrent Pods | 50 Concurrent Pods | +25% Pod Density |
| Headless Chrome (gVisor) | 80 Concurrent Pods | 160 Concurrent Pods | +100% Pod Density |
| Python Sandbox (gVisor) | 80 Concurrent Pods | 240 Concurrent Pods | +200% Pod Density |
1. Traditional workload: Linux kernel build
Before exploring specialized agentic architectures, swap was validated against classic batch workloads by running a complete Linux 6.1.1 kernel build. The kernel compilation process leverages concurrent worker threads, balloons in memory to hold compiled object files, and requires a large memory spike during the brief linking phase.
This workload mirrors the memory behavior of enterprise CI/CD pipelines. Because earlier compiled objects sit inactive in memory while the pipeline progresses, CI/CD jobs frequently hoard unused physical RAM, which makes them well suited to node swap compression.
On a baseline node without swap, the minimum memory limit to prevent an OOM crash during compilation was 600 MB. Routing swap to a Local SSD cut the container memory limit by 50% to 300 MB without incurring any execution slowdown (in fact, it ran cleanly in 374s vs the baseline 433s). However, as an explicit tradeoff, compressing the limit further to 200 MB forced the active working set into swap, causing long I/O wait times and increasing execution time by over 40%. This reinforces that swap serves as an insurance policy for burst memory, not a replacement for active RAM.
2. High-density agent workloads: headless browser runtimes
AI agent workloads frequently require manipulating headless browsers via Chromium. However, trusting external code execution often requires stricter security isolation than standard Linux namespaces. This benchmark cross-evaluated several container runtime environments. The raw logs and testing methodologies for the default runtime are available in the Agent Sandbox GKE Swap directory.
- Unsandboxed baseline limits (
runc): To test the limits of the environment without the overhead of security runtimes, plain runc containers were swept on a c4-standard-32 node (32 vCPU, 120 GB RAM). Without swap, the node exhausted physical memory and failed past 512 pods. Enabling Local SSD swap allowed the node to support 768 concurrent pods. - Advanced security runtimes (for example: gVisor, Kata Containers): Enabling strict security sandboxing increases memory overhead and normally reduces pod density. However, memory swap naturally absorbs this overhead penalty. Without swap, a gVisor environment hit a hard limit at 80 pods. Local SSD swap doubled that capacity, which allowed 160 concurrent gVisor pods on a single node. Similarly, Kata Containers microVMs exhausted physical RAM at 40 concurrent pods without swap, but using GCP Local SSD swap expanded this to 50 stable Kata microVMs before CPU saturation.
At these maximum densities, the per-pod latency increase is driven mainly by pods competing for CPU, not by swap I/O. An operator tuning for a specific latency target would run at a lower density than the peak numbers here and see a proportionally smaller latency cost.
For a comprehensive architectural breakdown and density metrics for gVisor and Kata, see the Agent Sandbox GKE Swap Runtimes directory.
3. Beyond browsers: sandboxed Python runtimes
The advantages of node swap also extend to untrusted, isolated code-execution environments. This sweep deployed simultaneous Python sandbox sessions analyzing 5 million rows of data from the MovieLens 20M dataset, requiring a ≃375 MiB resident memory footprint per execution. The in-depth scaling results and deployment code for this sweep can be reviewed in the Agent Sandbox GKE Swap Python Density directory.
Without swap, heavy concurrent bursts exhausted physical memory, causing the node to hit a hard RAM limit and fail at 80 concurrent sessions. Enabling Local SSD swap offloaded dormant anonymous memory, freeing up physical RAM and preserving the node's page cache. This allowed the node to scale to 240 concurrently isolated Python sandboxes—a 3× density improvement. As with the browser workloads, the latency rise at peak density comes mainly from the sandboxes competing for CPU rather than from swap itself.
Density Benchmarks with and without Node Swap
How to use it
If you manage Kubernetes infrastructure for developer environments, browser testing farms, JVM applications, or AI execution runtimes, leveraging Local SSD swap can multiply your density efficiency.
In Kubernetes v1.34+, node swap is Generally Available. You enable it via the kubelet configuration:
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
failSwapOn: false
memorySwap:
swapBehavior: LimitedSwap
Pairing this upstream configuration with your cloud provider's high-speed local disk gives you dynamic memory balancing. For example, this is natively supported on Google Kubernetes Engine via Node Memory Swap configured on Local SSD profiles.
To get the benefits, configure your workloads with Burstable QoS: set your container's memory limits higher than its requests. The node automatically rations fast swap space based on idle application memory usage while keeping active processes responsive.
Conclusion
As the Kubernetes ecosystem transitions into the agentic era, administrators face a growing conflict between finite hardware memory limits and the bursty behavior of AI workloads. Frameworks like Agent Sandbox provide the security isolation required for running untrusted agents, but that isolation traditionally demands large amounts of idle memory overhead.
By configuring the kubelet with LimitedSwap and routing it to Local SSDs, you can mitigate this conflict. Fast swap offloads the dormant states of idle agents, allowing you to increase pod density and node utilization on the same infrastructure without compromising security boundaries.
Spotlight on SIG Apps
As Kubernetes adoption has grown, the conversation has shifted beyond running containers to managing increasingly complex application lifecycles. Modern platforms support stateless web services, stateful databases, batch processing, AI workloads, and platform services. At the same time, they must remain reliable during upgrades, scaling events, and infrastructure failures.
Every Kubernetes user relies on SIG Apps, whether they realize it or not. Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs form the foundation of how applications are deployed, updated, scaled, and operated across the Kubernetes ecosystem.
SIG Apps is focused on improving workload resilience, refining application lifecycle management, and addressing the operational challenges that emerge when applications encounter node failures, rollout disruptions, and increasingly complex infrastructure environments.
In this spotlight, we sit down with two of the three SIG Apps chairs Janet Kuo and Maciej Szulik to discuss the evolution of Kubernetes workload management, the challenges of balancing application reliability with operational simplicity, and the future of application lifecycle management within one of Kubernetes’ most influential Special Interest Groups.
Introducing SIG Apps
Natalie Fisher: Can you introduce yourself, your role, and how you got involved in SIG Apps?
Janet Kuo: I'm a Senior Staff Software Engineer at Google and have been a Kubernetes maintainer since 2015, joining the community just as we were racing toward the 1.0 launch. In those early days, my focus was on building the core Workloads API, specifically developing controllers like Deployment, ReplicaSet, StatefulSet, and DaemonSet, defining their rollout behaviors, and bringing them from initial designs to GA. That hands-on work was my entry point into SIG Apps.
Since then, I've stayed deeply involved in both the technical and community sides of Kubernetes. I have led SIG Apps as Co-Chair and Tech Lead since 2019. Currently, in addition to maintaining the workloads API, I am driving new subprojects like the Agent Sandbox to ensure Kubernetes is ready for next-generation agentic and AI workloads.
Maciej Szulik: I started contributing to Kubernetes all the way back in 2014. Since then, I've worked across various areas of the project: controllers, kubectl, and apimachinery, which eventually led me to become one of the Chairs and Tech Leads for SIG Apps. My current focus is reliability of the workload controllers under the SIG Apps umbrella and stability and ease of use of kubectl as part of my SIG CLI Tech Lead role. I also care about overall community health and growth as part of my Steering Committee role. Outside of Kubernetes, I work as a Staff Platform Engineer at Defense Unicorns, where I'm helping make Kubernetes more airgap-native with a project called zarf.
The problem and the solution
SIG Apps is responsible for the core workload APIs that power how applications run on Kubernetes. From Deployments and StatefulSets to Jobs and CronJobs, these controllers determine how workloads are created, updated, scaled, and recovered when things go wrong.
As Kubernetes expands to support increasingly diverse workloads – including AI, batch processing, and large-scale distributed applications – SIG Apps continues to evolve these APIs while balancing reliability, backward compatibility, and operational simplicity.
NF: For readers who may not be familiar, what is SIG Apps, and what role does it play within the broader Kubernetes ecosystem?
MS: SIG Apps is the Kubernetes Special Interest Group responsible for the workloads APIs. CronJob and Job help running batch workloads, whereas DaemonSet, Deployment, ReplicaSet, and StatefulSet serve the majority of other applications. More broadly, SIG Apps owns the layer most developers actually touch day-to-day: the controllers that turn a workload specification into running, self-healing pods. It's the group deciding how Deployments roll out, how Jobs retry, how DaemonSets place a pod per node.
JK: Adding to what Maciej described, as the industry shifts, we are seeing a massive demand to run complex, non-traditional workloads like distributed AI training, batch computing, and dynamic agent environments. Our role is expanding: we aren't just maintaining the classic workloads API, but we are actively evolving it and establishing new patterns (like the Agent Sandbox) to make sure Kubernetes remains the best platform for the next generation of workloads, such as AI.
NF: Looking at the workload APIs owned by SIG Apps (Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs), which areas are receiving the most attention from maintainers and contributors?
MS: After a long stretch focused on making batch workloads run smoothly on Kubernetes, we’ve shifted attention to make sure serving workloads (DaemonSets, StatefulSets, etc) aren’t left behind. This means performance and high-scale improvements to rollout and scaling behavior, plus working through our backlog of user-reported issues, prioritizing the ones with the strongest support from the user base.
Current focus areas
As Kubernetes workloads grow in scale and complexity, the challenges facing workload controllers evolve as well. We asked the SIG Apps chairs where contributors are focusing their efforts today and which resilience problems they believe are the highest priorities.
NF: From your perspective, what are the most important workload resilience problems SIG Apps is trying to solve today?
MS: Node lifecycle challenges have come up repeatedly across SIG Apps, SIG Node, and SIG Autoscaling discussions. DaemonSets and Jobs are just where the pain is most visible, since they're the workloads most directly bound to node state. Rather than solve it piecemeal within one SIG, we've settled on spinning up a dedicated Node Lifecycle Working Group to focus on this properly and hopefully land long-term solutions instead of one-off patches.
JK: From an AI perspective, resilience is critical. When you are running a massive distributed LLM training job that spans hundreds of GPUs, a single node failure can halt the entire pipeline. Similarly, if a DaemonSet that runs your logging or GPU monitoring agent gets stuck on a bad node, it impacts the entire cluster's health.
In addition to the work in the Node Lifecycle WG to handle infrastructure-level degradation, SIG Apps is addressing this at the orchestration layer through subprojects like JobSet (for distributed training) and LeaderWorkerSet (LWS) (for sharded LLM inference). These APIs introduce patterns like "all-or-nothing" failure handling, where a single pod or job failure triggers a coordinated group-level restart to resume from the last clean checkpoint, rather than letting stuck workloads hang in an inconsistent state.
Real-world impact
The work happening within SIG Apps extends far beyond controller implementations and API design. We wanted to understand what these improvements mean in practice for platform teams operating Kubernetes clusters in production.
NF: For platform teams operating Kubernetes in production, what practical improvements would they notice if the node lifecycle and workload resilience work currently under discussion is successfully delivered?
MS: I’m mostly looking from the sidelines, the folks actually in the Node Lifecycle Working Group would give you a sharper answer. But from where I sit, I’m hoping their work translates into fewer 3am pages that turn out to be “a DaemonSet rollout got stuck because node X was flaky, and someone had to manually cordon/delete/restart to unstick it.”
JK: +1 to what Maciej said, and beyond reducing manual intervention, platform teams will also see much better resource predictability and cost efficiency. For example, in AI workloads where GPU idle time is extremely expensive, having Kubernetes automatically detect a degraded node and reschedule the training coordinator or agent before the job crashes means less wasted compute and more stable job execution.
Challenges and trade-offs
Evolving APIs that millions of workloads rely on requires careful engineering and even more careful decision-making. We asked the SIG Apps chairs about the technical and operational trade-offs they weigh when introducing changes to Kubernetes’ core workload controllers.
NF: What are some of the hardest technical or operational trade-offs SIG Apps encounters when evolving core workload controllers?
MS: Honestly, a few tensions keep coming up: how aggressively a controller should give up on stuck pods, and what signals it actually needs to make that call correctly. At the same time, we always have to think about backward compatibility. Deployment, DaemonSet, and Job behavior has been depended on for a decade [by Kubernetes users, tooling, automation, and higher-level controllers], so even a change that’s clearly “more correct” can break automation people built around the old behavior without meaning to.
JK: One of our hardest trade-offs is resisting the urge to make "elegant" design changes that break backward compatibility. Instead, we have to design opt-in features that let users adopt new behaviors without forcing them on legacy workloads. When we need to support completely new paradigms, we prefer introducing them as CRDs first rather than bloating the core APIs, like we are doing with Agent Sandbox, JobSet, and LWS.
Looking ahead
While much of SIG Apps’ work focuses on maintaining the stability of existing workload APIs, the group is also shaping the future of Kubernetes through new enhancements and proposals. We concluded by asking about one proposal that recently returned to active development and what it represents for the future of workload management.
NF: The SIG recently discussed reviving KEP-4443 with a target release of Kubernetes 1.38. What opportunities or challenges does this proposal aim to address, and why is now the right time to revisit it?
KEP-4443 addresses a small but real gap in the Job API: a PodFailurePolicy can be configured to add a condition reason to the JobFailed condition, but different pod failure policy rules targeting different container exit codes all produce that same generic reason. The proposal is simple: an optional Name field on each PodFailurePolicyRule, which gets appended to the JobFailed condition reason, so higher-level tools like JobSet can finally react differently depending on which rule triggered the failure.
As for timing, the answer is as simple as it always is in open source: we lost the original contributor who was driving this. Now we’ve got someone new interested in picking it up, that’s why we’re targeting the next release.
Getting Involved
NF: For someone interested in contributing to SIG Apps, where would you recommend they start, especially if they are not yet a Kubernetes maintainer?
MS: The best place to start is the #sig-apps slack channel and our regular SIG Apps meetings. We’ve all started there, and if it feels intimidating, or nobody replies right away, that’s completely normal. Everyone's busy. It's not personal.
JK: In addition to what Maciej answered, I'd suggest looking at our newer subprojects and initiatives. Contributing to stable APIs like Deployment or StatefulSet can be daunting because the barrier for making changes is very high due to backward compatibility, and there is much less low-hanging fruit.
If you are new to the community, projects like the Agent Sandbox are fantastic entry points. They are actively evolving, have a friendly group of maintainers, and offer plenty of greenfield development opportunities where you can make a significant impact quickly.
Summary
SIG Apps has shaped how Kubernetes applications are deployed and operated since the project’s earliest days. While users often interact with Deployments, StatefulSets, Jobs, and DaemonSets without thinking about the controllers behind them, the work within SIG Apps continues to shape the reliability and scalability of workloads across the Kubernetes ecosystem.
From improving workload resilience and node lifecycle behavior to enabling new patterns for AI and distributed computing, the SIG is evolving Kubernetes while remaining committed to one of the project’s core principles: preserving the stability and backward compatibility that users depend on. Whether you’re interested in core workload APIs, emerging projects like Agent Sandbox, or helping improve the operational experience of Kubernetes users everywhere, SIG Apps offers many opportunities to get involved.
Kubernetes v1.37: Tracking When a PersistentVolumeClaim Was Last Used (Beta)
Kubernetes v1.37 promotes the PersistentVolumeClaimUnusedSinceTime feature gate to Beta (enabled by
default). With this feature, the PersistentVolumeClaim (PVC) protection controller adds an Unused
condition to each PVC, telling you whether any running pod currently references it — no custom
tooling or cross-referencing required.
For the API definition of PVC conditions, see the
PersistentVolumeClaim API reference.
Read on to learn how the Unused condition works and how to use it.
Why track PVC usage?
In large-scale Kubernetes clusters, it is common for users to create PVCs and then delete the associated pods without cleaning up the storage, because Kubernetes does not automatically delete PVCs when their pods are removed (to protect against accidental data loss). Over time, these orphaned PVCs may