opensource.google.com

Menu
Showing posts with label Kubernetes. Show all posts
Showing posts with label Kubernetes. Show all posts

Kubernetes Python client gets a modern overhaul for the AI/ML era

Thursday, November 12, 2026

Google Open Source circuit-board banner featuring the blue Kubernetes helm wheel surrounded by colorful icons representing code, cloud infrastructure, configuration, and community collaboration.

When you train a large language model with Kubeflow, orchestrate distributed compute with Ray on Google Kubernetes Engine (GKE), serve inference traffic with KServe, or schedule thousands of ML pipeline pods with Apache Airflow, two technologies almost always meet under the hood: Python and Kubernetes.

Python is the undisputed lingua franca of artificial intelligence and machine learning. Kubernetes is the de facto operating system for AI infrastructure—scheduling GPUs and TPUs, orchestrating distributed training jobs, and scaling model serving endpoints. Connecting these two worlds is the official Kubernetes Python client (kubernetes-client/python).

Second only to Kubernetes's in-tree Go client (client-go) in popularity, the Kubernetes Python client is downloaded more than 73 million times per month on PyPI (~1.7 million times a day) and has earned over 7,500 GitHub stars.

Yet for the past several years, this critical bridge between Python and Kubernetes has operated in low-maintenance mode. As AI/ML workloads have exploded in scale and complexity, the gap between what modern AI infrastructure demands and what the legacy Python client was built to deliver has become impossible to ignore.

Today, we're sharing how Google Cloud engineers are partnering with open source contributors in Kubernetes SIG API Machinery to modernize the official Kubernetes Python client for the AI/ML era.

The historical problem: Legacy machinery meets hyperscale AI

As AI/ML clusters grew from dozens of nodes to 50,000-node hyperscale environments, the limitations of the Python client's aging architecture began surfacing as real-world production bottlenecks:

  1. Frozen on a decommissioned code generator: Up through Kubernetes 1.35, the Python client generated its ~400,000 lines of API and model code using OpenAPI Generator v4.3.0 (python-legacy)—a generator version that was deprecated in 2020 and completely removed from the upstream OpenAPI Generator project in May 2022. Staying pinned to v4.3.0 meant the client accumulated DeprecationWarning debt across newer Python releases, struggled with modern dependency resolution alongside PyTorch and TensorFlow, and couldn't easily receive upstream generator bug fixes.
  2. Synchronous-only execution in an asyncio world: Modern Python web frameworks, inference servers, and orchestrators have overwhelmingly shifted to asyncio to handle high-concurrency I/O without being bottlenecked by Python's Global Interpreter Lock (GIL). Because the official client lacked native asyncio support from the start, users were forced to turn to community clients like kubernetes-asyncio or third-party clients (kr8s, lightkube).
  3. Missing controller primitives (SharedInformer): In Go, controllers use SharedInformer to maintain a local, event-driven cache of cluster resources without repeatedly polling kube-apiserver. Without an equivalent primitive in Python, applications watching thousands of training pods or custom resources often hammered the API server or ran into watch disconnects.
  4. Slow model deserialization and HTTP/1.1 transport limits: In large clusters, listing pods or nodes in Python could take 30+ seconds due to excessive model reconfiguration during JSON deserialization (issue #1921). Combined with an underlying urllib3 transport layer restricted to HTTP/1.1, Python workloads missed out on the HTTP/2 multiplexing, flow control, and binary framing that make client-go fast at scale.

A principled approach: Built for AI/ML, not 1:1 parity

When we set out to modernize the Kubernetes Python client, we established two guiding principles to ensure our engineering investments deliver the highest possible return for the community:

  1. Target high-impact gaps, not 1:1 client-go parity: client-go is the in-tree Kubernetes client used by the control plane itself (kube-controller-manager, kube-scheduler, and in-tree controllers). Attempting to replicate every long-tail client-go feature in Python would yield diminishing returns. Instead, we focus strictly on the primitives Python developers and operators actually need in production.
  2. Prioritize AI/ML workload requirements: Because AI/ML training, inference, and batch frameworks are the primary drivers of Python on Kubernetes today, every architectural improvement—from async concurrency and CRD ergonomics to large-cluster list performance—is prioritized through the lens of AI/ML scale.

How we're modernizing the Kubernetes Python client

Our roadmap focuses on three technical pillars: Machinery, Features, and Performance.

1. Machinery: Upgrading to OpenAPI Generator v7.20+

Every improvement to the Python client starts with its code generation pipeline (kubernetes-client/gen). Over the past few years, multiple community attempts to migrate off the decommissioned v4.3.0 python-legacy generator stalled due to the sheer scale of template and schema changes across hundreds of Kubernetes API models.

We are actively driving the migration to OpenAPI Generator v7.20.0+ and establishing continuous tracking with upstream generator releases. Completing this foundational upgrade:

  • Replaces dead python-legacy templates with actively maintained v7+ Python code generation.
  • Modernizes type annotations and packaging metadata, easing pip dependency resolution alongside heavy ML frameworks.
  • Unlocks the ability to contribute performance and serialization fixes directly to upstream OpenAPI Generator templates.

2. Features: Native asyncio, Shared Informers, and WebSocket v5

To prevent ecosystem fragmentation and give Python developers first-class Kubernetes primitives out of the box, we are bringing three major capabilities to the official client:

  • Official asyncio support: Rather than forcing developers to choose between the official synchronous client and third-party async libraries, we are introducing official asyncio support directly into kubernetes-client/python. Async-first ML orchestrators, FastAPI-based model control planes, and notebooks will be able to issue concurrent, non-blocking Kubernetes API calls natively.
  • Shared Informers (kubernetes-client/python#2515): We are bringing the SharedInformer pattern to the Python client. With built-in list-watch caching, automatic reconnection, and resource event handlers (on_add, on_update, on_delete), Python controllers and AI job managers can monitor cluster state and Custom Resources (TrainJob, RayCluster, InferenceService) with a fraction of the memory footprint and API server load.
  • WebSocket v5 streaming (KEP-4006, kubernetes-client/python#2486): As upstream Kubernetes transitions remote command streaming (exec, attach, port-forward) away from legacy SPDY to standard WebSockets via the v5.channel.k8s.io subprotocol, we are adding native WebSocket v5 channel support so interactive debugging and remote job execution work seamlessly against modern control planes.

3. Performance: Slashing deserialization overhead

For AI/ML platforms operating on multi-thousand-node clusters, client performance is an availability requirement. Our performance track targets the largest bottlenecks in Python client execution:

  • Eliminating model reconfiguration overhead: When deserializing large JSON responses (such as listing tens of thousands of Pods across a training cluster), the legacy generated model layer repeatedly instantiates and reconfigures schema validation objects (issue #1921). With the move to OpenAPI Generator v7+, we are revamping model initialization so large cluster queries deserialize in a fraction of the time.

What's next and how to get involved

The intersection of Python and Kubernetes is where modern AI infrastructure is built. By modernizing the code generation machinery, adding native asyncio and SharedInformer primitives, and closing the deserialization performance gap, we aim to give every ML engineer and platform builder a fast, reliable, and Pythonic official client—whether they are running a single notebook experiment or orchestrating a 50,000-node GKE training fleet.

This effort is happening in the open under Kubernetes SIG API Machinery, and we welcome feedback, use cases, and contributions from the community:

Sandboxing Untrusted AI Agents at Scale: Google Donates Agent Substrate and gVisor to the CNCF

Thursday, November 5, 2026

When Google introduced Kubernetes twelve years ago, the Pod became the default unit of compute for the cloud. The design assumed a straightforward lifecycle: a program boots, binds to a port, serves traffic, and runs until you stop it.

AI agents work fundamentally differently.

An AI agent that’s running on a machine executes untrusted code, commands, and tool calls, and must be isolated from its host and its network resources. At the same time, these agents, whether they’re an interactive coding assistant, an SRE troubleshooter, or a multi-turn evaluation loop, do not run continuously. They execute a step, call a language model, and wait. Most of the time, the agent does nothing at all while it waits for a model to stream tokens or a human to review a pull request. In the same way that Kubernetes was created to address the needs of containerized applications, we introduced Agent Substrate as a new orchestrator for the emerging agentic applications. Secure by default, highly efficient and built for multi-million agent scale.

Agent Substrate was designed from the ground up for AI agent security and density and brings the best parts of Kubernetes together with a new control and data plane for fully-isolated, ultra-low-latency sandboxed compute and storage. The growing Agent Substrate community has been rapidly building core capabilities and stabilizing APIs, culminating in the latest 1.0 release.

This week at KubeCon North America in Salt Lake City, Google Cloud is donating Agent Substrate and gVisor to the Cloud Native Computing Foundation (CNCF) in Sandbox status, alongside OpenRL.

This post explains why traditional approaches struggle with stateful agents, how Agent Substrate solves the problem, how it connects to enterprise governance on Google Kubernetes Engine (GKE), and where to see it in action at KubeCon NA 2026.

Challenges of running AI agents at scale

AI agents introduce two infrastructure bottlenecks that standard Kubernetes was never built to handle: idle resource waste and control plane overload.

Diagram comparing standard Kubernetes dedicated Pods vs Agent Substrate stateful actor runtime with tiered storage

1. The idle resource problem

During a typical interactive session, an agent can spend 90% to 95% of its time waiting for model inference, external APIs, or human reviews.

During that wait, the agent consumes zero CPU. However, you cannot simply kill the container. The agent holds critical state in memory and on disk: active language runtimes, local Git repositories, uncommitted file edits, and tool sessions.

With standard Kubernetes, platform engineers face two poor options:

  1. Keep a dedicated Pod running all day: The agent responds quickly because it is already warm, but you pay for idle compute around the clock. At ten thousand active users, cluster costs become unsustainable.
  2. Scale to zero with serverless containers: You save compute costs while the agent sits idle, but resuming from cold storage forces the container to clone repositories, reinstall packages, and rebuild memory. Wakeup latency climbs past 20 seconds, breaking the interactive experience.

Platform teams need agents that wake up in under 500 milliseconds with full memory and disk state intact—without paying for running Pods while the agent sits idle.

2. Control plane overload

The opposite issue occurs when training or evaluating agents. Reinforcement learning loops and benchmark suites need to run untrusted agent code across thousands of tasks at the exact same time.

If every sandbox requires a full Kubernetes Pod, the control plane quickly grinds to a halt. Scheduling, creating, and destroying thousands of short-lived Pods every minute overwhelms etcd, saturates kube-apiserver, and exhausts subnet IP addresses. Kubernetes was built to manage stable application topologies, not to act as a microsecond sandbox allocator.

How Agent Substrate solves the problem

Agent Substrate adds an actor-based control and data plane built on top of Kubernetes. It solves the idle resource problem by decoupling Actors from Workers:

  • Actors hold the state: An Actor represents an individual agent session, complete with its own filesystem, environment variables, memory state, and network rules.
  • Workers provide the compute: A Worker is a pre-warmed Kubernetes Pod running in a shared pool (WorkerPool), equipped with container runtimes and sandbox boundaries.
Agent Substrate architecture diagram showing incoming requests routed via atenet-router, ateapi controller actor placement, tiered storage, and shared worker pool

The lifecycle of an agent turn

  1. Request parking: An incoming call hits the atenet-router L7 proxy. If the target Actor is suspended, the proxy holds the request in memory and alerts the Substrate controller.
  2. Fast placement: The controller assigns the Actor to an available slot in the warm Worker pool.
  3. Sub-second restore: The Actor’s memory and disk state restore from tiered storage, the process wakes up, and the proxy delivers the request. The agent resumes in under 500 milliseconds right where it paused.
  4. Suspension on idle: When the turn finishes, Substrate snapshots the Actor’s state to tiered storage and frees the Worker slot for other workloads.

Because Substrate manages Actor state directly, thousands of agents can activate, suspend, and resume every second without creating new Pod objects or stressing etcd.

Safe execution with gVisor and microVMs

Agents frequently execute model-generated, untrusted code. Standard Linux containers share the host kernel, which presents an unacceptable security risk on multi-tenant nodes.

Agent Substrate pairs with gVisor (runsc), Google's open source application kernel written in Go. gVisor intercepts system calls in user space, isolating untrusted code from the host kernel. Because gVisor virtualizes its own memory and process tables, Substrate can freeze a running environment, write memory pages to storage, and restore them on another node in under 500 milliseconds.

For workloads requiring hardware virtualization or custom kernel modules, Substrate also supports Cloud Hypervisor microVMs under the same unified API.

Why Google donated Agent Substrate to the CNCF

Over the past two years, engineering teams across the industry have run into the same agent scaling boundaries. Without a shared open standard, every organization ends up building custom schedulers on top of VMs or working around Kubernetes.

Donating Agent Substrate and gVisor to the CNCF gives the cloud-native community a neutral, vendor-independent foundation that works across any conformant Kubernetes cluster.

Together with community projects, they form an open stack for agentic infrastructure:

  • Layer 1: Secure Isolation (gVisor): Sandboxes untrusted code execution with user-space kernel isolation and fast memory checkpointing.
  • Layer 2: Agent Runtime (Agent Substrate): Multiplexes stateful actors onto warm worker pools with sub-second wakeups.
  • Layer 3: Training Orchestration (OpenRL): Coordinates Kubernetes-native reinforcement learning loops and evaluation sandboxes.

Where to find us at KubeCon NA 2026

If you are heading to Salt Lake City for KubeCon + CloudNativeCon North America from November 9 to 12, visit our sessions, workshops, and booth demos:

When

Session or Event

Speakers & Location

Monday,

Nov 9

Agent Substrate Day (ATEcon)

  • Welcome to ATECon, Tim Hockin, Eitan Yarmush, Ron Lev
  • Pleading the Agent: Proof of Agent identity, Greg Castle, Taahir Ahmed
  • Evolving The Substrate Dataplane, Benjamin Elder
  • Truly Sandboxed Networking: Moving beyond MVP, Bowei Du, Lior Lieberman
  • Using Substrate for anything but Agents, Mofi Rahman
  • Agent Substrate Environment, Jaana Dogan, Lior Lieberman
  • Closing Remarks, Tim Hockin, Eitan Yarmush, Ron Lev

Salt Palace

Monday,

Nov 9

Detonation and Isolation: Hardening Kubernetes Runtimes for Agents


Multi-agent orchestration and runtime governance

Syeda Anjum, Kaslin Fields, and Kanchana Rajendran


Agentics Day

Monday,

Nov 9

GKE Day


Announcements and hands-on agent infrastructure labs

Google Cloud GKE Team

Tuesday,

Nov 10

Flagship Fireside Chat: Kubernetes Act II


Why AI Agents Change Everything Under the Hood

Tim Hockin and Kelsey Hightower


Moderated by Savannah Peterson

Wednesday,

Nov 11

Hands-on Workshop: Building and Governing Agent Substrate on GKE


Hands-on lab deploying warm worker pools, testing snapshot restores, and configuring egress secrets

Mofi Rahman, Drew Brown, Casey West, Kaslin Fields, and Olivier Bourgeois


Schedule ID: 1247804

Wednesday,

Nov 11

Expo Hall Demo Theater: Inside the Show-Floor Agent Swarm


Live fleet multiplexing on GKE

Brandon Royal and Mofi Rahman

Nov 10 to 12

Google Cloud Booth #122

• Agent Swarm demo: See agent substrate in action.

• Partner demos with NVIDIA on OpenShell and GPU Actor infrastructure

• Filestore Agent Volumes in action

Booth #122 Expo Hall

Get started

Agent Substrate is open source and ready for testing today:

See you in Salt Lake City!

GKE Workspaces: Your notebook, on Google's AI infrastructure

Wednesday, November 4, 2026

If you're an ML engineer or researcher, the GPUs and TPUs you need increasingly live on Kubernetes, but Kubernetes should be an invisible layer. While your platform team sets up the GKE cluster and manages accelerator capacity and reservations, your job is to train and ship models—not to manage Kubernetes infrastructure.

Today we're introducing GKE Workspaces, a self-hosted AI/ML development platform for Google Kubernetes Engine, built on the open source Kubeflow Workspaces project. Deployed on your team's GKE cluster, it provides a dedicated web dashboard where you can pick an IDE and a machine, and start coding on cloud accelerators in minutes.

Screencast of signing into the GKE Workspaces UI and launching a JupyterLab workspace on cloud accelerators.

Why Kubeflow Workspaces

Kubeflow Workspaces gives every researcher a personal, persistent development environment on shared cluster hardware.

  • Self-service, in a few clicks. Sign in to your team's Workspaces web UI, choose JupyterLab or in-browser VS Code, pick an environment, and select a hardware size such as "Small CPU", "GPU", or "TPU". Researchers don't need Google Cloud Console access, YAML, or command-line setup.
  • Your work persists. Each workspace mounts its own home directory on a dedicated, encrypted Persistent Disk in your Google Cloud project—isolated to that workspace, so your files, notebooks, and installed packages stay safe across restarts, pauses, and machine changes.
  • Guardrails set by your platform team. Admins curate the approved images and hardware options, and decide which machines, reservations, and capacity types (such as on-demand or Spot) back them. Researchers only see choices that work.
  • Open source and portable. Workspaces is part of the Kubeflow community, empowering you with the freedom and flexibility of a fully open source notebook ecosystem.
  • Upstream integration and open licensing. GKE Workspaces builds directly on Kubeflow Workspaces with key enhancements for GKE environments—including standalone deployment without Istio dependencies, integrated authentication token flows, and stateful GPU/CPU pause and resume capabilities. Both GKE Workspaces and upstream Kubeflow Workspaces are released under the Apache 2.0 open source license, and we are actively working with the Kubeflow community to upstream these features.

What's unique about GKE Workspaces

GKE Workspaces brings Kubeflow Workspaces together with capabilities that are only available on Google Cloud.

1. Pause and resume—without losing your state

You've spent an hour loading a large model into GPU memory and building up results across notebook cells. Now it's time to go home. Normally you have two bad options: shut down and lose everything, or leave it running and pay for an idle GPU overnight.

With GKE Workspaces, you just pause by clicking a button from the Workspaces Dashboard. Built on Pod Snapshots, GKE takes a snapshot of your running notebook—including variables in memory and models loaded on the GPU—saves it to Cloud Storage, and releases the hardware so you stop paying. When you resume, you pick up exactly where you left off. Workspaces can also be paused automatically after a period of inactivity.

Screencast of stateful pause and resume via GKE Pod Snapshots preserving GPU memory.

2. An easy on-ramp to Cloud TPUs

Moving from a quick experiment to large-scale training on TPUs used to mean switching tools at every step. GKE Workspaces keeps you in the same notebook the whole way, with both JAX and PyTorch:

  1. Prototype on a single TPU chip, just like you would on a GPU.
  2. Scale to multiple chips on one machine by choosing a bigger hardware option.
  3. Scale out to multi-host TPU slices by sending the same training function from your notebook to a larger slice. Logs stream back to your notebook, and the TPUs are released as soon as the job finishes.
Screencast scaling a notebook workload from a single Cloud TPU chip to a multi-host TPU slice.

3. Use your local coding agents with cloud accelerators

Prefer using your own AI coding agents and editors locally? You can easily bridge them to cloud compute. In the Workspaces web UI, generate a short-lived, revocable link to run your notebooks and agent-generated code on a remote GPU or TPU. Traffic is secured over HTTPS via Google Identity-Aware Proxy and the cluster's access proxy, checking permissions on every request—so no SSH keys, local tunnels, or cluster credentials ever sit on your laptop.

Screencast connecting a local coding agent to a remote TPU workspace over Identity-Aware Proxy HTTPS.

4. A launchpad for distributed work

Your workspace doesn't have to be big to drive big jobs. From a small, inexpensive CPU notebook you can process data with Spark, launch multi-host training, serve your model, and scale Ray clusters—all from Python. Built-in Cloud Storage access, with no keys to manage, makes it easy to pass data between each step.

Screencast coordinating Spark and Ray distributed jobs from a lightweight CPU notebook.

5. A portal to safe elastic sandboxes for coding subagents

Running a fleet of coding agents directly on your local machine can drain your battery, expose sensitive files or credentials to agent-executed code, and halt progress as soon as you close your laptop. From a standard GKE Workspace, you can safely task your favorite coding agent to hand off parallel work—such as resolving failing tests across a repository—to background subagents. Crucially, while your primary workspace operates as a full-featured environment, these subagents are isolated within lightweight, in-cluster sandboxes using providers like Agent Substrate, requiring zero local setup. Each subagent executes independently in its own sandbox on shared cluster nodes, continues running even when you log off, snapshots to Cloud Storage at zero compute cost while idle, and outputs a clean patch for your review and merge.

Screencast spawning isolated coding subagents inside lightweight Agent Substrate cluster sandboxes.

6. Lightweight and secure by default

GKE Workspaces runs on its own on GKE Standard or Autopilot—you don't need to install the full Kubeflow platform. Users sign in with their Google accounts, traffic is encrypted with Google-managed certificates, and each team's work is isolated from the others.

Get started

GKE Workspaces lets ML teams focus on models, not infrastructure: one-click development environments, stateful pause and resume, a smooth path to Cloud TPUs, and the power of GKE behind every notebook.

Ask your platform team to deploy it using the GKE deployment guide, and learn more about the upstream project at Kubeflow Workspaces. We welcome feedback and contributions for both Apache 2.0-licensed projects.

In-place pod restarts: Boosting efficiency and workload reliability in Kubernetes v1.35

Thursday, June 18, 2026

Operational efficiency and system resilience are critical when running scaled platforms. Yet, in Kubernetes, recovering from software crashes remains a headache because you couldn't trigger a clean restart of a Pod's containers without recreating the entire Pod object, leading to some amount of resource waste.
To address this, Restart All Containers on Container Exits graduated to beta and is enabled by default in Kubernetes v1.36. Developed in close collaboration with the CNCF community, this capability represents Google's commitment to investing in the success of foundation-led open source projects. By sharing best practices from running large distributed systems internally, we are helping build a more resilient and efficient ecosystem. Letting containers restart while keeping the Pod's runtime identity provides a built-in way to perform in-place Pod recovery, boosting application reliability and saving resource costs.

The Problem: The High Cost of Pod Re-creation

Historically, Kubernetes managed failures using pod level restart policies. While sufficient for simple services, modern multi-container Pods often have complex dependencies. When a failure requires a full environment reset, your only option was deleting and recreating the entire Pod.
This introduces massive control plane churn, causing latency and pressure on the etcd backend during large failures:

  • Initialization Dependencies: If a main container corrupts a local environment, for example, single-use secrets that must be re-requested, restarting just that container is insufficient; the setup must run again.
  • Watcher Interoperability: If a watcher sidecar detects a fatal error, it must trigger a full recreate of the entire pod and its infrastructure, including the sandbox.
  • Stale States: If a database sidecar proxy restarts, the main application can get stuck attempting to use stale, broken connections.
  • Resource Race Conditions: When a large job finds a proper set of nodes, recreating Pods can lead to other pending Pods taking over those resources. In-place restarts eliminate this race condition risk.

Previously, resolving these failures required destroying the entire Pod. For large batch or AI/ML workloads, where thousands of Pods might fail simultaneously, this can lead to "Thundering Herd" scheduling requests, delaying recovery and wasting expensive GPU/TPU compute time.

Introducing In-Place Restarts: The RestartAllContainers Action

Kubernetes v1.35 introduces the RestartAllContainers action, enabled by the RestartAllContainersOnContainerExits feature gate, which graduated to beta in 1.36 alongside its dependencies ContainerRestartRules and NodeDeclaredFeatures. This lets a container's exit behavior trigger a fast, in-place restart of the entire Pod on its existing node.
The Kubelet halts all containers while keeping the Pod sandbox intact, preserving critical infrastructure:

  • Network Identity: Keeps the same IP, network namespace, and UID, completely bypassing IP reassignment.
  • Hardware and Devices: Keeps GPUs/TPUs bound, eliminating scheduling and re-allocation delays.
  • Storage Mounts: Volumes, including emptyDir and PVCs, remain fully mounted; their content is not cleared during restarts.

Once terminated, the Kubelet re-runs init containers (including sidecars, which are part of the init sequence) in order, guaranteeing a clean setup in a known-good environment.

A Native Pod Specification Example

You can implement this under the container's restartPolicyRules field. Here is a quick example of how a watcher sidecar can trigger an in-place restart of the entire Pod by exiting with code 88:
YAML
Note: Image names and paths in the YAML below are for illustrative purposes.

apiVersion: v1
kind: Pod
metadata:
  name: ml-worker-pod
spec:
  restartPolicy: Never
  initContainers:
    - name: setup-environment
      image: registry.k8s.io/ml-tools/setup-worker:v1.0
    - name: watcher-sidecar
      image: registry.k8s.io/ml-tools/watcher:v1.0
      restartPolicy: Always
      restartPolicyRules:
        - action: RestartAllContainers
          exitCodes:
            operator: In
            values: [88]
  containers:
    - name: main-application
      image: registry.k8s.io/ml-tools/training-app:v1.0

The Operational Impact of In-Place Restarts

For organizations running distributed workloads, RestartAllContainers provides serious operational advantages:

  • No Control Plane Overhead: By preserving identity, clusters avoid scheduling latency and DNS propagation. This was a key factor for JobSet using this feature to reduce recovery from minutes to seconds.
  • Node Locality Preservation: Since the Pod stays anchored to the same node, restarted containers can instantly access local, warm storage caches.
  • Maximized Hardware Efficiency: In distributed AI training, losing a single node halts the entire job. Keeping accelerators like GPUs/TPUs bound lets workloads resume training significantly faster, directly reducing compute costs.

Observability and SRE Best Practices

To support monitoring, Kubernetes v1.35 introduces the AllContainersRestarting Pod condition. Set to True during restarts, it alerts SREs and autoscalers, preventing false-positive alerts, while container restart counts increment to let Prometheus easily track recovery events.
To use in-place restarts successfully, shift your mental model to "persistent sandboxes" and follow three best practices:

  1. Ensure Reentrancy: Kubelet only guarantees "at least once" execution for init containers. Reentrancy is now a standard requirement, so your code must be fully idempotent.
  2. Plan for Termination Handling: Graceful termination (preStop hooks) is not supported for in-place restarts. SIGKILL is almost immediate, so applications must handle sudden exits gracefully.
  3. Prepare External Tooling: CD and observability tools should expect re-running init containers without interpreting them as new deployments.

What's Next?

This beta capability is a major step toward fluid workload management and serves as a building block for advanced community features like JobSet in-place restarts (KEP-467).
Our work on KEP-5532 reflects our commitment to transparent open source governance. Developed collaboratively within SIG Node, this feature shows how we hold ourselves to high citizenship standards; making our design, goals, and intentions transparent while building shared best practices that benefit everyone. We encourage you to experiment with Kubernetes v1.35 and share your feedback with the community!

Learn More

Introducing OpenRL: A self-hosted post-training API for fine-tuning LLMs

Thursday, June 11, 2026

We are pleased to share a research preview of OpenRL, a new open-source project coming out of GKE Labs. OpenRL is a self-hosted training API for fine-tuning LLMs on your own Kubernetes cluster.

Why we built it

If you look at agentic RL on LLMs, it is incredibly easy to get bogged down in system complexity. To run a single RL loop, you have to coordinate a dozen different things: selecting and cleaning datasets, choosing RL environments, debugging training loops, managing reward signals, handling inference mismatches, allocating hardware, and managing infrastructure. Picture looks something like this:

an AI researcher and an infrastructure engineer staring at the hurdles in post training along the way to the summit
Figure shows an AI researcher and an infrastructure engineer staring at the hurdles in post training along the way to the summit.

Each of these is a hard problem. But what makes it more complex is how tightly AI research and infrastructure concerns are mixed together in today's tooling and frameworks.

We believe decoupling the infrastructure from AI research can make these problems more tractable so that infrastructure engineers and AI researchers can independently tackle them. We have seen this pattern with Kubernetes where Kubernetes abstracted out the infrastructure and made application developers and SREs life easier.

So, can you abstract out post training infrastructure? We believe so and drew huge inspiration/validation from Tinker (from Thinking Machines). The Tinker APIs for post training hit that Goldilocks zone where it hides all the post training infrastructure behind four key APIs:

high level components and their interaction in a OpenRL based RL workflow
Figure shows high level components and their interaction in a OpenRL based RL workflow

So the end result of this abstraction is that AI Researchers get full flexibility on their RL loop and infrastructure engineers can focus on scaling, orchestration, and reliability. OpenRL allows you to run the same training APIs but on your own infrastructure. And this decoupling has other interesting benefits.

Sharing GPUs

Traditional RL loops are strictly sequential. The trainer waits for the sampler to finish rollouts, the sampler waits for the environment to score rewards (which is often bound by slow CPU/network tasks), and the whole loop sits blocked. Your expensive GPUs spend a lot of time doing nothing. The abstraction allows running multiple RL jobs and allows infrastructure engineers to pack the training/sampling steps to utilize more of their GPUs. The graph below shows the GPU consumption in OpenRL for running one, two, and three RL jobs concurrently.

The figure shows the trainer/sampler duty cycle in OpenRL for scenarios with 1 RL job, 2RL jobs and 3 RL jobs respectively
The figure shows the trainer/sampler duty cycle in OpenRL for scenarios with 1 RL job, 2RL jobs and 3 RL jobs respectively.

Better UX

Once you separate out the infrastructure behind the APIs, you start to see the gains in user experience of developing the RL loop because AI researchers no longer have to wrangle the complex python dependencies like cuda. When you are doing R&D, you do not have to run the RL loop directly on the machines with GPUs, you can simply run your RL loop on your Mac pointing to the training APIs running on a Kubernetes cluster/VMs.

Autoresearch

We believe that frontier AI research will get more and more automated in the future and abstracting out infrastructure as a building block is key to that. To demonstrate that, we added an autoresearch recipe inspired heavily by karpathy's work. The recipe demonstrates how to conduct parallel experiments to conduct parameter sweep, and improve the reward signal for our text-to-sql recipe for Gemma models.

Figure showing autoresearch UI with multiple AI researchers conducting experiments in parallel in OpenRL
Figure showing autoresearch UI with multiple AI researchers conducting experiments in parallel in OpenRL

What OpenRL is not

  • A managed service. OpenRL is self-hosted and not a managed service. We aim to make it easy for users to deploy and operate it on their Kubernetes clusters.
  • An RL framework. OpenRL gives AI researchers full control over their RL loop.

Get started

We have made it easy to run OpenRL on your Mac, Nvidia GPUs, or on GKE. This allows you to test your RL loop on Mac and when you are ready to scale, you can point the RL loop to the OpenRL endpoint running in the GKE cluster.

Try out our text-to-SQL example for teaching the latest Gemma model SQL here: guides.

One of the benefits of a Tinker compatible endpoint is that you can use Tinker-Cookbook with OpenRL. Tinker-cookbook is one of the best resources for post training infrastructure for RL.

Future steps

We have started with a simple architecture focussing on LoRA fine-tuning and plan to evolve the project in the coming months, so please give it a try and share your feedback. A few things we are very excited to work on:

  • Full parameter fine-tuning
  • Multitenancy (simultaneous RL on different types of base models)

Acknowledgement

We have been inspired by the work done by various open source projects in AI communities, so huge thank you to Thinking Machines, vLLM, PyTorch, prime-rl, verl, SkyRL, and llm-d.

Disrupting the presentation layer using autonomous workflows

Thursday, May 21, 2026

Empowering every engineer to do more with Kubernetes

Kubernetes is the gold standard for container orchestration. Its power, flexibility, and rich API surface are exactly why it has become the foundation of modern cloud-first infrastructure. Today, engineers express that power through the K8s API, declarative YAML manifests, and cloud consoles, a remarkably expressive toolbox.
We believe the next step is to expand how engineers interact with that toolbox. A Kubernetes expert should be able to converse with a deep-domain peer that speaks fluent control-plane and can reason about cluster state in real time. An engineer who isn't a Kubernetes specialist should be able to express higher-order intent, such as "deploy my application," "rebalance this workload" and have it carried out safely against the same powerful APIs. Both audiences get more leverage out of the platform they already trust.
This is the vision behind Kube-Agents: a system of intelligent, autonomous, and human-in-the-loop agents that act as a new, intent-driven presentation layer for Kubernetes. We are moving from declarative intent via API to higher-order, human intent-driven operations while preserving everything that makes Kubernetes great underneath.

The Vision: Expanding the Presentation Layer

Today, engineers do impressive work stitching together metrics, alerts, and multi-step commands to keep clusters healthy. Agents extend that work, not replace it. By complementing existing interfaces with autonomous agents, engineers can choose the level of abstraction that fits the task: drop down to kubectl and YAML when precision matters, or describe intent in plain language when speed and clarity matter more. The agents continuously observe system state and can execute complex operations in real time on the engineer's behalf. This isn't about hiding Kubernetes. It's about giving every engineer a more capable collaborator on top of it.

Meet the Agents: A Specialized Team

Our architecture is currently built upon three core specialized agents, each acting as a new kind of intent-driven collaborator for different stakeholders:

  1. The Platform Agent
    • Role: A partner for the central governance layer, your management plane.
    • Focus: Codifying best practices and keeping platform blueprints evergreen and synchronized across the entire fleet.
    • Example: When a new egress policy is defined at the org level, the Platform Agent propagates it to the Dev Team agents and confirms enforcement, giving platform teams confidence in compliance while letting developers stay focused on their applications.
  2. The Cluster Operator Agent
    • Role: A trusted teammate for your infrastructure operators.
    • Focus: Global concerns like multi-cluster balancing, automated provisioning, security patching, and zero-downtime version upgrades.
    • Example: It can detect a degrading node and proactively migrate workloads before application latency spikes, expanding what a single operator can safely manage at scale.
  3. The Development Team Agent
    • Role: A production-savvy peer for developers.
    • Focus: The primary collaborator for developers. It supports the full workload lifecycle — reconciling manifest drift, right-sizing resources, and assisting with real-time debugging.
    • Example: When a developer asks "Why is my service failing?" in chat, the agent responds with relevant logs, correlated metrics, and a diagnosis of recent config changes — meeting a Kubernetes expert at depth and meeting a less specialized developer at intent.

Leveraging Industry Benchmarks

DevOps Bench is a comprehensive suite of benchmarks. These specialized agents learn from those results so they are equipped with the context to make well-reasoned decisions when autonomously supporting infrastructure work.

The First Demo

To be truly useful at the presentation layer, these agents can't be short-lived request/response scripts. They need to be persistent, long-running "team members" capable of continuous learning and collaboration.
As a first step, we've launched a set of workspaces compatible with OpenClaw for a demo, installable into your OpenClaw environment, leveraging existing out-of-the-box capabilities around identity, storage persistence, and memory. The agents included are: the Platform Agent, the Cluster Operator Agent, and the Development Team Agent.

  • Autonomous GitOps & JIT Probing (Dev Team Agent): Demonstrates prompt-driven staging deployments and dynamically generated, context-aware probers. The agent adheres strictly to GitOps workflows by opening PRs for infrastructure updates (such as node failure tolerance) and actively prevents configuration drift by reconciling manual manifest edits upon merge.
  • Self-Healing Infrastructure (Dev Team Agent): Showcases automated troubleshooting when a manifest is deployed with an image name typo. The agent executes a complete, autonomous 5-step remediation loop—Notification, Learning, Recommendation, Mutation, and Validation—to detect, fix, and verify the deployment without human intervention.
  • Multi-Agent Governance & Policy Coordination (Cluster Operator & Dev Team Agents): Highlights cross-agent negotiation when the Cluster Operator attempts to downscale underutilized resources for cost savings. The Dev Team Agent steps in to enforce minimum capacity policies, successfully prioritizing application reliability and governance over financial savings.
Animated walkthrough showcasing three OpenClaw agent demos in sequence: first, the Platform Agent configuring core infrastructure and identity; second, the Cluster Operator Agent managing cluster health and scaling; and finally, the Development Team Agent deploying applications and managing developer workflows.

Going forward, we will further productize this pattern, building on open standards to define agents and their capabilities (AGENTS.md, skills, MCP), and provide an out-of-the-box harness to orchestrate these agents.

Redefining the Stack

An intent-driven presentation layer is just the beginning. With an agentic interface in place, we can keep evolving the underlying infrastructure — adopting new components or integrating directly with additional infrastructure APIs — while engineers continue to interact with the system the way they already do. The interface stays intent-driven and stable; the agents adapt to the evolving stack underneath, so investment in how teams work today carries forward.

Call to Action

We're building Kube-Agents in the open because we believe the best infrastructure solutions are built collaboratively. Our goal is to use our expertise to give back to the open source community, while also actively learning from the ecosystem's real-world challenges. By working together, we can define best practices that benefit everyone.
If you're interested in helping shape the future of Kubernetes management, check out the Kube Agents repo.

We are seeking engagement on two fronts:

  • Share your use cases: What would you most like an autonomous teammate to help with? We want to learn from your unique operational needs—whether it's multi-cluster balancing, specific debugging scenarios, or policy enforcement—to ensure we're building tools that provide real leverage.
  • Define future roles: What new specialized agents should exist? We value your input on the roles these agents should fulfill to best serve diverse team structures and operational requirements.

Join the conversation, contribute your ideas, and help us build a self-driving cloud that works for everyone. Check out the Kube Agents project to open an issue or start a discussion.

Kubernetes goes AI-First: Unpacking the new AI conformance program

Monday, April 6, 2026

As AI workloads move from experimental notebooks into massive production environments, the industry is rallying around a new standard to ensure these workloads remain portable, reliable, and efficient.

At the heart of this shift is the launch of the Certified Kubernetes AI Conformance program.

This initiative represents a significant investment in common, accessible, industry-wide standards, ensuring that the benefits of AI-first Kubernetes are available to everyone.

How Kubernetes is Evolving for an AI-First World

Traditional Kubernetes was built for stateless, cloud-first applications. However, AI workloads introduce unique complexities that standard conformance doesn't fully cover:

  • Specific Hardware Demands: AI models require precise control over accelerators like GPUs and TPUs.
  • Networking and Latency: Inference and distributed training require low-latency networking and specialized configurations.
  • Stateful Nature: Unlike traditional web apps, AI often relies on complex, stateful data pipelines.

The AI Conformance program acts as a superset of standard Kubernetes conformance. To be AI-conformant, a platform must first pass all standard Kubernetes tests and then meet additional requirements specifically for AI.

Key Pillars of the AI Conformance Program

The Kubernetes AI Conformance program is being driven in the open via the AI Conformance program. This cross-company effort is led by industry experts Janet Kuo (Google), Mario Fahlandt (Kubermatic GmbH), Rita Zhang (Microsoft), and Yuan Tang (RedHat). This program is a collaborative effort within the open source ecosystem, involving multiple organizations and individuals. By developing this program in the open, the community ensures the standard is built on trust and directly addresses the diverse needs of the global ecosystem. The program establishes a verified set of capabilities that platforms across the industry, like Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS) are already adopting.

Dynamic Resource Allocation (DRA)

DRA is the cornerstone of the new standard. It shifts resource allocation from simple accelerator quantity to fine-grained hardware control via attributes. For data scientists, this means they can now request specific hardware based on characteristics such as memory capacity or specialized capabilities, ensuring the environment perfectly matches the model's needs.

All-or-Nothing Scheduling

Distributed training jobs often face "deadlocks" where some pods start while others wait for resources, wasting expensive GPU time. AI Conformance mandates support for solutions like Kueue, allowing developers to ensure a job only begins when all required resources are available, improving cluster efficiency.

Intelligent Autoscaling for AI Workloads

Conformant clusters must support Horizontal Pod Autoscaling (HPA) based on custom AI metrics, such as GPU or TPU utilization, rather than just standard CPU/memory. This allows clusters to scale up for heavy inference demand and scale down to save costs when idle.

Standardized Observability for High Performance

To manage AI at scale, you need deep visibility. The program requires platforms to expose rich accelerator performance metrics directly, enabling teams to monitor inference latency, throughput, and hardware health in a standardized way.

What's Next?

The launch of AI Conformance is just the beginning. As we head further into 2026, the community is adding automated testing for certification and expanding the standard to include more advanced inference patterns and stricter security requirements.

The ultimate goal? Making "AI-readiness" an inherent, invisible part of the Kubernetes standard.

To get involved and help shape the future of AI on Kubernetes, consider joining AI Conformance in Open Source Kubernetes. We welcome diverse perspectives, as your expertise and feedback are crucial to building a robust and inclusive standard for all.

The End of an Era: Transitioning Away from Ingress NGINX

Thursday, February 12, 2026

An AI generated image that depicts an old crumbling building labeled Ingress NGINX displays a banner saying Retiring March 2026. An equally crumbling road leads away from the building to a glowing, nice, new archway with the Kubernetes Logo at the top, with Gateway API across the top arch. Within the arch at the center of the light is another Kubernetes-inspired logo with the words Gateway API below it. Several light paths of varying colors, with labels like HTTP Route and TCP Route lead away from the archway, some of them off the side of the frame, and some of them to a set of servers. Some of the servers have clouds with up or down arrows above them, indicating cloud traffic moving in or out.

For many of us, the first time we successfully routed traffic into a Kubernetes cluster, we did it using Ingress NGINX. It was the project that turned a complex networking API into something we could actually use.

However, the Kubernetes community recently announced that Ingress NGINX is officially entering retirement. Maintenance will cease in March 2026.

Here is what you need to know about why this is happening, what comes next, and why this "forced" migration is actually a great opportunity for your infrastructure.

Clarifying Terminology

First, there are some confusing, overlapping terms here. Let's clarify.

  1. Ingress API - Kubernetes introduced the Ingress API as a Generally Available (GA) feature in 2020 with the release of Kubernetes version 1.19. This API is still available in Kubernetes with no immediate plans for deprecation or removal. However, it is "feature-frozen" meaning it is no longer being actively worked on or updated. The community has instead moved to Gateway API, which we'll talk more about later in this post.
  2. Ingress NGINX - "Ingress" is an API object available by default in Kubernetes, as described above. You can define your Ingress needs as an Ingress resource. But that resource won't actually do anything without a controller. Ingress NGINX is a very popular controller that uses NGINX as a reverse proxy and load balancer. This will no longer be maintained as of March 2026.
  3. As it says in the What You Need To Know blog from the Kubernetes project, "Existing deployments of Ingress NGINX will continue to function and installation artifacts will remain available." However "there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered."
  4. NGINX Ingress Controller - to make things more confusing, there is another controller called "NGINX Ingress." This controller to implement ingress for your Kubernetes resources via NGINX and NGINX Plus, owned and maintained by F5 / NGINX Inc. This will continue to be maintained and available in both its Open Source and Commercial forms.

In this blog post,we are going to talk about "Ingress NGINX," the controller being deprecated. We will also talk about "Ingress" or the "Ingress API", which is still around, but feature-frozen.

What Problem Did Ingress NGINX Solve?

In the early days of Kubernetes, getting external traffic to your pods was a nightmare. You either had to use expensive, cloud-specific LoadBalancers for every single service or manage complex NodePorts.

While the Kubernetes Ingress API was introduced as a standard specification for Layer 7 routing (HTTP/HTTPS), it was inherently limited, designed for a simpler time in Kubernetes' history, and offered minimal features. Features like advanced routing, traffic splitting, and non-HTTP protocols were not natively supported by the API

Ingress NGINX solved this problem by serving as a robust Ingress controller that executed the API's rules. Leveraging the widely adopted NGINX reverse proxy, the controller provided a powerful, provider-agnostic entry point for cluster traffic. It was able to:

  • Consolidate multiple services under a single IP address.
  • Provide robust Layer 7 capabilities, including SSL/TLS termination and basic load balancing.
  • Use familiar NGINX configuration logic inside a cloud-native environment.
  • Extend the basic Ingress API to support advanced features, such as rate limiting, custom headers, and sophisticated traffic management, by allowing users to inject familiar, raw NGINX configuration logic using custom nginx.ingress.kubernetes.io annotations (often called "snippets").

This flexibility, achieved by translating standard Ingress objects into feature-rich NGINX configurations, made Ingress NGINX the de-facto controller and the "Swiss Army Knife" of Kubernetes networking.

Why is it Retiring?

If it's so popular, why kill it? The very flexibility that made it so popular also (at least partially) led to its demise. The announcement points to two primary "silent killers":

  • The "Snippet" Security Debt: Ingress NGINX gained popularity through its flexibility, specifically "snippets" that let users inject raw NGINX config via annotations. Today, these are viewed as major security risks, as they can allow for configuration injection attacks. Fixing this architectural "feature" has become an insurmountable task.
  • The Maintainership Gap: Despite having millions of users, the project was sustained by only one or two people working in their spare time. In an industry where security vulnerabilities move fast, "best-effort" maintenance isn't enough to protect the global ecosystem.

Time for Gateway API

The removal of the popular NGINX ingress implementation opens up an opportunity to transition to the Gateway API. While the Ingress API in Kubernetes is not going anywhere (just the NGINX variant of it), development on it is frozen, and there are reasons for that.

Think of Gateway API as "Ingress 2.0." While the Ingress API is a single, limited resource, Gateway API is role-oriented. It separates the concerns of the Infrastructure Provider (who sets up the LB), the Cluster Operator (who defines policies), and the Application Developer (who routes the traffic).

For the Kubernetes Podcast from Google, we've interviewed Kubernetes maintainers working on Gateway API (Like in this episode featuring Lior Lieberman), and they tell a great story about why it was developed. In the early days of Kubernetes, the maintainers & contributors weren't sure exactly what users would need with regard to ingress management for workloads running on Kubernetes. The early Kubernetes Ingress object was an attempt to address the problems the maintainers thought users would need to solve, and they didn't get it all right. The annotations Ingress-NGINX supported on top of the Ingress API helped cover the many gaps in the Kubernetes API, but the annotations tied you to Ingress-NGINX. Those gaps have now been largely closed by Gateway API, and the API is supported by many conformant implementations, so you can have confidence in the portability of the API.

An important feature of Gateway API's design is that it is an API standard defined by the community, but implemented by your infrastructure or networking solution provider. Networking ultimately boils down to cables transmitting electrical signals between machines. What kind of machines and how they're connected has a big impact on the types of ingress capabilities available to you- or at least in how they're actually implemented. Gateway API provides a standard set of capabilities that you can access in a standardized way, while allowing for the reality of different networking implementations across providers. It's meant to help you get the most out of your infrastructure- regardless of what that infrastructure actually is.

How Gateway API solves the old problems with Ingress NGINX:

  • Security by Design: No more "configuration snippets." Features are built into the API natively, reducing the risk of accidental misconfiguration.
  • Standardization: Unlike the old Ingress API, which required custom annotations for almost everything (like traffic splitting), Gateway API builds these features into the spec, offering greater portability.
  • Extensibility: It is designed to handle more than just HTTP—it brings the same power to TCP, UDP, and gRPC.

The Challenges of Transitioning

Migration is rarely "click and play." Users moving away from Ingress NGINX should prepare for:

  • Annotation Mapping: Most of your nginx.ingress.kubernetes.io annotations won't work on new controllers. You'll need to map these to the new Gateway API "HTTPRoute" logic.
  • Learning Curve: Gateway API has more "objects" to manage (Gateways, GatewayClasses, Routes). It takes a moment to wrap your head around the hierarchy, but it was implemented that way based on experience - these objects should help you manage your workloads' ingress needs more efficiently.
  • Feature Parity: If you rely on very specific, obscure NGINX modules, you'll need to verify that your new controller (be it Envoy-based like Emissary or Cilium, or a different NGINX-based provider) supports them.

Why It's Worth It

The retirement of Ingress NGINX is not just a chore; it is a forcing function for adopting more sustainable architecture. By migrating to Gateway API, you gain:

  • Stability and Active Development: Gateway API is a General Availability (GA) networking standard that has maintained a "standard channel" without a single breaking change or API version deprecation for over two years. Unlike many Ingress controllers where development has largely paused, most Gateway controllers are far more actively maintained and continue to add new features like CORS and timeouts.
  • Portability: Choosing a different Ingress controller might seem easier, but if you rely on Ingress-NGINX annotations, you will likely have to migrate to another set of implementation-specific annotations. Gateway API provides more portable features directly in the core API and ensures a consistent experience across different implementations. When you select an implementation that is conformant with the latest v1.4 release, you can be confident that the behavior of these features will be consistent.
  • Future-Proof Extensibility: While Gateway API supports many more features than the core Ingress API, if you find a needed feature missing, an implementation is likely to provide a similar or equivalent feature as an implementation-specific extension. For example, GKE Gateway and Envoy Gateway extend the API with their own custom policies.

Next Steps

Start your migration planning today to capitalize on the opportunity and meet the deadline.

  1. Audit Your Usage: Run kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx to see where you are still using the legacy controller.
  2. Utilize Automation: Check out the ingress2gateway project. A lot of work is going into this tool to make the migration experience better, including adding support for the most widely used Ingress-NGINX annotations.
  3. Experiment and Provide Feedback: Give Gateway API a try! Start a PoC with a conformant Gateway API implementation (like GKE Gateway, Cilium, or Envoy Gateway). The community welcomes help and feedback on ingress2gateway and encourages users to share feedback on what Gateway API is getting right and wrong.
  4. Adhere to the Timeline: You have until March 2026 before the security updates stop. Start your migration planning sooner rather than later!

For more details on migrating from Ingress to Gateway API refer to our documentation.

.