Pod startup speed is often overlooked in cloud-native environments, yet its impact extends across multiple dimensions of system performance and cost. Consider a scenario where traffic suddenly surges and the auto-scaling system needs to quickly spin up new Pods to handle the load. If each Pod takes tens of seconds or even minutes to become fully operational, those incoming requests during the startup window will likely be dropped, degrading user experience. This isn’t merely a performance issue—it’s a cost issue, as idle compute resources consume expenses every single second.

Why Pod Startup Speed Matters

Pod startup performance touches several critical concerns. First is the need for rapid scaling. When applications require autoscaling, startup speed determines how quickly the system can respond to traffic fluctuations. Second is resource efficiency. Faster startup means less wasted idle resources. Third is user experience, especially in serverless architectures where cold start latency directly impacts the application response time users perceive. Finally, from a cost perspective, reducing Pod startup time significantly lowers infrastructure expenses.

The Four Key Points of Pod Startup

Understanding the various stages of Pod startup is essential to optimization efforts. The Pod startup process can be divided into four major phases: API Server processing, scheduling, pod startup on node, and application start-to-ready.

The API Server processing phase involves Pod object creation, validation, and persistence. During this phase, the control plane must handle the request, execute admission control policies, and write the Pod object to etcd. While typically fast, this process can become a bottleneck in high-concurrency scenarios.

The scheduling phase spans from Pod creation to being scheduled on a specific node. The scheduler must evaluate all available nodes and select the most suitable target. The duration depends on cluster size and scheduler configuration. In large-scale clusters, this can become a significant source of latency.

The node startup phase encompasses pulling images, creating containers, and starting processes on the selected node. This is usually the longest phase in the entire Pod startup process. It includes network image pulls, storage volume initialization, application startup, and health check completion.

The ready phase, while not strictly part of the “startup” process, affects how the system perceives Pod readiness. If health checks are misconfigured, a Pod might be running but considered unready, affecting overall startup time metrics.

Optimizing the API Server Processing Stage

At the API Server level, the focus is on improving throughput and reducing latency. A straightforward but effective optimization is adjusting the API Server’s concurrent request handling capacity. Increasing the --max-requests-inflight and --max-mutating-requests-inflight parameters allows the API Server to handle more Pod creation requests simultaneously.

Another crucial optimization is streamlining admission controllers. Some controllers might perform expensive operations, such as accessing external services or executing complex validations. Consider disabling unnecessary admission controllers or configuring them for maximum efficiency. Similarly, ensuring excellent etcd performance is vital, as the API Server ultimately must persist Pod objects to etcd.

Optimizing the Scheduling Phase

The scheduler’s performance directly impacts the time from Pod creation to scheduling. Leveraging various optimization techniques provided by the scheduling framework can accelerate this process. For instance, the pre-filter and filter phases can quickly eliminate unsuitable nodes, reducing the number of candidates for subsequent scoring phases.

Another key optimization involves judiciously using node affinity and Pod affinity rules while avoiding overly complex rules that increase scheduling latency. Additionally, for specific workloads, using priority and preemption features can ensure critical Pods are scheduled faster.

In large-scale clusters, consider deploying multiple scheduler instances to distribute the load. Kubernetes natively supports running multiple scheduler instances concurrently, which can significantly boost scheduling throughput.

Optimizing the Node Startup Phase

This phase offers the most substantial optimization opportunities. First, image pulling is a major bottleneck. Image warming is a proven optimization strategy. Pre-pull commonly used images to nodes during startup or scheduled maintenance windows. When Pods actually launch, they won’t need to fetch images from remote registries, drastically reducing startup time.

There are many ways to improve image pull speed. These include best practices for reducing image size, configuring Kubelet to enable concurrent image pulls, and adopting P2P distribution (such as Dragonfly or Uber/Kraken) or lazy loading mechanisms (such as stargz-snapshotter and Nydus). How to accelerate Pod startup under large-scale concurrency has been discussed in the context of VKE’s practices using Dragonfly and Nydus.

Container runtime performance is also critical. Different runtimes (such as containerd and Docker) have different performance characteristics. containerd is generally considered more lightweight and efficient, especially at large scale. Regularly upgrading the runtime to the latest version can also bring performance improvements. Slow docker create / docker run behavior is very common: it is usually caused by too many containers or images on a node (which can be mitigated by scheduled cleanup jobs). The legacy devicemapper storage driver is significantly slower in scenarios with frequent container creation and deletion; overlayfs performs somewhat better. In addition, older Docker versions contain many minor bugs that can cause docker ps or docker run to hang.

Application startup time itself also deserves attention. Some applications perform a large amount of initialization work at startup, such as database migrations or cache warm-up. Making these tasks asynchronous or deferring them until after the application has started can significantly reduce perceived startup latency.

CPU throttling during startup is another common issue. For a detailed discussion of CPU throttling in Kubernetes, possible mitigations include increasing CPU limits, pinning CPUs, or bypassing the issue via VPA-based approaches. (For cold-start scenarios with VPA, see: After waiting six years, Kubernetes 1.35 finally reaches GA with in-place resizing, boosting Java startup speed by 70%!)

Init containers can be used to perform necessary setup before the main container starts. However, init containers run sequentially, so it is important to avoid excessive initialization steps. Only perform what is strictly necessary and parallelize where possible. If feasible, use PostStart hooks to avoid delaying the main container startup. Complete preparation as early as possible: can these tasks be done during image build time? Can they be handled by a DaemonSet or Pod on the node? All of these approaches help reduce the preparation time before the Pod’s main container starts.

Optimizing Observability and Health Checks

Health check configuration has a significant impact on startup time metrics. If a StartupProbe is configured, the initialDelaySeconds of the ReadinessProbe should be reduced.

If no StartupProbe is configured, an appropriate initialDelaySeconds must be set for the ReadinessProbe.

At the same time, health checks must remain sufficiently strict to ensure that unhealthy Pods are not mistakenly considered ready.

Startup probes allow applications enough time to complete initialization without being killed or restarted during the startup phase.

Checkpointing and Snapshots

The Kubernetes community is exploring the use of checkpointing techniques to accelerate Pod startup. Checkpointing allows the state of a running container to be saved and later restored quickly, thereby skipping the application’s normal startup process. This is particularly beneficial for applications with long startup times.

For example, CRIU (Checkpoint/Restore In Userspace) has been integrated into the container runtimes used by Kubernetes. By saving a container’s state at a certain point in time—including memory and filesystem state—it can be rapidly restored when needed, effectively enabling a “warm start.” This approach is especially promising for serverless computing and batch workloads.

Kubelet configuration

Kubernetes 1.27: updates on speeding up Pod startup

https://kubernetes.io/blog/2023/05/15/speed-up-pod-startup/

This post points out a common issue in versions prior to 1.27, where Pods could start slowly and node-side events such as volume mount failures would occur.

To speed up Pod startup on nodes hosting multiple Pods—especially during sudden scale-up or scale-down events—the kubelet needs to synchronize Pod state and prepare ConfigMaps, Secrets, or volumes. This requires high-bandwidth access to the kube-apiserver.

In versions prior to v1.27, the default value of kubeAPIQPS was 5 and kubeAPIBurst was 10. Starting from v1.27, to improve Pod startup performance, the kubelet increased these defaults to 50 and 100 respectively. It is worth noting that raising the kubelet API QPS limits is not the only factor contributing to the performance improvement.

Comprehensive Optimization Strategy

Pod startup optimization isn’t an isolated effort but requires a systematic, layered approach. From the API Server to the scheduler, container runtime, and application layer, every level offers optimization opportunities.

Establishing clear Pod startup time metrics is an essential first step. Clearly defining what constitutes startup time (from Pod creation to container running, or to Pod readiness?) is important. Using Prometheus or other monitoring tools to collect detailed startup metrics helps identify where the real bottlenecks are.

Priorities differ based on specific business needs and cluster characteristics. For high-traffic services requiring rapid scaling, image warming and startup probe tuning might yield the best results. For applications with long startup times, checkpoint technology might provide more value. For large-scale clusters, scheduler performance optimization and multiple scheduler instances might be key.

Finally, remember that optimization is a continuous process. Regularly reviewing and testing new optimization strategies, along with performance improvements from new Kubernetes versions, ensures your cluster maintains optimal performance.

Related Resources

【KubeCon China 2023】How Can Pod Start-up Be Accelerated on Nodes in Large Clusters? – Paco Xu, DaoCloud & Byron Wang https://www.youtube.com/watch?v=UfjSphSD1Uk&pp=2AYD

https://github.com/pacoxu/AI-Infra/blob/main/docs/kubernetes/pod-lifecycle.md

Agones brings dedicated game server hosting to Kubernetes, enabling multiplayer gaming infrastructure with cloud-native scalability and management. This blog explores Agones as it applies to join CNCF Sandbox.

Introduction

As the gaming industry grows rapidly, the demand for scalable, reliable dedicated game server infrastructure has become critical. Agones is an open-source platform built on Kubernetes that addresses this need by providing a specialized solution for hosting, running, and scaling dedicated game servers.

Agones, derived from the Greek word “agōn” meaning “contest” or “competition at games”, transforms Kubernetes into a powerful platform for managing game server workloads with the same cloud-native principles used for traditional applications.

Project Status: Agones has applied to join the CNCF Sandbox (github.com/cncf/sandbox/issues/440), marking an important step in bringing gaming workloads into the cloud-native ecosystem.

What is Agones?

Agones is a library for hosting, running, and scaling dedicated game servers on Kubernetes. It replaces bespoke or proprietary cluster management solutions with Kubernetes-native APIs and controllers.

Core Concept: Dedicated game servers are stateful, ephemeral workloads that differ significantly from typical web applications. Each game session requires its own isolated server process, must maintain consistent network identity, and needs specialized lifecycle management. Agones extends Kubernetes to handle these unique requirements through Custom Resource Definitions (CRDs) and controllers.

Key Features

• GameServer CRD: Define individual game servers declaratively using YAML or the Kubernetes API, complete with health checking and connection information
• Fleet Management: Manage large groups of game servers as Fleets, similar to Kubernetes Deployments but optimized for game server workloads
• Autoscaling: Native integration with Kubernetes cluster autoscaling, allowing Fleets to scale based on game server demand
• Client SDKs: SDKs for multiple languages (Go, C#, C++, Rust, Node.js, REST) enabling game servers to communicate with the Agones control plane
• Lifecycle Management: Automatic health checks, graceful shutdown handling, and state management for game server processes
• Metrics and Observability: Game server-specific metrics exports and dashboards for operations teams

Architecture and Design

Agones extends Kubernetes with custom controllers and resources specifically designed for game server workloads:

Custom Resources

• GameServer: Represents a single dedicated game server instance with health status, network ports, and connection information
• Fleet: Manages groups of GameServers, providing replica management, rolling updates, and scaling capabilities
• FleetAutoscaler: Automates Fleet scaling based on buffer policies, webhook policies, or counter/list-based policies
• GameServerAllocation: Enables matchmakers to atomically allocate Ready GameServers from a Fleet for player connections

How It Works

1. Deployment: Operators define GameServers or Fleets using Kubernetes manifests
2. Lifecycle Management: Agones controllers create pods and manage their lifecycle based on game server state
3. Ready State: Game servers use the Agones SDK to mark themselves Ready when accepting connections
4. Allocation: Matchmaking systems request GameServer allocation via the Kubernetes API
5. Session Management: Game servers notify Agones when sessions end, triggering cleanup
6. Autoscaling: FleetAutoscalers monitor Fleet status and adjust replicas to maintain desired buffer or respond to custom policies

Use Cases and Production Adoption

Agones is designed for multiplayer gaming scenarios requiring dedicated game servers:

• Session-based multiplayer games: FPS, MOBA, battle royale games where each match runs on a dedicated server
• Persistent game worlds: MMO game zones or shards that require long-lived server processes
• Match-based esports: Competitive gaming infrastructure requiring consistent server performance
• Cross-platform gaming: Unified infrastructure for console, PC, and mobile multiplayer experiences

The project is already used in production by major gaming companies and has proven its reliability at scale. The CNCF sandbox application notes that “this project is already used in production by many” organizations.

Why CNCF?

According to the CNCF Sandbox application:

Since Agones is tightly coupled to Kubernetes, CNCF is the logical home for the project. Agones being in the CNCF allows for a broader community contributor ecosystem.

Agones brings a new gaming offering to the CNCF landscape, representing a specific but important use case for Kubernetes. As cloud-native technologies expand into specialized domains, gaming infrastructure represents a significant workload category with unique requirements.

Cloud-Native Integration

Agones integrates directly with core CNCF projects:

• Kubernetes: Built as a Kubernetes controller with CRDs
• Prometheus: Metrics exports for monitoring game server health and performance
• Helm: Installation and configuration via Helm charts
• Container runtimes: Works with any Kubernetes-compatible container runtime

Project Governance and Community

Agones operates as a vendor-neutral open-source project:

• License: Apache 2.0
• Code of Conduct: Contributor Covenant
• Governance: Clear contribution guidelines and ownership model
• Community Channels: Active Slack workspace, mailing list, regular community meetings
• Maintained by: Originally created by Google Cloud, now community-driven with multiple maintainers

The project has comprehensive documentation, quickstart guides, and example implementations for developers getting started with game server hosting on Kubernetes.

Similar Projects and Ecosystem

Within the Kubernetes gaming ecosystem, OpenKruise’s kruise-game (github.com/openkruise/kruise-game) provides similar capabilities. Both projects demonstrate growing interest in gaming workloads on Kubernetes.

Agones’ application to CNCF Sandbox represents an opportunity to establish standards and best practices for game server orchestration across the cloud-native community.

Vision and Roadmap

Agones continues active development with regular releases following a documented release process. The project roadmap focuses on:

• Enhancing autoscaling capabilities with more sophisticated policies
• Improving observability and debugging tools for game server operations
• Expanding SDK support for additional programming languages and engines
• Performance optimizations for larger-scale deployments
• Better integration with matchmaking and lobby systems

The project aims to make dedicated game server hosting as straightforward and reliable as deploying stateless web applications, while respecting the unique requirements of real-time gaming workloads.

Getting Started

For developers interested in exploring Agones:

1. Documentation: Comprehensive guides at agones.dev/site/docs/
2. Quick Start: Install Agones on a Kubernetes cluster and deploy a simple game server
3. Examples: Multiple example game server implementations in the repository
4. Community: Join the Agones Slack and mailing list for support and discussion

Agones represents the maturation of gaming infrastructure into the cloud-native era, bringing the operational benefits of Kubernetes to one of the most demanding real-time workload types.

Conclusion

Agones transforms Kubernetes into a powerful platform for dedicated game server hosting, addressing the unique challenges of multiplayer gaming infrastructure. As it applies to join the CNCF Sandbox, the project demonstrates how cloud-native technologies can adapt to specialized workload requirements while maintaining Kubernetes-native principles.

For gaming companies building multiplayer experiences and infrastructure teams managing game servers, Agones provides a proven, production-ready solution that leverages the full ecosystem of cloud-native tools and practices.

References:

• Agones GitHub: github.com/googleforgames/agones
• Official Website: agones.dev/site/
• CNCF Sandbox Application: github.com/cncf/sandbox/issues/440
• Announcement Blog: cloud.google.com/blog/products/containers-kubernetes/ introducing-agones-open-source-multiplayer-dedicated-game-server-hosting- built-on-kubernetes

Note: The content in this article is based on currently available public information and is intended for technical reference only. The effectiveness of each solution depends heavily on your specific workload, infrastructure, and ecosystem integration. The architectural affiliations and early design choices mentioned here do not determine their future direction. In practice, community activity, openness, and long-term evolution are often more important factors. Please evaluate and choose based on your own scenario.

Introduction

The landscape of open-source inference orchestration for Large Language Models (LLMs) has evolved rapidly in 2025. Multiple projects have emerged to address the challenges of deploying and scaling LLM inference workloads on Kubernetes, each with its own approach to workload management, resource orchestration, and performance optimization.

This blog post provides an overview of the current inference orchestration solutions, examines the convergence trends in the ecosystem, and raises important questions about when Prefill-Decode (PD) disaggregation truly provides value.

The Current Landscape

Rapid Development, Gradual Convergence

The inference orchestration space is characterized by:

• Many implementations: Multiple projects solving similar problems
• Different architectural choices: Varying approaches to workload management
• Shared goals: All aim to optimize LLM inference at scale
• Emerging patterns: Common solutions beginning to emerge

Despite the diversity, we’re seeing convergence around key patterns: LeaderWorkerSet (LWS)-based architectures, intelligent routing, and disaggregated serving models.

Workload Orchestration Solutions

1. Dual LWS Architecture

llm-d implements a dual LeaderWorkerSet architecture for Prefill-Decode disaggregation:

• Two LWS instances: Separate LWS for prefill and decode workers
• KServe integration: Deep integration with KServe for model serving
• LMCache support: Efficient KV cache management across workers
• Routing sidecar: Intelligent request routing and cache optimization

Client → Routing Sidecar → Prefill LWS → KV Cache → Decode LWS → Response

Why dual LWS? This architecture enables independent scaling and resource optimization for each phase while maintaining coordination through the leader-worker pattern.

2. Serving Group: Volcano Kthena

Kthena takes a different approach with its Serving Group concept:

• No dual LWS: Kthena intentionally avoids the dual LWS pattern
• Gang scheduling integration: Leverages Volcano’s gang scheduling capabilities
• Reduced layering: Eliminates the StatefulSet/Pod layer complexity
• Direct integration: Native integration with Volcano scheduler

Why not LWS? The Kthena team found that integrating with Volcano’s gang scheduling required a different architecture. The dual LWS, StatefulSet, and Pod layering added complexity without clear benefits for their use case.

This design choice reflects a key insight: the best orchestration solution depends on your existing infrastructure and scheduling requirements.

3. StormService: AIBrix

AIBrix StormService provides specialized container lifecycle management for P/D disaggregation:

• P/D lifecycle management: Fine-grained control over prefill and decode containers
• Multi-mode support: TP, PP, single GPU, and P/D disaggregation
• StormService and RoleSet CRDs: Custom resources for P/D orchestration
• Enterprise features: Multi-tenancy, routing, and observability

Architecture:

AIBrix Control Plane
    ├── StormService Controller
    │   ├── RoleSet (Prefill)
    │   └── RoleSet (Decode)
    ├── Gateway & Routing
    └── Autoscaler

4. NVIDIA Dynamo: Two Modes

Dynamo offers two distinct deployment modes:

Grove Mode: https://github.com/ai-dynamo/dynamo/blob/be67f67b1a8d0837291ac7033af6edbc146f6995/docs/kubernetes/grove.md

• High-performance inference
• NVIDIA-native deployment
• Optimized for pure NVIDIA infrastructure
  • “GPU support depends on the engine: Dynamo uses backends vllm, sglang and trt-llm. Dynamo is the layer above that.” quota

LWS Mode:

• Kubernetes-native deployment using LeaderWorkerSet
• Multi-node disaggregated serving
• Integration with Kubernetes ecosystem

This dual-mode approach allows users to choose the right level of abstraction for their infrastructure.

5. SGLang RBG: LWS-Inspired

RBG (Resource-Aware Batch Scheduler) learned from and reused design patterns from LWS:

• LWS-inspired: Incorporates proven patterns from LeaderWorkerSet
• Resource-aware scheduling: Optimizes batch scheduling based on resources
• Batch optimization: Intelligent batching strategies for throughput
• P/D support: Enables disaggregated prefill and decode workloads

Convergence Trends

Common Patterns Emerging

Despite different implementations, several patterns are converging:

Why So Many Implementations?

The diversity reflects different optimization goals:

1. Scheduling integration: Kthena needs Volcano gang scheduling directly
2. Enterprise features: AIBrix focuses on multi-tenancy and observability
3. Performance focus: Dynamo optimizes for NVIDIA hardware
4. Simplicity: RBG provides a lightweight LWS-inspired approach
5. Production-readiness: llm-d demonstrates a complete reference implementation

The PD Disaggregation Question

At KCD Hangzhou 2025, Wen Yuan Yu’s keynote “Kubernetes Is Born for Service Resource Orchestration—MaaS Changes Everything” raised an important question about PD-separation:

“Achieving strong production gains from PD-separation is very difficult.
While stress testing can show great results, in real dynamic environments it becomes much harder.
Over-provisioning Decode introduces significant challenges.”

This observation directly challenges the assumption that PD-separation is always beneficial.

Does PD Disaggregation Always Provide Value?

At KCD Hangzhou 2025, Yu Wen Yuan’s keynote “Kubernetes Was Built for Service-Resource Orchestration. MaaS Changes Everything” raised important questions about PD disaggregation:

“PD-Disaggregate Role Scheduling • Not So Sure? (Our answer is Data Plane!)”

This challenges the assumption that PD disaggregation is always beneficial.

When PD Disaggregation Helps

PD disaggregation provides clear benefits when:

• Long prefill, short decode: Input prompts are much longer than outputs
• High concurrency: Many simultaneous requests need serving
• Heterogeneous hardware: Different GPU types for different phases
• SLA-driven scheduling: Different latency requirements (TTFT vs TPOT)

When PD Disaggregation May Not Help

Consider alternatives when:

• Short contexts: Both prefill and decode are fast
• Low concurrency: Few simultaneous requests
• Homogeneous hardware: Same GPUs for all workloads
• Complexity costs: Operational overhead outweighs benefits
• KV cache transfer overhead: Network latency exceeds computation savings

The Data Plane Perspective

The “Data Plane” answer suggests that the value of PD disaggregation depends on where bottlenecks actually exist. Before implementing complex orchestration:

1. Profile your workload: Understand where time is spent
2. Measure KV cache transfer costs: Network overhead matters
3. Consider simpler alternatives: TP/DP without disaggregation
4. Evaluate operational complexity: More components = more failure modes

Configuration Optimization: AIConfigurator

Choosing the right P/D configuration is complex. NVIDIA’s AIConfigurator helps optimize disaggregated deployment configurations:

What AIConfigurator Does

• Configuration space search: Evaluates thousands of P/D combinations
• SLA-constrained optimization: Finds configurations meeting TTFT/TPOT targets
• Hardware-specific tuning: Supports H100, H200, B200 with collected data
• xPyD planning: Determines optimal prefill/decode worker ratios

Example Usage

# Find optimal configuration for Qwen3-32B on 32 H200 GPUs
# with SLA targets: TTFT ≤ 300ms, TPOT ≤ 10ms
aiconfigurator cli default \
  --model QWEN3_32B \
  --total_gpus 32 \
  --system h200_sxm \
  --isl 4000 \
  --osl 500 \
  --ttft 300 \
  --tpot 10

Why AIConfigurator Matters

Traditional autoscaling (HPA/KPA) doesn’t understand LLM-specific characteristics. AIConfigurator provides:

• Informed decisions: Data-driven configuration choices
• Predictive optimization: Estimate performance before deployment
• Resource efficiency: Maximize GPU utilization with SLA guarantees

Recommendations

For New Deployments

1. Start simple: Begin with monolithic serving (no P/D disaggregation)
2. Profile first: Understand your workload characteristics
3. Use AIConfigurator: Let data guide configuration decisions
4. Add complexity gradually: Introduce P/D only when benefits are clear

For Existing Infrastructure

Key Questions Before Adopting PD Disaggregation

1. Is your prefill time >> decode time? If not, disaggregation may not help.
2. Can your network handle KV cache transfer? Network overhead can eliminate gains.
3. Do you need independent scaling? If P and D scale together, keep them together.
4. Is operational complexity acceptable? More components = more failure modes.

Conclusion

The inference orchestration landscape is diverse but converging. Key takeaways:

• Multiple solutions exist because different infrastructure has different needs
• LWS-based patterns are popular but not universal (Kthena’s Serving Group shows alternatives)
• PD disaggregation is not always valuable – profile your workload first
• Tools like AIConfigurator help navigate the complex configuration space
• Start simple, add complexity when needed based on actual measurements

The future will likely see further consolidation around proven patterns, but the current diversity reflects healthy experimentation in a rapidly evolving field.

References

Workload Orchestration Projects

• llm-d – Dual LWS architecture for P/D
• Kthena – Volcano-based Serving Group
• AIBrix – StormService for P/D
• Dynamo – NVIDIA inference platform
• RBG – LWS-inspired batch scheduler

Configuration Tools

• AIConfigurator – P/D configuration optimizer

Related Documentation

• PD Disaggregation Overview
• Inference Guide
• LWS (LeaderWorkerSet)

Presentations

• KCD China 2025: Kubernetes Was Built for Service-Resource Orchestration. MaaS Changes Everything
• PDF Slides

Agent Sandbox provides a secure, isolated, and efficient execution environment
for AI agents. This blog explores the project, its integration with gVisor and
Kata Containers, and future trends.

Introduction

As AI agents become more prevalent in enterprise applications, the need for
secure execution environments has become critical. Agent Sandbox is a new
Kubernetes project under SIG Apps
that addresses this challenge by providing a standardized, declarative API for
managing isolated, stateful, singleton workloads—ideal for AI agent runtimes.

Key Features:

• Kubernetes Primitive Sandbox CRD and Controller: A native Kubernetes
  abstraction for managing sandboxed workloads
• Ready to Scale: Support for thousands of concurrent sandboxes while
  achieving sub-second latency
• Developer-Focused SDK: Easy integration into agent frameworks and tools

Project Overview

Core: Sandbox CRD

The Sandbox Custom Resource Definition (CRD) is the heart of agent-sandbox.
It provides a declarative API for managing a single, stateful pod with:

• Stable Identity: Each Sandbox has a stable hostname and network identity
• Persistent Storage: Sandboxes can be configured with persistent storage
  that survives restarts
• Lifecycle Management: The controller manages pod lifecycle including
  creation, scheduled deletion, pausing, and resuming

Extensions

The project provides additional CRDs for advanced use cases:

• SandboxTemplate: Reusable templates for creating Sandboxes
• SandboxClaim: Allows users to create Sandboxes from templates
• SandboxWarmPool: Manages a pool of pre-warmed Sandbox Pods for fast
  allocation (achieving sub-second startup latency)

Architecture

                              ┌─────────────────┐
                              │   K8s API       │
                              │   Server        │
                              └────────┬────────┘
                                       │
                              ┌────────▼────────┐     ┌─────────────┐
                              │  Agent Sandbox  │────▶│  Replenish  │
                              │   Controller    │     │    Pool     │
                              └────────┬────────┘     └─────────────┘
                                       │
                                       │ Allocate from Pool
                                       ▼
┌────────────────────────────────────────────────────────────────────┐
│                        Agent Sandbox                               │
│            Executing Isolated, Low Latency Tasks                   │
│ ┌──────────────────┐   ┌──────────┐   ┌──────────────────────────┐ │
│ │ Agent Orchestrator│──▶│ Executor │──▶│  Task Execution         │ │
│ │       Pod         │   │ (API/SDK)│   │  Agent Sandbox          │ │
│ │                   │   │          │   │ ┌──────────────────────┐│ │
│ │ Agent app/framework   │ iStream  │   │ │Execution Process     ││ │
│ │ requesting sandboxed  │          │   │ │  (gVisor/Kata)       ││ │
│ │ execution environment │          │   │ ├──────────────────────┤│ │
│ │                   │   │          │   │ │ Ephemeral Storage    ││ │
│ │                   │   │          │   │ ├──────────────────────┤│ │
│ │                   │   │          │   │ │ Network Policy       ││ │
│ └──────────────────┘   └──────────┘   │ └──────────────────────┘│ │
│                                        └──────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘

Runtime Integration: gVisor and Kata Containers

Agent Sandbox is designed to be vendor-neutral, supporting various runtimes
to provide enhanced security and isolation. The two primary implementations are
gVisor and Kata Containers.

gVisor Integration (GKE)

图片来源：KubeCon 北美 Keynote 演讲，Jago Macleod （谷歌）

gVisor is an application kernel that provides an
additional layer of isolation between container applications and the host
kernel. It intercepts application system calls and implements them in user
space.

GKE Integration Status:

• Production Ready: gVisor is available as a runtime option in Google
  Kubernetes Engine (GKE) via the gvisor RuntimeClass
• Snapshot and Resume: GKE supports snapshotting and resuming sandboxes,
  enabling infrastructure efficiency and sophisticated parallel executions
• Performance Optimized: The gVisor team at Google has optimized the
  runtime for AI agent workloads with minimal overhead

Example Configuration:

apiVersion: agents.x-k8s.io/v1alpha1
kind: Sandbox
metadata:
  name: ai-agent-sandbox
spec:
  podTemplate:
    spec:
      runtimeClassName: gvisor
      containers:
      - name: agent-runtime
        image: my-ai-agent:latest

Kata Containers Integration

From: KCD hangzhou 2025

Kata Containers provides lightweight virtual
machines that behave like containers but offer the security isolation of VMs.
Each container runs in its own lightweight VM with a dedicated kernel.

Integration Status:

• Active Development: The Kata Containers community is actively working on
  Agent Sandbox integration
• VM-Level Isolation: Provides strong isolation through hardware
  virtualization
• GPU Support: Kata supports GPU passthrough for AI/ML workloads

Example with Kata on GKE:

apiVersion: agents.x-k8s.io/v1alpha1
kind: Sandbox
metadata:
  name: kata-ai-sandbox
spec:
  podTemplate:
    spec:
      runtimeClassName: kata-qemu-nvidia-gpu
      containers:
      - name: agent-runtime
        image: my-ai-agent:latest

Key Resources:

• Kata Containers Agent Sandbox Blog
• GKE with Kata Containers Example

Comparison

Desired Characteristics

The Agent Sandbox project aims to achieve:

• Strong Isolation: Support for gVisor and Kata Containers for kernel and
  network isolation
• Deep Hibernation: Save state to persistent storage and archive Sandbox
  objects
• Automatic Resume: Resume sandboxes on network connection
• Efficient Persistence: Elastic and rapidly provisioned storage
• Memory Sharing: Explore sharing memory across Sandboxes on the same host
• Rich Identity & Connectivity: Dual user/sandbox identities and efficient
  traffic routing
• Programmable: Applications and agents can programmatically consume the
  Sandbox API

Use Cases

Agent Sandbox is designed for:

1. AI Agent Runtimes: Isolated environments for executing untrusted,
   LLM-generated code
2. Development Environments: Persistent, network-accessible cloud
   environments for developers
3. Notebooks and Research Tools: Persistent sessions for tools like Jupyter
   Notebooks
4. Stateful Single-Pod Services: Hosting single-instance applications
   needing stable identity

Getting Started

Installation

# Replace "vX.Y.Z" with a specific version tag
export VERSION="v0.1.0"

# Install core components
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/manifest.yaml

# Install extensions (optional)
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/extensions.yaml

Create Your First Sandbox

apiVersion: agents.x-k8s.io/v1alpha1
kind: Sandbox
metadata:
  name: my-sandbox
spec:
  podTemplate:
    spec:
      containers:
      - name: my-container
        image: your-agent-image:latest

Trends and Future Directions

Industry Trends

1. Growing AI Agent Adoption: As AI agents become more autonomous and
   capable, secure execution environments become essential
2. Zero-Trust Security: Agent Sandbox aligns with zero-trust principles by
   providing isolated execution environments
3. Cloud-Native AI Infrastructure: Integration with Kubernetes ecosystem
   tools (Kueue, Gateway API, etc.)

Future Development

The project roadmap includes:

• Enhanced Runtime Support: Continued improvements for gVisor and Kata
  integration
• Better Warm Pool Management: More sophisticated allocation strategies
• Observability Integration: Native support for monitoring and tracing
• Multi-Cluster Support: Managing sandboxes across clusters

Resources

• GitHub Repository:
  kubernetes-sigs/agent-sandbox
• Documentation: agent-sandbox.sigs.k8s.io
• Slack: #sig-apps
• Mailing List:
  [email protected]

Conclusion

Agent Sandbox represents an important step forward in providing secure,
efficient execution environments for AI agents on Kubernetes. With support for
multiple isolation runtimes (gVisor and Kata Containers), standardized APIs,
and a focus on developer experience, it addresses the growing need for
sandboxed AI workloads in enterprise environments.

The project is actively developing under SIG Apps, and contributions from the
community are welcome. Whether you’re building AI agents, development
environments, or any workload requiring isolated execution, Agent Sandbox
provides a Kubernetes-native solution.

Co-Evolving: When Kubernetes Features Empower the Ecosystem

In the rapidly evolving AI infrastructure landscape, a beautiful synergy is
emerging: the Kubernetes community develops foundational capabilities, and
downstream projects like JobSet,
Ray, and
LeaderWorkerSet (LWS) adopt these
features to dramatically improve their efficiency. We call this Co-Evolving
(协同演进) — the entire ecosystem advancing together.

Kubernetes has been introducing more AI-related capabilities recently, but
realizing their full potential in AI workloads requires adaptation by other
projects. Today, we’ll explore a prime example: JobSet leveraging Kubernetes In-Place Container Restart to achieve 92% faster restart times.

The Problem: Slow JobSet Restart

When a distributed training job running on
JobSet needs to restart (due to
transient failures, configuration updates, or checkpoint recovery), the
traditional approach involves:

1. Delete all pods in the JobSet
2. Wait for pod termination to complete
3. Reschedule all pods through the Kubernetes scheduler
4. Wait for pod startup (including image pulls, init containers, etc.)

In a large-scale cluster with 5000 nodes, this process takes approximately
2 minutes and 10 seconds. For AI/ML workloads where fast recovery is
critical, this overhead is significant.

The Solution: In-Place Container Restart

Kubernetes has introduced capabilities that allow containers to restart without
pod recreation:

KEP-5307: Container Restart Policy (Kubernetes 1.34)

KEP-5307
introduces fine-grained control over individual container restart behavior
within pods. This allows:

• Specifying restart policies per container (not just per pod)
• Triggering container restarts without affecting the entire pod
• Maintaining pod identity, IP, and volumes during container restarts

KEP-5532: Restart All Containers on Container Exits (Kubernetes 1.35)

KEP-5532
extends this capability to enable coordinated restarts:

• Restart all containers in a pod when specific containers exit
• Restart init containers and sidecars as part of the pod lifecycle
• Enable pod-level restart coordination without pod recreation

Real-World Results: JobSet In-Place Restart

The JobSet team has developed an
in-place restart prototype
that demonstrates remarkable performance improvements:

For detailed design information, see the
JobSet in-place restart design document.

Why This Matters for AI Workloads

1. Distributed Training Recovery

Large-scale distributed training jobs (PyTorch DDP, TensorFlow MultiWorkerMirroredStrategy)
are particularly sensitive to restart latency:

• Checkpoint recovery: After a failure, all workers need to restart from
  the latest checkpoint. In-place restart gets workers back online 12x faster.
• Gradient synchronization: All workers must be running for training to
  proceed. Faster restarts mean less wasted GPU time.
• Cost savings: On expensive GPU clusters ($2-10/GPU-hour), 2 minutes saved
  per restart adds up significantly.

2. Job Dependencies

Many AI pipelines have complex job dependencies. When a job restarts:

• Downstream jobs wait for upstream completion
• Gang scheduling constraints require all workers to be present
• Network connectivity must be maintained for collective operations

In-place restart preserves pod identity and network connectivity, minimizing
disruption to the overall pipeline.

3. Resource Efficiency

Traditional restart involves:

• Scheduler load: Finding nodes for potentially thousands of pods
• API server load: Creating/deleting pod objects
• Node preparation: Image pulls, volume mounts, init containers

In-place restart eliminates all of this overhead, keeping resources available
for actual workloads.

How It Works

Before: Traditional Restart Flow

Job Restart Triggered
    ↓
Delete All Pods → Wait for Termination (30s+)
    ↓
Create New Pods → Wait for Scheduling (30s+)
    ↓
Pull Images (if needed) → Start Containers (60s+)
    ↓
Total: ~2m10s

After: In-Place Restart Flow

Job Restart Triggered
    ↓
Signal Container Exit → Container Restarts In-Place (10s)
    ↓
Total: ~10s

The key differences:

1. No pod deletion: Pod objects remain, preserving identity
2. No rescheduling: Pods stay on their current nodes
3. No image pulls: Images are already cached on nodes
4. Immediate restart: Container process simply restarts

Implementation Considerations

When to Use In-Place Restart

• Transient failures: Container crashes, OOM kills, network timeouts
• Configuration updates: Restart to pick up new environment variables
• Checkpoint recovery: Resume training from saved state
• Rolling updates: Graceful restart of workers in sequence

When Traditional Restart is Needed

• Node failures: Pod must move to a healthy node
• Resource changes: Pod needs more/less resources (consider VPA)
• Image updates: New container image required
• Topology changes: Pod needs different placement

Integration with JobSet

JobSet can leverage in-place restart through:

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: distributed-training
spec:
  replicatedJobs:
  - name: workers
    replicas: 8
    template:
      spec:
        template:
          spec:
            restartPolicy: Always  # Enables in-place restart
            containers:
            - name: trainer
              image: pytorch/pytorch:latest

The Broader Co-Evolving Pattern

This JobSet improvement exemplifies the Co-Evolving pattern in cloud-native AI:

As Kubernetes continues to add AI-friendly features, we expect more projects
to adopt them, creating a virtuous cycle of improvement.

Getting Started

Prerequisites

• Kubernetes 1.34+ (for KEP-5307)
• Kubernetes 1.35+ (for KEP-5532 pod-level restart)
• JobSet with in-place restart support (check latest releases)

Enable Feature Gates

# On kubelet for KEP-5307 (Container Restart Policy, 1.34+)
--feature-gates=ContainerRestartPolicy=true

# On kubelet for KEP-5532 (Restart All Containers, 1.35+)
--feature-gates=RestartAllContainersOnContainerExits=true

Test In-Place Restart

1. Deploy a JobSet with restartPolicy: Always
2. Trigger a container restart (e.g., kubectl exec ... -- kill -TERM 1)
3. Observe the restart time compared to pod recreation

Future Roadmap

The in-place restart capability continues to evolve:

• KEP-5307 graduation: Moving toward Beta/GA
• KEP-5532 graduation: Enhanced pod-level restart control
• JobSet integration: Native support for in-place restart policies
• Monitoring: Better observability for restart events
• Kueue integration: Workload-aware restart handling

Conclusion

The JobSet in-place restart optimization demonstrates the power of Co-Evolving
in the Kubernetes ecosystem. By adopting upstream Kubernetes capabilities,
projects can achieve dramatic performance improvements:

• 92% faster restart (2m10s → 10s)
• No scheduling overhead
• Preserved pod identity and network
• Reduced API server load

This is just one example of how the Kubernetes community and downstream
projects work together to improve AI workload efficiency. As more AI-related
features land in Kubernetes, we can expect even more optimizations from
projects like JobSet, Ray, LWS, and others.

The future of AI infrastructure is Co-Evolving — and it’s happening now.

References

KEPs and Documentation

• KEP-5307: Container Restart Policy
• KEP-5532: Restart All Containers on Container Exits
• KEP-1287: In-Place Pod Vertical Scaling
• JobSet In-Place Restart Design Document
• JobSet In-Place Restart Prototype

Related Projects

• JobSet – Kubernetes SIG Apps
• LeaderWorkerSet – Kubernetes SIG Apps
• Kueue – Kubernetes SIG Scheduling
• Volcano – CNCF Incubating

Related Blog Posts

• Gang Scheduling in Kubernetes
• Scheduling Optimization

Why Topology? Why Now?

At KubeCon NA 2025, one theme dominated conversations in the AI/ML space:
topology. Everyone is talking about topology-aware scheduling because it’s
critical for optimizing AI workload performance.

Source: Lightning Talk: Mind the Topology – Roman Baron, NVIDIA

Modern AI workloads, especially distributed training and high-performance
inference, are extremely sensitive to hardware topology. When GPUs, NICs, CPUs,
and memory are not properly aligned within the same NUMA node, PCIe root, or
network fabric, performance can degrade by 30-50% or more.

Background: Current Topology Scheduling Support

Device Plugin: The Traditional Approach

Kubernetes Device Plugins have been the standard mechanism for managing
hardware resources like GPUs. The Device Plugin API provides:

Source: KubeCon NA 2025: Device Management

Key Components:

• GetDevicePluginOptions: Plugin configuration
• ListAndWatch: Report available devices to kubelet
• GetPreferredAllocation: Suggest optimal device allocation (topology hint)
• Allocate: Perform device allocation for containers
• PreStartContainer: Pre-container-start hooks

Device Plugin supports:

• Basic GPU counting (e.g., nvidia.com/gpu: 8)
• MIG (Multi-Instance GPU) partitioning
• Time-slicing for GPU oversubscription

Limitations of Device Plugin

However, Device Plugins have significant limitations for topology-aware
scheduling:

Source: KubeCon NA 2025: Device Management

1. Static isolation config: MIG configurations must be pre-defined
2. Static slicing config: Time-slicing ratios are fixed at deployment
3. Only even sharing expected: Limited sharing granularity
4. Requires secondary scheduler: Complex topologies need additional
   schedulers like Volcano or Kueue

Kueue: Topology-Aware Scheduling

Kueue provides topology-aware
scheduling through node labels. It uses hierarchical topology levels like:

# Node labels for rack/block topology
cloud.google.com/gce-topology-block: "block-1"
cloud.google.com/gce-topology-subblock: "subblock-1"
cloud.google.com/gce-topology-host: "host-1"
kubernetes.io/hostname: "node-1"

Kueue supports:

• TopologyAwareScheduling: Place workload pods on nodes with matching
  topology
• Cohort-based resource sharing: Share resources within topology groups
• Gang scheduling with topology: Ensure all gang members are
  topology-aligned

Kueue Topology Configuration Example:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: gpu-topology
spec:
  nodeLabels:
    cloud.google.com/gce-topology-block: "block-1"
  nodeTaints:
  - effect: NoSchedule
    key: nvidia.com/gpu
    value: "present"

Volcano: Gang Scheduling with Topology

Volcano provides advanced scheduling
features including:

• Gang scheduling: All-or-nothing scheduling for distributed workloads
• Topology plugin: Consider GPU topology in scheduling decisions
• Network-aware scheduling: RDMA/InfiniBand fabric awareness

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: distributed-training
spec:
  minMember: 8
  minResources:
    nvidia.com/gpu: "8"
  queue: training-queue
  # Topology affinity for NVLink connectivity
  topologyPolicy: "best-effort"

DRA: The Next Generation of Topology Management

Dynamic Resource Allocation (DRA)
represents a fundamental shift in how Kubernetes handles device topology. DRA
provides structured parameters that enable rich topology expression and
constraint specification.

How DRA Handles Topology-Aware Scheduling

DRA uses attributes and constraints with CEL (Common Expression
Language) to express topology requirements. The key mechanisms include:

1. Device Attributes: Each device publishes topology information

• pcieRoot: PCIe hierarchy identifier
• numaNode: NUMA node association
• nvlinkDomain: NVLink fabric identifier
• rdmaDevice: Associated RDMA NIC

1. Constraints: CEL expressions that enforce topology rules

• Same PCIe root for GPU and NIC
• Same NUMA node for CPU and memory
• NVLink connectivity between GPUs

1. SharedID: Devices on the same topology domain get a shared identifier

GPU + NIC Topology Coordination

The most powerful use case for DRA topology is coordinating GPU and NIC
allocation on the same PCIe root. This is critical for RDMA-based distributed
training where GPU-Direct is used.

ResourceClaimTemplate with PCIe Topology Constraint Example:

apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
  name: gpu-nic-topology
spec:
  spec:
    devices:
      requests:
      - name: gpu
        deviceClassName: nvidia-gpu
        count: 1
      - name: rdma-nic
        deviceClassName: rdma-nic
        count: 1
      constraints:
      # GPU and NIC must be on the same PCIe root
      - requests: ["gpu", "rdma-nic"]
        matchAttribute: pcieRoot

How this works:

1. The DRA scheduler evaluates available GPUs and NICs
2. For each candidate GPU, it finds NICs on the same PCIe root
3. Only allocations satisfying the constraint are considered
4. The matchAttribute: pcieRoot ensures both devices share the same
   PCIe topology

DRANET: Network Device DRA

DRANET is Google’s DRA implementation for
network devices. It integrates with Kueue’s topology-aware scheduling using
node labels:

# DRANET uses these labels for topology awareness
cloud.google.com/gce-topology-block
cloud.google.com/gce-topology-subblock
cloud.google.com/gce-topology-host
kubernetes.io/hostname

DRANET + NVIDIA GPU DRA can coordinate:

• RDMA NICs allocated with GPUs on same PCIe fabric
• Multi-NIC configurations for distributed training
• Network isolation using SR-IOV VFs

CPU Micro-Topology Support

The dra-driver-cpu
project is adding CPU micro-topology support including:

• NUMA-aware CPU allocation
• CPU pinning with topology alignment
• Coordination with GPU NUMA placement

DRAConsumableCapacity: New in Kubernetes 1.34

A major advancement in DRA is the DRAConsumableCapacity feature:

Source: KubeCon NA 2025: Device Management

Key Capabilities:

• Alpha feature introduced in Kubernetes 1.34
• Recommended to start using from Kubernetes 1.35 (still in Alpha)

Core abilities:

• Allow multiple allocations over multiple resource requests
• Consumable capacity: Guaranteed resource sharing

Potential use cases:

• Virtual GPU Memory Partitioning
• Virtual NIC (vNIC) Sharing
• Bandwidth-limited Network Allocation
• I/O Bandwidth Smart Storage Device Sharing
• Native Resource Request (CPU)

This enables much more flexible resource sharing while maintaining topology
awareness.

Challenges: Device Plugin to DRA Migration

Many organizations have invested heavily in Device Plugin-based solutions.
Migrating to DRA presents several challenges:

1. Existing Device Plugin Investments

Organizations may have:

• Custom Device Plugins with topology logic
• Integration with monitoring and observability tools
• Operator workflows depending on Device Plugin APIs

2. Coexistence Problems

Running Device Plugin and DRA together can cause:

• Resource conflicts: Same device managed by both systems
• Topology inconsistency: Different topology views between systems
• Scheduling confusion: Scheduler doesn’t have unified view

3. Feature Gaps

Some Device Plugin features don’t have DRA equivalents yet:

• Device health monitoring: Device Plugin has built-in health checks
• Hot-plug support: Device Plugin supports dynamic device addition
• Metrics integration: Prometheus metrics from Device Plugins

Solutions and Workarounds

DRA Extension Capabilities:

• DRA drivers can implement compatibility layers
• NVIDIA’s DRA driver supports Device Plugin migration path
• NRI integration can bridge runtime-level gaps

Recommended Migration Path:

1. Deploy DRA driver alongside existing Device Plugin
2. Use node taints to partition workloads
3. Gradually migrate workloads to DRA-based resource claims
4. Phase out Device Plugin once all workloads migrated

Related KubeCon Talks

Several excellent talks from KubeCon NA 2025 cover these topics:

Lightning Talk: Mind the Topology

Mind the Topology: Smarter Scheduling for AI Workloads on Kubernetes
by Roman Baron, NVIDIA

Key topics:

• Why topology matters for AI workloads
• NVIDIA KAI Scheduler for topology-aware scheduling
• NVIDIA KAI-Scheduler

Device Management Deep Dive

Deep dive into DRA and Device Plugin

Key topics:

• Evolution from Device Plugin to DRA
• DRAConsumableCapacity feature
• Multi-device topology coordination

Best Practices for Topology-Aware Scheduling

1. Understand your topology requirements

• Profile workloads to identify topology sensitivity
• Map hardware topology (PCIe, NUMA, NVLink, RDMA)

1. Choose the right scheduling approach

• Simple GPU workloads: Device Plugin + Topology Manager
• Complex multi-device: DRA with constraints
• Distributed training: Kueue or Volcano + DRA

1. Label nodes with topology information

• Use consistent labeling scheme
• Include rack, block, and host-level topology

1. Test topology impact

• Benchmark with and without topology alignment
• Measure latency and throughput differences

1. Plan for migration

• Start with new workloads on DRA
• Create compatibility tests
• Document topology requirements

Conclusion

Topology-aware scheduling has evolved from a nice-to-have feature to a critical
requirement for AI workloads. The transition from Device Plugin to DRA
represents a fundamental shift in how Kubernetes manages hardware topology:

• Device Plugin: Simple, established, but limited topology support
• DRA: Rich topology expression, multi-device coordination, future of
  Kubernetes device management

As AI workloads continue to grow in complexity, the need for sophisticated
topology-aware scheduling will only increase. Whether you’re using Kueue,
Volcano, or native Kubernetes scheduling, understanding topology and planning
for DRA adoption is essential for optimizing your AI infrastructure.

Resources

Projects

• DRA – Dynamic Resource Allocation
• NVIDIA DRA GPU Driver
• NVIDIA KAI Scheduler
• Kueue
• Volcano
• DRANET
• dra-driver-cpu

Documentation

• DRA Kubernetes Documentation
• GCE Topology Policies
• Kubernetes Topology Manager

Videos

• Mind the Topology – Roman Baron, NVIDIA
• Device Management Deep Dive

中文 https://mp.weixin.qq.com/s/EO0yfdVQMNgKI7nqkJ18Yw Kubernetes 支持原生 Gang Scheduling ： 适应 AI/ML 工作负载

Introduction

Scheduling large workloads in Kubernetes has always been challenging. When you need to run distributed training jobs, batch processing tasks, or other multi-pod applications, the traditional pod-by-pod scheduling approach can lead to resource wastage, deadlocks, and inefficiencies. Today, we’re excited to share insights about the Workload Aware Scheduling initiative that’s transforming how Kubernetes handles multi-pod workloads.

The Problem with Traditional Pod Scheduling

In traditional Kubernetes scheduling, each pod is scheduled independently. For distributed workloads like:

• Distributed ML training (e.g., PyTorch, TensorFlow multi-worker jobs)
• Batch processing (e.g., Apache Spark, Ray clusters)
• High-performance computing (e.g., MPI applications)

This independent scheduling creates several problems:

1. Partial scheduling deadlocks: Some pods get scheduled while others wait indefinitely for resources
2. Resource wastage: Scheduled pods consume resources but can’t start work until all peers are ready
3. Poor cluster utilization: Resources are tied up by incomplete workloads
4. Unpredictable job completion times: Jobs may wait hours or days in partially-scheduled states

Kubernetes v1.35: Workload Aware Scheduling

The Kubernetes community has introduced Workload Aware Scheduling in v1.35, featuring three major components:

1. Workload API (Alpha)

The new Workload API resource in scheduling.k8s.io/v1alpha1 provides a structured way to define scheduling requirements for multi-pod applications.

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: training-job-workload
  namespace: ml-workloads
spec:
  podGroups:
  - name: workers
    policy:
      gang:
        # All-or-nothing: schedule only if 4 pods can run together
        minCount: 4

Link your pods to the workload:

apiVersion: v1
kind: Pod
metadata:
  name: worker-0
  namespace: ml-workloads
spec:
  workloadRef:
    name: training-job-workload
    podGroup: workers
  containers:
  - name: trainer
    image: my-ml-framework:latest
    resources:
      requests:
        nvidia.com/gpu: 1

2. Gang Scheduling (Alpha)

Gang scheduling implements the all-or-nothing placement strategy:

How it works:

1. Waiting Phase: When pods arrive, the scheduler blocks them until minCount pods are pending
2. Evaluation Phase: The scheduler attempts to find suitable nodes for all pods in the gang
3. Decision Phase:
   • ✅ Success: If all pods can be placed, they’re bound to nodes together
   • ❌ Failure: If any pod can’t be placed within timeout (5 minutes), ALL pods are rejected and requeued

This prevents resource waste and ensures your distributed workload either runs completely or waits for sufficient resources.

Key benefits:

• Eliminates partial scheduling deadlocks
• Improves cluster utilization by freeing resources for runnable workloads
• Provides predictable behavior for distributed applications
• Works seamlessly with pod preemption and autoscaling

3. Opportunistic Batching (Beta)

Opportunistic Batching is a performance optimization that speeds up scheduling of identical pods without requiring any configuration changes.

How it works:

When the scheduler processes pods with identical scheduling requirements (same resources, images, affinities, etc.), it can reuse feasibility calculations and scoring results for subsequent pods in the queue.

Performance impact:

• Dramatically reduces scheduling latency for large homogeneous workloads
• Can improve scheduling throughput by 5-10x for batch workloads
• Works transparently – no user configuration needed
• Enabled by default in Kubernetes v1.35 (Beta)

Current restrictions:

• Disabled for pods using topology spread constraints
• Disabled for pods using Dynamic Resource Allocation (DRA)
• All scheduling-relevant pod fields must be identical

Real-World Use Cases

Distributed ML Training

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: pytorch-training
spec:
  podGroups:
  - name: workers
    policy:
      gang:
        minCount: 8  # Need 8 GPUs for distributed training

Your PyTorch distributed training job only starts when all 8 workers can be scheduled, preventing wasted GPU resources.

Apache Spark on Kubernetes

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: spark-job
spec:
  podGroups:
  - name: executors
    policy:
      gang:
        minCount: 10  # 1 driver + 9 executors minimum

Spark jobs with gang scheduling avoid the common problem where the driver starts but executors can’t be scheduled.

Ray Clusters

Ray applications benefit from gang scheduling by ensuring the head node and worker nodes start together, enabling immediate distributed computation.

The Roadmap: What’s Coming in 1.36 and Beyond

The Workload Aware Scheduling effort has an ambitious roadmap for Kubernetes 1.36:

Planned for v1.36

• Expanding Workload API: Enhanced capabilities and refinements based on alpha feedback
• Auto-workload for Job, StatefulSet, JobSet: Automatic workload creation for common Kubernetes resources
• Topology Aware Scheduling: Consider network and hardware topology when placing gang members
• Single-cycle workload scheduling: Schedule entire gangs in a single scheduling cycle for better performance
• Tree-based workload scheduling algorithm: More efficient gang placement decisions
• Improved binding process: Better handling of kubelet races using nominations
• Delayed preemption: Introduce nominating victims before actual eviction
• Workload-level preemption: Preempt entire gangs rather than individual pods

Long-term Vision

The ultimate goal is to make Kubernetes natively understand and optimize for workload-level operations, including:

• Deep integration with cluster autoscaling
• Workload-aware resource quotas and limits
• Better support for mixed workload types (batch + serving)
• Enhanced observability for multi-pod applications

Upcoming Official Blog Post

The Kubernetes community is preparing an official blog post about Workload Aware Scheduling that will be published soon on the Kubernetes blog. Watch for kubernetes/website#53012 to be merged for the official announcement.

Getting Started

Prerequisites

• Kubernetes v1.35 or later
• Feature gates configured on kube-apiserver and kube-scheduler

Enable Workload API and Gang Scheduling

# On kube-apiserver
--feature-gates=GenericWorkload=true
--runtime-config=scheduling.k8s.io/v1alpha1=true

# On kube-scheduler
--feature-gates=GenericWorkload=true,GangScheduling=true

Enable Opportunistic Batching

Opportunistic Batching is enabled by default in v1.35 as a Beta feature. To disable it:

# On kube-scheduler
--feature-gates=OpportunisticBatching=false

Testing Gang Scheduling

1. Create a Workload resource
2. Create pods with workloadRef pointing to the Workload
3. Observe scheduling behavior in kube-scheduler logs
4. Monitor metrics for gang scheduling success/failure rates

Best Practices

1. Set appropriate minCount: Consider your application’s minimum viable size
2. Use resource requests accurately: Gang scheduling depends on accurate resource requirements
3. Monitor scheduling metrics: Track gang scheduling success rates and timeout events
4. Test with cluster autoscaling: Ensure your autoscaler can provision nodes for gangs
5. Plan for failure scenarios: Understand timeout behavior and retry logic

Comparison with Existing Solutions

Before native gang scheduling, users relied on:

• Volcano: CNCF incubating project with gang scheduling
• Kueue: Kubernetes SIG project for queue and quota management
• YuniKorn: Apache project with gang scheduling support
• Custom schedulers: In-house solutions for specific use cases

Why use native gang scheduling?

• Maintained by Kubernetes SIG Scheduling
• Integrated with core scheduler features (preemption, autoscaling)
• No additional components to deploy and maintain
• Part of the Kubernetes conformance suite (eventually)

When to use external schedulers?

• Need production-ready gang scheduling today (use Volcano or Kueue)
• Require features beyond current Kubernetes roadmap
• Have existing investments in specific schedulers

Resources and References

KEPs and Documentation

• KEP-4671: Gang Scheduling
• KEP-5598: Opportunistic Batching
• Workload Aware Scheduling Tracking Issue
• Kubernetes Website PR #53012

Related Projects

Several projects currently support gang scheduling:

• Volcano Scheduler – CNCF Incubating
  • Full gang scheduling support
  • Recently added LeaderWorkerSet (LWS) gang scheduling in v1.13 release
• Koordinator – Alibaba Open Source
  • Basic gang scheduling capabilities
  • Workload orchestration and resource scheduling enhancements
• Kueue – Kubernetes SIG Project
  • CoScheduling support (a lighter version of gang scheduling)
  • Focus on job queueing and quota management
• YuniKorn – Apache Project
  • Gang scheduling and resource scheduling capabilities

Community

• SIG Scheduling: https://github.com/kubernetes/community/tree/master/sig-scheduling
• Slack: #sig-scheduling on Kubernetes Slack

Conclusion

Gang Scheduling and Workload Aware Scheduling represent a major step forward for Kubernetes in supporting AI/ML, HPC, and batch processing workloads. The v1.35 alpha release provides a foundation for native multi-pod scheduling, with an exciting roadmap for v1.36 and beyond.

We encourage the community to:

• Test these features in development environments
• Provide feedback through GitHub issues
• Share use cases and requirements
• Contribute to the ongoing development

The future of Kubernetes scheduling is workload-aware, and the journey has just begun!

As v1.35 will announce the cgroup v1 deprecation, kubelet will fail on cgroup v1 with default configuration. FailCgroupV1 will be set to true by default. See more in coming blog https://github.com/kubernetes/website/pull/52814. Blow is what I wrote after cgroup v1 was announced to enter maintenance mode. As I linked a lot and can not finish is pretty complete, I stopped update https://github.com/kubernetes/website/pull/47342. Just publish it here for users who want to know more about why we should shift from cgroup v1 to v2 and the difference.

cgroups (control groups) are a Linux kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications run smoothly without interfering with each other. With the release of Kubernetes v1.31, cgroups v1 has been moved into [maintenance mode]/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/). For cgroups v2, it graduated in v1.25 2 years ago.

Top FAQs are why we should migrate, what’s the benifits and lost, and what needs to be noticed when using cgroups v2.

cgroups v1 problem, and solutions in cgroups v2

cgroups v1 and cgroups official doc can be found in

• v1 doc
• v2 doc

Let’s enumerate some known issues.

active_file memory is not considered as available memory

There is a known issue of page cache: #43916.

• In cgroups v1, we have no native solutions. Workarounds are setting larger memory limit for Pods or using some external projects to drop cache or throttling memory allocating when memory is beyond a threshold.
• In cgroups v2, we can use memory.high to throttle.

Support for Memory QoS was initially added in Kubernetes v1.22, and later some limitations around the formula for calculating memory.high were identified. These limitations are addressed in Kubernetes v1.27.

However, until v1.31, the feature gate is still alpha due to another known issue that application pod may be hanging forever due to heavy memory reclaiming.

Container aware OOM killer and better OOM handling strategies

In cgroups v2, one process of a multi-processes Pod could be killed by the OOM killer. In this case, Pod has to use runit or supervisord to manage multi processes lifecycle.

cgroups v2 uses cgroup.kill file. Writing “1” to the file causes the cgroups and all descendant cgroups to be killed. This means that all processes located in the affected cgroup tree will be killed via SIGKILL. Pod may run multiple processes, and all processes can be killed simultaneously.

As mentioned above, cgroups v2 memory.high can throttle the new memory allocation and cgroups can be aware of the OOM earsiler. Besides, PSI can also help to know the memory load. oomd is a good example using PSI to implement a userspace out-of-memory killer.

Rootless support

In cgroups v1, delegating cgroups v1 controllers to less privileged containers may be dangerous.

Unlike cgroups v1, cgroups v2 officially supports delegation. Most Rootless Containers implementations rely on systemd for delegating v2 controllers to non-root users.

User Namespace minimal kernel version is 6.5, according to KEP-127.

What’s more?

1. eBPF stories:
   • In cgroups v1, the device access control are defined in the static configuration/.
   • cgroups v2 device controller has no interface files and is implemented on top of cgroup BPF.
   • Cilium will automatically mount cgroups v2 filesystem required to attach BPF cgroup programs by default at the path /run/cilium/cgroupv2 .
2. PSI is planned in a future release KEP-4205, but pending due to runc 1.2.0 release delay.
3. monitoring tools support, like Cadvisor. Currently, cgroups v2 features are not fully-supported yet.

Adopting cgroup version 2

Requirements

Here’s what you need to use cgroup v2 with Kubernetes. First up, you need to be using a version of Kubernetes with support for v2 cgroup management; that’s been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support.

• OS distribution enables cgroups v2
• Linux Kernel version is 5.8 or later
• Container runtime supports cgroups v2. For example:
  • containerd v1.4 or later (at the time of writing, containerd releases v1.6 and later are within that project’s support period)
  • CRI-O v1.20 or later
• The kubelet and the container runtime are configured to use the systemd cgroup driver

kernel updates around cgroups v2

cgroups v2 first appeared in Linux Kernel 4.5 in 2016.

• In Linux 4.5, cgroups v2 io, memory & pid cgroups management were supported.
• Linux 4.15 added support for cgroups v2 cpu management
• Pressure Stall Information (PSI) support began with Linux 4.20.
• The Kubernetes project does not recommend using cgroups v2 with a Linux kernel older than 5.2 due to lack of cgroup-level task freezer support.
• In Kubernetes, 5.8 is the minimal kernel version for cgroups v2 as root cpu.stat file on cgroupv2 was only added on kernel 5.8.
• memory.peak is added in 5.19.

Use systemd as cgroup driver

Configure the kubelet’s cgroup driver to match the container runtime cgroup driver.

The Container runtimes page explains that the systemd driver is recommended for kubeadm based setups instead of the kubelet’s default cgroupfs driver, because kubeadm manages the kubelet as a systemd service.

A minimal example of configuring the field explicitly:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

In v1.31, KEP-4033 is beta to extend CRI API for the kubelet to discover the cgroup driver from the container runtime. This will help installer and kubelet to autodetect

Tools and commands for troubleshooting

Tools and commands that you should know about cgroups:

• stat -fc %T /sys/fs/cgroup/: Check if cgroups v2 is enabled which will return cgroup2fs
• systemctl list-units kube* --type=slice or --type=scope: List kube related units that systemd currently has in memory.
• bpftool cgroup list /sys/fs/cgroup/*: List all programs attached to the cgroup CGROUP.
• systemd-cgls /sys/fs/cgroup/*: Recursively show control group contents.
• systemd-cgtop: Show top control groups by their resource usage.
• tree -L 2 -d /sys/fs/cgroup/kubepods.slice: Show Pods’ related cgroups directories.

How to check if a Pod CPU or memory limit is successfully applied to the cgroup file?

• Kubernetes Pod Spec: check limits spec.containers[*].resources.limits.{cpu,memory} and requests spec.containers[*].resources.requests.{cpu,memory}
• CRI: cpu_period, cpu_quota, cpu_shares for CPU and memory_limit_in_bytes for memory limit
• OCI Spec: memorry.limit, cpu.shares, cpu.quota, cpu.period
• Systemd Scope Unit: CPUWeight, CPUQuotaPerSecUSec, CPUQuotaPeriodUSec, MemoryMax
• Cgroupfs value: /sys/fs/cgroup/../cpu.weight, /sys/fs/cgroup/../cpu.max, /sys/fs/cgroup/../memory.max

Further reading

• Kubernetes 1.31: Moving cgroups v1 Support into Maintenance Mode
• cgroups v2 in Kubernetes
• Kubernetes 1.27: Quality-of-Service for Memory Resources (alpha)
• Kubernetes 1.25: cgroups v2 graduates to GA
• KubeCon NA 2022 Cgroups V2: Before You Jump In by Tony Gosselin & Mike Tougeron, Adobe Systems
• KubeCon NA 2022 Cgroupv2 Is Coming Soon To a Cluster Near You – David Porter, Google & Mrunal Patel, RedHat
• KubeCon EU 2020 Kubernetes On cgroups v2 by Giuseppe Scrivano, Red Hat.
• Note, this blog will only include the basic requirments and configurations in Kubernetes components. It will not include how to enable cgroup fs in OS distributions. For migration, you can refer to migrating cgroups v2