KEP-4671: Implement Gang scheduling in kube-scheduler by macsko · Pull Request #134722 · kubernetes/kubernetes

macsko · 2025-10-20T10:18:14Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds gang scheduling plugin and basic workload awareness (using dedicated manager) to kube-scheduler based on KEP-4671. This is a second PR for this KEP. First one, with the API, is #134564.

Which issue(s) this PR is related to:

Part of #134471
Fixes #134475
Fixes #134476
Fixes #134477

KEP: kubernetes/enhancements#4671

Special notes for your reviewer:

This PR is dedicated for kube-scheduler changes. For API review, please post the comments in #134564. Six first commits are part of #134564, so please filter changes of this PR appropriately.

This PR may be merged with the #134564 depending on review efficiency.

Does this PR introduce a user-facing change?

Introduced GangScheduling kube-scheduler plugin to enable "all-or-nothing" scheduling. Workload API in scheduling.k8s.io/v1alpha1 is used to express the desired policy.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4671-gang-scheduling

k8s-ci-robot · 2025-10-20T10:18:23Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

macsko · 2025-10-20T10:18:30Z

/hold

ingvagabund · 2025-11-03T11:56:24Z

+
+// deletePod completely deletes the pod from this group.
+// It returns true when the group is empty after removal.
+func (pgs *podGroupInfo) deletePod(podUID types.UID) bool {


The intuitive interpretation of the return value is whether a pod got deleted or not. If anything I'd suggest to return two values. One for the actual deletion (always true in this case, can change in the future?) and the other as a check of whether a group is empty. Or, how much more expensive would it be to create a new function for "isEmpty" check?

Right, added new method

ingvagabund · 2025-11-03T14:07:14Z

 	logger.V(3).Info("Add event for unscheduled pod", "pod", klog.KObj(pod))
 	sched.SchedulingQueue.Add(logger, pod)
+	if utilfeature.DefaultFeatureGate.Enabled(features.GangScheduling) {
+		sched.SchedulingQueue.MoveAllToActiveOrBackoffQueue(logger, framework.EventUnscheduledPodAdd, nil, pod, nil)


+1 for a mechanism that will avoid putting irrelevant pods into the queue

macsko · 2025-11-04T09:39:08Z

/test pull-kubernetes-integration
Non-related flake

sanposhiho

only one nit, others lgtm

sanposhiho · 2025-11-04T15:00:38Z

 		{Event: fwk.ClusterEvent{Resource: fwk.StorageClass, ActionType: fwk.All}},
 		{Event: fwk.ClusterEvent{Resource: fwk.ResourceClaim, ActionType: fwk.All}},
 		{Event: fwk.ClusterEvent{Resource: fwk.DeviceClass, ActionType: fwk.All}},
+		{Event: fwk.ClusterEvent{Resource: fwk.Workload, ActionType: fwk.All}},


register it only if the gate is enabled?

Good point, added

sanposhiho

/lgtm
/approve

k8s-ci-robot · 2025-11-04T19:07:27Z

LGTM label has been added.

Details

Git tree hash: 18e3fbeb2d8866222f5234175b70e75d16899f50

k8s-ci-robot · 2025-11-06T10:21:02Z

@macsko: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-apidiff-client-go	`50fa39b`	link	false	`/test pull-kubernetes-apidiff-client-go`
pull-kubernetes-e2e-kind-alpha-beta-features	`50fa39b`	link	false	`/test pull-kubernetes-e2e-kind-alpha-beta-features`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

wojtek-t · 2025-11-06T11:23:48Z

/lgtm
/approve

k8s-ci-robot · 2025-11-06T11:23:56Z

LGTM label has been added.

Details

Git tree hash: a6e5193895a298c8cce734359287f61dc06ae321

k8s-ci-robot · 2025-11-06T11:23:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: macsko, sanposhiho, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/scheduler/OWNERS~~ [macsko,sanposhiho,wojtek-t]
~~plugin/pkg/auth/authorizer/OWNERS~~ [wojtek-t]
~~staging/src/k8s.io/kube-scheduler/OWNERS~~ [macsko,sanposhiho,wojtek-t]
~~test/OWNERS~~ [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wojtek-t · 2025-11-06T11:25:45Z

/hold cancel

k8s-triage-robot · 2025-11-06T12:28:04Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

dom4ha · 2025-11-06T13:15:43Z

Congrats Maciek, great job on Gang and Workload API!

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Oct 20, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2025

k8s-ci-robot requested review from brianpursley and cici37 October 20, 2025 10:18

github-project-automation Bot added this to SIG Apps Oct 20, 2025

k8s-ci-robot added sig/etcd Categorizes an issue or PR as relevant to SIG Etcd. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 20, 2025

github-project-automation Bot moved this to Needs Triage in SIG Apps Oct 20, 2025

github-project-automation Bot added this to SIG CLI Oct 20, 2025

k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Oct 20, 2025

github-project-automation Bot moved this to Needs Triage in SIG CLI Oct 20, 2025

k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Oct 20, 2025

github-project-automation Bot added this to SIG Scheduling Oct 20, 2025

k8s-ci-robot requested a review from dom4ha November 3, 2025 13:48

ingvagabund reviewed Nov 3, 2025

View reviewed changes

macsko force-pushed the gang_scheduling_scheduler branch 2 times, most recently from 3802f5b to 9f9ff21 Compare November 4, 2025 08:46

sanposhiho reviewed Nov 4, 2025

View reviewed changes

macsko force-pushed the gang_scheduling_scheduler branch from 9f9ff21 to 525ca9c Compare November 4, 2025 18:46

sanposhiho approved these changes Nov 4, 2025

View reviewed changes

k8s-ci-robot assigned sanposhiho Nov 4, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 4, 2025

ania-borowiec mentioned this pull request Nov 5, 2025

KEP-4671: Add Workload API. Copy of PR #134564 #135140

Closed

wojtek-t mentioned this pull request Nov 5, 2025

[WIP] KEP-4671: Add Workload API. Copy of PR #134564 #135143

Closed

dom4ha mentioned this pull request Nov 5, 2025

KEP-4671: Implement Gang scheduling in kube-scheduler. Copy of PR #134722 #135138

Closed

macsko added 4 commits November 6, 2025 10:47

Implement Gang scheduling in kube-scheduler

8d67173

Add integration tests for gang queueing

c086bde

Add read Workload permissions to kube-scheduler

4a067cc

Add integration test for gang scheduling

049cb85

macsko mentioned this pull request Nov 6, 2025

KEP-4671: Remove Basic pod group policy from Workload API #135179

Closed

pacoxu mentioned this pull request Nov 11, 2025

Gang Scheduling Support in Kubernetes kubernetes/enhancements#4671

Open

24 tasks

dom4ha mentioned this pull request Nov 14, 2025

Make native scheduling workload-aware #132192

Open

carlory mentioned this pull request Nov 24, 2025

add Workload permissions to view, edit and admin clusterroles #135418

Merged

renovate Bot mentioned this pull request Jan 20, 2026

Update Python Dependency Updates for Next (next) (major) nautobot/nautobot#8000

Open

1 task

Conversation

macsko commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR is related to:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Oct 20, 2025

Uh oh!

macsko commented Oct 20, 2025

Uh oh!

ingvagabund Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

macsko Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

ingvagabund Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

macsko commented Nov 4, 2025

Uh oh!

sanposhiho left a comment

Choose a reason for hiding this comment

Uh oh!

sanposhiho Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

macsko Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

sanposhiho left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Nov 4, 2025

Uh oh!

k8s-ci-robot commented Nov 6, 2025

Uh oh!

wojtek-t commented Nov 6, 2025

Uh oh!

k8s-ci-robot commented Nov 6, 2025

Uh oh!

k8s-ci-robot commented Nov 6, 2025

Uh oh!

wojtek-t commented Nov 6, 2025

Uh oh!

k8s-triage-robot commented Nov 6, 2025

Uh oh!

dom4ha commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

macsko commented Oct 20, 2025 •

edited

Loading