add KEP for cgroups v2 support by giuseppe · Pull Request #1370 · kubernetes/enhancements

giuseppe · 2019-11-18T14:53:32Z

Signed-off-by: Giuseppe Scrivano [email protected]

k8s-ci-robot · 2019-11-18T14:53:39Z

Welcome @giuseppe!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-11-18T14:53:40Z

Hi @giuseppe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

AkihiroSuda

Let's unshare cgroupns by default

AkihiroSuda · 2019-11-18T15:51:47Z

Do you think we should open a separate KEP for rootless?

giuseppe · 2019-11-18T16:11:08Z

Do you think we should open a separate KEP for rootless?

yes let's keep it separate from cgroups v2, which is already touching enough components :-)

derekwaynecarr

we need to add more detail on kubelet changes where it interacts with cgroups directly

derekwaynecarr · 2019-11-18T17:36:14Z

+
+## User Stories
+
+*   The Kubelet can run on a cgroups v2 host


Clarify that kubelet will support v1 or v2

derekwaynecarr · 2019-11-18T17:42:06Z

+
+Kubelet PR: [https://github.com/kubernetes/kubernetes/pull/85218](https://github.com/kubernetes/kubernetes/pull/85218)
+
+### Option 2: Use cgroups v2 throughout the stack


For option 1, kubelet still has to write cgroups v2 for its interactions with cgroup. Please add some detail around kubelet / cm package and eviction manager.

derekwaynecarr · 2019-11-18T17:44:19Z

+
+## Graduation Criteria
+
+- Alpha: Basic support for running Kubernetes on a cgroups v2 host.


Worth discussing if we should add test coverage on fedora as a good candidate upstream.

derekwaynecarr · 2019-11-18T17:45:39Z

/assign

squeed · 2019-11-22T18:33:16Z

+_net_prio_ are not available with the new version.  The alternative to
+these controllers is to use eBPF.
+
+The lack of the network controllers probably affects some CNI


I don't currently know of any CNI plugins that use cgroups for network management. (My list is not exhaustive, of course...). Bandwidth limiting, if implemented, is done via a tc filter / qdisc for all of the plugins I'm aware of.

yujuhong · 2019-12-03T18:53:51Z

+
+
+With this approach cAdvisor would have to read back values from
+cgroups v2 files (already done).


So, with this option, CRI implementations will need to map the cgroup v1 fields to cgroup v2 values, right?

Could you list what each layer needs to do for each option to make it clear?

user (pod spec)

kubelet

CRI implementation

OCI runtime

the changes in the CRI implementation and OCI runtime might be implemented in a different way, deciding where to draw the line.
For CRI-O+crun, most of the logic is in the OCI runtime itself, but that is not the only way to achieve it.
For the Kubelet, the changes in this patch should be enough (at least until we have not hugetlb available): kubernetes/kubernetes#85218

In the second phase though, when cgroup v2 is fully supported through the stack there is need to change both pod specs+CRI.

can you clarify why you feel the pod spec needs to change? i see no major reason to change the pod spec or resource representation.

I was assuming we'd like to expose all/most of the cgroup v2 features.

e.g. the memory controller on cgroup v1 allows to configure:

memory.soft_limit_in_bytes

memory.limit_in_bytes

while on cgroup v2 we have:

memory.high

memory.low

memory.max

memory.min

but that is probably out of the scope as each new feature (if needed) must go through its own KEP?

@derekwaynecarr are future improvements based on what cgroup 2 offers out of scope for the current KEP?

@giuseppe future improvements for what cgroup v2 offers should be out of scope of this kep. i would keep this kep focused on kubelet is tolerant of cgroup v2 host. adding new cgroup v2 specific features to resource model would be a separate enhancement.

To address @yujuhong question, I think we are saying the following:

no change required to pod spec

kubelet cgroup manager uses v1 or v2 mode based on its introspection of sys/fs on startup

kubelet qos and pod level cgroup creation on a v2 host uses the mapping table specified

cri implementers that support cgroup v2 hosts must use a similar mapping to ensure that pod bounding cgroup values are consistent with the container cgroup values.

oci runtime spec is not required to change as cri implementer provides mapping in the transitional period.

@giuseppe agree with above?

yujuhong · 2019-12-03T19:19:11Z

/cc @Random-Liu for containerd

k8s-ci-robot · 2019-12-03T19:19:13Z

@yujuhong: GitHub didn't allow me to request PR reviews from the following users: for, containerd.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @Random-Liu for containerd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

The rootless patches has been merged into the upstream Docker since early 2019 and no longer needs to be incubated in this repo. Another reason to remove Docker from this repo is that dockershim might not be going to support cgroup v2: kubernetes/enhancements#1370 (comment) Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda · 2020-02-18T14:18:57Z

@derekwaynecarr @yujuhong @dchen1107 PTAL?

sftim · 2020-02-22T23:37:58Z

/retest

derekwaynecarr · 2020-02-26T16:09:24Z

per dawn's comment and discussion on sig-node, we can merge this kep, and update as we find issues with implementation in 1.19. there is no disagreement on need to iterate here.

see: #1370 (comment)

/approve
/lgtm

Signed-off-by: Giuseppe Scrivano <[email protected]>

mrunalp · 2020-02-26T16:27:21Z

/lgtm

k8s-ci-robot · 2020-02-26T16:27:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, giuseppe, mrunalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~keps/sig-node/OWNERS~~ [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hectorj2f · 2020-02-27T10:50:50Z

awesome work!

crosbymichael · 2020-08-11T18:08:00Z

LGTM :)

smileusd · 2021-05-17T07:41:32Z

+| period | cpu.max | y = x| period and quota are written together|
+| quota | cpu.max | y = x| period and quota are written together|
+
+**blkio controller**


Any kuberente versions and plans to support blkio controller ?

Any kuberente versions and plans to support blkio controller ?

kubernetes/enhancements#1907 the KEP is WIP.

AkihiroSuda · 2025-12-03T11:17:36Z

+new namespace.  It was not enabled by default on a cgroup v1 system as
+older kernel lacked support for it.
+
+Privileged pods will still use the host cgroup namespace so to have


Let's make this customizable, especially for nested containers:

KEP-5714: Allow specifying whether to unshare cgroup namespaces #5715

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 18, 2019

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 18, 2019

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. labels Nov 18, 2019

k8s-ci-robot requested review from dchen1107 and derekwaynecarr November 18, 2019 14:53

giuseppe mentioned this pull request Nov 18, 2019

kubelet: add initial support for cgroupv2 kubernetes/kubernetes#85218

Merged

AkihiroSuda reviewed Nov 18, 2019

View reviewed changes

Comment thread keps/sig-node/20191118-cgroups-v2.md

Comment thread keps/sig-node/20191118-cgroups-v2.md Outdated

derekwaynecarr requested changes Nov 18, 2019

View reviewed changes

k8s-ci-robot assigned derekwaynecarr Nov 18, 2019

AkihiroSuda mentioned this pull request Nov 18, 2019

sig-node: Kubelet-in-UserNS, aka Rootless mode #1371

Merged

AkihiroSuda reviewed Nov 18, 2019

View reviewed changes

Comment thread keps/sig-node/20191118-cgroups-v2.md Outdated

giuseppe force-pushed the cgroupsv2 branch 2 times, most recently from 83faa61 to c0c871d Compare November 20, 2019 07:49

squeed reviewed Nov 22, 2019

View reviewed changes

PatrickLang reviewed Dec 3, 2019

View reviewed changes

Comment thread keps/sig-node/20191118-cgroups-v2.md Outdated

yujuhong reviewed Dec 3, 2019

View reviewed changes

k8s-ci-robot requested a review from Random-Liu December 3, 2019 19:19

alban mentioned this pull request Dec 4, 2019

Gardenlet gardener/gardener#1601

Merged

AkihiroSuda mentioned this pull request Dec 4, 2019

support cgroup2 containerd/containerd#3799

Merged

giuseppe force-pushed the cgroupsv2 branch 2 times, most recently from 23ddf34 to a87c972 Compare December 9, 2019 14:54

giuseppe force-pushed the cgroupsv2 branch from 2fc1d40 to cba0e9c Compare February 25, 2020 18:38

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 26, 2020

keps/sig-node: add KEP for cgroups v2

5b4d3d5

Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe force-pushed the cgroupsv2 branch from cba0e9c to 5b4d3d5 Compare February 26, 2020 16:22

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 26, 2020

k8s-ci-robot assigned mrunalp Feb 26, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 26, 2020

k8s-ci-robot merged commit 4abcc67 into kubernetes:master Feb 26, 2020

k8s-ci-robot added this to the v1.18 milestone Feb 26, 2020

AkihiroSuda mentioned this pull request Feb 27, 2020

cgroup2: privileged and cgroupns containers/podman#5341

Closed

h-vetinari mentioned this pull request Mar 7, 2020

Tracker for cgroups v2 opencontainers/runtime-spec#1002

Closed

6 tasks

yashbhutwala mentioned this pull request Apr 14, 2020

cool utils. ask some use case for clear facebookincubator/senpai#1

Closed

neolit123 mentioned this pull request May 5, 2020

Kubelet on Fedora 31+ doesn't start without kernel parameter systemd.unified_cgroup_hierarchy = 0 kubernetes/kubernetes#90710

Closed

harche mentioned this pull request May 19, 2020

Ability to use CRI-O as runtime in kube-up (on Ubuntu for now) kubernetes/kubernetes#91209

Closed

giuseppe mentioned this pull request May 28, 2020

REQUEST: New membership for giuseppe kubernetes/org#1905

Closed

6 tasks

giuseppe mentioned this pull request Jan 12, 2021

cgroup v2 #2254

Closed

4 tasks

smileusd reviewed May 17, 2021

View reviewed changes

alban mentioned this pull request May 24, 2023

Don't use hostNetwork=true or hostPID=true inspektor-gadget/inspektor-gadget#1665

Closed

3 tasks

iholder101 mentioned this pull request Apr 10, 2025

Conversion of cgroup v1 CPU shares to v2 CPU weight causes workloads to have low CPU priority kubernetes/kubernetes#131216

Closed

AkihiroSuda reviewed Dec 3, 2025

View reviewed changes


		Kubelet PR: [https://github.com/kubernetes/kubernetes/pull/85218](https://github.com/kubernetes/kubernetes/pull/85218)

		### Option 2: Use cgroups v2 throughout the stack


		## Graduation Criteria

		- Alpha: Basic support for running Kubernetes on a cgroups v2 host.



		With this approach cAdvisor would have to read back values from
		cgroups v2 files (already done).

Conversation

giuseppe commented Nov 18, 2019

Uh oh!

k8s-ci-robot commented Nov 18, 2019

Uh oh!

k8s-ci-robot commented Nov 18, 2019

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda commented Nov 18, 2019

Uh oh!

giuseppe commented Nov 18, 2019

Uh oh!

derekwaynecarr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr commented Nov 18, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yujuhong commented Dec 3, 2019

Uh oh!

k8s-ci-robot commented Dec 3, 2019

Uh oh!

AkihiroSuda commented Feb 18, 2020

Uh oh!

sftim commented Feb 22, 2020

Uh oh!

derekwaynecarr commented Feb 26, 2020

Uh oh!

mrunalp commented Feb 26, 2020

Uh oh!

k8s-ci-robot commented Feb 26, 2020

Uh oh!

hectorj2f commented Feb 27, 2020

Uh oh!

crosbymichael commented Aug 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees