kubelet: add initial support for cgroupv2 by giuseppe · Pull Request #85218 · kubernetes/kubernetes

giuseppe · 2019-11-13T15:22:09Z

/kind feature

do a conversion from the cgroups v1 limits to cgroups v2.

e.g. cpu.shares on cgroups v1 has a range of [2-262144] while the
equivalent on cgroups v2 is cpu.weight that uses a range [1-10000].

Signed-off-by: Giuseppe Scrivano [email protected]

Does this PR introduce a user-facing change?:

NONE

Support for running on a host that uses cgroups v2 unified mode

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2019-11-13T15:22:17Z

Hi @giuseppe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mattjmcnaughton

/uncc

Looking forward to reading the KEP :) Removing myself as a reviewer while this diff is WIP.

mrunalp · 2019-11-14T17:45:19Z

/ok-to-test

giuseppe · 2019-11-18T14:54:21Z

Looking forward to reading the KEP :) Removing myself as a reviewer while this diff is WIP.

KEP here: kubernetes/enhancements#1370

WIP backport of: kubernetes/kubernetes#85218 Signed-off-by: Giuseppe Scrivano <[email protected]>

derekwaynecarr · 2020-03-12T14:48:08Z

/assign

derekwaynecarr

i want to review this a little closer.

no conversion was needed for pids and hugepages, correct?

derekwaynecarr · 2020-03-12T14:51:33Z

-			Cgroups: cgroups,
-			Paths:   paths,
-		}, nil
+		f, err := cgroupsystemd.NewSystemdCgroupsManager()


does this manager work on v1 and v2 hosts?

i need to check the runc code, but is this abstracing legacy from unified manager?

yes, NewSystemdCgroupsManager() checks for the cgroup version used:

https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L118-L136

If we are running on cgroup v1, it will use cgroupsystemd.LegacyManager

thanks for pointer, and noting that i confirmed this is in the vendored version.

giuseppe · 2020-03-12T15:01:48Z

no conversion was needed for pids and hugepages, correct?

yes correct, both pids and hugetlb don't need any conversion. I'll need to add support for hugetlb to cAdvisor though, once Linux 5.6 is out (it is still at 5.6-rc5), support in libcontainer is already being added.

mattjmcnaughton · 2020-03-14T16:04:11Z

Cool stuff!
It will take me a couple of days to give this a full review (gotta read up on cgroup v2 a bit first :) ).

@giuseppe I haven't forgotten about this :) I just haven't had a lot of time to review larger PRs recently :) Will hopefully have thoughts by end of week. Thanks for your patience.

/uncc

Whoops, I didn't get to this before @derekwaynecarr did. Tbh, I don't see myself adding anything that his review wouldn't, so going to defer to him on the review :)

derekwaynecarr · 2020-03-26T15:08:46Z

+	if libcontainercgroups.IsCgroup2UnifiedMode() {
+		if resourceConfig.CpuShares != nil {
+			// Convert from the range [2-262144] to [1-10000]
+			resources.CpuWeight = (1 + ((*resourceConfig.CpuShares-2)*9999)/262142)


minor nit: may be useful to move this conversion into a helper that we can then unit test in a follow-on.

PR here: #89567

derekwaynecarr

great to see this progress.

/approve
/lgtm

derekwaynecarr · 2020-03-26T15:09:50Z

/milestone v1.19

k8s-ci-robot · 2020-03-26T15:14:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, giuseppe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/kubelet/cm/OWNERS~~ [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mattjmcnaughton · 2020-03-26T23:39:20Z

Wow, exciting stuff @giuseppe @derekwaynecarr - big thank you for all your work!

follow-up for kubernetes#85218 Signed-off-by: Giuseppe Scrivano <[email protected]>

k8s-ci-robot requested review from dashpole and mattjmcnaughton November 13, 2019 15:23

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 13, 2019

AkihiroSuda reviewed Nov 13, 2019

View reviewed changes

Comment thread pkg/kubelet/cm/helpers_linux.go Outdated

AkihiroSuda reviewed Nov 13, 2019

View reviewed changes

Comment thread pkg/kubelet/cm/helpers_linux.go Outdated

giuseppe force-pushed the cgroupv2 branch from 83eb7a5 to 1e1cbdf Compare November 13, 2019 15:44

mattjmcnaughton reviewed Nov 14, 2019

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 14, 2019

alban mentioned this pull request Dec 4, 2019

Gardenlet gardener/gardener#1601

Merged

giuseppe force-pushed the cgroupv2 branch from 1e1cbdf to 4969d96 Compare December 4, 2019 17:54

giuseppe mentioned this pull request Dec 9, 2019

add KEP for cgroups v2 support kubernetes/enhancements#1370

Merged

giuseppe force-pushed the cgroupv2 branch from 4969d96 to 0704ef6 Compare December 19, 2019 19:43

giuseppe force-pushed the cgroupv2 branch from 0704ef6 to 1413203 Compare January 28, 2020 10:44

giuseppe added a commit to giuseppe/origin that referenced this pull request Jan 28, 2020

UPSTREAM: <carry>: cgroupv2 support

5b598c3

WIP backport of: kubernetes/kubernetes#85218 Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe mentioned this pull request Jan 28, 2020

[WIP] UPSTREAM: <carry>: cgroupv2 support openshift/origin#24467

Closed

giuseppe force-pushed the cgroupv2 branch from 1413203 to 4cd7cf1 Compare January 28, 2020 14:27

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 12, 2020

k8s-ci-robot assigned derekwaynecarr Mar 12, 2020

derekwaynecarr reviewed Mar 12, 2020

View reviewed changes

neolit123 mentioned this pull request Mar 14, 2020

Add guide to switch to cgroup v1 kubernetes/website#19632

Closed

derekwaynecarr reviewed Mar 26, 2020

View reviewed changes

derekwaynecarr approved these changes Mar 26, 2020

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 26, 2020

k8s-ci-robot added this to the v1.19 milestone Mar 26, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 26, 2020

k8s-ci-robot merged commit 34c8b26 into kubernetes:master Mar 26, 2020

giuseppe mentioned this pull request Mar 27, 2020

kubelet: add tests for cgroup v2 conversions #89567

Merged

giuseppe added a commit to giuseppe/kubernetes that referenced this pull request Mar 27, 2020

kubelet: add tests for cgroup v2 conversions

c4429d8

follow-up for kubernetes#85218 Signed-off-by: Giuseppe Scrivano <[email protected]>

AkihiroSuda mentioned this pull request Mar 28, 2020

update components rootless-containers/usernetes#153

Merged

harche mentioned this pull request Mar 30, 2020

Add a crio tests kubernetes/test-infra#17014

Merged

neolit123 mentioned this pull request Apr 1, 2020

Support for cgroupv2 kubernetes/system-validators#12

Merged

neolit123 mentioned this pull request May 5, 2020

Kubelet on Fedora 31+ doesn't start without kernel parameter systemd.unified_cgroup_hierarchy = 0 #90710

Closed

giuseppe mentioned this pull request Jan 12, 2021

cgroup v2 kubernetes/enhancements#2254

Closed

4 tasks

BenTheElder mentioned this pull request Feb 23, 2021

Kubelet for versions prior to 1.19 do not start under unified cgroups kubernetes-sigs/kind#2082

Closed

DennisGlindhart mentioned this pull request Apr 26, 2021

Cluster join fails due to insufficient ephemeral-storage #99305

Closed

kakkoyun mentioned this pull request Sep 2, 2021

Add support for cgroup v2 parca-dev/parca-agent#23

Closed

Conversation

giuseppe commented Nov 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Nov 13, 2019

Uh oh!

Uh oh!

Uh oh!

mattjmcnaughton left a comment

Choose a reason for hiding this comment

Uh oh!

mrunalp commented Nov 14, 2019

Uh oh!

giuseppe commented Nov 18, 2019

Uh oh!

derekwaynecarr commented Mar 12, 2020

Uh oh!

derekwaynecarr left a comment

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr Mar 12, 2020

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr Mar 12, 2020

Choose a reason for hiding this comment

Uh oh!

giuseppe Mar 12, 2020

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr Mar 26, 2020

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Mar 12, 2020

Uh oh!

mattjmcnaughton commented Mar 14, 2020

Uh oh!

derekwaynecarr Mar 26, 2020

Choose a reason for hiding this comment

Uh oh!

giuseppe Mar 27, 2020

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr left a comment

Choose a reason for hiding this comment

Uh oh!

derekwaynecarr commented Mar 26, 2020

Uh oh!

k8s-ci-robot commented Mar 26, 2020

Uh oh!

mattjmcnaughton commented Mar 26, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

giuseppe commented Nov 13, 2019 •

edited

Loading