Skip to content

containerd-stress: add support for running through CRI#6931

Merged
samuelkarp merged 1 commit intocontainerd:mainfrom
egernst:cri-stress
Aug 11, 2022
Merged

containerd-stress: add support for running through CRI#6931
samuelkarp merged 1 commit intocontainerd:mainfrom
egernst:cri-stress

Conversation

@egernst
Copy link
Copy Markdown
Contributor

@egernst egernst commented May 12, 2022

Introduce a --cri flag, which'll run containerd-stress using the CRI,
instead of containerd's task api.

In doing so, introduce cri_worker, rename existing worker to ctr_worker and introduce
a worker interface that each of these implement.

Signed-off-by: Eric Ernst [email protected]

@k8s-ci-robot
Copy link
Copy Markdown

Hi @egernst. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@egernst egernst force-pushed the cri-stress branch 2 times, most recently from 19fbb4d to d811b8c Compare May 12, 2022 00:58
Comment thread cmd/containerd-stress/main.go Outdated
@estesp
Copy link
Copy Markdown
Member

estesp commented May 12, 2022

Looks like the branch you started from is missing some recent commits from main that fix a CI issue on Vagrant. I know your PR content has nothing to do with any of those test flows, but will be easier to get "green" CI if you rebase on main.

@egernst
Copy link
Copy Markdown
Contributor Author

egernst commented May 12, 2022

Thanks Phil (and Sam). Rebased accordingly.

@egernst egernst requested a review from samuelkarp June 1, 2022 16:42
@kzys
Copy link
Copy Markdown
Member

kzys commented Jun 10, 2022

@samuelkarp Can you take a look?

@samuelkarp
Copy link
Copy Markdown
Member

I tried running this, but I'm getting a bunch of errors like this:

ERRO[0006] running container 0-13130                     error="rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for \"io.containerd.runc.v2\" is configured"
E0610 13:08:02.923645  254737 remote_runtime.go:135] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "io.containerd.runc.v2" is configured

Here's how I invoked it:

$ sudo bin/containerd-stress --cri

Meanwhile, the io.containerd.runc.v2 runtime appears to work when I use ctr:

$ sudo ctr run --runtime io.containerd.runc.v2 --rm docker.io/library/debian:latest test cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

@egernst
Copy link
Copy Markdown
Contributor Author

egernst commented Jun 11, 2022

@samuelkarp thanks for giving it a try. crictl/CRI doesn't use the containerd fully qualified runtime name. Instead, it just uses the handler name. For runc, it'd be runc. In Kata case, in my testing I just use kata.

Example:

sudo ./containerd-stress --duration 24h --runtime runc --cri  --concurrent 30

I updated the usage of --cri in order to help clarify this change in expectations:

...
   --cri                         utilize CRI to create pods for the stress test. This requires a runtime that matches CRI runtime handler. Example: --runtime runc
...

Introduce a --cri flag, which will enable running container-stress using the CRI,
instead of containerd's task API.

In doing so, we introduce cri_worker, rename the existing worker to ctr_worker, and introduce
a worker interface that each of these implement.

Signed-off-by: Eric Ernst <[email protected]>
@samuelkarp
Copy link
Copy Markdown
Member

crictl/CRI doesn't use the containerd fully qualified runtime name.

I didn't specify a runtime name, that was the default when running containerd-stress with just the --cri option specified. Can you make it so that the default behavior works (i.e., containerd-stress itself knows to use "runc" instead of "io.containerd.runc.v2" when constructing requests) or document that --runtime must also be specified when using --cri?

With that said, when I run the command you provided I get further errors as follows:

ERRO[0001] running container 24-20                       error="rpc error: code = Unknown desc = failed to setup network for sandbox \"cb9858cad06ba2349ba0e335074ba6ab1d0c976848c808d19b402f4e5609bccc\": cni plugin not initialized"

Can you document the requirements for running containerd-stress in this new mode? I assume it'd be something like:

  • enable the CRI plugin
  • install CNI plugins, configure containerd to use them
  • specify --runtime runc

@egernst
Copy link
Copy Markdown
Contributor Author

egernst commented Jun 13, 2022

I updated the --cri usage to provide more details on expectations of --runtime flag -- does that meet your needs?

I think we'll probably want to introduce a README for containerd-stress if we want to augment and have more details around usage, expectations? WDYT?

@samuelkarp
Copy link
Copy Markdown
Member

I updated the --cri usage to provide more details on expectations of --runtime flag -- does that meet your needs?

I don't see a new change yet (maybe it didn't get pushed?), but that would probably work.

@egernst
Copy link
Copy Markdown
Contributor Author

egernst commented Jun 13, 2022

Goodness gracious, my fault. actually pushed now. Thanks.

Copy link
Copy Markdown
Member

@mikebrow mikebrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have not reviewed yet.. just a heads up there is a project called cri-tools where we usually create cri tools :-) there may also be a stress bucket or two over in k8s e2e.. Some may not want to put a CRI dependency here..

@egernst
Copy link
Copy Markdown
Contributor Author

egernst commented Aug 9, 2022

Hey Mike -- I did consider this initially. I want the same ux as existing containerd-stress, and frankly, the containerd-stress codebase was much easier to work with. We already have a (nice) package for interacting with CRI in containerd, and we are exercising the CRI handling which is implemented within containerd, so I figured it is pretty reasonable to include here.

Copy link
Copy Markdown
Member

@samuelkarp samuelkarp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I finally had the chance to come back and test this and it does work as long as the CNI plugins are also installed. LGTM.

@samuelkarp samuelkarp merged commit f87a1b0 into containerd:main Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants