Welcome @shannonxtreme!

It looks like this is your first PR to kubernetes-sigs/cri-tools 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cri-tools has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

looks like all the bases are covered. thanks @shannonxtreme

https://cs.k8s.io/?q=dockershim.sock&i=nope&files=&excludeFiles=&repos=kubernetes-sigs/cri-tools

/assign @adisky

one thing though ... for hack/run-dockershim-critest.sh we could add some comments that i can only work with https://github.com/Mirantis/cri-dockerd and not with any other CRI implementation. (either in this PR or in a follow up, i don't mind)

@dims: GitHub didn't allow me to assign the following users: adisky.

Note that only kubernetes-sigs members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Do we want crictl to be able to talk to earlier versions of k8s clusters while they are still supported? Once crictl is moved to use v1 of CRI API only, this will be totally reasonable PR. Before that, I'd suggest we keep it.

Do we want crictl to be able to talk to earlier versions of k8s clusters while they are still supported?

One thing is that the matrix is not up to date: https://github.com/kubernetes-sigs/cri-tools#current-status

Do we want crictl to be able to talk to earlier versions of k8s clusters while they are still supported? Once crictl is moved to use v1 of CRI API only, this will be totally reasonable PR. Before that, I'd suggest we keep it.

@SergeyKanzhelev doesn't users expected to use matching version of crictl & k8s, if K8s 1.24 removed dockershim crictl 1.24 also expected to do the same? possiblty we should release a 1.23 tag for crictl then merge this PR?

Also this isn't essentially removing any support, only removing dockershim from default endpoints users can still configure unix:///var/run/dockershim.sock manually

@SergeyKanzhelev doesn't users expected to use matching version of crictl & k8s

as a general statement, this wouldn't be ideal. crictl may be used alongside the containerd. For instance, it is installed on COS. And it works independently on the version of k8s installed on that COS image.

Since the dockershim.sock is only exposed by k8s itself, this statement has more sense. Also, defaults are deprecated and customers are encouraged to specify the socket explicitly (#599). So even if we will break customer scenario here, where the fallback to default is expected, it will not be completely unexpected. It also will be a good notification for customers to migrate :-).

Let's discuss the release related bits in #856

We would have hate having to use different versions of crictl in minikube, for different versions of kubernetes.

Removing dockershim support just for the sake of it seems useless (or premature), if it is not there it is not there ?

EDIT: Just to be clear, minikube does set up the crictl.yaml. Even for docker, where it not even used or required

But still using crictl 1.21 for k8s 1.23, so requiring a closer match would still be a big change compared with today

Since the defaults are deprecated anyway, isn't this loss of functionality more of an incentive to move off relying on just the defaults, and instead specifying explicit endpoints?

But then you could remove all endpoints, instead of just one.

I'm not sure why that hasn't happened. Maybe a deprecation policy thing?

The majority of users have not even installled (much less configured) these still undocumented debugging tools, since it was not recommended for their runtime. So I guess it made sense to have it try with dockershim, if they dared to install the tool ?

But now that is required by all container runtimes, maybe it will end up being documented under "container runtimes" and not just under "node debugging" and it will see some better installations - such as packaging and configuring (setting up crictl.yaml)

• https://kubernetes.io/docs/setup/production-environment/container-runtimes/

• https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/

as a general statement, this wouldn't be ideal. crictl may be used alongside the containerd. For instance, it is installed on COS. And it works independently on the version of k8s installed on that COS image.

Same thing for minikube, currently using a random version 1.21.0 for the crictl installation in the OS

If it will be versioned with kubernetes, we need to move it to runtime bootstrapping installation instead.

It also will be a good notification for customers to migrate :-).

They don't have to "migrate", they just have to start using CRI

Is it that the defaults currently do not work?

The project doesn't dare to pick a new favorite, so recommends using any container runtime that follows the CRI

"Items on this page refer to third party products or projects that provide functionality required by Kubernetes.
The Kubernetes project authors aren't responsible for those third-party products or projects."

And if crictl needs to try all the possible locations, that introduces a latency when checking the various sockets...

But checking all 4 locations "works", I thought.

• Add support for cri-dockerd #869

It also will be a good notification for customers to migrate :-).

They don't have to "migrate", they just have to start using CRI

I meant migrate away from using dockershim.

@mikebrow

Currently (v1.24.0), it is just failing. It never gets around to trying unix:///var/run/cri-dockerd.sock

WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 
E0510 08:00:35.184124   12046 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 

It works fine with the -r parameter. I think it is done that way in kubeadm, instead of relying on CRI config.

$ crictl -r unix:///run/cri-dockerd.sock version
Version:  0.1.0
RuntimeName:  docker
RuntimeVersion:  20.10.12
RuntimeApiVersion:  1.41.0

make a rebase again?

@mikebrow

Currently (v1.24.0), it is just failing. It never gets around to trying unix:///var/run/cri-dockerd.sock

WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 
E0510 08:00:35.184124   12046 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 

It works fine with the -r parameter. I think it is done that way in kubeadm, instead of relying on CRI config.

$ crictl -r unix:///run/cri-dockerd.sock version
Version:  0.1.0
RuntimeName:  docker
RuntimeVersion:  20.10.12
RuntimeApiVersion:  1.41.0

ok.. thx for the heads up.. need some more debug so we can see which container runtime on which socket doesn't have a matching runtime.v1alpha2.RuntimeService... From your log going to guess you have a very old version of containerd running?

or.... see below

It was either 1.4 or 1.5 (whatever docker used), should be easy to reproduce

There were some weird ghost reports of it "failing" even with containerd 1.6, since the error message can be misleading if the daemon is not answering/running for some reason...

By providing it explicitly, the error is more clear.

And by using configuration, it is less cluttered than having to lug that -r parameter around. So I think that kubeadm should require /etc/crictl.yaml use well as crictl

It was either 1.4 or 1.5 (whatever docker used), should be easy to reproduce

There were some weird ghost reports of it "failing" even with containerd 1.6, since the error message can be misleading if the daemon is not answering/running for some reason...

By providing it explicitly, the error is more clear.

And by using configuration, it is less cluttered than having to lug that -r parameter around. So I think that kubeadm should require /etc/crictl.yaml use well as crictl

If containerd is installed but you have disabled the cri plugin on purpose (standard for docker install's version of the containerd config.toml) the containerd sock will not have any cri services to serve and crictl will see that as a failure. It, crictl, shouldn't stop the loop rather it should continue on reporting that the containerd sock does not have cri enabled and going to the next default runtime in the loop.

The error here https://github.com/kubernetes-sigs/cri-tools/blob/master/cmd/crictl/main.go#L82 isn't very useful when the err returned does not mention the endpoint that failed...

IMO the loop thing should be: .. you don't have a container runtime configured in your crictl config and you have not set one at the command line or in the ENV.. here are your options... we've tried the following container runtimes with at the typical defaults (show the options in the loop and status).. ask which they would like to set as their default in the crictl config...: grab stdin wait for response for 30sec.. no input fail to prompt

Or we can fix the somewhat rude disabling of the containerd cri plugin :-) maybe just output the instructions in the error message. "container config default > /etc/containerd/config.toml".. "service containerd restart"

Or we can just remove the loop entirely, it's deprecation time has expired?

I tried again, with the latest minikube beta - but with k8s version 1.24.0 and crictl version 1.24.0:

minikube start --kubernetes-version=v1.24.0

docker@minikube:~$ sudo rm /etc/crictl.yaml 
docker@minikube:~$ sudo ./crictl --version
crictl version 1.24.0
docker@minikube:~$ sudo ./crictl version
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 
E0525 18:30:45.757607   51352 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 
docker@minikube:~$ containerd --version
containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
docker@minikube:~$ dockerd --version
Docker version 20.10.15, build 4433bf6
docker@minikube:~$ cri-dockerd --version
cri-dockerd 0.2.0 (a4d1895)
docker@minikube:~$ sudo ./crictl -r unix:///var/run/cri-dockerd.sock version
Version:  0.1.0
RuntimeName:  docker
RuntimeVersion:  20.10.15
RuntimeApiVersion:  1.41.0

So we always set up the /etc/crictl.yaml config.

Ok so I rebased, but I'd really like someone to take another look at these changes to make sure this is right

I tried again, with the latest minikube beta - but with k8s version 1.24.0 and crictl version 1.24.0:

minikube start --kubernetes-version=v1.24.0

docker@minikube:~$ sudo rm /etc/crictl.yaml 
docker@minikube:~$ sudo ./crictl --version
crictl version 1.24.0
docker@minikube:~$ sudo ./crictl version
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 
E0525 18:30:45.757607   51352 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 
docker@minikube:~$ containerd --version
containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
docker@minikube:~$ dockerd --version
Docker version 20.10.15, build 4433bf6
docker@minikube:~$ cri-dockerd --version
cri-dockerd 0.2.0 (a4d1895)
docker@minikube:~$ sudo ./crictl -r unix:///var/run/cri-dockerd.sock version
Version:  0.1.0
RuntimeName:  docker
RuntimeVersion:  20.10.15
RuntimeApiVersion:  1.41.0

So we always set up the /etc/crictl.yaml config.

you may have tried it again but did you fix the broken config.toml yet and restart containerd?

It using docker, not containerd

The config is not broken, it just doesn't need the cri plugin (since that is handled by cri dockerd)

Even if so, it would be better to clear the config than to copy an old default config (to be obsoleted)

rm /etc/containerd/config.toml

Here is the output from the version in this PR:

docker@minikube:~$ sudo ./crictl --version
crictl version 1.24.0-38-g43477844
docker@minikube:~$ sudo ./crictl --debug version
DEBU[0000] get runtime connection                       
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock unix:///var/run/dockershim.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
DEBU[0000] Note that performance maybe affected as each default connection attempt takes n-seconds to complete before timing out and going to the next in sequence. 
DEBU[0000] Connect using endpoint "unix:///run/containerd/containerd.sock" with "2s" timeout 
DEBU[0000] Connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] VersionRequest: &VersionRequest{Version:v1,} 
E0526 10:31:27.234639     568 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
DEBU[0000] VersionResponse: nil                         
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 
docker@minikube:~$ 

It never tries the other endpoints.

If I enable the cri plugin, then it will think that it should use the containerd runtime (instead of the docker runtime)

root@minikube:/home/docker# containerd config default > /etc/containerd/config.toml
root@minikube:/home/docker# systemctl restart containerd
root@minikube:/home/docker# exit
exit
docker@minikube:~$ sudo ./crictl --debug version
DEBU[0000] get runtime connection                       
WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock unix:///var/run/dockershim.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
DEBU[0000] Note that performance maybe affected as each default connection attempt takes n-seconds to complete before timing out and going to the next in sequence. 
DEBU[0000] Connect using endpoint "unix:///run/containerd/containerd.sock" with "2s" timeout 
DEBU[0000] Connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] VersionRequest: &VersionRequest{Version:v1,} 
DEBU[0000] VersionResponse: &VersionResponse{Version:0.1.0,RuntimeName:containerd,RuntimeVersion:1.6.4,RuntimeApiVersion:v1,} 
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.6.4
RuntimeApiVersion:  v1

So then it won't match the CRI configuration from kubeadm, which will confuse things later.

/var/lib/kubelet/kubeadm-flags.env:KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --node-ip=192.168.49.2 --pod-infra-container-image=k8s.gcr.io/pause:3.7"

(the last kubelet argument doesn't actually do anything now, but is a different issue than crictl)

Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock

So I think it is better to leave the default docker configuration for containerd, and configure crictl ?

/etc/containerd/config.toml

disabled_plugins = ["cri"]

/etc/crictl.yaml

runtime-endpoint: unix:///var/run/cri-dockerd.sock

It never tries the other endpoints.

correct if docker's edit of the containerd config has disabled the cri services the containerd sock won't serve cri services, not moving on seems like a bug in crictl

If I enable the cri plugin, then it will think that it should use the containerd runtime (instead of the docker runtime)

this is your view.. makes sense to you.. may not make sense to people who are not using a dockershim version of kubelet on their workernode or development environment

crictl has a config option and a cli option to let users specify which container runtime they want to use

for ages the defaulting sequence list in crictl was dockershim first and the list was deprecated .. I believe with the intention of removing it and letting people choose manually via config or cli.. given the number of options available for choosing it makes sense to let users choose which container runtime they want.

choices: either delete the default code and force manual config, or keep it as it is with a fix to continue to the next when the sock is there but docker has disabled the cri api, or get docker to stop disabling the cri api in containerd, and/or give crictl users a list to choose from and ask them which one they want..

So then it won't match the CRI configuration from kubeadm, which will confuse things later.

why does kubeadm require containerd not be configured correctly for working with kubeadm? I believe kubeadm also works with containerd/cri-o?

/var/lib/kubelet/kubeadm-flags.env:KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --node-ip=192.168.49.2 --pod-infra-container-image=k8s.gcr.io/pause:3.7"
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock
So I think it is better to leave the default docker configuration for containerd, and configure crictl ?

it is better to have containerd, and the other container runtimes, configured correctly for their clients, and configure crictl for your specific use case. There are a great number of clients of the container runtimes... not just this particular kubeadm use case.

/etc/crictl.yaml

runtime-endpoint: unix:///var/run/cri-dockerd.sock

yup and

crictl config --get runtime-endpoint 
unix:///run/containerd/containerd.sock

or

crictl config --set runtime-endpoint unix:///var/run/cri-dockerd.sock

for ages the defaulting sequence list in crictl was dockershim first and the list was deprecated ..

And currently it is broken, ever since cri-dockerd.sock was moved to the end of the deprecated list.

why does kubeadm require containerd not be configured correctly for working with kubeadm?

It uses a separate flag / config item for kubeadm, and then uses crictl -r with it for all the calls.

@shannonxtreme: PR needs rebase.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Reopen this PR with /reopen
• Mark this PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-triage-robot: Closed this PR.