That's actually the problem with limiting IOPS: with the current Linux kernel it's not possible to limit it reliably and the reason we don't have it yet in Kubernetes.

As docker supports it, it seems to be a general requirements for storage, especially block volumes. As I mentioned, the disk device that docker is running on, is so important that we need to protect it from being overused by some pods.

--device-read-bps value       Limit read rate (bytes per second) from a device (default [])
      --device-read-iops value      Limit read rate (IO per second) from a device (default [])
      --device-write-bps value      Limit write rate (bytes per second) to a device (default [])
      --device-write-iops value     Limit write rate (IO per second) to a device (default [])

if docker-shim is deprecated, I would look into other cri like containerd. Whether this is supported?

It looks like there is some level of cgroups v2 support in containerd? containerd/containerd#3726

containerd is supported well, it's available on GKE and is becoming the default.

https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd

We don't use alpha annotations in Kubernetes, we directly declare fields in structs (as beta proposal).

will only include the final design later.

1. This assumes that cluster admin can trust users to create pods with the right IOPS requirements. A rogue user can claim all / most of the IOPS for itself. In Kubernetes, we don't trust users (much) and we have quotas and similar concepts to limit them. I don't know how to apply IOPS there, perhaps via StorageClass & IOPS defined there? Then the cluster admin can define quota of volumes to each user / namespace.

2. How does it work when user sets different IOPS for volumes based by the same device (e.g. each Secret / EmptyDir / HostPath volume having different IOPS)?

3. Not all volume types support IOPS, e.g. NFS / GlusterFS / CephFS / tmpfs. Will be IOPS setting silently ignored for them?

1. To limit user, we may add it to limitrange of namespace.
   IOPS can be applied to storage class as well, but this may be a limitation from storage side. This may be a physical limitation.

2. IOPS limitation may only apply to volume, which is a block device.
   cgroups can be applied to only device id.
   If configmap and secrets are mounted on which is on the rootfs of the host, the rootfs iops limitation will take effect.

3. That's the limitation.
   Just ignore those in kubelet. kubelet can get mount type.

Would it make sense to instead of adding absolute IOPS limits, add more of a QoS class to the storageclass. eg,

• unlimited, no restriction on IOPS, which is what we currently do

• fair, all pods using the PVC/volume split IOPS fairly. This does require a pretty complicated controller to track PV usage and update all the nodes with how usage should split, especially if we want to base throttling on actual usage (eg, pods that aren't hammering the device don't get the same share as a more active pod), but this may be what users actually want

• priority, pods with some sort of annotation get unlimited IOPS, all other pods get a fixed share

OK, I'll admit this is all very vague and not well thought-out, but a direction like this has a few advantages:

• Current state of the art means we're not going to be able to hit all IOPS limits anyway AFAIU

• This gives leeway for implementation differences and means we wouldn't have to change the API when cgroups v2 comes out or do all sort of complicated docker vs containerd differences

• Managing IOPS at a pod level is not going to scale for large deployments, but it's (probably) only in large deployments where this sort of thing is really needed. So a higher-level API is maybe ultimately much more useful.

#27000 already was discussing along these lines.

Maybe another way to see this is that QoS is really a volume characteristic and not a pod characteristic. That is, we are concerned about guaranteeing a certain performance for all users of a volume more than we are concerned about setting what performance a pod sees. So setting limits at some higher level than the pod is probably the right way to go about this.

Given this limitation, how useful is this feature for node stability? In my experience, most writes are buffered, so this would only limit some pods.

How does this feature improve speed? It seems to only address limiting speed.

It is advantage of local volume. I may put it in the wrong place. thanks.

Windows has been GA for a while now. Given that you are proposing a user-facing API, we should at minimum evaluate feasibility of implementing this feature in windows.

nit: This is the container cgroup, but you say below we aren't managing containers. Can you remove <Container_ID> from the path?

ok

We have discussed having both container-level and pod-level control for other resources in the past, but it can be quite complex: #1592. I would recommend assuming we will only get either pod or container-level limiting.

Does containerd have something similar?

I see blkio-based stats but am not familiar in general with it. https://sourcegraph.com/github.com/containerd/containerd@a615a6fe5dd3a6a6689ccb119b0a9632100c7a50/-/blob/metrics/cgroups/blkio.go

This is already a key problem, I think we need to design something for that here.

eg with GCP PD, IOPS are pooled across all devices attached to a node so that limiting IOPS on a single device may not work as expected (eg, accessing the rootfs may end up throttling an attached PV).

• cgroups can limit iops for local block volume and rootfs.

I may change to goal to this. For all types of volumes' QoS, it may be a much bigger topic.

Will cgroups v2 change any of this?

If so we should mention that and link in (or start...) related work in docker/containerd.

I think if cgroups is fundamentally limited we want to be careful about adding in a feature that won't work very well but will get baked into deployments and require long-term support.

If this is to be volume-based, why not use the pod volume name rather than the PVC? That would incidentally allow for rootfs throttling.

This might not be effective for all kinds of volumes depending on how they're mounted, I suppose, but since the throttling has to be best-effort anyway that might not be a problem in practice.

Would it make sense to instead of adding absolute IOPS limits, add more of a QoS class to the storageclass. eg,

• unlimited, no restriction on IOPS, which is what we currently do

• fair, all pods using the PVC/volume split IOPS fairly. This does require a pretty complicated controller to track PV usage and update all the nodes with how usage should split, especially if we want to base throttling on actual usage (eg, pods that aren't hammering the device don't get the same share as a more active pod), but this may be what users actually want

• priority, pods with some sort of annotation get unlimited IOPS, all other pods get a fixed share

OK, I'll admit this is all very vague and not well thought-out, but a direction like this has a few advantages:

• Current state of the art means we're not going to be able to hit all IOPS limits anyway AFAIU

• This gives leeway for implementation differences and means we wouldn't have to change the API when cgroups v2 comes out or do all sort of complicated docker vs containerd differences

• Managing IOPS at a pod level is not going to scale for large deployments, but it's (probably) only in large deployments where this sort of thing is really needed. So a higher-level API is maybe ultimately much more useful.

#27000 already was discussing along these lines.

Maybe another way to see this is that QoS is really a volume characteristic and not a pod characteristic. That is, we are concerned about guaranteeing a certain performance for all users of a volume more than we are concerned about setting what performance a pod sees. So setting limits at some higher level than the pod is probably the right way to go about this.