-
|
I wanted to start a discussion around what it may look like to implement a feature in containerd to better image registry mirroring around airgapped environments. The closest I saw to this was a PR put in by the Rancher folks - #5171. This added mirror registry regex rewrites, allowing the Sadly that didn't go anywhere, and the underlying system was refactored to use hosts.toml instead, which would require completely changing the implementation of the PR. It should be noted they're still using a patched version of containerd for k3s and RKE2 with this in it to help support airgapped environments. Similarly I'd like to support rewriting all image pull requests to go to a single registry, with a single set of credentials. This registry could differentiate based on the Host header, a namespace, or possibly a custom header; which registry to pull from. A proxy wouldn't suffice here because it'd require either a custom CA be used, or caching/scanning/monitoring of images wouldn't be possible. I'd love to hear thoughts on how something like this might be implemented, within containerd, or outside of it. One obvious alternative would be a k8s mutator hook, but that wouldn't work for containers started before/during k8s bootstrap, so it definitely has some limitations. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 5 replies
-
|
Having a global airgapped configuration would be easy to implement and hosts.toml could easily support it. What is missing is a configuration in the CRI plugin to enable this. As we transition to move the pull logic out of CRI to the new transfer service, a new configuration option would belong there. The underlying hosts.toml system you mentioned is already designed to support this. The CRI configuration today only looks up based on registry host name, it needs a fallback to a global configuration to support the use case you mentioned. See CRI logic for getting mirror configs https://github.com/containerd/containerd/blob/main/pkg/cri/server/image_pull.go#L341 The PR you linked to included functionality to alter an image name provided by a client, which we have tried to avoid as much as possible, only supporting the narrow "short name" conversion logic inherited by K8s through their Docker support. We have always maintained that image names in the containerd backend should not change but the registry host resolution should be flexible and easily configurable. So changing |
Beta Was this translation helpful? Give feedback.
-
|
I hadn't seen the open PR, that actually looks like exactly what I was thinking! Am I wrong in saying that the current hosts.toml doesn't currently support wildcards, so redirecting all repositories (instead of just an explicit subset) isn't currently possible with hosts.toml? I see in the open PR they have the concept of |
Beta Was this translation helpful? Give feedback.
-
The hosts.toml just provides a set of hosts configurations from a directory, it doesn't specify how that directory is chosen. Today the CRI chooses that directory by looking for a directory which has the same name as the image host. CRI could easily be configured to fallback to a default directory when the more specific directory does not exist, that would satisfy the wildcard functionality. I think it would be OK to accept a PR with a new configuration into CRI and backport that into 1.6. That configuration may end up being moved in 2.0 out of the CRI plugin, but we can handle that along with other config migrations. |
Beta Was this translation helpful? Give feedback.
-
|
True, it would be a simple and backwards compatible change to the hosts.toml system. There'd still be the problem of communicating the source image registry to the proxying image registry though. Rewriting the repository was one workaround for that, or using an |
Beta Was this translation helpful? Give feedback.
-
|
The ns parameter is already added by containerd when talking to a registry which has a hostname different from the given image host. |
Beta Was this translation helpful? Give feedback.
-
|
Oh! I had no idea. Thanks for all the information, I think I have a good path forward now |
Beta Was this translation helpful? Give feedback.
Having a global airgapped configuration would be easy to implement and hosts.toml could easily support it. What is missing is a configuration in the CRI plugin to enable this. As we transition to move the pull logic out of CRI to the new transfer service, a new configuration option would belong there. The underlying hosts.toml system you mentioned is already designed to support this. The CRI configuration today only looks up based on registry host name, it needs a fallback to a global configuration to support the use case you mentioned.
See CRI logic for getting mirror configs https://github.com/containerd/containerd/blob/main/pkg/cri/server/image_pull.go#L341
The PR you linked to include…