Update CRI to use transfer service for image pull by default #8515

fangn2 · 2023-05-15T03:01:38Z

This PR is to update CRI to optionally use transfer service for image pull.

Part of #8227

Create a new CRI config ImagePullWithTransferService to specify whether to use transfer service for image pull
Move CRI image pull related configs to server side, i.e. transfer service.

The PR is still in progress, put out to for early review and utilize CI for integration tests.

k8s-ci-robot · 2023-05-15T03:01:47Z

Hi @fangn2. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mikebrow · 2023-05-17T00:51:24Z

/ok-to-test

pkg/cri/config/config.go

pkg/cri/server/image_pull.go

fuweid · 2023-06-12T05:35:32Z

pkg/cri/server/image_pull.go

+			return nil, fmt.Errorf("failed to pull and unpack image %q: %w", ref, err)
+		}
+
+		return &runtime.PullImageResponse{ImageRef: ref}, nil


need to update the image store.

Thanks for review! Will address it in the coming changes.

Done.
Just curious CRI currently maintain an in-memory image cache(there could be situations that the CRI image store and containerd image store are out of sync, like other clients e.g. ctr removed an image). Can we get rid of it by directly using contained image store?

estesp · 2023-10-03T18:37:23Z

CRI test failures seem related to the new (optional) image verifier service merged in #8493; the image verifier plugin is a required plugin for transfer plugin now and the error seems to be related to that? Seems you rebased, so I'm not sure what else has to change to make that work properly?

Error:

2023-10-03T02:31:08.9836244Z === NAME  TestCRIImagePullTimeout/HoldingContentOpenWriter
2023-10-03T02:31:08.9837268Z     build_local_containerd_helper_test.go:109: 
2023-10-03T02:31:08.9838989Z         	Error Trace:	/home/runner/work/containerd/containerd/integration/build_local_containerd_helper_test.go:109
2023-10-03T02:31:08.9840452Z         	            				/home/runner/work/containerd/containerd/integration/image_pull_timeout_test.go:78
2023-10-03T02:31:08.9841049Z         	Error:      	Received unexpected error:
2023-10-03T02:31:08.9842280Z         	            	no plugins registered for io.containerd.image-verifier.v1: not found
2023-10-03T02:31:08.9843017Z         	Test:       	TestCRIImagePullTimeout/HoldingContentOpenWriter
2023-10-03T02:31:08.9843552Z     build_local_containerd_helper_test.go:120: 
2023-10-03T02:31:08.9844427Z         	Error Trace:	/home/runner/work/containerd/containerd/integration/build_local_containerd_helper_test.go:120
2023-10-03T02:31:08.9845822Z         	            				/home/runner/work/containerd/containerd/integration/image_pull_timeout_test.go:78
2023-10-03T02:31:08.9846406Z         	Error:      	Received unexpected error:
2023-10-03T02:31:08.9847983Z         	            	failed to get "io.containerd.transfer.v1" plugin: no plugins registered for io.containerd.image-verifier.v1: not found
2023-10-03T02:31:08.9848807Z         	Test:       	TestCRIImagePullTimeout/HoldingContentOpenWriter
2023-10-03T02:31:08.9861131Z === NAME  TestCRIImagePullTimeout/NoDataTransferred
2023-10-03T02:31:08.9862011Z     build_local_containerd_helper_test.go:109: 
2023-10-03T02:31:08.9863128Z         	Error Trace:	/home/runner/work/containerd/containerd/integration/build_local_containerd_helper_test.go:109
2023-10-03T02:31:08.9864564Z         	            				/home/runner/work/containerd/containerd/integration/image_pull_timeout_test.go:216
2023-10-03T02:31:08.9865157Z         	Error:      	Received unexpected error:
2023-10-03T02:31:08.9866373Z         	            	no plugins registered for io.containerd.image-verifier.v1: not found
2023-10-03T02:31:08.9867085Z         	Test:       	TestCRIImagePullTimeout/NoDataTransferred
2023-10-03T02:31:08.9867605Z     build_local_containerd_helper_test.go:120: 
2023-10-03T02:31:08.9868487Z         	Error Trace:	/home/runner/work/containerd/containerd/integration/build_local_containerd_helper_test.go:120
2023-10-03T02:31:08.9870257Z         	            				/home/runner/work/containerd/containerd/integration/image_pull_timeout_test.go:216
2023-10-03T02:31:08.9871126Z         	Error:      	Received unexpected error:
2023-10-03T02:31:08.9872738Z         	            	failed to get "io.containerd.transfer.v1" plugin: no plugins registered for io.containerd.image-verifier.v1: not found
2023-10-03T02:31:08.9873513Z         	Test:       	TestCRIImagePullTimeout/NoDataTransferred
2023-10-03T02:31:08.9874021Z --- FAIL: TestCRIImagePullTimeout (0.01s)

- adds a transfer service progress reporter to handle timeouts. Also other test fixes - fallback to local image pull when configuration conflict Signed-off-by: Tony Fang <[email protected]> Co-authored-by: Swagat Bora <[email protected]>

swagatbora90 · 2025-04-23T18:50:37Z

Skipping Registry.Mirrors test when running with Transfer Service.

swagatbora90 · 2025-04-23T20:22:04Z

There are additional test failures, but does not appear to be related to the PR changes:

Run script/test/test2annotation.sh *-gotest.json
jq: error: Could not open file *-gotest.json: No such file or directory

swagatbora90 · 2025-04-23T20:28:50Z

I don't see any report of this particular error, so may be related to my changes. Taking a look.

estesp · 2025-04-23T20:34:08Z

There are additional test failures, but does not appear to be related to the PR changes:
Run script/test/test2annotation.sh *-gotest.json
jq: error: Could not open file *-gotest.json: No such file or directory

That's a common problem after a previous stage failure on the Windows CI jobs; in this case a 503 from registry.k8s.io killed the image tests in the first integration run. We're not having the best of luck with CI stability these days

containerd 2.1.0 Welcome to the v2.1.0 release of containerd! The first minor release of containerd 2.x focuses on continued stability alongside new features and improvements. This is the first time-based released for containerd. Most the feature set and core functionality has long been stable and harderened in production environments, so now we transition to a balance of timely delivery of new functionality with the same high confidence in stability and performance. * Add no_sync option to boost boltDB performance on ephemeral environments ([containerd#10745](containerd#10745)) * Add content create event ([containerd#11006](containerd#11006)) * Erofs snapshotter and differ ([containerd#10705](containerd#10705)) * Update CRI to use transfer service for image pull by default ([containerd#8515](containerd#8515)) * Support multiple cni plugin bin dirs ([containerd#11311](containerd#11311)) * Support container restore through CRI/Kubernetes ([containerd#10365](containerd#10365)) * Add OCI/Image Volume Source support ([containerd#10579](containerd#10579)) * Enable Writable cgroups for unprivileged containers ([containerd#11131](containerd#11131)) * Fix recursive RLock() mutex acquisition ([containerd/go-cni#126](containerd/go-cni#126)) * Support CNI STATUS Verb ([containerd/go-cni#123](containerd/go-cni#123)) * Retry last registry host on 50x responses ([containerd#11484](containerd#11484)) * Multipart layer fetch ([containerd#10177](containerd#10177)) * Enable HTTP debug and trace for transfer based puller ([containerd#10762](containerd#10762)) * Add support for unpacking custom media types ([containerd#11744](containerd#11744)) * Add dial timeout field to hosts toml configuration ([containerd#11106](containerd#11106)) * Expose Pod assigned IPs to NRI plugins ([containerd#10921](containerd#10921)) * Support multiple uid/gid mappings ([containerd#10722](containerd#10722)) * Fix race between serve and immediate shutdown on the server ([containerd/ttrpc#175](containerd/ttrpc#175)) * Update FreeBSD defaults and re-organize platform defaults ([containerd#11017](containerd#11017)) * Postpone cri config deprecations to v2.2 ([containerd#11684](containerd#11684)) * Remove deprecated dynamic library plugins ([containerd#11683](containerd#11683)) * Remove the support for Schema 1 images ([containerd#11681](containerd#11681)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. * Derek McGowan * Phil Estes * Akihiro Suda * Maksym Pavlenko * Jin Dong * Wei Fu * Sebastiaan van Stijn * Samuel Karp * Mike Brown * Adrien Delorme * Austin Vazquez * Akhil Mohan * Kazuyoshi Kato * Henry Wang * Gao Xiang * ningmingxiao * Krisztian Litkey * Yang Yang * Archit Kulkarni * Chris Henzie * Iceber Gu * Alexey Lunev * Antonio Ojea * Davanum Srinivas * Marat Radchenko * Michael Zappa * Paweł Gronowski * Rodrigo Campos * Alberto Garcia Hierro * Amit Barve * Andrey Smirnov * Divya * Etienne Champetier * Kirtana Ashok * Philip Laine * QiPing Wan * fengwei0328 * zounengren * Adrian Reber * Alfred Wingate * Amal Thundiyil * Athos Ribeiro * Brian Goff * Cesar Talledo * ChengyuZhu6 * Chongyi Zheng * Craig Ingram * Danny Canter * David Son * Fupan Li * HirazawaUi * Jing Xu * Jonathan A. Sternberg * Jose Fernandez * Kaita Nakamura * Kohei Tokunaga * Lei Liu * Marco Visin * Mike Baynton * Qiyuan Liang * Sameer * Shiming Zhang * Swagat Bora * Teresaliu * Tony Fang * Tõnis Tiigi * Vered Rosen * Vinayak Goyal * bo.jiang * chriskery * luchenhan * mahmut * zhaixiaojuan * **github.com/Microsoft/hcsshim** v0.12.9 -> v0.13.0-rc.3 * **github.com/cilium/ebpf** v0.11.0 -> v0.16.0 * **github.com/containerd/cgroups/v3** v3.0.3 -> v3.0.5 * **github.com/containerd/containerd/api** v1.8.0 -> v1.9.0 * **github.com/containerd/continuity** v0.4.4 -> v0.4.5 * **github.com/containerd/go-cni** v1.1.10 -> v1.1.12 * **github.com/containerd/imgcrypt/v2** v2.0.0-rc.1 -> v2.0.1 * **github.com/containerd/otelttrpc** ea5083fda723 -> v0.1.0 * **github.com/containerd/platforms** v1.0.0-rc.0 -> v1.0.0-rc.1 * **github.com/containerd/ttrpc** v1.2.6 -> v1.2.7 * **github.com/containerd/typeurl/v2** v2.2.2 -> v2.2.3 * **github.com/containernetworking/cni** v1.2.3 -> v1.3.0 * **github.com/containernetworking/plugins** v1.5.1 -> v1.7.1 * **github.com/containers/ocicrypt** v1.2.0 -> v1.2.1 * **github.com/davecgh/go-spew** d8f796af33cc -> v1.1.1 * **github.com/fsnotify/fsnotify** v1.7.0 -> v1.9.0 * **github.com/go-jose/go-jose/v4** v4.0.4 -> v4.0.5 * **github.com/google/go-cmp** v0.6.0 -> v0.7.0 * **github.com/grpc-ecosystem/grpc-gateway/v2** v2.22.0 -> v2.26.1 * **github.com/klauspost/compress** v1.17.11 -> v1.18.0 * **github.com/mdlayher/socket** v0.4.1 -> v0.5.1 * **github.com/moby/spdystream** v0.4.0 -> v0.5.0 * **github.com/moby/sys/user** v0.3.0 -> v0.4.0 * **github.com/opencontainers/image-spec** v1.1.0 -> v1.1.1 * **github.com/opencontainers/runtime-spec** v1.2.0 -> v1.2.1 * **github.com/opencontainers/selinux** v1.11.1 -> v1.12.0 * **github.com/pelletier/go-toml/v2** v2.2.3 -> v2.2.4 * **github.com/petermattis/goid** 4fcff4a6cae7 **_new_** * **github.com/pmezard/go-difflib** 5d4384ee4fb2 -> v1.0.0 * **github.com/prometheus/client_golang** v1.20.5 -> v1.22.0 * **github.com/prometheus/common** v0.55.0 -> v0.62.0 * **github.com/sasha-s/go-deadlock** v0.3.5 **_new_** * **github.com/smallstep/pkcs7** v0.1.1 **_new_** * **github.com/stretchr/testify** v1.9.0 -> v1.10.0 * **github.com/tchap/go-patricia/v2** v2.3.1 -> v2.3.2 * **github.com/urfave/cli/v2** v2.27.5 -> v2.27.6 * **github.com/vishvananda/netlink** v1.3.0 -> 0e7078ed04c8 * **github.com/vishvananda/netns** v0.0.4 -> v0.0.5 * **go.etcd.io/bbolt** v1.3.11 -> v1.4.0 * **go.opentelemetry.io/auto/sdk** v1.1.0 **_new_** * **go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc** v0.56.0 -> v0.60.0 * **go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp** v0.56.0 -> v0.60.0 * **go.opentelemetry.io/otel** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/metric** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/sdk** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/trace** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/proto/otlp** v1.3.1 -> v1.5.0 * **golang.org/x/crypto** v0.28.0 -> v0.36.0 * **golang.org/x/exp** aacd6d4b4611 -> 2d47ceb2692f * **golang.org/x/mod** v0.21.0 -> v0.24.0 * **golang.org/x/net** v0.30.0 -> v0.38.0 * **golang.org/x/oauth2** v0.22.0 -> v0.27.0 * **golang.org/x/sync** v0.8.0 -> v0.14.0 * **golang.org/x/sys** v0.26.0 -> v0.33.0 * **golang.org/x/term** v0.25.0 -> v0.30.0 * **golang.org/x/text** v0.19.0 -> v0.23.0 * **golang.org/x/time** v0.3.0 -> v0.7.0 * **google.golang.org/genproto/googleapis/api** 5fefd90f89a9 -> 56aae31c358a * **google.golang.org/genproto/googleapis/rpc** 324edc3d5d38 -> 56aae31c358a * **google.golang.org/grpc** v1.67.1 -> v1.72.0 * **google.golang.org/protobuf** v1.35.1 -> v1.36.6 * **k8s.io/api** v0.31.2 -> v0.32.3 * **k8s.io/apimachinery** v0.31.2 -> v0.32.3 * **k8s.io/apiserver** v0.31.2 -> v0.32.3 * **k8s.io/client-go** v0.31.2 -> v0.32.3 * **k8s.io/cri-api** v0.31.2 -> v0.32.3 * **k8s.io/kubelet** v0.31.2 -> v0.32.3 * **k8s.io/utils** 18e509b52bc8 -> 3ea5e8cea738 * **sigs.k8s.io/json** bc3834ca7abd -> 9aa6b5e7a4b3 * **sigs.k8s.io/structured-merge-diff/v4** v4.4.1 -> v4.4.2 * **tags.cncf.io/container-device-interface** v0.8.0 -> v1.0.1 * **tags.cncf.io/container-device-interface/specs-go** v0.8.0 -> v1.0.0 Previous release can be found at [v2.0.0](https://github.com/containerd/containerd/releases/tag/v2.0.0) * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.

fangn2 marked this pull request as draft May 15, 2023 03:01

k8s-ci-robot added the needs-ok-to-test label May 15, 2023

fangn2 changed the title ~~[WIP]Update CRI to optional use transfer service for image pull~~ [WIP]Update CRI to optionally use transfer service for image pull May 15, 2023

fangn2 force-pushed the cri-image-transfer branch from a62e6ed to ed26de0 Compare May 15, 2023 03:07

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels May 17, 2023

mikebrow reviewed May 17, 2023

View reviewed changes

pkg/cri/config/config.go Outdated Show resolved Hide resolved

mikebrow reviewed May 17, 2023

View reviewed changes

pkg/cri/config/config.go Outdated Show resolved Hide resolved

fangn2 force-pushed the cri-image-transfer branch from ed26de0 to 3ed06cd Compare May 30, 2023 16:56

dmcgowan self-assigned this May 30, 2023

dmcgowan reviewed Jun 11, 2023

View reviewed changes

pkg/cri/server/image_pull.go Outdated Show resolved Hide resolved

fangn2 force-pushed the cri-image-transfer branch from 3ed06cd to 88d975b Compare June 12, 2023 03:12

fuweid reviewed Jun 12, 2023

View reviewed changes

fangn2 force-pushed the cri-image-transfer branch 2 times, most recently from fd0d0c3 to 113293b Compare August 9, 2023 14:41

fangn2 force-pushed the cri-image-transfer branch 6 times, most recently from 45eaa5d to 88e6abe Compare August 16, 2023 16:52

mxpv added the status/needs-update Awaiting contributor update label Sep 7, 2023

fangn2 force-pushed the cri-image-transfer branch from 88e6abe to b47bf45 Compare September 21, 2023 21:34

fangn2 force-pushed the cri-image-transfer branch from b47bf45 to 95e3822 Compare October 3, 2023 01:58

fangn2 force-pushed the cri-image-transfer branch from 95e3822 to 3faa50c Compare October 4, 2023 15:09

fangn2 mentioned this pull request Oct 25, 2023

Enhance CRI to download image by image transfer service #9291

Closed

fangn2 force-pushed the cri-image-transfer branch from 919d22b to 97fd85e Compare October 30, 2023 01:58

swagatbora90 force-pushed the cri-image-transfer branch from 8d36ef3 to b694be2 Compare April 23, 2025 18:49

estesp approved these changes Apr 23, 2025

View reviewed changes

estesp added this pull request to the merge queue Apr 23, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 23, 2025

estesp added this pull request to the merge queue Apr 23, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 23, 2025

dmcgowan approved these changes Apr 23, 2025

View reviewed changes

dmcgowan added this pull request to the merge queue Apr 23, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 23, 2025

dmcgowan added this pull request to the merge queue Apr 23, 2025

Merged via the queue into containerd:main with commit 116b987 Apr 24, 2025
59 checks passed

github-project-automation bot moved this from Review In Progress to Done in Pull Request Review Apr 24, 2025

estesp added the impact/changelog label Apr 24, 2025

This was referenced Apr 24, 2025

Add release notes for v2.1.0-rc.0 #11752

Merged

[transfer] CRI update to use transfer service #8227

Closed

smira mentioned this pull request May 7, 2025

containerd 2.1: use transfer service to pull images siderolabs/talos#10921

Closed

dmcgowan mentioned this pull request May 7, 2025

Prepare release notes for v2.1.0 #11819

Merged

clementnuss mentioned this pull request Jun 6, 2025

400 error with artifactory since containerd v2.1.0-rc.0 #11953

Closed

BobyMCbobs mentioned this pull request Jul 13, 2025

feat: add containerd-image-verifier-sigstore extension siderolabs/extensions#597

Draft

AkihiroSuda mentioned this pull request Oct 31, 2025

Refactor the image management code to use the Transfer service containerd/nerdctl#4573

Closed

KCSesh mentioned this pull request Nov 10, 2025

Image pull problems on Bottlerocket OS 1.49.0 (aws-k8s-1.34) bottlerocket-os/bottlerocket#4677

Open

This was referenced Nov 18, 2025

Bottlerocket EKS v1.34 ECR Image Pull Fails for the private ECR images bottlerocket-os/bottlerocket#4707

Closed

Bottlerocket Image Pull Issues on aws-k8s-1.34* and aws-k8s-1.33* variants. bottlerocket-os/bottlerocket#4709

Closed

Update CRI to use transfer service for image pull by default #8515

Update CRI to use transfer service for image pull by default #8515

Uh oh!

Conversation

fangn2 commented May 15, 2023

Uh oh!

k8s-ci-robot commented May 15, 2023

Uh oh!

mikebrow commented May 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fuweid Jun 12, 2023

Choose a reason for hiding this comment

Uh oh!

fangn2 Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

fangn2 Aug 11, 2023

Choose a reason for hiding this comment

Uh oh!

estesp commented Oct 3, 2023

Uh oh!

swagatbora90 commented Apr 23, 2025

Uh oh!

Uh oh!

swagatbora90 commented Apr 23, 2025

Uh oh!

swagatbora90 commented Apr 23, 2025

Uh oh!

estesp commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants