Add content create event #11006

phillebaba · 2024-11-14T23:31:04Z

This change adds a new create event to the API and publishes an event when a blob is successfully committed to the store.

Fixes #10508

k8s-ci-robot · 2024-11-14T23:31:14Z

Hi @phillebaba. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dmcgowan · 2024-11-15T23:46:28Z

api/events/content.proto

 option (containerd.types.fieldpath_all) = true;

+message ContentCreate {
+	string digest = 1;


We should include size here as well

It would be nice to include size on ContentDelete, but that would cause extra effort on Delete that isn't worth it. The Commit calls will have the size anywhere it has the digest.

We could probably add it to the delete event but it would require calling info on every blob we delete. It might be good to leave that for a separate PR?

dmcgowan · 2024-11-16T00:00:04Z

Make sure to run make vendor after changes in /api

dmcgowan

Run make vendor to get the change picked up by the main module

phillebaba · 2024-11-19T19:42:38Z

Would vendor pick up the change when it points to a specific version of containerd/api? My understanding is that we would have to cut a release of the api module first before we could use the new event type.

dmcgowan · 2024-11-19T20:21:45Z

make vendor should do the right thing. It should add a replace statement and vendor the change in. Tagging and removing the replace is part of the release process, but you don't need to handle that now.

phillebaba · 2024-11-19T23:09:28Z

Ok I understand the issue now. I was having problems generating the protos initially which lead to a partial state. And because replace is set conditionally when generating protos it wasn't done the second time around when I ran it. I fixed it now and generated vendors so things should be good.

dmcgowan · 2024-11-19T23:52:44Z

core/metadata/content.go

-	return innerErr
+	if publisher := nw.db.Publisher(ctx); publisher != nil {
+		if err := publisher.Publish(ctx, "/content/create", &eventstypes.ContentCreate{
+			Digest: expected.String(),


Either dgst should be declared to stay in scope here or expected updated after nw.commit. expected is not required to be set though.

Signed-off-by: Philip Laine <[email protected]>

phillebaba · 2025-04-11T08:20:34Z

@estesp rebased and fixed the merge conflict. There are now tests that are now passing but the errors seem to be unrelated to my changes, are these flakes that are known to occur? I ran some of them locally without any issues. I can't retrigger them, so I can't figure out if they will pass on a second run.

estesp · 2025-04-11T13:27:22Z

@estesp rebased and fixed the merge conflict. There are now tests that are now passing but the errors seem to be unrelated to my changes, are these flakes that are known to occur? I ran some of them locally without any issues. I can't retrigger them, so I can't figure out if they will pass on a second run.

Yeah, there is a known GH actions image runner issue currently impacting a lot of project tests. No action needed on your part, but there should be no further changes required and we'll follow up on the CI flakes and get this merged when we can.

containerd 2.1.0 Welcome to the v2.1.0 release of containerd! The first minor release of containerd 2.x focuses on continued stability alongside new features and improvements. This is the first time-based released for containerd. Most the feature set and core functionality has long been stable and harderened in production environments, so now we transition to a balance of timely delivery of new functionality with the same high confidence in stability and performance. * Add no_sync option to boost boltDB performance on ephemeral environments ([containerd#10745](containerd#10745)) * Add content create event ([containerd#11006](containerd#11006)) * Erofs snapshotter and differ ([containerd#10705](containerd#10705)) * Update CRI to use transfer service for image pull by default ([containerd#8515](containerd#8515)) * Support multiple cni plugin bin dirs ([containerd#11311](containerd#11311)) * Support container restore through CRI/Kubernetes ([containerd#10365](containerd#10365)) * Add OCI/Image Volume Source support ([containerd#10579](containerd#10579)) * Enable Writable cgroups for unprivileged containers ([containerd#11131](containerd#11131)) * Fix recursive RLock() mutex acquisition ([containerd/go-cni#126](containerd/go-cni#126)) * Support CNI STATUS Verb ([containerd/go-cni#123](containerd/go-cni#123)) * Retry last registry host on 50x responses ([containerd#11484](containerd#11484)) * Multipart layer fetch ([containerd#10177](containerd#10177)) * Enable HTTP debug and trace for transfer based puller ([containerd#10762](containerd#10762)) * Add support for unpacking custom media types ([containerd#11744](containerd#11744)) * Add dial timeout field to hosts toml configuration ([containerd#11106](containerd#11106)) * Expose Pod assigned IPs to NRI plugins ([containerd#10921](containerd#10921)) * Support multiple uid/gid mappings ([containerd#10722](containerd#10722)) * Fix race between serve and immediate shutdown on the server ([containerd/ttrpc#175](containerd/ttrpc#175)) * Update FreeBSD defaults and re-organize platform defaults ([containerd#11017](containerd#11017)) * Postpone cri config deprecations to v2.2 ([containerd#11684](containerd#11684)) * Remove deprecated dynamic library plugins ([containerd#11683](containerd#11683)) * Remove the support for Schema 1 images ([containerd#11681](containerd#11681)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. * Derek McGowan * Phil Estes * Akihiro Suda * Maksym Pavlenko * Jin Dong * Wei Fu * Sebastiaan van Stijn * Samuel Karp * Mike Brown * Adrien Delorme * Austin Vazquez * Akhil Mohan * Kazuyoshi Kato * Henry Wang * Gao Xiang * ningmingxiao * Krisztian Litkey * Yang Yang * Archit Kulkarni * Chris Henzie * Iceber Gu * Alexey Lunev * Antonio Ojea * Davanum Srinivas * Marat Radchenko * Michael Zappa * Paweł Gronowski * Rodrigo Campos * Alberto Garcia Hierro * Amit Barve * Andrey Smirnov * Divya * Etienne Champetier * Kirtana Ashok * Philip Laine * QiPing Wan * fengwei0328 * zounengren * Adrian Reber * Alfred Wingate * Amal Thundiyil * Athos Ribeiro * Brian Goff * Cesar Talledo * ChengyuZhu6 * Chongyi Zheng * Craig Ingram * Danny Canter * David Son * Fupan Li * HirazawaUi * Jing Xu * Jonathan A. Sternberg * Jose Fernandez * Kaita Nakamura * Kohei Tokunaga * Lei Liu * Marco Visin * Mike Baynton * Qiyuan Liang * Sameer * Shiming Zhang * Swagat Bora * Teresaliu * Tony Fang * Tõnis Tiigi * Vered Rosen * Vinayak Goyal * bo.jiang * chriskery * luchenhan * mahmut * zhaixiaojuan * **github.com/Microsoft/hcsshim** v0.12.9 -> v0.13.0-rc.3 * **github.com/cilium/ebpf** v0.11.0 -> v0.16.0 * **github.com/containerd/cgroups/v3** v3.0.3 -> v3.0.5 * **github.com/containerd/containerd/api** v1.8.0 -> v1.9.0 * **github.com/containerd/continuity** v0.4.4 -> v0.4.5 * **github.com/containerd/go-cni** v1.1.10 -> v1.1.12 * **github.com/containerd/imgcrypt/v2** v2.0.0-rc.1 -> v2.0.1 * **github.com/containerd/otelttrpc** ea5083fda723 -> v0.1.0 * **github.com/containerd/platforms** v1.0.0-rc.0 -> v1.0.0-rc.1 * **github.com/containerd/ttrpc** v1.2.6 -> v1.2.7 * **github.com/containerd/typeurl/v2** v2.2.2 -> v2.2.3 * **github.com/containernetworking/cni** v1.2.3 -> v1.3.0 * **github.com/containernetworking/plugins** v1.5.1 -> v1.7.1 * **github.com/containers/ocicrypt** v1.2.0 -> v1.2.1 * **github.com/davecgh/go-spew** d8f796af33cc -> v1.1.1 * **github.com/fsnotify/fsnotify** v1.7.0 -> v1.9.0 * **github.com/go-jose/go-jose/v4** v4.0.4 -> v4.0.5 * **github.com/google/go-cmp** v0.6.0 -> v0.7.0 * **github.com/grpc-ecosystem/grpc-gateway/v2** v2.22.0 -> v2.26.1 * **github.com/klauspost/compress** v1.17.11 -> v1.18.0 * **github.com/mdlayher/socket** v0.4.1 -> v0.5.1 * **github.com/moby/spdystream** v0.4.0 -> v0.5.0 * **github.com/moby/sys/user** v0.3.0 -> v0.4.0 * **github.com/opencontainers/image-spec** v1.1.0 -> v1.1.1 * **github.com/opencontainers/runtime-spec** v1.2.0 -> v1.2.1 * **github.com/opencontainers/selinux** v1.11.1 -> v1.12.0 * **github.com/pelletier/go-toml/v2** v2.2.3 -> v2.2.4 * **github.com/petermattis/goid** 4fcff4a6cae7 **_new_** * **github.com/pmezard/go-difflib** 5d4384ee4fb2 -> v1.0.0 * **github.com/prometheus/client_golang** v1.20.5 -> v1.22.0 * **github.com/prometheus/common** v0.55.0 -> v0.62.0 * **github.com/sasha-s/go-deadlock** v0.3.5 **_new_** * **github.com/smallstep/pkcs7** v0.1.1 **_new_** * **github.com/stretchr/testify** v1.9.0 -> v1.10.0 * **github.com/tchap/go-patricia/v2** v2.3.1 -> v2.3.2 * **github.com/urfave/cli/v2** v2.27.5 -> v2.27.6 * **github.com/vishvananda/netlink** v1.3.0 -> 0e7078ed04c8 * **github.com/vishvananda/netns** v0.0.4 -> v0.0.5 * **go.etcd.io/bbolt** v1.3.11 -> v1.4.0 * **go.opentelemetry.io/auto/sdk** v1.1.0 **_new_** * **go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc** v0.56.0 -> v0.60.0 * **go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp** v0.56.0 -> v0.60.0 * **go.opentelemetry.io/otel** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/metric** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/sdk** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/otel/trace** v1.31.0 -> v1.35.0 * **go.opentelemetry.io/proto/otlp** v1.3.1 -> v1.5.0 * **golang.org/x/crypto** v0.28.0 -> v0.36.0 * **golang.org/x/exp** aacd6d4b4611 -> 2d47ceb2692f * **golang.org/x/mod** v0.21.0 -> v0.24.0 * **golang.org/x/net** v0.30.0 -> v0.38.0 * **golang.org/x/oauth2** v0.22.0 -> v0.27.0 * **golang.org/x/sync** v0.8.0 -> v0.14.0 * **golang.org/x/sys** v0.26.0 -> v0.33.0 * **golang.org/x/term** v0.25.0 -> v0.30.0 * **golang.org/x/text** v0.19.0 -> v0.23.0 * **golang.org/x/time** v0.3.0 -> v0.7.0 * **google.golang.org/genproto/googleapis/api** 5fefd90f89a9 -> 56aae31c358a * **google.golang.org/genproto/googleapis/rpc** 324edc3d5d38 -> 56aae31c358a * **google.golang.org/grpc** v1.67.1 -> v1.72.0 * **google.golang.org/protobuf** v1.35.1 -> v1.36.6 * **k8s.io/api** v0.31.2 -> v0.32.3 * **k8s.io/apimachinery** v0.31.2 -> v0.32.3 * **k8s.io/apiserver** v0.31.2 -> v0.32.3 * **k8s.io/client-go** v0.31.2 -> v0.32.3 * **k8s.io/cri-api** v0.31.2 -> v0.32.3 * **k8s.io/kubelet** v0.31.2 -> v0.32.3 * **k8s.io/utils** 18e509b52bc8 -> 3ea5e8cea738 * **sigs.k8s.io/json** bc3834ca7abd -> 9aa6b5e7a4b3 * **sigs.k8s.io/structured-merge-diff/v4** v4.4.1 -> v4.4.2 * **tags.cncf.io/container-device-interface** v0.8.0 -> v1.0.1 * **tags.cncf.io/container-device-interface/specs-go** v0.8.0 -> v1.0.0 Previous release can be found at [v2.0.0](https://github.com/containerd/containerd/releases/tag/v2.0.0) * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.

k8s-ci-robot added needs-ok-to-test size/L labels Nov 14, 2024

dosubot bot added the kind/feature label Nov 14, 2024

phillebaba force-pushed the feature/api-content-create branch from 8dc2504 to fbae431 Compare November 15, 2024 20:10

dmcgowan reviewed Nov 15, 2024

View reviewed changes

phillebaba force-pushed the feature/api-content-create branch 6 times, most recently from 9185550 to f557de7 Compare November 19, 2024 19:01

dmcgowan approved these changes Nov 19, 2024

View reviewed changes

dmcgowan requested changes Nov 19, 2024

View reviewed changes

phillebaba force-pushed the feature/api-content-create branch from f557de7 to 76e2481 Compare November 19, 2024 23:08

phillebaba changed the title ~~Add content create event to api~~ Add content create event Nov 19, 2024

dmcgowan reviewed Nov 19, 2024

View reviewed changes

phillebaba force-pushed the feature/api-content-create branch from 76e2481 to 6ef2af0 Compare November 20, 2024 06:30

phillebaba requested a review from dmcgowan November 20, 2024 07:37

dmcgowan approved these changes Nov 23, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase label Jan 6, 2025

phillebaba force-pushed the feature/api-content-create branch from 6ef2af0 to 1f647ee Compare January 6, 2025 09:05

k8s-ci-robot removed the needs-rebase label Jan 6, 2025

k8s-ci-robot added the needs-rebase label Mar 6, 2025

dmcgowan added this to the 2.1 milestone Mar 6, 2025

phillebaba force-pushed the feature/api-content-create branch from 1f647ee to a95cc8a Compare March 7, 2025 21:18

k8s-ci-robot removed the do-not-merge/contains-merge-commits label Apr 11, 2025

phillebaba force-pushed the feature/api-content-create branch from 0a38e82 to c8dc7cc Compare April 11, 2025 07:36

k8s-ci-robot added size/L and removed size/XXL labels Apr 11, 2025

phillebaba force-pushed the feature/api-content-create branch from c8dc7cc to 9a91e7d Compare April 11, 2025 07:39

Add content create event to api

752914b

Signed-off-by: Philip Laine <[email protected]>

phillebaba force-pushed the feature/api-content-create branch from 9a91e7d to 752914b Compare April 11, 2025 07:40

k8s-ci-robot added size/XL and removed size/L labels Apr 11, 2025

estesp approved these changes Apr 11, 2025

View reviewed changes

github-project-automation bot moved this from Merge on Green to Review In Progress in Pull Request Review Apr 11, 2025

estesp added this pull request to the merge queue Apr 11, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 11, 2025

estesp added this pull request to the merge queue Apr 11, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 11, 2025

estesp added this pull request to the merge queue Apr 14, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 14, 2025

estesp added this pull request to the merge queue Apr 17, 2025

Merged via the queue into containerd:main with commit 448490b Apr 17, 2025
56 checks passed

github-project-automation bot moved this from Review In Progress to Done in Pull Request Review Apr 17, 2025

dmcgowan mentioned this pull request Apr 23, 2025

Add release notes for api v1.9.0-rc.0 #11751

Merged

1 task

dmcgowan added the impact/changelog label Apr 24, 2025

dmcgowan mentioned this pull request Apr 25, 2025

Add release notes for v2.1.0-rc.0 #11752

Merged

This was referenced May 7, 2025

Prepare release notes for api/v1.9.0 #11812

Merged

Prepare release notes for v2.1.0 #11819

Merged

estesp mentioned this pull request Jul 31, 2025

Include /dev/shm for single container pods in checkpoint archive #12143

Open

Add content create event #11006

Add content create event #11006

Uh oh!

Conversation

phillebaba commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Nov 14, 2024

Uh oh!

dmcgowan Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

phillebaba Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

dmcgowan commented Nov 16, 2024

Uh oh!

dmcgowan left a comment

Choose a reason for hiding this comment

Uh oh!

phillebaba commented Nov 19, 2024

Uh oh!

dmcgowan commented Nov 19, 2024

Uh oh!

phillebaba commented Nov 19, 2024

Uh oh!

dmcgowan Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

phillebaba commented Apr 11, 2025

Uh oh!

estesp commented Apr 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

phillebaba commented Nov 14, 2024 •

edited

Loading