Skip to content

Containerd unable to launch images created with buildpacks (label key and value greater than maximum size (4096 bytes), key: io.buildpa: invalid argument) #6123

@sambhav

Description

@sambhav

Description

https://buildpacks.io/ is a container building tool that relies on OCI image labels to store some configuration metadata in the container image in order to do various operations related to caching/rebuilding and storing provenance metadata about the built container. These labels can sometimes be larger than 4096B which is fine given that no limits are placed on the image label size as per the OCI image spec per https://github.com/opencontainers/image-spec/blob/main/config.md and https://github.com/opencontainers/image-spec/blob/main/annotations.md#rules.

With a recent change, image labels were propagated to the container being run in PR https://github.com/containerd/containerd/pull/6012/files

This causes various issues with running both the builder images used by buildpacks and the output images generated by buildpacks. For eg. buildpacks/pack#1309

Would it be possible for containerd to throw a warning and drop large labels in the container rather than failing to run it whatsoever.

NOTE: This might impact a fairly large number of images as buildpacks are used by a variety of different platforms -

https://github.com/buildpacks/community/blob/main/ADOPTERS.md

Steps to reproduce the issue

  1. Try running the image paketobuildpacks/builder:full with containerd 1.5.6 or later fails
  2. The same works with previous versions

Describe the results you received and expected

See buildpacks/pack#1309

When containerd tries to start a container which was built with the paketobuildpacks/builder:full image, it fails with an error:

label key and value greater than maximum size (4096 bytes), key: io.buildpa: invalid argument
This seems to originate here: jessvalarezo/containerd@18c4322/labels/validate.go

Everything works fine on containerd v1.4.11. I had a look at changelogs and I think it's this one (introduced in 1.5.6):

#5938
#6012 (files)

What version of containerd are you using?

1.5.7

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions