Skip to content

compression: add support for the zstd algorithm#4809

Merged
estesp merged 1 commit intocontainerd:masterfrom
giuseppe:zstd-support
Dec 7, 2020
Merged

compression: add support for the zstd algorithm#4809
estesp merged 1 commit intocontainerd:masterfrom
giuseppe:zstd-support

Conversation

@giuseppe
Copy link
Copy Markdown
Contributor

@giuseppe giuseppe commented Dec 4, 2020

zstd is a compression algorithm that has a very fast decoder, while
providing also good compression ratios. The fast decoder makes it
suitable for container images, as decompressing the tarballs is a very
expensive operation.

opencontainers/image-spec#788 added support
for zstd to the OCI image specs.

Signed-off-by: Giuseppe Scrivano [email protected]

@k8s-ci-robot
Copy link
Copy Markdown

Hi @giuseppe. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Dec 4, 2020

Build succeeded.

@thaJeztah
Copy link
Copy Markdown
Member

does this depend on it being in an image spec release, to not be out of spec? opencontainers/image-spec#803

@giuseppe
Copy link
Copy Markdown
Contributor Author

giuseppe commented Dec 6, 2020

does this depend on it being in an image spec release, to not be out of spec? opencontainers/image-spec#803

not sure it should block on a new release of the image specs and if there is any value in it (we should probably try to get a new tag, last release was 3 years ago 😞). It won't affect existing images and how they are handled. The only difference will be that a zstd layer will be decompressed instead of returning an error.

The main reason for such PR is that I am experimenting with zstd (containers/image#1084) and I'd like to not break interoperability with Moby/containerd.

@dmcgowan dmcgowan added this to the 1.5 milestone Dec 6, 2020
@dmcgowan
Copy link
Copy Markdown
Member

dmcgowan commented Dec 6, 2020

Adding this to the 1.5 milestone, I think adding default support for zstd makes sense without having to rely on configuring a stream processor.

I do have one question on this change though, where was application/vnd.docker.image.rootfs.diff.tar.zstd introduced as a valid/known media type? I couldn't find this referenced in any docker/moby code. What is the scenario where this is necessary over just using the OCI type?

+1 on not waiting for an image-spec since this is only relying on a common sense addition of +zstd

@fuweid
Copy link
Copy Markdown
Member

fuweid commented Dec 7, 2020

+1 to support zstd without stream-processor

@giuseppe
Copy link
Copy Markdown
Contributor Author

giuseppe commented Dec 7, 2020

I do have one question on this change though, where was application/vnd.docker.image.rootfs.diff.tar.zstd introduced as a valid/known media type? I couldn't find this referenced in any docker/moby code. What is the scenario where this is necessary over just using the OCI type?

I've added it only for completeness as it is was done for gzip. I agree it is not needed now, if we'll need it in future then we can add.

Pushed a new version without the application/vnd.docker.image.rootfs.diff.tar.zstd change.

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Dec 7, 2020

Build succeeded.

zstd is a compression algorithm that has a very fast decoder, while
providing also good compression ratios.  The fast decoder makes it
suitable for container images, as decompressing the tarballs is a very
expensive operation.

opencontainers/image-spec#788 added support
for zstd to the OCI image specs.

Signed-off-by: Giuseppe Scrivano <[email protected]>
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Dec 7, 2020

Build succeeded.

Copy link
Copy Markdown
Member

@dmcgowan dmcgowan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Member

@estesp estesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants