Add StreamProcessors for handling media types of content streams by crosbymichael · Pull Request #3482 · containerd/containerd

crosbymichael · 2019-08-02T19:32:20Z

StreamProcessors are used with the content operations that will allow people to create binaries that accept a content stream with a media type and output a content stream for another media type. This can be used with the encryption code to keep encryption primitives out of the daemon and extend the daemon for custom media types without having to do a full release.

Example config where content types are pipelined.

[[stream_processors]]
        id = "cat"
        accepts = ["application/vnd.docker.image.rootfs.diff.tar.gzip"]
        returns = "test"
        path = "cat"

[[stream_processors]]
        id = "pigz"
        accepts = ["test"]
        returns = "application/vnd.oci.image.layer.v1.tar"
        path = "unpigz"
        args = ["-d", "-c"]

theopenlab-ci · 2019-08-02T19:37:03Z

Build succeeded.

containerd-build-arm64 : FAILURE in 3m 53s (non-voting)

theopenlab-ci · 2019-08-02T20:17:45Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 41s (non-voting)

codecov-io · 2019-08-02T21:28:21Z

Codecov Report

Merging #3482 into master will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #3482      +/-   ##
==========================================
- Coverage   44.27%   44.26%   -0.01%     
==========================================
  Files         124      124              
  Lines       13672    13675       +3     
==========================================
  Hits         6053     6053              
- Misses       6684     6687       +3     
  Partials      935      935

Flag	Coverage Δ
#linux	`48.04% <0%> (-0.01%)`	⬇️
#windows	`39.74% <0%> (-0.02%)`	⬇️

Impacted Files	Coverage Δ
services/server/server.go	`1.22% <0%> (-0.02%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ff1ef4...552a0b1. Read the comment docs.

stevvooe

LGTM on design

AkihiroSuda · 2019-08-03T09:24:11Z

cmd.Env = os.Environ()?

also needs Pdeathsig?

theopenlab-ci · 2019-08-05T14:42:37Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 20s (non-voting)

theopenlab-ci · 2019-08-05T15:14:50Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 20s (non-voting)

theopenlab-ci · 2019-08-05T19:53:24Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 46s (non-voting)

theopenlab-ci · 2019-08-05T20:50:56Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 37s (non-voting)

theopenlab-ci · 2019-08-06T18:25:45Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 42s (non-voting)

theopenlab-ci · 2019-08-06T18:43:44Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 25s (non-voting)

theopenlab-ci · 2019-08-06T19:11:57Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 15s (non-voting)

theopenlab-ci · 2019-08-06T19:49:17Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 28s (non-voting)

theopenlab-ci · 2019-08-06T21:17:19Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 40s (non-voting)

stefanberger · 2019-08-06T21:57:23Z

works for me

crosbymichael · 2019-08-07T17:25:11Z

ping @dmcgowan

AkihiroSuda · 2019-08-07T18:16:12Z

LGTM but cmd.Env = append(os.Environ(), ...) seems needed

Signed-off-by: Michael Crosby <[email protected]>

crosbymichael · 2019-08-07T18:38:10Z

@AkihiroSuda updated

theopenlab-ci · 2019-08-07T18:45:25Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 7m 29s (non-voting)

estesp

LGTM

crosbymichael force-pushed the apply-remote branch 2 times, most recently from 5576674 to 3c15b31 Compare August 2, 2019 20:01

stevvooe approved these changes Aug 2, 2019

View reviewed changes

AkihiroSuda reviewed Aug 3, 2019

View reviewed changes

Comment thread diff/stream.go Outdated

AkihiroSuda reviewed Aug 3, 2019

View reviewed changes

crosbymichael force-pushed the apply-remote branch from 3c15b31 to de7b67d Compare August 5, 2019 14:33

crosbymichael force-pushed the apply-remote branch from af4ce3c to d4b54d7 Compare August 5, 2019 20:41

crosbymichael force-pushed the apply-remote branch from d4b54d7 to 941f083 Compare August 6, 2019 18:16

crosbymichael force-pushed the apply-remote branch from 941f083 to 60d83f4 Compare August 6, 2019 18:34

crosbymichael force-pushed the apply-remote branch from 60d83f4 to 4ec5842 Compare August 6, 2019 19:02

crosbymichael force-pushed the apply-remote branch from 4ec5842 to 2ec0b0d Compare August 6, 2019 19:40

crosbymichael added 4 commits August 7, 2019 18:35

Add StreamProcessor for apply

97a9877

Signed-off-by: Michael Crosby <[email protected]>

Add server config for stream processors

3668237

Signed-off-by: Michael Crosby <[email protected]>

Use fds and pass Payloads over diff api

f867401

Signed-off-by: Michael Crosby <[email protected]>

Add windows apply code

134d3c8

Signed-off-by: Michael Crosby <[email protected]>

crosbymichael added 4 commits August 7, 2019 18:35

Use named pipes for windows processors

e1489f9

Signed-off-by: Michael Crosby <[email protected]>

Pass apply opts through rootfs/* code

26b9061

Signed-off-by: Michael Crosby <[email protected]>

Add unpack opts

3fded74

Signed-off-by: Michael Crosby <[email protected]>

Handle stderr in stream processors

552a0b1

Signed-off-by: Michael Crosby <[email protected]>

crosbymichael force-pushed the apply-remote branch from 378b3ee to 552a0b1 Compare August 7, 2019 18:37

AkihiroSuda approved these changes Aug 7, 2019

View reviewed changes

estesp approved these changes Aug 7, 2019

View reviewed changes

estesp merged commit 4e6e61c into containerd:master Aug 7, 2019

crosbymichael deleted the apply-remote branch August 7, 2019 20:27

lumjjb mentioned this pull request Sep 18, 2019

Taming media types opencontainers/image-spec#791

Open

lumjjb mentioned this pull request Oct 9, 2019

Proposal to add Encrypted Layer Mediatype opencontainers/image-spec#775

Open

estesp mentioned this pull request Oct 18, 2019

Add non-core project "imgcrypt" to containerd organization containerd/project#37

Closed

13 tasks

Conversation

crosbymichael commented Aug 2, 2019

Uh oh!

theopenlab-ci Bot commented Aug 2, 2019

Uh oh!

theopenlab-ci Bot commented Aug 2, 2019

Uh oh!

codecov-io commented Aug 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stevvooe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AkihiroSuda Aug 3, 2019

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Aug 3, 2019

Choose a reason for hiding this comment

Uh oh!

theopenlab-ci Bot commented Aug 5, 2019

Uh oh!

theopenlab-ci Bot commented Aug 5, 2019

Uh oh!

theopenlab-ci Bot commented Aug 5, 2019

Uh oh!

theopenlab-ci Bot commented Aug 5, 2019

Uh oh!

theopenlab-ci Bot commented Aug 6, 2019

Uh oh!

theopenlab-ci Bot commented Aug 6, 2019

Uh oh!

theopenlab-ci Bot commented Aug 6, 2019

Uh oh!

theopenlab-ci Bot commented Aug 6, 2019

Uh oh!

theopenlab-ci Bot commented Aug 6, 2019

Uh oh!

stefanberger commented Aug 6, 2019

Uh oh!

crosbymichael commented Aug 7, 2019

Uh oh!

AkihiroSuda commented Aug 7, 2019

Uh oh!

crosbymichael commented Aug 7, 2019

Uh oh!

theopenlab-ci Bot commented Aug 7, 2019

Uh oh!

estesp left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov-io commented Aug 2, 2019 •

edited

Loading