Add reader/writer to for oci-archive multi image #1072

QiWang19 · 2020-11-02T15:08:36Z

Add read/writer with helpers to allow podman save/load multi oci-archive images.
work with PR: containers/podman#8218
close: containers/podman#4646

Signed-off-by: Qi Wang [email protected]

vrothberg · 2020-11-03T11:44:52Z

oci/archive/reader.go

+	tempDirOCIRef
+}
+
+func CreateUntarTempDirReader(src string, sys *types.SystemContext) (*reader, error) {


Docs are missing

oci/archive/reader.go

vrothberg · 2020-11-03T11:54:37Z

oci/archive/reader.go

+
+// List returns a list of ReferenceWrapper for images in the reader.
+// the ImageReferences of the ReferenceWrapper are valid only until the Reader is closed.
+func (r *reader) List() ([]ReferenceWrapper, error) {


Looking at docker-archive, I think we should not create a custom ReferenceWrapper but return a [][]types.ImageReference. With the first array determining the amount of images and the second one the "names".

docker-archive supports loading by index and by name and I think that's what we should do here as well to be consistent. The docker-archive parts are in docker/archive/reader.go.

@vrothberg Could you elaborate on what's loading by index and name?
lloading by index, the usage is load -i images.tar:@source-index, the source-index should be one of the digests from manifests in index.json, like sha256:e2cb60c4b307f3253254276da1d93e5edb32c3ddc0c40b40333def88eb48bf5f?

A poadman load -i always expects a file, so we can't use the transport syntax. But we can individually pull/extract images from an archive. To extract an image (note we can do this with podman pull as well), we need to specify which one. We can specify the image either via a docker-reference (without digest) or by it's index in the list of manifests.

Examples are the man page (https://github.com/containers/image/blob/master/docs/containers-transports.5.md#docker-archivepathdocker-referencesource-index) or the tests in podman that exercise this code (containers/podman@7fea46752cbf#diff-36d4ee0b4d8927b38ddbff8a5648df51ebf0e9289fcb0d37e0dedc2ffa3ee161R173).

QiWang19 · 2020-11-16T19:00:21Z

@vrothberg @mtrmac PTAL

vrothberg

Nice work, @QiWang19

vrothberg · 2020-11-17T09:03:52Z

oci/archive/oci_transport.go

+	if strings.HasPrefix(image, "@") {
+		idx, err := strconv.Atoi(image[1:])
+		if err != nil {
+			return nil, errors.Wrapf(err, "Invalid source index %s", image)


Suggested change

return nil, errors.Wrapf(err, "Invalid source index %s", image)

return nil, errors.Wrapf(err, "Invalid source index @%s: not an integer", image[1:])

vrothberg · 2020-11-17T09:08:13Z

oci/layout/oci_transport.go

 // We do not expose an API supplying the resolvedDir; we could, but recomputing it
 // is generally cheap enough that we prefer being confident about the properties of resolvedDir.
-func NewReference(dir, image string) (types.ImageReference, error) {
+func NewReference(dir, image string, sourceIndex int) (types.ImageReference, error) {


That's technically breaking the API. Can we let this default to 0?

Maybe a new NewReferenceWithIndex?

rhatdan · 2020-11-19T20:59:14Z

oci/archive/writer.go

+	"os"
+
+	"github.com/containers/image/v5/internal/tmpdir"
+


Should not have blank line.

rhatdan · 2020-11-19T21:01:33Z

oci/layout/oci_transport.go

+//
+// We do not expose an API supplying the resolvedDir; we could, but recomputing it
+// is generally cheap enough that we prefer being confident about the properties of resolvedDir.
+func NewReferenceWithIndex(dir, image string, sourceIndex int) (types.ImageReference, error) {


NewReference() Should be calling this function, rather then duplicating the code.

QiWang19 · 2020-11-23T15:49:34Z

@vrothberg @mtrmac PTAL

mtrmac

Thanks, this is a definitely a welcome feature.

oci/archive/reader.go

oci/layout/oci_transport.go

oci/archive/oci_transport.go

oci/layout/oci_transport.go

QiWang19 · 2020-12-01T21:59:08Z

@mtrmac PTAL

QiWang19 · 2020-12-03T19:54:01Z

@mtrmac PTAL

Add read/writer with helpers to allow podman save/load multi oci-archive images. Allow read oci-archive using source_index to point to the an inedx from oci-archive manifest. Signed-off-by: Qi Wang <[email protected]>

QiWang19 · 2020-12-09T19:42:58Z

@mtrmac PTAL.

mtrmac

Just a quick partial read for now I’m afraid, focusing at least on the public API.

I’d still prefer to re-implement ociArchiveImage{Source,Destination} in terms of the Reader/Writer, rather than making Reader/Writer a completely separate path that creates oci/layout references, per #1072 (comment) .

docs/containers-transports.5.md

oci/archive/oci_dest.go

oci/archive/oci_transport.go

oci/archive/reader.go

oci/archive/writer.go

oci/layout/oci_transport.go

mtrmac · 2020-12-10T23:14:33Z

oci/archive/writer.go

+
+// Close converts the data about images in the temp directory to the archive and
+// deletes temporary files associated with the Writer
+func (w *Writer) Close() error {


Should this have a context.Context, to eventually allow aborting that write? OTOH that would make this not conform to io.Closer.

Ugh.

Alternatively, pass a context.Context to NewWriter would be an option, violating the usual rules of use of context.Context (OTOH it wouldn’t be the first time IIRC).

*shrug* Maybe we should defer solving that for now, we can always add a CloseWithContext or something like that when actually adding a cancellable implementation.

mtrmac

A full review now.

oci/archive/oci_transport.go

oci/archive/reader.go

oci/layout/oci_transport.go

oci/layout/oci_transport_test.go

Signed-off-by: Qi Wang <[email protected]>

QiWang19 · 2021-01-04T16:22:59Z

@mtrmac PTAL

rhatdan · 2021-03-07T11:31:39Z

@QiWang19 @mtrmac Any movement on this?

rhatdan · 2021-03-23T18:31:14Z

@vrothberg PTAL
@QiWang19 Could you rebase and push.

vrothberg reviewed Nov 3, 2020

View reviewed changes

QiWang19 force-pushed the multi-descriptor branch 4 times, most recently from 194951a to 3ef46bb Compare November 13, 2020 20:06

vrothberg reviewed Nov 17, 2020

View reviewed changes

QiWang19 force-pushed the multi-descriptor branch from 3ef46bb to 52ca48d Compare November 17, 2020 19:51

rhatdan reviewed Nov 19, 2020

View reviewed changes

oci/archive/writer.go Outdated

"os"

"github.com/containers/image/v5/internal/tmpdir"

Copy link

Member

rhatdan Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not have blank line.

rhatdan reviewed Nov 19, 2020

View reviewed changes

rhatdan approved these changes Nov 19, 2020

View reviewed changes

QiWang19 force-pushed the multi-descriptor branch from 52ca48d to 1e6dc9b Compare November 20, 2020 21:28

mtrmac reviewed Nov 24, 2020

View reviewed changes

oci/layout/oci_transport.go Show resolved Hide resolved

QiWang19 force-pushed the multi-descriptor branch 2 times, most recently from bf2318e to 94e6444 Compare December 1, 2020 21:57

QiWang19 force-pushed the multi-descriptor branch 3 times, most recently from 85815ec to 2196004 Compare December 1, 2020 22:57

QiWang19 force-pushed the multi-descriptor branch from 2196004 to 8f570d1 Compare December 8, 2020 15:55

Add reader/writer to for oci-archive multi image

e3f17d3

Add read/writer with helpers to allow podman save/load multi oci-archive images. Allow read oci-archive using source_index to point to the an inedx from oci-archive manifest. Signed-off-by: Qi Wang <[email protected]>

QiWang19 force-pushed the multi-descriptor branch from 8f570d1 to e3f17d3 Compare December 9, 2020 19:42

mtrmac reviewed Dec 10, 2020

View reviewed changes

mtrmac reviewed Dec 11, 2020

View reviewed changes

mtrmac mentioned this pull request Dec 19, 2020

RFE: create multi-arch manifests containers/skopeo#1136

Open

QiWang19 force-pushed the multi-descriptor branch from 633420a to e8670de Compare December 21, 2020 20:55

QiWang19 force-pushed the multi-descriptor branch from e8670de to b7df36a Compare December 23, 2020 00:29

reimplmeemnt ociArchiveImage{Source,Destination}

bdf582f

Signed-off-by: Qi Wang <[email protected]>

QiWang19 force-pushed the multi-descriptor branch from b7df36a to bdf582f Compare December 23, 2020 17:26

mtrmac mentioned this pull request Jan 14, 2021

Support saving multiple images to oci-archive: containers/container-libs#257

Open

mtrmac mentioned this pull request Apr 27, 2021

Add oci-archive-uncompressed-fd:5 #1209

Closed

QiWang19 mentioned this pull request May 5, 2021

[WIP] load/save oci-archive multi images containers/podman#8218

Closed

QiWang19 closed this Aug 9, 2021

umohnani8 mentioned this pull request Sep 20, 2021

Add reader/writer for oci-archive multi image support #1381

Closed

	return nil, errors.Wrapf(err, "Invalid source index %s", image)
	return nil, errors.Wrapf(err, "Invalid source index @%s: not an integer", image[1:])

Add reader/writer to for oci-archive multi image #1072

Add reader/writer to for oci-archive multi image #1072

Uh oh!

Conversation

QiWang19 commented Nov 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QiWang19 Nov 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QiWang19 commented Nov 16, 2020

Uh oh!

vrothberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QiWang19 commented Nov 23, 2020

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QiWang19 commented Dec 1, 2020

Uh oh!

QiWang19 commented Dec 3, 2020

Uh oh!

QiWang19 commented Dec 9, 2020

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QiWang19 commented Nov 2, 2020 •

edited

Loading

QiWang19 Nov 4, 2020 •

edited

Loading