Skip to content

Adding Docker API endpoint to inspect image manifest#32061

Merged
thaJeztah merged 4 commits intomoby:masterfrom
nishanttotla:engine-api-manifest
May 10, 2017
Merged

Adding Docker API endpoint to inspect image manifest#32061
thaJeztah merged 4 commits intomoby:masterfrom
nishanttotla:engine-api-manifest

Conversation

@nishanttotla
Copy link
Contributor

@nishanttotla nishanttotla commented Mar 24, 2017

The goal of this PR is to make it possible for the client to figure out the image manifest, which will allow the client to construct the spec with the full digest (instead of doing it in the daemon like #28173) and platform information (#31348).

The client can make a call to the Engine to retrieve the manifest.

Here are more details about the change:

- What I did
Added /distribution/{name:*}/json endpoint in the API, that accesses the registry and pulls the manifest. This is required to pull digest and platform information.

- How I did it
I added a new distributionRouter, and a function called getDistributionInfo in distribution_routes.go that accesses the registry to get the digest, and pulls the image manifest to get more information. A DistributionInspect struct is returned.

- How to verify it
Send a curl request to the engine, at http:/distribution/alpine:latest/json

- Description for the changelog

Adding Docker API endpoint to inspect image manifest

- A picture of a cute animal (not mandatory but encouraged)
(\__/)
(='.'=)
(")_(")

I'm on a weak wifi, so here's a text bunny.

What I will do after this PR:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how to retrieve platform information from a schema2.DeserializedManifest. Working on figuring this out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is a little unclear based on the backed.GetRepository function as it is right now. Maybe it should be possible for it to just take a reference.Named object instead of reference.NamedTagged, since it doesn't seem to depend on having the latter.

@nishanttotla
Copy link
Contributor Author

@nishanttotla nishanttotla changed the title [WIP] Adding endpoint to contact registry from engine-api Adding endpoint to contact registry from engine-api Mar 29, 2017
@nishanttotla
Copy link
Contributor Author

Bumping up this PR. It works in the current form, and the goal is for the client to be able to use this endpoint to get manifest information to construct an appropriate spec.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it a post ?

Copy link
Contributor Author

@nishanttotla nishanttotla Mar 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made it POST because we needed to send registry auth token for private repos. But since we send it in the header instead of the payload, is GET fine?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe GET is fine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vieux changed it to GET.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on the GET

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch 2 times, most recently from 2bb96d1 to 6d04d48 Compare March 30, 2017 01:36
@nishanttotla
Copy link
Contributor Author

Architecture information is updated for all types of manifests now.

cc @aaronlehmann

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about returning the top-level manifest itself instead of (or in addition to) the digest? I could imagine this being useful for some things.

We could also return an optional config blob if one is referenced by the top-level manifest.

Then the platform determination could be done on the client side, making this /manifest more generic.

It's still not really ideal, because the daemon is deciding what items to fetch from the registry and return to the client. I kind of like the idea of proxying the registry API through the daemon, but that's a pretty big project and probably way out of scope for what we're trying to do here.

That said, I don't feel strongly about any of this, and I could be okay with just returning specific fields such as Platforms in the API response if that's what people prefer.

Copy link
Contributor Author

@nishanttotla nishanttotla Mar 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaronlehmann I'm inclined towards making it as general as possible. So returning the top-level manifest (and config blob) sounds like a better idea, especially because then the client can deal with it as it likes.

Also, then any format updates have to be dealt with by the client only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaronlehmann as an example, what do you think about

{
  Digest       digest.Digest,
  Manifest     distribution.Manifest,
  ConfigJSON   []bytes,
}

I still think we'd have to separate cases (fat, v2, v1 manifests) to get the right config blob, but then the client could Unmarshal it as desired.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's basically what I had in mind. Although maybe Manifest should be []byte so that we aren't introducing distribution types into the API, and also so that the digest can be verified. It's ugly to put JSON inside JSON though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean marshaling the manifest into []byte before sending? I have a couple of questions

  • Why does the digest need to be verified at the client?
  • Why does sending []bytes help with it?

It's ugly to put JSON inside JSON though

This applies to ConfigJSON as well, right?

Copy link

@aaronlehmann aaronlehmann Mar 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The manifest is a JSON document. DeserializedManifest is just a convenience for working with it in Go code. If you marshal it to JSON, you'll get the same set of bytes back, so the hash won't change (DeserializedManifest implements a MarshalJSON method that ensures this).

However, I've thought about it some more, and I don't think using []byte is worth the ugliness. Let's do this instead:

{
  Digest       digest.Digest
  Manifest     *json.RawMessage
  ConfigJSON   *json.RawMessage
}

Manifest would contain the original manifest JSON returned by (*DeserializedManifest).MarshalJSON. This is similar to using []byte, but it causes the JSON to be inserted directly instead of as a string. We lose the ability to verify the digest (because of possible whitespace added around the value during the marshalling process), but we avoid the ugliness off putting JSON inside a string inside JSON.

Why does the digest need to be verified at the client?

I don't think there's an immediate reason to verify the digest at the client, but I was thinking that if we're going to send the manifest to the client, we might as well send the exact manifest.

Why does sending []bytes help with it?

Because the hash is computed over these bytes.

It's ugly to put JSON inside JSON though

This applies to ConfigJSON as well, right?

Yes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaronlehmann PTAL, does this look reasonable?

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch 2 times, most recently from 73bc289 to c7c2b74 Compare March 31, 2017 20:20

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably this should return an error

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: it might be slightly more readable to use a type switch here. It's purely a cosmetic suggestion, though.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess since we're potentially returning different kinds of manifests in this field (schema1, schema2, manifest list...), we should have another field containing the media type. Depending which kind of manifest is being returned, this field would either be set to schema1.MediaTypeSignedManifest, schema2.MediaTypeManifest, or manifestlist.MediaTypeManifestList.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I intended to add this but missed it. Doing it now.

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch 2 times, most recently from 69e56a7 to 8181648 Compare April 1, 2017 00:00
@nishanttotla
Copy link
Contributor Author

@aaronlehmann addressed all comments.

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch from 8181648 to a23ae7b Compare April 1, 2017 00:32

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe ManifestMediaType?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding the omitempty flag, as this field won't always be present.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move the lines related to ConfigJSON into the *schema2.DeserializedManifest case so that manifestInspect.ConfigJSON remains nil when other types of manifests are returned.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaronlehmann I wasn't sure if there was a ConfigJSON we could return for FAT manifests, so I put it here. It doesn't seem to be the case, so I'll move this.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for an error

@thaJeztah thaJeztah removed the status/failing-ci Indicates that the PR in its current state fails the test suite label May 9, 2017
@nishanttotla nishanttotla force-pushed the engine-api-manifest branch 2 times, most recently from a9262a9 to 4d336db Compare May 9, 2017 23:03
@nishanttotla
Copy link
Contributor Author

@aaronlehmann PTAL.

@aaronlehmann
Copy link

Just one comment... it may be a good idea to override the descriptor's media type always, not just in the descriptorFromPayload case, because we know the one the registry give us is going to be misleading. Other than that, it looks good.

@nishanttotla
Copy link
Contributor Author

@aaronlehmann sure, updating the PR. If we're going to call Payload() always, then descriptorFromPayload isn't really needed, and we can update size if it's 0.

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch from 4d336db to 409fc20 Compare May 10, 2017 00:01
Copy link

@aaronlehmann aaronlehmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible cleanup: dscrptr is no longer necessary because distributionInspect.Descriptor can be set directly. Not important though.

LGTM

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch from 409fc20 to 12e232e Compare May 10, 2017 00:25
@nishanttotla
Copy link
Contributor Author

@aaronlehmann thanks, I cleaned that up.

Features:
type: "array"
items:
type: "string"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should add an example response here, because the current output doesn't provide a lot of information to the reader 😄
screen shot 2017-05-10 at 19 13 22

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thaJeztah updated. PTAL.

screen shot 2017-05-10 at 11 02 24 am

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like empty array definitions are still causing an error, I'm fixing those.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we come up with some sample URL's and/or fill the OSVersion / Variants? Or do you think it's ok to keep them empty?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thaJeztah In the examples I saw, these weren't filled, so I think it's okay. I can try to search for better examples later.

type: "string"
Size:
type: "integer"
format: "int64"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondered if this should be an uint64, but I see it's coming from an existing type

@nishanttotla nishanttotla force-pushed the engine-api-manifest branch 2 times, most recently from 8babced to 2878fda Compare May 10, 2017 18:18
@nishanttotla nishanttotla force-pushed the engine-api-manifest branch from 2878fda to 4a81204 Compare May 10, 2017 18:20
@nishanttotla
Copy link
Contributor Author

@thaJeztah PTAL.

@thaJeztah
Copy link
Member

I think We're almost there; left one comment on the example, also (sorry, forgot to mention); this needs an entry in the API history; https://github.com/moby/moby/blob/master/docs/api/version-history.md#v130-api-changes

@nishanttotla
Copy link
Contributor Author

@thaJeztah updated API history here: #33147

Copy link
Contributor

@mdlinville mdlinville left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs LGTM

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@thaJeztah thaJeztah merged commit 28d428f into moby:master May 10, 2017
@nishanttotla nishanttotla deleted the engine-api-manifest branch May 10, 2017 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.