Skip to content

Pin images by digest#28173

Merged
aluzzardi merged 2 commits intomoby:masterfrom
nishanttotla:pin-images-by-digest
Nov 10, 2016
Merged

Pin images by digest#28173
aluzzardi merged 2 commits intomoby:masterfrom
nishanttotla:pin-images-by-digest

Conversation

@nishanttotla
Copy link
Contributor

@nishanttotla nishanttotla commented Nov 8, 2016

- What I did
When a Docker Swarm mode service is created using the image and tag, the agents pull image with the specified tag. This creates an issue for non-static tags like latest, which keep updating. This might cause multiple versions of the image running for the same service. The solution to this problem is to pin images by digest.

For example, on docker service create alpine:latest top, the manifest for the alpine:latest image is retrieved, and the digest is added to the image description before passing on the spec to the SwarmKit manager. In this case, alpine:latest is changed to alpine:latest@sha256:1354db23ff5478120c980eca1611a51c9f2b88b61f24283ee8200bf9a54f2e5c that specifies an image precisely. If at a future time, the image under alpine:latest were to have a different digest, this would be detected at the next service update, and all images would be pinned to the new digest.

Related issues: #24066, #27508, moby/swarmkit#1334

- How I did it
The change is fairly simple. A new daemon function ResolveTagToDigest is used to retrieve a digest for a reference.NamedTagged object, by making a request to the registry. This function is used by docker service create and docker service update to resolve image to digest, if the image used to create the service didn't already have it.

The service spec is modified before passing to the SwarmKit manager to always contain a digest string, thereby ensuring that a service image is always precisely specified. On image update, a digest is retrieved again, and if different from the existing one, will result in a service update.

One thing that remains to be done is to enable pinning by digest for private images (this is a minor addition which I will shortly finish) by passing AuthConfig to ResolveTagToDigest.

- How to verify it
Try the following commands:

docker service create --name testsvc alpine:latest top
docker service ls

This should show the image as alpine:latest@sha256:1354db23ff5478120c980eca1611a51c9f2b88b61f24283ee8200bf9a54f2e5c

docker service update --image alpine:edge testsvc
docker service ls

This will show the image updated to alpine:edge@sha256:cd9c03c2d382fcf00c31dc1635445163ec185dfffb51242d9e097892b3b0d5b4

NOTE: It is by choice that the operation won't fail if the digest retrieval fails for some reason. It will only print a warning and move on. This is to make sure that the use case where a locally built image might be used to create a service still works.

- Description for the changelog

Pin images by digest for docker service create and update

- A picture of a cute animal (not mandatory but encouraged)

screen shot 2016-11-08 at 10 07 29 am

This is Mutsu the kitten. For more of Mutsu, you can check here

Copy link

@aaronlehmann aaronlehmann Nov 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return reference.WithDigest(ref, dgst).String(), nil

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@nishanttotla
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest ignoring confirmedV2 here and attempting the Tags(ctx).Get() anyway. Worst case, it fails.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds fair. The check doesn't provide any particular advantages anyway.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return the Digest object rather than the stringified version.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The if is not necessary - you can just do ref = reference.WithDefaultTag(ref) since WithDefaultTag is defined as follows:

// WithDefaultTag adds a default tag to a reference if it only has a repo name.
func WithDefaultTag(ref Named) Named {
        if IsNameOnly(ref) {
                ref, _ = WithTag(ref, DefaultTag)
        }
        return ref
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I think I missed this after some refactoring.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

namedTaggedRef, ok := ref.(reference.NamedTagged)

Return an error if ok is false (should not be possible currently, but the reference package could change in the future).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a Debugf, and ideally should provide more context about what was pinned. I'd suggest creating a logger at the top of this function with logrus.WithFields. Then if you use that logger for the log messages, they can have the service name, image reference, and so on filled in.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved these log messages outside this function, which now only returns an error, and the caller decides what to do with it.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this above the if encodedAuth != "" { so cntr can be reused there.

@nishanttotla nishanttotla force-pushed the pin-images-by-digest branch 2 times, most recently from ef09483 to 288fe63 Compare November 8, 2016 18:45
@nishanttotla nishanttotla force-pushed the pin-images-by-digest branch 4 times, most recently from c543b7b to 869ed54 Compare November 8, 2016 20:42
@nishanttotla
Copy link
Contributor Author

@aaronlehmann I've addressed all your comments. Please let me know if this looks acceptable. The only thing I haven't quite decided is what fields to add to logrus.Debugf, though I moved the log statements to the caller so that all logging related to digest pinning is in one place.

Also cc @tonistiigi.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would log an error message if the user already provided a digest. How about making imageWithDigestString return image, nil if the image already has a digest specified, and then this code can only log "pinning image..." if digestImage != newCntr.Image?

@aluzzardi aluzzardi added this to the 1.13.0 milestone Nov 8, 2016
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't make the error path in the main path. Get in the habit of making an early return.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wouldn't we return the canonical reference if it is already canonical?

if canonical, ok := ref.(reference.Canonical); ok {
  return canonical, nil
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encodedAuth should really be cross-cutting. Perhaps, store in the backed or preconfigure an object with the auth before calling this method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevvooe if I understand correctly, passing the types.AuthConfig struct to this function would be acceptable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is probably a first step. It is hard to tell, since it is not actually being used. I have no clue why we would bury an unencoded value so deeply in the call stack. The auth should probably be decoded as soon as the request is received. Then, I would put it in the context and access as needed, without passing it down the entire call chain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than add yet another method to daemon, why not export access to the configured repository? As it is, this method is very narrow and not very flexible. Ideally, we give it an image reference and get back fully configured repository.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will make this function return a fully configured repository to expand its usability.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if err != nil {
log warning
} else if digestImage != ctnr.Image {
log debug(pinning)
ctnr.Image = digestImage
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this expected not to error out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any operations related to pinning by digest aren't supposed to error out right now, because some people use local images in dev, and this would break that workflow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should return lastError instead of nil, it is possible that the last endpoint returned an error and caused repository to be nil.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, good catch @dmcgowan

@aaronlehmann
Copy link

One thing I just thought of is that we should avoid re-pulling the image when the task starts if the image is pinned by digest and it already exists locally. This will make task startup faster and reduce unnecessary registry traffic.

This should be checked in the Prepare function in daemon/cluster/executor/container/controller.go. I think we may need to add LookupImage to Backend in daemon/cluster/executor/backend.go and use that function to check if the image already exists locally.

I'd suggest doing this in a separate PR before the release.

@nishanttotla
Copy link
Contributor Author

nishanttotla commented Nov 10, 2016

@aaronlehmann
Copy link

aaronlehmann commented Nov 10, 2016

Tested and this works for me.

LGTM

Thanks for that list of action items. It looks good to me. I would add one more - add an integration test that covers pinning by digest.

@tonistiigi
Copy link
Member

LGTM

1 similar comment
@aluzzardi
Copy link
Member

LGTM

@nishanttotla
Copy link
Contributor Author

@thaJeztah this needs a docs/revisit label, I'm working on updating docs for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.