Add Windows secrets support#32208
Conversation
|
+1 to get to a ramdisk (or similar) backend so we don't have to persist these on disk. as long as we ensure the permissions and access is limited this sgtm as a start. design LGTM |
|
@ehazlett Since secrets are stored in the Docker runtime root ( However, for a custom runtime root (e.g. |
|
Thanks for the PR @johnstep.
|
|
+1 for documenting differences between Windows and Linux
Yes +1. I say we do this regardless of the Docker root to be sure nothing would accidentally leak and to make sure at minimum the secrets directory has the proper access restrictions. |
|
Can we disable |
|
@AkihiroSuda Windows does not support commit of a running container, and Docker deletes secrets when stopping a container. If the daemon terminates abnormally, it deletes the secrets on restarting. |
|
@mstanleyjones We need to document that secrets for Windows will be persisted on the host, in the container directory. The secrets directory is deleted when the container is stopped, or when the daemon is restarted, in the case of abnormal termination. In the short term, customers can mitigate this concern by enabling BitLocker on the volume containing the Docker runtime root. Separately, we are working on a RAM disk driver for consistency with Linux. |
9b51ab0 to
cb73b2c
Compare
|
Updated to always secure secrets directories explicitly, and enabled integration tests. |
There was a problem hiding this comment.
I'd really like to see much of this combined with the (almost) identical code in container_operations_unix.go
There was a problem hiding this comment.
I looked at this, and definitely can move some of it out into a shared file, but would rather keep it as is for now, and factor it out in a future change.
There was a problem hiding this comment.
This is identical to SecretMount() in container_unix.go. Can it be combined?
There was a problem hiding this comment.
True, there are at least a couple of these simple functions that could be shared. It might make sense to move them to container.go but I think I'd like to keep them separate for now.
|
@johnstep needs a rebase |
There was a problem hiding this comment.
Wondering under what circumstances this condition would be hit?
There was a problem hiding this comment.
IIRC File is in a oneof in the swarmkit gRPC API. If we add other possible types of secret targets, to allow exposing them in other ways than as files, File will be nil.
There was a problem hiding this comment.
The comment here should probably be updated.
There was a problem hiding this comment.
setupSecretDir is doing a mount/unmount, and then that's being done in the loop at line 38 immediately after. Mount/Unmount is a very expensive (relatively) operation compared to Linux (we had strong feedback from the perf team when I was doing an accidental mount/unmounts a while back and had to refactor), so I'd really like to see it limited to at most 1 mount/unmounts pair to support secrets.
There was a problem hiding this comment.
Good catch; the mount in setupSecretDir is actually not required; I will remove it.
There was a problem hiding this comment.
It looks like the this would be logged if system.MkdirAllWIthACL failed.
There was a problem hiding this comment.
I will skip the log message if the error is non-existence.
mlaventure
left a comment
There was a problem hiding this comment.
LGTM
ping @jhowardmsft
There was a problem hiding this comment.
Is it OK to have TODO comments in the code?
There was a problem hiding this comment.
Consider this one as a hint ;-)
There was a problem hiding this comment.
To clarify; yes, we use TODO's in the code; usually not for "unfinished" code, but more as reminders that (e.g.) something needs to be removed after release X, or after feature Y is implemented. The TODO (name) format also works in some editors/IDE's to show a todo list for one person specific 😄 https://confluence.jetbrains.com/display/PhpStorm/Working+with+todo+comments+and+the+todo+tool+window
There was a problem hiding this comment.
t.b.h., I don't think this TODO is really useful 😄 ...
There was a problem hiding this comment.
Yes, perhaps not all that useful, and now in 4 separate places...
There was a problem hiding this comment.
these should be fine to remove. there were some timeline and roadmap adjustments that delayed this a bit so I think it's fine to remove for now.
There was a problem hiding this comment.
I know we've talked offline about this, but I would still really like a comment here to explain why not been started before
There was a problem hiding this comment.
I am not sure this is the best place to explain when to mount containers on Windows, in general, but perhaps you can give me a rough idea of the comment you would like to see. Thanks for pointing me to HasBeenStartedBefore; this check fixes an issue where the code was trying to recreate the symlinks for a container that had already been started, which fails.
There was a problem hiding this comment.
Something like "Ensure that we do not attempt to mount the filesystem of a Hyper-V container that has previously been started to stop an attack vector."
There was a problem hiding this comment.
Again, as per offline, if we can avoid this and not rely on reference counting, that would be far better and more explicit. Note that as also mentioned, this should be co-ordinated with the configs PR
There was a problem hiding this comment.
Is there a guarantee that the container file system is mounted here in the non-Hyper-V case? It seems strange to me to start modifying the container file system without first ensuring it is mounted. Is that guarantee documented anywhere, or just implicit in the design?
There was a problem hiding this comment.
Yes, for Windows Server Containers, you are guaranteed that the container FS is mounted here. In start.go:
if err := daemon.conditionalMountOnStart(container); err != nil {
return err
}
if err := daemon.initializeNetworking(container); err != nil {
return err
}
spec, err := daemon.createSpec(container)
if err != nil {
return err
}Where conditionalMountOnStart always mounts Windows Server Containers, but not Hyper-V containers:
// We do not mount if a Hyper-V container
if !daemon.runAsHyperVContainer(container.HostConfig) {
return daemon.Mount(container)
}|
It's much closer, but I'd still like a few changes. I'd also like to see some docs here explaining the nuances of what all this implies before giving a 'LGTM' |
|
I will work on the documentation. This Windows differences will definitely be documented for 17.06. The main differences between secrets on Linux and Windows are:
@jhowardmsft Are there any other important differences you can think of? Thanks! |
|
@johnstep ^^ differences sounds good. |
Signed-off-by: John Stephens <[email protected]>
| } | ||
| err := c.CreateSecretSymlinks() | ||
| if isHyperV { | ||
| daemon.Unmount(c) |
There was a problem hiding this comment.
Minor nit, talked offline. When the configs PR is ready after this, change to a defer in the previous check.
lowenna
left a comment
There was a problem hiding this comment.
LGTM. Thanks for addressing the feedback. One minor nit which can be addressed in the configs PR, but let's get this merged.
|
Thanks for reviewing @jhowardmsft And whoop, @johnstep, happy to see this merged! |
Signed-off-by: John Stephens [email protected]
This change implements secrets support for Windows by writing to persistent files on the host, due to the lack of a RAM disk driver built in to Windows. A separate change will add RAM disk support later.
@jhowardmsft @PatrickLang @ehazlett @diogomonica @cyli