Skip to content

Add Windows secrets support#32208

Merged
lowenna merged 1 commit intomoby:masterfrom
johnstep:windows-secrets
May 16, 2017
Merged

Add Windows secrets support#32208
lowenna merged 1 commit intomoby:masterfrom
johnstep:windows-secrets

Conversation

@johnstep
Copy link
Copy Markdown
Member

Signed-off-by: John Stephens [email protected]

This change implements secrets support for Windows by writing to persistent files on the host, due to the lack of a RAM disk driver built in to Windows. A separate change will add RAM disk support later.

@jhowardmsft @PatrickLang @ehazlett @diogomonica @cyli

@ehazlett
Copy link
Copy Markdown
Contributor

+1 to get to a ramdisk (or similar) backend so we don't have to persist these on disk. as long as we ensure the permissions and access is limited this sgtm as a start.

design LGTM

@johnstep
Copy link
Copy Markdown
Member Author

johnstep commented Mar 29, 2017

@ehazlett Since secrets are stored in the Docker runtime root (%ProgramData%\Docker by default), the secrets directories inherit those permissions, only granting administrators and system access:

$ icacls %ProgramData%\Docker
C:\ProgramData\Docker BUILTIN\Administrators:(OI)(CI)(F)
                      NT AUTHORITY\SYSTEM:(OI)(CI)(F)

Successfully processed 1 files; Failed processing 0 files

However, for a custom runtime root (e.g. --graph A:\docker), security is based on that custom root directory. We could explicitly force stricter permissions on each secrets directory.

@diogomonica
Copy link
Copy Markdown
Contributor

diogomonica commented Mar 29, 2017

Thanks for the PR @johnstep.

  • We need a commitment on the timeline for the RamFS before we merge this
  • We need the docs to reflect that secrets will not be as secure on windows as they are on Linux.

@ehazlett
Copy link
Copy Markdown
Contributor

+1 for documenting differences between Windows and Linux

@johnstep

We could explicitly force stricter permissions on each secrets directory.

Yes +1. I say we do this regardless of the Docker root to be sure nothing would accidentally leak and to make sure at minimum the secrets directory has the proper access restrictions.

@AkihiroSuda
Copy link
Copy Markdown
Member

Can we disable docker commit when secret is mounted on the rootfs rather than on RAM disk?

@johnstep
Copy link
Copy Markdown
Member Author

@AkihiroSuda Windows does not support commit of a running container, and Docker deletes secrets when stopping a container. If the daemon terminates abnormally, it deletes the secrets on restarting.

@johnstep
Copy link
Copy Markdown
Member Author

@mstanleyjones We need to document that secrets for Windows will be persisted on the host, in the container directory. The secrets directory is deleted when the container is stopped, or when the daemon is restarted, in the case of abnormal termination. In the short term, customers can mitigate this concern by enabling BitLocker on the volume containing the Docker runtime root.

Separately, we are working on a RAM disk driver for consistency with Linux.

@johnstep
Copy link
Copy Markdown
Member Author

Updated to always secure secrets directories explicitly, and enabled integration tests.

Comment thread daemon/container_operations_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd really like to see much of this combined with the (almost) identical code in container_operations_unix.go

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at this, and definitely can move some of it out into a shared file, but would rather keep it as is for now, and factor it out in a future change.

Comment thread container/container_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is identical to SecretMount() in container_unix.go. Can it be combined?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, there are at least a couple of these simple functions that could be shared. It might make sense to move them to container.go but I think I'd like to keep them separate for now.

@ehazlett
Copy link
Copy Markdown
Contributor

@johnstep needs a rebase

Comment thread container/container_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering under what circumstances this condition would be hit?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC File is in a oneof in the swarmkit gRPC API. If we add other possible types of secret targets, to allow exposing them in other ways than as files, File will be nil.

Comment thread container/container_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment here should probably be updated.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

Comment thread daemon/oci_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setupSecretDir is doing a mount/unmount, and then that's being done in the loop at line 38 immediately after. Mount/Unmount is a very expensive (relatively) operation compared to Linux (we had strong feedback from the perf team when I was doing an accidental mount/unmounts a while back and had to refactor), so I'd really like to see it limited to at most 1 mount/unmounts pair to support secrets.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch; the mount in setupSecretDir is actually not required; I will remove it.

Comment thread daemon/container_operations_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the this would be logged if system.MkdirAllWIthACL failed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will skip the log message if the error is non-existence.

@johnstep johnstep requested review from cpuguy83 and thaJeztah May 11, 2017 21:11
Copy link
Copy Markdown
Contributor

@mlaventure mlaventure left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ping @jhowardmsft

Comment thread daemon/container_operations_windows.go Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it OK to have TODO comments in the code?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider this one as a hint ;-)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify; yes, we use TODO's in the code; usually not for "unfinished" code, but more as reminders that (e.g.) something needs to be removed after release X, or after feature Y is implemented. The TODO (name) format also works in some editors/IDE's to show a todo list for one person specific 😄 https://confluence.jetbrains.com/display/PhpStorm/Working+with+todo+comments+and+the+todo+tool+window

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

t.b.h., I don't think this TODO is really useful 😄 ...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, perhaps not all that useful, and now in 4 separate places...

Copy link
Copy Markdown
Contributor

@ehazlett ehazlett May 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should be fine to remove. there were some timeline and roadmap adjustments that delayed this a bit so I think it's fine to remove for now.

Copy link
Copy Markdown
Contributor

@ehazlett ehazlett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code LGTM

Comment thread daemon/oci_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we've talked offline about this, but I would still really like a comment here to explain why not been started before

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this is the best place to explain when to mount containers on Windows, in general, but perhaps you can give me a rough idea of the comment you would like to see. Thanks for pointing me to HasBeenStartedBefore; this check fixes an issue where the code was trying to recreate the symlinks for a container that had already been started, which fails.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like "Ensure that we do not attempt to mount the filesystem of a Hyper-V container that has previously been started to stop an attack vector."

Comment thread daemon/oci_windows.go Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, as per offline, if we can avoid this and not rely on reference counting, that would be far better and more explicit. Note that as also mentioned, this should be co-ordinated with the configs PR

Copy link
Copy Markdown
Member Author

@johnstep johnstep May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a guarantee that the container file system is mounted here in the non-Hyper-V case? It seems strange to me to start modifying the container file system without first ensuring it is mounted. Is that guarantee documented anywhere, or just implicit in the design?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, for Windows Server Containers, you are guaranteed that the container FS is mounted here. In start.go:

	if err := daemon.conditionalMountOnStart(container); err != nil {
		return err
	}

	if err := daemon.initializeNetworking(container); err != nil {
		return err
	}

	spec, err := daemon.createSpec(container)
	if err != nil {
		return err
	}

Where conditionalMountOnStart always mounts Windows Server Containers, but not Hyper-V containers:

	// We do not mount if a Hyper-V container
	if !daemon.runAsHyperVContainer(container.HostConfig) {
		return daemon.Mount(container)
	}

@lowenna
Copy link
Copy Markdown
Member

lowenna commented May 16, 2017

It's much closer, but I'd still like a few changes. I'd also like to see some docs here explaining the nuances of what all this implies before giving a 'LGTM'

@johnstep
Copy link
Copy Markdown
Member Author

johnstep commented May 16, 2017

I will work on the documentation. This Windows differences will definitely be documented for 17.06.

The main differences between secrets on Linux and Windows are:

  1. Windows stores temporary secret files for containers directly in the data root instead of in a tmpfs file system, since Windows does not have a built-in RAM disk driver. The primary mitigation for this is to enable BitLocker on the volume containing the data root.

  2. Windows secret files with custom targets are not directly bind-mounted into the container, since Windows does not support non-directory file bind-mounts. Instead, secrets for a container are all mounted in C:\ProgramData\Docker\internal\secrets (within the container, and an implementation detail), and symlinks are created that point from the target, to the secret in the internal directory.

  3. On Windows, when creating a service, the options to specify UID, GID, and mode are not supported for secrets; secrets are currently only accessible by administrators and system in a container.

@jhowardmsft Are there any other important differences you can think of? Thanks!

@lowenna
Copy link
Copy Markdown
Member

lowenna commented May 16, 2017

@johnstep ^^ differences sounds good.

Signed-off-by: John Stephens <[email protected]>
Comment thread daemon/oci_windows.go
}
err := c.CreateSecretSymlinks()
if isHyperV {
daemon.Unmount(c)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit, talked offline. When the configs PR is ready after this, change to a defer in the previous check.

Copy link
Copy Markdown
Member

@lowenna lowenna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for addressing the feedback. One minor nit which can be addressed in the configs PR, but let's get this merged.

@thaJeztah
Copy link
Copy Markdown
Member

Thanks for reviewing @jhowardmsft

And whoop, @johnstep, happy to see this merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.