Skip to content

Windows: Client: Allow process isolation [RS5+]#38000

Merged
thaJeztah merged 1 commit intomoby:masterfrom
microsoft:jjh/processandiot
Oct 10, 2018
Merged

Windows: Client: Allow process isolation [RS5+]#38000
thaJeztah merged 1 commit intomoby:masterfrom
microsoft:jjh/processandiot

Conversation

@lowenna
Copy link
Copy Markdown
Member

@lowenna lowenna commented Oct 10, 2018

fixes #27524
fixes #37873
fixes docker/for-win#1822

closes #37065
closes #27730

Signed-off-by: John Howard 🇬🇧 [email protected]

A lot of people will like this! As of Windows 10 (i.e. client) version 1809 (aka RS5), we are removing the restriction limiting docker to Hyper-V isolation only. With this fix, when running on RS5/later, client SKUs will be able to add --isolation=process to run a container with process isolation. Note that the support statement is that we allow it for dev/test, but don't support it for production usage. I'll let @taylorb-microsoft/@scooley and folks on that team figure where this should be documented/announced/marketed, and to make that change as a follow-up. I'm staying well clear of that conversation 😇 😆

It also removes the IoTCore check, which was nonsense previously and didn't work. That needs re-evaluation at a later date, if ever (there is a separate workaround currently).

@taylorb-microsoft - Can you do some validation that this operates as you expect and as we discussed?

ping @thaJeztah @simonferquel @johnstep.

@lowenna lowenna changed the title Windows:Allow process isolation Windows:Allow process isolation [RS5+] Oct 10, 2018
@lowenna lowenna changed the title Windows:Allow process isolation [RS5+] Windows: Client: Allow process isolation [RS5+] Oct 10, 2018
@lowenna
Copy link
Copy Markdown
Member Author

lowenna commented Oct 10, 2018

dockerd.zip

@taylorb-microsoft - Binary attached for your validation.

@simonferquel
Copy link
Copy Markdown
Contributor

You made my day!
(Lgtm)

@thaJeztah
Copy link
Copy Markdown
Member

Note that the support statement is that we allow it for dev/test, but don't support it for production usage.

I assume people don't run Windows client in "production" (as in; to host their containers) 😅, but maybe some do.

@jhowardmsft Are there known issues? Do you think a warning should be included in docker info's output if the default is set to process isolation on a Windows client? (see #37502)

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!!

Probably needs some slight touch-ups in the documentation.

Also, @jhowardmsft

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 10, 2018

Codecov Report

❗ No coverage uploaded for pull request base (master@82a4797). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #38000   +/-   ##
=========================================
  Coverage          ?   36.11%           
=========================================
  Files             ?      610           
  Lines             ?    45159           
  Branches          ?        0           
=========================================
  Hits              ?    16307           
  Misses            ?    26610           
  Partials          ?     2242

Copy link
Copy Markdown
Member

@vdemeester vdemeester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐸

@Skybladev2
Copy link
Copy Markdown

I assume people don't run Windows client in "production" (as in; to host their containers) sweat_smile, but maybe some do.

You may be surprised, but we use Windows 10 where possible because of the Windows Server licence cost.
Specifically, we run Jenkins on Windows 10 inside Linux container. It run really nice and smooth. However, it runs inside Linux VM, which we have no control over.
So, this change can save us more than gigabyte of memory considering native container performance will be comparable to Linux version. It will also remove 2 GB memory limitation for Linux VM and 3 GB memory limitation for Hyper-V container.
As you can see, I'm very very happy because of this change. Thank you very much, this is a critical update for Docker ecosystem.

@MikhailTymchukDX
Copy link
Copy Markdown

Is this change go live? I mean Docker for Windows release, not Windows 10.

@thaJeztah
Copy link
Copy Markdown
Member

@MikhailTymchukDX not yet; it's been merged to the 18.09.1 Docker CE codebase in docker-archive#81, and after that will find its way into the 18.09.1 Docker EE codebase. It will be available once Docker 18.09.1 EE is released

@tfenster
Copy link
Copy Markdown

tfenster commented Nov 30, 2018

@thaJeztah I tried with the latest master builds from https://master.dockerproject.com/ but process isolation just hangs. Here is what I did:

To make sure everything works I did

C:\WINDOWS\system32>docker run mcr.microsoft.com/windows/nanoserver:1809
Microsoft Windows [Version 10.0.17763.134]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>

and also

C:\WINDOWS\system32>docker run -ti --isolation=process mcr.microsoft.com/windows/nanoserver:1809
docker: Error response from daemon: Windows client operating systems only support Hyper-V containers.
See 'docker run --help'.

That is as expected as I am on 18.09.0

C:\WINDOWS\system32>docker version
Client: Docker Engine - Community
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:47:51 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:56:41 2018
  OS/Arch:          windows/amd64
  Experimental:     false

Now i downloaded the latest master builds which seems to update as expected:

C:\WINDOWS\system32>docker version
Client:
 Version:           master-dockerproject-2018-11-29
 API version:       1.40
 Go version:        go1.11.1
 Git commit:        504cecf2
 Built:             Thu Nov 29 23:50:48 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          master-dockerproject-2018-11-29
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.11.2
  Git commit:       baab736
  Built:            Thu Nov 29 23:59:33 2018
  OS/Arch:          windows/amd64
  Experimental:     false

docker run still works:

C:\WINDOWS\system32>docker run mcr.microsoft.com/windows/nanoserver:1809
Microsoft Windows [Version 10.0.17763.134]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>

Process isolation also doesn't show the immediate error, so the PR seems to be merged in that binary but it just hangs here:

C:\WINDOWS\system32>docker run -ti --isolation=process mcr.microsoft.com/windows/nanoserver:1809

Daemon output looks like this and then also just hangs

DEBU[2018-11-30T17:41:55.826424900+01:00] Calling GET /_ping
DEBU[2018-11-30T17:41:55.828426800+01:00] Calling POST /v1.40/containers/create
DEBU[2018-11-30T17:41:55.828945200+01:00] form data: {"AttachStderr":true,"AttachStdin":true,"AttachStdout":true,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[30,120],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"process","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"mcr.microsoft.com/windows/nanoserver:1809","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":true,"Tty":true,"User":"","Volumes":{},"WorkingDir":""}
DEBU[2018-11-30T17:41:55.832462500+01:00] hcsshim::GetLayerMountPath path C:\ProgramData\docker\windowsfilter\ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b
DEBU[2018-11-30T17:41:55.832462500+01:00] Calling proc (1)
DEBU[2018-11-30T17:41:55.833955300+01:00] Calling proc (2)
DEBU[2018-11-30T17:41:55.834484400+01:00] hcsshim::GetLayerMountPath succeeded path=C:\ProgramData\docker\windowsfilter\ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b mountPath=C:\ProgramData\docker\windowsfilter\ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b
DEBU[2018-11-30T17:41:55.835987900+01:00] hcsshim::CreateScratchLayer path C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:55.836428300+01:00] hcsshim::NameToGuid name:ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b guid:010df3e1-7918-5616-b7ed-f330ad22e0d8
DEBU[2018-11-30T17:41:55.837448500+01:00] hcsshim::NameToGuid name:7be5fa817a7ea239fa1b0e1a18f4587794dc8a043761eb25b7659026a9bdf0f1 guid:44ee543c-e9f7-5ac3-a463-d6dc23dde7ce
DEBU[2018-11-30T17:41:55.858979500+01:00] hcsshim::CreateScratchLayer - succeeded path=C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:55.894773700+01:00] Calling POST /v1.40/containers/c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668/attach?stderr=1&stdin=1&stdout=1&stream=1
DEBU[2018-11-30T17:41:55.895270900+01:00] attach: stderr: begin
DEBU[2018-11-30T17:41:55.895270900+01:00] attach: stdout: begin
DEBU[2018-11-30T17:41:55.895270900+01:00] attach: stdin: begin
DEBU[2018-11-30T17:41:55.898773300+01:00] Calling POST /v1.40/containers/c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668/wait?condition=next-exit
DEBU[2018-11-30T17:41:55.899770000+01:00] Calling POST /v1.40/containers/c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668/start
DEBU[2018-11-30T17:41:55.899770000+01:00] WindowsGraphDriver Get() id c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 mountLabel
DEBU[2018-11-30T17:41:55.900773200+01:00] hcsshim::ActivateLayer path C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:55.949367600+01:00] hcsshim::ActivateLayer  - succeeded path=C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:55.949868400+01:00] hcsshim::PrepareLayer path C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:55.960868200+01:00] hcsshim::NameToGuid name:ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b guid:010df3e1-7918-5616-b7ed-f330ad22e0d8
DEBU[2018-11-30T17:41:55.962865900+01:00] hcsshim::NameToGuid name:7be5fa817a7ea239fa1b0e1a18f4587794dc8a043761eb25b7659026a9bdf0f1 guid:44ee543c-e9f7-5ac3-a463-d6dc23dde7ce
DEBU[2018-11-30T17:41:56.025371000+01:00] hcsshim::PrepareLayer succeeded path=C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:56.025869300+01:00] hcsshim::GetLayerMountPath path C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:56.028867900+01:00] Calling proc (1)
DEBU[2018-11-30T17:41:56.032366700+01:00] Calling proc (2)
DEBU[2018-11-30T17:41:56.036873900+01:00] hcsshim::GetLayerMountPath succeeded path=C:\ProgramData\docker\windowsfilter\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 mountPath=\\?\Volume{31306e05-d5d7-4817-a68b-b130040e464e}
DEBU[2018-11-30T17:41:56.038868100+01:00] container mounted via layerStore: &{\\?\Volume{31306e05-d5d7-4817-a68b-b130040e464e} 0x30a3660 0x30a3660}
DEBU[2018-11-30T17:41:56.041371300+01:00] Assigning addresses for endpoint pensive_mclaren's interface on network nat
DEBU[2018-11-30T17:41:56.041869300+01:00] RequestAddress(172.21.16.0/20, <nil>, map[])
DEBU[2018-11-30T17:41:56.042370400+01:00] endpointStruct.EnableInternalDNS =[false]
DEBU[2018-11-30T17:41:56.043369700+01:00] [POST]=>[/endpoints/] Request : {"VirtualNetwork":"5A07FB7B-7843-457C-A526-06488B0CEDB2","EnableInternalDNS":true}
DEBU[2018-11-30T17:41:56.064871200+01:00] Network Response : {"ActivityId":"0ACFD905-8FF9-4529-8A72-BB1C4022C5A7","AdditionalParams":{},"CreateProcessingStartTime":131880697160558684,"DNSServerList":"172.21.16.1,192.168.1.1","DNSSuffix":"fritz.box","EnableInternalDNS":true,"EnableLowInterfaceMetric":true,"GatewayAddress":"172.21.16.1","Health":{"LastErrorCode":0,"LastUpdateTime":131880697160503678},"ID":"E21F82A3-5D83-499C-BBD8-82F17C4CC000","IPAddress":"172.21.17.9","MacAddress":"00-15-5D-D6-F0-69","Name":"Ethernet","Policies":[],"PrefixLength":20,"Resources":{"AdditionalParams":{},"AllocationOrder":0,"Health":{"LastErrorCode":0,"LastUpdateTime":131880697160503678},"ID":"0ACFD905-8FF9-4529-8A72-BB1C4022C5A7","PortOperationTime":0,"State":1,"SwitchOperationTime":0,"VfpOperationTime":0,"parentId":"84CD3469-F8E3-4173-A661-62168B8B249B"},"SharedContainers":[],"State":1,"Type":"nat","Version":38654705665,"VirtualNetwork":"5A07FB7B-7843-457C-A526-06488B0CEDB2","VirtualNetworkName":"nat"}
DEBU[2018-11-30T17:41:56.082369200+01:00] Assigning addresses for endpoint pensive_mclaren's interface on network nat
DEBU[2018-11-30T17:41:56.097867700+01:00] hcsshim::OpenComputeSystem ID=c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
DEBU[2018-11-30T17:41:56.106369700+01:00] Programming external connectivity on endpoint pensive_mclaren (5e65471111572ed913df6eee44e2cea33caaa379d06614a57c3447aff3fa2b58)
DEBU[2018-11-30T17:41:56.106369700+01:00] EnableService c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 START
DEBU[2018-11-30T17:41:56.117368200+01:00] EnableService c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 DONE
DEBU[2018-11-30T17:41:56.133868500+01:00] hcsshim::NameToGuid name:ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b guid:010df3e1-7918-5616-b7ed-f330ad22e0d8
DEBU[2018-11-30T17:41:56.144368900+01:00] hcsshim::NameToGuid name:7be5fa817a7ea239fa1b0e1a18f4587794dc8a043761eb25b7659026a9bdf0f1 guid:44ee543c-e9f7-5ac3-a463-d6dc23dde7ce
DEBU[2018-11-30T17:41:56.145376400+01:00] hcsshim::CreateComputeSystem ID=c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 config={"SystemType":"Container","Name":"c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668","Owner":"docker","VolumePath":"\\\\?\\Volume{31306e05-d5d7-4817-a68b-b130040e464e}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668","Layers":[{"ID":"010df3e1-7918-5616-b7ed-f330ad22e0d8","Path":"C:\\ProgramData\\docker\\windowsfilter\\ecca0170f401bd3132a4b0fa72abea915e0f6ef94003dff5c00e6d3f93fc149b"},{"ID":"44ee543c-e9f7-5ac3-a463-d6dc23dde7ce","Path":"C:\\ProgramData\\docker\\windowsfilter\\7be5fa817a7ea239fa1b0e1a18f4587794dc8a043761eb25b7659026a9bdf0f1"}],"HostName":"c80bd23ea18a","HvPartition":false,"EndpointList":["E21F82A3-5D83-499C-BBD8-82F17C4CC000"],"AllowUnqualifiedDNSQuery":true}
DEBU[2018-11-30T17:41:56.548869900+01:00] hcsshim::CreateComputeSystem succeeded id=c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 handle=56164288
DEBU[2018-11-30T17:41:56.549369900+01:00] starting container                            container=c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668 module=libcontainerd namespace=moby
DEBU[2018-11-30T17:41:56.553868900+01:00] hcsshim::ComputeSystem::Start ID=c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668
WARN[2018-11-30T17:45:56.555017400+01:00] StartComputeSystem c80bd23ea18aac62251e61762e2781365ac8d2bfd7b7f5a2c5707332a0fa3668:: Did not complete within 4m0s. This may indicate a platform issue. If it appears to be making no forward progress, obtain the stacks and see is there is a syscall stuck in the platform API for a significant length of time.

Any idea what might go wrong? Am I missing something? I am happy to provide more information if needed but I unfortunately don't know where to start looking

@lowenna lowenna deleted the jjh/processandiot branch November 30, 2018 19:39
@lowenna
Copy link
Copy Markdown
Member Author

lowenna commented Nov 30, 2018

@tfenster Can you open a new issue with this information rather than a comment against the PR. FWIW, I am not able to repro this, and I know many other people who are successfully running process isolated containers on RS5 after this PR. Thanks.

@tfenster
Copy link
Copy Markdown

@jhowardmsft I created #38306, thanks for looking at it. Might very well be a problem with my setup, would appreciate any hints where I might start looking

@mback2k
Copy link
Copy Markdown

mback2k commented Dec 27, 2019

As mentioned in docker/for-win#1822 (comment) and docker/for-win#3571 it seems like Hyper-V and Hardware Virtualization support are still hard requirements for Docker Desktop on Windows 10? I will open up a new issue for that, but I wanted to mention here that I am unable to make process isolation work without Hardware Virtualization support due to the Host Network Service not being able to fulfil networking requests to the Docker daemon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

9 participants