Skip to content

Conversation

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Apr 8, 2021

Follow-up to #39846

Addresses #43395

@thaJeztah
Copy link
Member Author

Failure on first run:

[2021-04-08T23:31:08.761Z] INFO: Building busybox
[2021-04-08T23:31:08.761Z] Sending build context to Docker daemon   5.12kB
[2021-04-08T23:31:08.761Z] 
[2021-04-08T23:31:08.761Z] Step 1/13 : ARG WINDOWS_BASE_IMAGE=mcr.microsoft.com/windows/servercore
[2021-04-08T23:31:08.761Z] Step 2/13 : ARG WINDOWS_BASE_IMAGE_TAG=ltsc2019
[2021-04-08T23:31:08.761Z] Step 3/13 : ARG BUSYBOX_VERSION=FRP-3329-gcf0fa4d13
[2021-04-08T23:31:08.761Z] Step 4/13 : ARG BUSYBOX_SHA256SUM=bfaeb88638e580fc522a68e69072e305308f9747563e51fa085eec60ca39a5ae
[2021-04-08T23:31:08.761Z] Step 5/13 : FROM ${WINDOWS_BASE_IMAGE}:${WINDOWS_BASE_IMAGE_TAG}
[2021-04-08T23:31:08.761Z]  ---> 39d157a84080
[2021-04-08T23:31:08.761Z] Step 6/13 : RUN mkdir C:\tmp && mkdir C:\bin
[2021-04-08T23:31:09.216Z]  ---> Running in 6c0628c195a0
[2021-04-08T23:31:09.216Z] hcsshim::CreateComputeSystem 6c0628c195a0ebf0106e570a595837a3f58abbbed65ec0101eb3688ba49e3d16: The request is not supported.
[2021-04-08T23:31:09.216Z] (extra info: {"SystemType":"Container","Name":"6c0628c195a0ebf0106e570a595837a3f58abbbed65ec0101eb3688ba49e3d16","Owner":"docker","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"D:\\CI\\PR-4
[2021-04-08T23:31:09.216Z] 2277\\2\\daemon\\windowsfilter\\6c0628c195a0ebf0106e570a595837a3f58abbbed65ec0101eb3688ba49e3d16","Layers":[{"ID":"15e76a7c-63d7-52bb-85c7-f8abe95e977d","Path":"D:\\CI\\PR-42277\\2\\daemon\\wi
[2021-04-08T23:31:09.216Z] ndowsfilter\\c96bf8429a680429141028847769128c365eb39a4d1c31d787527f38d512e36b"}],"HostName":"6c0628c195a0","HvPartition":true,"EndpointList":["d617e37c-0793-4d03-9225-270e5ed6017e"],"HvRuntime
[2021-04-08T23:31:09.216Z] ":{"ImagePath":"D:\\CI\\PR-42277\\2\\daemon\\windowsfilter\\c96bf8429a680429141028847769128c365eb39a4d1c31d787527f38d512e36b\\UtilityVM"},"AllowUnqualifiedDNSQuery":true})

@thaJeztah thaJeztah force-pushed the jenkinsfile_windows_2022_hyperv branch from 2eb8c88 to 1d3113b Compare April 9, 2021 08:45
@thaJeztah
Copy link
Member Author

I tried updating the busybox Dockerfile (use mkdir -p instead of mkdir - in case the problem was caused by directories already existing), but looks like it's not that; it's hcsshim failing to create the container;

11:17:27  Step 6/13 : RUN powershell New-Item -ItemType "Directory" -Force -Path "C:\tmp", "C:\bin"
11:17:27   ---> Running in 8408ec410b3a
11:17:27  powershell.exe : hcsshim::CreateComputeSystem 8408ec410b3a8d8d1e6f5ee8bd56499878db602c38c60b912df653b5656ad245: The request is not supported.
11:17:27  At D:\gopath\src\github.com\docker\docker@tmp\durable-7e9a38ae\powershellWrapper.ps1:3 char:1
11:17:27  + & powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -Comm ...
11:17:27  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11:17:27      + CategoryInfo          : NotSpecified: (hcsshim::Create... not supported.:String) [], RemoteException
11:17:27      + FullyQualifiedErrorId : NativeCommandError
11:17:27   
11:17:27  (extra info: {"SystemType":"Container","Name":"8408ec410b3a8d8d1e6f5ee8bd56499878db602c38c60b912df653b5656ad245","Owner":"docker","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"D:\\CI\\PR-4
11:17:27  2277\\6\\daemon\\windowsfilter\\8408ec410b3a8d8d1e6f5ee8bd56499878db602c38c60b912df653b5656ad245","Layers":[{"ID":"2968fe41-273b-52a3-84dc-9ddb2f685d03","Path":"D:\\CI\\PR-42277\\6\\daemon\\wi
11:17:27  ndowsfilter\\5b1892bb219225b50c37bc089028d58193d1533ea17ba4c51ee289a43e3f3a85"}],"HostName":"8408ec410b3a","HvPartition":true,"EndpointList":["34258b16-6c65-4c64-a543-115932293402"],"HvRuntime
11:17:27  ":{"ImagePath":"D:\\CI\\PR-42277\\6\\daemon\\windowsfilter\\5b1892bb219225b50c37bc089028d58193d1533ea17ba4c51ee289a43e3f3a85\\UtilityVM"},"AllowUnqualifiedDNSQuery":true})

Cleaning up that "extra info" JSON;

{
  "SystemType": "Container",
  "Name": "8408ec410b3a8d8d1e6f5ee8bd56499878db602c38c60b912df653b5656ad245",
  "Owner": "docker",
  "IgnoreFlushesDuringBoot": true,
  "LayerFolderPath": "D:\\CI\\PR-42277\\6\\daemon\\windowsfilter\\8408ec410b3a8d8d1e6f5ee8bd56499878db602c38c60b912df653b5656ad245",
  "Layers": [
    {
      "ID": "2968fe41-273b-52a3-84dc-9ddb2f685d03",
      "Path": "D:\\CI\\PR-42277\\6\\daemon\\windowsfilter\\5b1892bb219225b50c37bc089028d58193d1533ea17ba4c51ee289a43e3f3a85"
    }
  ],
  "HostName": "8408ec410b3a",
  "HvPartition": true,
  "EndpointList": [
    "34258b16-6c65-4c64-a543-115932293402"
  ],
  "HvRuntime": {
    "ImagePath": "D:\\CI\\PR-42277\\6\\daemon\\windowsfilter\\5b1892bb219225b50c37bc089028d58193d1533ea17ba4c51ee289a43e3f3a85\\UtilityVM"
  },
  "AllowUnqualifiedDNSQuery": true
}

@thaJeztah
Copy link
Member Author

Related struct from hcsshim

// ContainerConfig is used as both the input of CreateContainer
// and to convert the parameters to JSON for passing onto the HCS
type ContainerConfig struct {
	SystemType                  string              // HCS requires this to be hard-coded to "Container"
	Name                        string              // Name of the container. We use the docker ID.
	Owner                       string              `json:",omitempty"` // The management platform that created this container
	VolumePath                  string              `json:",omitempty"` // Windows volume path for scratch space. Used by Windows Server Containers only. Format \\?\\Volume{GUID}
	IgnoreFlushesDuringBoot     bool                `json:",omitempty"` // Optimization hint for container startup in Windows
	LayerFolderPath             string              `json:",omitempty"` // Where the layer folders are located. Used by Windows Server Containers only. Format  %root%\windowsfilter\containerID
	Layers                      []Layer             // List of storage layers. Required for Windows Server and Hyper-V Containers. Format ID=GUID;Path=%root%\windowsfilter\layerID
	Credentials                 string              `json:",omitempty"` // Credentials information
	ProcessorCount              uint32              `json:",omitempty"` // Number of processors to assign to the container.
	ProcessorWeight             uint64              `json:",omitempty"` // CPU shares (relative weight to other containers with cpu shares). Range is from 1 to 10000. A value of 0 results in default shares.
	ProcessorMaximum            int64               `json:",omitempty"` // Specifies the portion of processor cycles that this container can use as a percentage times 100. Range is from 1 to 10000. A value of 0 results in no limit.
	StorageIOPSMaximum          uint64              `json:",omitempty"` // Maximum Storage IOPS
	StorageBandwidthMaximum     uint64              `json:",omitempty"` // Maximum Storage Bandwidth in bytes per second
	StorageSandboxSize          uint64              `json:",omitempty"` // Size in bytes that the container system drive should be expanded to if smaller
	MemoryMaximumInMB           int64               `json:",omitempty"` // Maximum memory available to the container in Megabytes
	HostName                    string              `json:",omitempty"` // Hostname
	MappedDirectories           []MappedDir         `json:",omitempty"` // List of mapped directories (volumes/mounts)
	MappedPipes                 []MappedPipe        `json:",omitempty"` // List of mapped Windows named pipes
	HvPartition                 bool                // True if it a Hyper-V Container
	NetworkSharedContainerName  string              `json:",omitempty"` // Name (ID) of the container that we will share the network stack with.
	EndpointList                []string            `json:",omitempty"` // List of networking endpoints to be attached to container
	HvRuntime                   *HvRuntime          `json:",omitempty"` // Hyper-V container settings. Used by Hyper-V containers only. Format ImagePath=%root%\BaseLayerID\UtilityVM
	Servicing                   bool                `json:",omitempty"` // True if this container is for servicing
	AllowUnqualifiedDNSQuery    bool                `json:",omitempty"` // True to allow unqualified DNS name resolution
	DNSSearchList               string              `json:",omitempty"` // Comma seperated list of DNS suffixes to use for name resolution
	ContainerType               string              `json:",omitempty"` // "Linux" for Linux containers on Windows. Omitted otherwise.
	TerminateOnLastHandleClosed bool                `json:",omitempty"` // Should HCS terminate the container once all handles have been closed
	MappedVirtualDisks          []MappedVirtualDisk `json:",omitempty"` // Array of virtual disks to mount at start
	AssignedDevices             []AssignedDevice    `json:",omitempty"` // Array of devices to assign. NOTE: Support added in RS5
}

And for comparison, examples from libcontainerd/local/local_windows.go#L95-L145;

Isolation=Process example:

{
	"SystemType": "Container",
	"Name": "5e0055c814a6005b8e57ac59f9a522066e0af12b48b3c26a9416e23907698776",
	"Owner": "docker",
	"VolumePath": "\\\\\\\\?\\\\Volume{66d1ef4c-7a00-11e6-8948-00155ddbef9d}",
	"IgnoreFlushesDuringBoot": true,
	"LayerFolderPath": "C:\\\\control\\\\windowsfilter\\\\5e0055c814a6005b8e57ac59f9a522066e0af12b48b3c26a9416e23907698776",
	"Layers": [{
		"ID": "18955d65-d45a-557b-bf1c-49d6dfefc526",
		"Path": "C:\\\\control\\\\windowsfilter\\\\65bf96e5760a09edf1790cb229e2dfb2dbd0fcdc0bf7451bae099106bfbfea0c"
	}],
	"HostName": "5e0055c814a6",
	"MappedDirectories": [],
	"HvPartition": false,
	"EndpointList": ["eef2649d-bb17-4d53-9937-295a8efe6f2c"]
}

Isolation=Hyper-V example:

{
	"SystemType": "Container",
	"Name": "475c2c58933b72687a88a441e7e0ca4bd72d76413c5f9d5031fee83b98f6045d",
	"Owner": "docker",
	"IgnoreFlushesDuringBoot": true,
	"Layers": [{
		"ID": "18955d65-d45a-557b-bf1c-49d6dfefc526",
		"Path": "C:\\\\control\\\\windowsfilter\\\\65bf96e5760a09edf1790cb229e2dfb2dbd0fcdc0bf7451bae099106bfbfea0c"
	}],
	"HostName": "475c2c58933b",
	"MappedDirectories": [],
	"HvPartition": true,
	"EndpointList": ["e1bb1e61-d56f-405e-b75d-fd520cefa0cb"],
	"DNSSearchList": "a.com,b.com,c.com",
	"HvRuntime": {
		"ImagePath": "C:\\\\control\\\\windowsfilter\\\\65bf96e5760a09edf1790cb229e2dfb2dbd0fcdc0bf7451bae099106bfbfea0c\\\\UtilityVM"
	}
}

@thaJeztah
Copy link
Member Author

Interesting bit from the above (comparing to the examples);

  • We create a hyper-v container ("HvPartition": true)
  • But also set options that (according to the examples) only (should be?) set for process-isolation containers;
    • LayerFolderPath

But, perhaps LayerFolderPath is only set if --data-root is non-standard (which looks to be the case in our CI; we use the D:\ drive for storage, not C:\

@thaJeztah
Copy link
Member Author

Actually; Go doc says; Windows Server Containers only (which is process isolation?)

Where the layer folders are located. Used by **Windows Server Containers only**. Format  %root%\windowsfilter\containerID

Windows Defender interferes with some of our integration tests
and causes the tests to take a long time to run. Other Windows
machines are configured with Windows Defender disabled (for this
reason), but these Windows machine's configuration has to be updated.

Disable Windows Defender as part of the Jenkins pipeline while the
machine configuration is not yet updated.

Signed-off-by: Sebastiaan van Stijn <[email protected]>
This may catch a regression in Windows 1903 and up, where setting a custom size
causes `docker build` to fail (possibly because `docker build` overrides the size
with a custom size of 127GB)

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@TBBle
Copy link
Contributor

TBBle commented Mar 26, 2022

Looking back, the outcome hasn't changed. The presence of LayerFolderPath does stick out as likely the problem and #42277 (comment) is correct. I don't know the HCS v1 API beyond what's here, but I guess that %root% value is sourced elsewhere, and is used to automatically determine the directory for the VHDX underneath the scratch layer.

It might be worth adding a Hyper-V-under-ContainerD parallel as well, in case "You can only put layer images on C:" turns out out to be a HCS v1 limitation (although I'd think someone would have hit this with dockerd at some point. Moving the datadir on Windows is not uncommon.)

@olljanat
Copy link
Contributor

FYI. I tested to run CI on self-hosted GitHub runner based on Windows 2022 + Hyper-V isolation + HCS v1 by using olljanat@371dd21 and these looks to be tests which fails nowadays:

=== FAIL: github.com/docker/docker/integration/container TestNetworkLocalhostTCPNat (24.85s)
    nat_test.go:50: assertion failed: error is not nil: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.

=== FAIL: github.com/docker/docker/integration/container TestContainerWithAutoRemoveCanBeRestarted/kill (40.81s)
    restart_test.go:198: assertion failed: error is not nil: Error response from daemon: Cannot restart container 8d2bc287ede4c58687d4441769dc01bda905888ca07eae9fd54fa42d739a4b2e: container 8d2bc287ede4c58687d4441769dc01bda905888ca07eae9fd54fa42d739a4b2e encountered an error during hcs::System::Start: failure in a Windows system call: A communication protocol error has occurred between the virtual machine or container and the host. (0xc0370111)
    restart_test.go:193: Cleaning up test container failed with error: Error response from daemon: No such container: 8d2bc287ede4c58687d4441769dc01bda905888ca07eae9fd54fa42d739a4b2e
    --- FAIL: TestContainerWithAutoRemoveCanBeRestarted/kill (40.81s)

=== FAIL: github.com/docker/docker/integration/container TestContainerWithAutoRemoveCanBeRestarted/stop (40.40s)
    restart_test.go:198: assertion failed: error is not nil: Error response from daemon: Cannot restart container 1895e0159c2af7f152db676117aa3a54b4767ba45409347cac42321e679311e9: container 1895e0159c2af7f152db676117aa3a54b4767ba45409347cac42321e679311e9 encountered an error during hcs::System::Start: failure in a Windows system call: A communication protocol error has occurred between the virtual machine or container and the host. (0xc0370111)
    restart_test.go:193: Cleaning up test container failed with error: Error response from daemon: No such container: 1895e0159c2af7f152db676117aa3a54b4767ba45409347cac42321e679311e9
    --- FAIL: TestContainerWithAutoRemoveCanBeRestarted/stop (40.40s)

=== FAIL: github.com/docker/docker/integration/container TestContainerWithAutoRemoveCanBeRestarted (81.23s)

=== FAIL: github.com/docker/docker/integration/container TestWaitRestartedContainer/default (44.79s)
    wait_test.go:208: assertion failed: error is not nil: Error response from daemon: Cannot restart container 66d543226927bb2fd03f7b3613e1a6ecd8e5f5f42c938cfb4d1a51cff116c696: container 66d543226927bb2fd03f7b3613e1a6ecd8e5f5f42c938cfb4d1a51cff116c696 encountered an error during hcs::System::Start: failure in a Windows system call: A communication protocol error has occurred between the virtual machine or container and the host. (0xc0370111)
    --- FAIL: TestWaitRestartedContainer/default (44.79s)

=== FAIL: github.com/docker/docker/integration/container TestWaitRestartedContainer/not-running (45.23s)
    wait_test.go:208: assertion failed: error is not nil: Error response from daemon: Cannot restart container d52f6b1f66ac0094f3956c56a9b90992873783c71b228426dbbddbbddc225805: container d52f6b1f66ac0094f3956c56a9b90992873783c71b228426dbbddbbddc225805 encountered an error during hcs::System::Start: failure in a Windows system call: A communication protocol error has occurred between the virtual machine or container and the host. (0xc0370111)
    --- FAIL: TestWaitRestartedContainer/not-running (45.23s)

=== FAIL: github.com/docker/docker/integration/container TestWaitRestartedContainer/next-exit (45.34s)
    wait_test.go:208: assertion failed: error is not nil: Error response from daemon: Cannot restart container 8bb50d6938d0e19c65bd135cd84d6a426f0717ca3ee4313ffbb2f07fb0b50a06: container 8bb50d6938d0e19c65bd135cd84d6a426f0717ca3ee4313ffbb2f07fb0b50a06 encountered an error during hcs::System::Start: failure in a Windows system call: A communication protocol error has occurred between the virtual machine or container and the host. (0xc0370111)
    --- FAIL: TestWaitRestartedContainer/next-exit (45.34s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants