Docker fails to create Windows Containers with a file larger than 8gB

**Description**

When trying to create a container on Windows with a file larger than 8gB (8,589,934,592 bytes of random data is my test-case), Docker fails to commit the layer, giving an error along the lines of [`re-exec error: exit status 1: output: write \\?\C:\ProgramData\docker\tmp\hcs816749287\Files\UnrealEngine\LocalBuilds\Engine\Windows\Engine\Plugins\Experimental\BlastPlugin\Binaries\Win64\UE4Editor-BlastAuthoring.pdb: There is not enough space on the disk.`](https://github.com/adamrehn/ue4-docker/issues/63#issuecomment-580964306) or [`hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)`](https://github.com/adamrehn/ue4-docker/issues/63#issuecomment-580585672).

**Steps to reproduce the issue:**

For better debug logging:
* Set a system env-var `DOCKER_WINDOWSFILTER_NOREEXEC=true`
* Ensure dockerd has both debug enabled, and log-level debug set:
```
  "debug": true,
  "log-level": "debug",
```
* Start/restart dockerd.exe with those settings.

1. Given an 8gB file, e.g.,
```powershell
$writer = [System.IO.File]::OpenWrite('8gfile')
$random = new-object Random
$blockSize = 1073741824
$bytes = new-object byte[] $blockSize
for ($i=0; $i -lt 8; $i++)
{
 $random.NextBytes($bytes)
 $writer.Write($bytes, 0, $blockSize)
}
$writer.Close()
```
2. And the following Dockerfile
```
FROM mcr.microsoft.com/windows/nanoserver:1909
COPY 8gfile 8gfile
```
3. `docker build .`

Alternative repro, using the [`ue4-docker` tool](https://github.com/adamrehn/ue4-docker/issues/63#issuecomment-580585672):
1. Given Python 3 (or a virtual env)
2. `pip install ue4-docker`
3. `ue4-docker diagnostics 8gig`

This runs the file-creation script inside the container, rather than using `COPY`, and triggers the same failure. See the [Dockerfile](https://github.com/adamrehn/ue4-docker/blob/master/ue4docker/dockerfiles/diagnostics/8gig/windows/Dockerfile) for reference.

**Describe the results you received:**

The CLI may simply report `archive/tar: invalid tar header` and terminate.

Errors as described above in the logs, on the various times I've tried. There's a lot of diagnostics at https://github.com/adamrehn/ue4-docker/issues/44 where I originally hit the issue with Docker EE 18.09.5 for Windows Server.

The current log output for the `COPY` stage in my repro looks like this, with blank lines around the `hcsshim::ImportLayer` span which is failing for some reason.
```
[09:27:11.061][WindowsDaemon     ][Info   ] debug: [BUILDER] Cache miss: [cmd /S /C #(nop) COPY file:6eacbaee636d1db8e156690cfb8319f516a3dc7aed07881788d2b01bc5f63c42 in 8gfile ]
[09:27:11.061][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:11.061][WindowsDaemon     ][Info   ] debug: Calling proc (1) [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:11.062][WindowsDaemon     ][Info   ] debug: Calling proc (2) [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:11.062][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath - succeeded [mountPath=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:11.062][WindowsDaemon     ][Info   ] debug: hcsshim::CreateScratchLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.062][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:11.062][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid - succeeded [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f guid=9a520dd8-e003-5ac9-aafa-90282d69bead]
[09:27:11.068][WindowsDaemon     ][Info   ] debug: hcsshim::CreateScratchLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.072][WindowsDaemon     ][Info   ] debug: WindowsGraphDriver Get() id 3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc mountLabel 
[09:27:11.073][WindowsDaemon     ][Info   ] debug: hcsshim::ActivateLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.084][WindowsDaemon     ][Info   ] debug: hcsshim::ActivateLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.084][WindowsDaemon     ][Info   ] debug: hcsshim::PrepareLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.085][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:11.086][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid - succeeded [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f guid=9a520dd8-e003-5ac9-aafa-90282d69bead]
[09:27:11.111][WindowsDaemon     ][Info   ] debug: hcsshim::PrepareLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.111][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.111][WindowsDaemon     ][Info   ] debug: Calling proc (1) [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.113][WindowsDaemon     ][Info   ] debug: Calling proc (2) [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:11.114][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc mountPath=\\?\Volume{1c31fe35-e5f2-4858-b1bf-93dd6de9e54a}]
[09:27:11.115][WindowsDaemon     ][Info   ] debug: CopyFileWithTar(\\?\C:\ProgramData\Docker\tmp\docker-builder002875846\8gfile, \\?\Volume{1c31fe35-e5f2-4858-b1bf-93dd6de9e54a}\8gfile)
[09:27:31.096][WindowsDaemon     ][Info   ] debug: hcsshim::UnprepareLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:31.098][WindowsDaemon     ][Info   ] debug: hcsshim::UnprepareLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:31.098][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.099][WindowsDaemon     ][Info   ] debug: Calling proc (1) [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.099][WindowsDaemon     ][Info   ] debug: hcsshim::ExportLayer [exportFolderPath=C:\ProgramData\Docker\tmp\hcs621940270 path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:27:31.099][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.100][WindowsDaemon     ][Info   ] debug: Calling proc (2) [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.100][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid - succeeded [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f guid=9a520dd8-e003-5ac9-aafa-90282d69bead]
[09:27:31.100][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f mountPath=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.100][WindowsDaemon     ][Info   ] debug: hcsshim::CreateLayer [path=C:\ProgramData\Docker\windowsfilter\e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9 parent=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.101][WindowsDaemon     ][Info   ] debug: hcsshim::CreateLayer - succeeded [parent=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f path=C:\ProgramData\Docker\windowsfilter\e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9]
[09:27:31.102][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.102][WindowsDaemon     ][Info   ] debug: Calling proc (1) [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.102][WindowsDaemon     ][Info   ] debug: Calling proc (2) [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:31.103][WindowsDaemon     ][Info   ] debug: hcsshim::GetLayerMountPath - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f mountPath=C:\ProgramData\Docker\windowsfilter\3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:53.663][WindowsDaemon     ][Info   ] debug: hcsshim::ExportLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc exportFolderPath=C:\ProgramData\Docker\tmp\hcs621940270]

[09:27:53.669][WindowsDaemon     ][Info   ] debug: hcsshim::ImportLayer [importFolderPath=C:\ProgramData\Docker\tmp\hcs438461328 path=\\?\C:\ProgramData\Docker\windowsfilter\e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9]
[09:27:53.669][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:27:53.669][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid - succeeded [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f guid=9a520dd8-e003-5ac9-aafa-90282d69bead]
[09:27:53.676][WindowsDaemon     ][Error  ] hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3) [path=\\?\C:\ProgramData\Docker\windowsfilter\e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9 importFolderPath=C:\ProgramData\Docker\tmp\hcs438461328 error=hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)]

[09:28:49.409][WindowsDaemon     ][Info   ] debug: Cleaning up layer e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9: archive/tar: invalid tar header
[09:28:49.409][WindowsDaemon     ][Info   ] debug: hcsshim::GetComputeSystems - Begin Operation
[09:28:49.410][WindowsDaemon     ][Info   ] debug: HCS ComputeSystem Query [json={}]
[09:28:49.412][WindowsDaemon     ][Info   ] debug: hcsshim::GetComputeSystems - End Operation - Success
[09:28:49.421][WindowsDaemon     ][Info   ] debug: hcsshim::DestroyLayer [path=C:\ProgramData\Docker\windowsfilter\e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9-removing]
[09:28:49.422][WindowsDaemon     ][Info   ] debug: hcsshim::DestroyLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\e2c4b4241777c2f707a7e75fdfde0b5e17570bb92c915e1176f82e01b795a2e9-removing]
[09:28:49.423][WindowsDaemon     ][Info   ] debug: hcsshim::PrepareLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:28:49.423][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f]
[09:28:49.423][WindowsDaemon     ][Info   ] debug: hcsshim::NameToGuid - succeeded [name=3c5bfe2daef2c14ab2185e679ccf26891f306db3c8af06cd6a0b06eecc3a0f1f guid=9a520dd8-e003-5ac9-aafa-90282d69bead]
[09:28:49.423][WindowsDaemon     ][Info   ] debug: hcsshim::PrepareLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:28:49.423][WindowsDaemon     ][Info   ] debug: WindowsGraphDriver Put() id 3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc
[09:28:49.423][WindowsDaemon     ][Info   ] debug: hcsshim::UnprepareLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:28:49.424][WindowsDaemon     ][Info   ] debug: hcsshim::UnprepareLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:28:49.424][WindowsDaemon     ][Info   ] debug: hcsshim::DeactivateLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:28:49.698][WindowsDaemon     ][Info   ] debug: hcsshim::DeactivateLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc]
[09:28:49.698][WindowsDaemon     ][Info   ] debug: hcsshim::GetComputeSystems - Begin Operation
[09:28:49.699][WindowsDaemon     ][Info   ] debug: HCS ComputeSystem Query [json={}]
[09:28:49.699][WindowsDaemon     ][Info   ] debug: hcsshim::GetComputeSystems - End Operation - Success
[09:28:49.700][WindowsDaemon     ][Info   ] debug: hcsshim::DestroyLayer [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc-removing]
[09:28:49.701][WindowsDaemon     ][Info   ] debug: hcsshim::DestroyLayer - succeeded [path=C:\ProgramData\Docker\windowsfilter\3c0a616503a2de2ed418f49aabcf18e54e7809d5340a0c859f877c45a5c6c9bc-removing]
```


**Describe the results you expected:**

The container commits its layer successfully.

**Additional information you deem important (e.g. issue happens only occasionally):**

In case it's useful, I did a diagnostics upload from Docker Desktop: 98C09E8E-11CF-4AA0-AA5F-9160DDC01FE8/20200201223719

As well as reproducing with Docker Desktop's bundled 19.03.5 Docker Engine, I was also able to reproduce the problem with a build of dockerd.exe from 513b207b002dfe070705d790326f731a803cf68d, and also reproduced it (but with less-useful logs) from 513b207b002dfe070705d790326f731a803cf68d with Microsoft/hcsshim revendored to 99f064b70e88824d6653da19160d4ae02fd35a72.

**Output of `docker version`:**

```
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:37 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:36:50 2019
  OS/Arch:          windows/amd64
  Experimental:     false
```

**Output of `docker info`:**

```
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 19.03.5
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 18363 (18362.1.amd64fre.19h1_release.190318-1202)
 Operating System: Windows 10 Pro Version 1909 (OS Build 18363.592)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.96GiB
 Name: KEITARO
 ID: NDQJ:UAVJ:AHVY:OAXE:FYZU:4JGK:FV2M:PCET:ZEBX:AGSH:7L7H:GHCF
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 27
  System Time: 2020-02-02T10:08:41.0300877+11:00
  EventsListeners: 1
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
```

**Additional environment details (AWS, VirtualBox, physical, etc.):**

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker fails to create Windows Containers with a file larger than 8gB #40444

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Docker fails to create Windows Containers with a file larger than 8gB #40444

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions