Description
When using larger image layers, gzip can perform fairly slowly in serial operation. It's a quick fix to replace golang gzip with native parallel gzip (pigz/unpigz) to dramatically speed things up while still maintaining backwards compatibility with standard gzip. I've done some tests in Moby and find it's well worth enabling pigz via environment variable USE_PIGZ=Y added to /etc/sysconfig/docker (EnvironmentFile in systemd unit). A decompression comparison of performance from debug logs can be found below, showing about twice the performance for images > 1MB composed of more than one gzip block. As compressed gzip blocks are variable length without length in header, decompression is fairly limited in performance as blocks are scanned for.
Layer compression scales much better - almost linearly with CPU count up to memory and bus limitations (10x faster on my 12 core machine) but I haven't yet implemented it into CompressStream.
The following log excerpts are taken with -D (debug) and devicemapper thinpool as storage engine. overlay2, etc don't seem to show untar time but do reflect similar results.
format:
docker command (compressed size, expanded size)
Standard gzip runtime:
USE_PIGZ=Y runtime:
docker pull fedora (75MB expands to 231MB)
dockerd[22526]: time="2017-09-06T12:45:07.768143343+01:00" level=debug msg="Untar time: 6.99320561s"
dockerd[17897]: time="2017-09-06T12:43:52.809042830+01:00" level=debug msg="Untar time: 3.675086276s"
docker pull shoop/oel7 (120MB expands to 352MB)
dockerd[3836]: time="2017-09-07T15:11:01.935045814+01:00" level=debug msg="Untar time: 9.368929673s"
dockerd[8738]: time="2017-09-07T15:13:48.979072649+01:00" level=debug msg="Untar time: 4.869739837s"
docker pull bash
2017-09-06T12:53:33.857064636+01:00" level=debug msg="Untar time: 0.056180032s"
2017-09-06T12:52:35.118040346+01:00" level=debug msg="Untar time: 0.052013559s"
Describe the results you currently experience:
The container equivalent of coffee breaks while booting.
Describe the results you expected:
Half the time spent waiting for images to be ready.
Additional information you deem important (e.g. issue happens only occasionally):
In my test code, I default back to normal gzip decompression for any error, such as no unpigz present or no exec permissions on it. This seems to work OK and add pigz/unpigz as soft depedencies.
Output of docker version:
Additional environment details (AWS, VirtualBox, physical, etc.):
Physical dev box - HP Z600 2x6 Xeon. All tests run from disk cache to eliminate I/O variable.
Description
When using larger image layers, gzip can perform fairly slowly in serial operation. It's a quick fix to replace golang gzip with native parallel gzip (pigz/unpigz) to dramatically speed things up while still maintaining backwards compatibility with standard gzip. I've done some tests in Moby and find it's well worth enabling pigz via environment variable USE_PIGZ=Y added to /etc/sysconfig/docker (EnvironmentFile in systemd unit). A decompression comparison of performance from debug logs can be found below, showing about twice the performance for images > 1MB composed of more than one gzip block. As compressed gzip blocks are variable length without length in header, decompression is fairly limited in performance as blocks are scanned for.
Layer compression scales much better - almost linearly with CPU count up to memory and bus limitations (10x faster on my 12 core machine) but I haven't yet implemented it into CompressStream.
The following log excerpts are taken with -D (debug) and devicemapper thinpool as storage engine. overlay2, etc don't seem to show untar time but do reflect similar results.
format:
docker command (compressed size, expanded size)
Standard gzip runtime:
USE_PIGZ=Y runtime:
docker pull fedora (75MB expands to 231MB)
dockerd[22526]: time="2017-09-06T12:45:07.768143343+01:00" level=debug msg="Untar time: 6.99320561s"
dockerd[17897]: time="2017-09-06T12:43:52.809042830+01:00" level=debug msg="Untar time: 3.675086276s"
docker pull shoop/oel7 (120MB expands to 352MB)
dockerd[3836]: time="2017-09-07T15:11:01.935045814+01:00" level=debug msg="Untar time: 9.368929673s"
dockerd[8738]: time="2017-09-07T15:13:48.979072649+01:00" level=debug msg="Untar time: 4.869739837s"
docker pull bash
2017-09-06T12:53:33.857064636+01:00" level=debug msg="Untar time: 0.056180032s"
2017-09-06T12:52:35.118040346+01:00" level=debug msg="Untar time: 0.052013559s"
Describe the results you currently experience:
The container equivalent of coffee breaks while booting.
Describe the results you expected:
Half the time spent waiting for images to be ready.
Additional information you deem important (e.g. issue happens only occasionally):
In my test code, I default back to normal gzip decompression for any error, such as no unpigz present or no exec permissions on it. This seems to work OK and add pigz/unpigz as soft depedencies.
Output of
docker version:Additional environment details (AWS, VirtualBox, physical, etc.):
Physical dev box - HP Z600 2x6 Xeon. All tests run from disk cache to eliminate I/O variable.