Repository navigation
[Feature request] Improve Docker image performance with an alternative base image instead of Alpine #2299
Description
Which version are you referring to
3.1dev
Please check this repo whether this is a known feature request
No existing issue came up after a few searches and browsing of results. The closest I found were some 2019 observations with an Alpine image.
Describe your feature request (if it's a technical feature)
It'd be nice if a Debian base image could be offered. It does not have the minimal disk usage that an alpine base image offers, but performance can be notably better.
If your feature request otherwise is related to a usage problem, please describe it
There are reports about DNS performance being problematic on Alpine images, but I did not notice a difference compared to testing against an IP address directly. So I assume it's more to do with musl vs glibc.
I have observed a repeatable difference between the current Alpine image and a Debian base image alternative of about 50% in time to perform a test with --preference, and alpine being 2-3x slower without --preference. The Debian image performed at native (outside of docker) or very close to.
Describe the solution you'd like
I adapted the current alpine Dockerfile to one for Debian:
# syntax=docker.io/docker/dockerfile:1
# HereDoc (EOF) feature (avoids needing `&& \`) requires BuildKit:
# https://docs.docker.com/engine/reference/builder/#here-documents
FROM debian:bullseye-slim
ARG DEBIAN_FRONTEND=noninteractive
RUN <<EOF
## Prepare package install
apt-get -qq update
apt-get -qq upgrade
# Install packages
apt-get install -qq --no-install-recommends bsdmainutils procps dnsutils
# Cleanup
apt-get -qq clean
rm -rf /var/lib/apt/lists/*
# Create testssl user
adduser --system --group --uid 1000 --gecos "testssl user" --shell /bin/bash --disabled-password testssl
# Prepare for COPY
ln -s /home/testssl/testssl.sh /usr/local/bin/
mkdir -m 755 -p /home/testssl/etc /home/testssl/bin
EOF
USER testssl
WORKDIR /home/testssl/
COPY --chown=testssl:testssl etc/ /home/testssl/etc/
COPY --chown=testssl:testssl bin/ /home/testssl/bin/
COPY --chown=testssl:testssl testssl.sh /home/testssl/
ENTRYPOINT ["testssl.sh"]
CMD ["--help"]- Image weight 138MB vs 41MB of current Alpine image.
- I have also tried with Fedora as another base image where performance was likewise better. It's very likely a
glibcvsmuslissue or something else going on with Alpine base images. - There is also this caveat mentioned about DNS on alpine. A related DNS issue implies it sends out A + AAAA record queries in parallel which apparently may be a cause.
Comparison
Here are some observations I noticed when testing against a mailserver container (with custom network and also with host network, not performance change), with a docker image and native install of testssl.sh:3.1dev.
Single test
# ~60 seconds to complete, same as native. Alpine ~150s:
$ docker run --rm -it --network test-network testssl:debian example.test:465
# 11-15 seconds to complete, likewise competitive with native (10-13s). Alpine 25-37s:
$ docker run --rm -it --network test-network testssl:debian example.test:465Parallel tests
These results are from running against the docker-mailserver test suite:
-
--quiet --preference --mode paralleloptions are used, along with the ENVADDTL_CA_FILESto verify chain of trust. -
Time in milliseconds is reported at the end of each test case.
-
Each test case has the mail-server offer RSA, ECDSA, or both as the certificate, and with one of two cipher suite configs.
-
Each test case is running
testssl.shwith--fileto perform tests across 7 ports from two services (Postfix and Dovecot):--jsonfile-pretty port_25.json --starttls smtp example.test:25 --jsonfile-pretty port_587.json --starttls smtp example.test:587 --jsonfile-pretty port_465.json example.test:465 --jsonfile-pretty port_110.json --starttls pop3 example.test:110 --jsonfile-pretty port_995.json example.test:995 --jsonfile-pretty port_143.json --starttls imap example.test:143 --jsonfile-pretty port_993.json example.test:993
# Debian base image:
$ make clean generate-accounts test/tls_cipherlists
tls_cipherlists.bats
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=intermediate' + RSA [45863]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=intermediate' + ECDSA [36584]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=intermediate' + ECDSA with RSA fallback [55790]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=modern' + RSA [43647]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=modern' + ECDSA [36443]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=modern' + ECDSA with RSA fallback [52742]
6 tests, 0 failures in 274 seconds
# Alpine base image (worse case was around 330 sec):
$ make clean generate-accounts test/tls_cipherlists
tls_cipherlists.bats
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=intermediate' + RSA [52541]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=intermediate' + ECDSA [42963]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=intermediate' + ECDSA with RSA fallback [62151]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=modern' + RSA [45966]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=modern' + ECDSA [45853]
✓ [Security - TLS] (cipher lists) 'TLS_LEVEL=modern' + ECDSA with RSA fallback [57875]
6 tests, 0 failures in 311 secondsThe benefit was less apparent with --mode parallel, about 40 seconds or so difference. That is on a system with an old i5-6500 (4 cores / 4 threads) that would be under full load by either base image while running.
Activity
polarathene commentedon Jan 29, 2023
Good news is that it seems something happened with the newer 3.17 release of Alpine that greatly reduces the performance delta (all tests report version
3.2rc2):Still not quite as good as changing the base image to Debian (or another base image with
glibc), but pretty decent.drwetter commentedon Jan 30, 2023
Thanks @polarathene !
You were faster as I could respond during the weekend. If you're sure 3.17 works 100% I'd first appreciate for both branches a PR. The remaining 10% we can worry about afterwards.
jauderho commentedon Jan 30, 2023
Alpine 3.17.1 is the latest so that's what should be used.
drwetter commentedon Jan 30, 2023
With all due respect @jauderho but that is nonsense. Try
docker run alpine:3.17 cat /etc/os-release.3.17.1 will be automatically chosen.
You shall not fix the patchlevel of the Alpine version otherwise you forget it and you have security issues.
drwetter commentedon Jan 30, 2023
@polarathene : if you want to help with a PR, there's also a "Dockerfile.git"
jauderho commentedon Jan 30, 2023
@drwetter That way is the implicit method which is not a best practice. The "correct" way is to explicitly specify the image including the hash and use Dependabot to keep things up to date.
So
FROM alpine:3.17.1@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753apolarathene commentedon Jan 30, 2023
Yes, but someone needs to create a PR to do so, if that's something you'd like to see, perhaps you want to contribute what you can via PR and delegate anything else that requires @drwetter to configure for the repo?
Explicit version bumping is simpler until then, and infrequent enough that I'm not bothered by it. I wouldn't normally expect a big difference in the performance between releases of Alpine, this was just an observation from investigating why the current image was much worse than a
glibcbase image that performed like runs without Docker involved.@jauderho I've not set up version pinning in CI before, so if you'd like to tackle that, awesome 👍
@drwetter I'm happy to open a PR to bump to 3.17 for now. Did you want that for both
DockerfileandDockerfile.git, or just one of those?Which branches?
3.1devand3.0?I wouldn't expect these base image upgrades to cause much risk of regression.
Presently we're (
docker-mailserver) only usingtestssl.shin our test suite to test multiple service ports with--preferenceand output JSON to check the cipher suites results are equal to what we expect them to be.I did test without
--preferenceagainst Github and saw no issues with the terminal output there 👍I have noticed that using the
--openssl /usr/bin/openssloption has changed from version1.1.1sto3.0.7(both released in Nov 2022 with 94 ciphers), otherwise they both default to the openssl binary you bundle into the image from this repo.polarathene commentedon Jan 31, 2023
Also, in mentioning
--openssl /usr/bin/openssl, at least for my use-case, the performance improves a fair bit, especially in the parallel test runs:--opensslI still see the 3.16 Alpine image hit 300+ seconds running those tests.--openssloption is a bit slower and seems to manage around 220 sec, however if I install the OpenSSL1.1.1scompat package (instead of using3.0.7) and use--openssl /usr/bin/openssl1.1- it performs just as well in the parallel tests, one result was 189 sec. Likewise the test against Github completed in 104 sec.drwetter commentedon Jan 31, 2023
To be honest: I don't like that. It makes it depended on dependabot. with no reason. It worked before. And it causes more PRs.
Do you want to amend the PRs or file new ones? Otherwise I'll close them and I'll make new ones. Just let me know.
polarathene commentedon Feb 1, 2023
@drwetter I've opened a new PR as requested.
If the other changes don't interest you at all, feel free to close the PR and just bump the Alpine version 👍
Upgrade Alpine version for both Dockerfiles
Upgrade Alpine version for both Dockerfiles (3.0)
drwetter commentedon Mar 14, 2023
It´s as you said 138 MB includes lot's of unnecessary stuff because of the debian eco system:
maxmind?? libxml, + libjson?? :-(
With ldnsutils it's smaller: 99.5MB and not using that stuff:
ncal? libpcap0?
drwetter commentedon Mar 14, 2023
Probably we can't get it any smaller, unless we're using a multistage image.
drwetter commentedon Mar 14, 2023
This was w/o openssl and with libidn. Using a similar package selection under alpine I am @ 28.5MB.
For 3.1dev we would need for both openssl, socat, curl and libidn.
polarathene commentedon Mar 16, 2023
TL;DR: I managed to get a glibc image with your required dependencies down to
60 MiB(Current 3.1 Alpine image is36.5 MiB).I have looked into minimizing the images with Fedora and OpenSUSE as their package managers have an
--installrootoption that could be used to make a smaller image.I am not familiar with how to do that with Debian or if it would be any better size wise. That said Google does offer a
gcr.io/distroless/base-nossl-debian11image (orbase-debian11with openssl) that is 15 MB in size (20MB with openssl). That's without a package manager, and from what I've seen with these other packages the weight tends to increase a fair bit.Fedora was also including a bunch of extra dependencies for some packages that didn't seem necessary. Especially with
util-linux(forhexdump) andvim-common(forxxd), both binaries likely not needing the bulk of it.With OpenSUSE it was better:
ldns+libidn2-0fordrilland theidnsupport adds about13 MiB, which can be replaced withbusybox-bind-utilsto recover most of that (but then loses the benefit ofdrill+idn).socatwas added, it had a bunch of overlap withldns, only adding approx1 MiB.grep,gawk,sed,coreutilsall bring inprocpsandbash(30 MiBweight if these are the first packages considered, otherwise about13.5 MiB, with1 MiBof that forbash).busybox-util-linux(provideshexdump) andbusybox-vi(providesxxd), only about2 MiB(on Fedora either one of the equivalents added about30 MiB).testssl.shdetects busyboxpsanddd, otherwise the busybox equivalent packages could be used and bring down the weight by about10 MiBmore.curlthen adds close to5 MiB. I think Fedora offered acurl-minimal.opensslonly adds about1.5 MiBafter all this./usr/sharerecovers4-9MiB.This results in an image that is
46 MiB(originalopensuse/leapbase image is113 MiB), and then the14 MiBfrom this git repo that is added viaCOPYbrings total image size to60 MiB(uncompressed). The current3.1devalpine image is36.5 MiB.It's probably possible to reduce that further (might have some luck with the Slim project), but this is as far as I'll go 😅
Less than
30 MiBextra isn't that bad for a non-alpine image I think?Here's the
Dockerfile, it is using some BuildKit features, but this works as-is with the Docker Engine v23 release out of the boxdocker build --tag testssl-leap:builderstage withdocker build --target builder --tag testssl-leap-builder .diveis a useful tool to inspect layers (although the HereDocEOFfeature inRUNseems to cause some issues).divethat is actively updated:docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock jauderho/dive testssl-leap-builderhexdump/xxd, and there are some interesting ones likehexylbut they're not drop-in replacements. If you really want to push down the image size, having support intestssl.shfor Go or Rust based alternatives forhexdump,socat,drilland perhaps evencoreutils(there's a Rust one that I'm aware of) could probably shave off a good bulk of the weight.opensslis that relevant going forward? If it's generally not likely to provide much benefit testing modern servers, you could consider dropping it or providing it as an image variant (13.5MiB copied into image: 1.2MiBtestssl.sh, 10.3MiBbin/openssl+.i686&.krbvariants, 2MiBetc/)..i686file isn't useful in the image (I assume the same for.krb), while thex86_64build wouldn't be useful on ARM hosts/containers?drwetter commentedon Mar 16, 2023
cool!
drwetter commentedon Mar 19, 2023
For 3.0 I rather went to openssl on alpine, see #2340. For 3.1dev let's change the container image.
For 3.1dev I am not sure how the multistage works in reality on the command line, and I am not sure whether the whole image size pulled is smaller.
Let's finalize this or another image next week. Any Dockerfile is probably better to be not in HEREDOC format. And pulling automatically latest is not I'd like to do. If we'll go for opensuse opensuse/leap:15.4 is what is the best.
The static build will probably be not the primary binary in 3.3dev but I won't change this in the release candidate phase.
polarathene commentedon Mar 20, 2023
Hmm? I did the multi-stage in a Fedora 37 VM, installed Docker via the Docker Install Fedora docs, so that it uses v23 (Github Actions should also have that). I think it should work on older releases too.
I made sure to add some comments in the
Dockerfileabove to explain some things if helpful.builder).builderstage is done first.FROM scratch) copies over the minimal OS install we made in/buildfolder of the first stage.testssluser probably isn't necessary, but as we don't have a package installed providing theaddusercommand, I am using a feature ofRUN --mountto temporarily provide theaddusercommand from another docker image (busybox). That could be done in thebuilderimage, but needs to account for/build/etcinstead of/etc, I think openSUSEaddusercommand has an option to support that, the symlink likewise needs to be added to the/builddir.If you meant how to build it, you shouldn't need to change anything. I cited the
docker buildcommand to try it out, same as without multi-stage.If you are interested in building an earlier stage instead of the full file, you can reference the stage name, so
docker build --target builder --tag this-is-the-builder-stage ., afterwardsdocker run --rm -it this-is-the-builder-stage bash.Likewise, to get into the final image, since the
ENTRYPOINTis set to thetestssl.shcommand, you'd need to override that instead to use bash,docker run --rm -it --entrypoint bash drwetter/testssl.sh:3.1dev(adjust image tag to whatever you used withdocker build).The final image is the last stage which uses
scratch(nothing) as the base image.You'll observe that
docker image ls/docker image history image-name-herewill show you the image size on disk.If you want to inspect a breakdown of that further, look at the
divetool I mentioned earlier which lets you navigate the image layers and their impact on the images filesystem showing where all the file weight is located.With the
RUN --mount, we used for caching the package manager, that content is no longer stored in the image layers, so it's just like we cleared it manually afterwards, except locally when you build it you can re-use that cache. That is not exported with Dockers build cache, thus doesn't persist across CI builds, non-issue here.The builder stage itself will still be stored on disk as part of your build cache (locally or in CI), and CI can persist that if you've configured it to (like we do on
docker-mailserver), probably not too useful fortestssl.shthough 👍As a user, you don't have to worry about that, and you'd only pull down the final image that was built. Which should be approx 60 MiB on disk, but compressed should be less than 30 MiB on Dockerhub I think, which is how much network traffic will be used to pull it, and it'll be uncompressed once stored on disk.
That is up to you, it's rather nice for
Dockerfileto embed shell commands into a singleRUNinstead of the noisy&& \chaining on each line.In this case since it's only 4 commands in
builderyou could do that, it'd be better than individualRUNs (which would also be fine inbuilderstage as that doesn't affect the final image weight, but you'd get noise from the--mountcache instead).Another option you have, is to extract it out of the
Dockerfileand have a separate.shfile that youCOPYover.That would still pull updates to 15.4 tag, but absolutely a good idea 👍
I forgot to be more specific there while experimenting 😅
drwetter commentedon Mar 21, 2023
Sorry that I confused you. I actually can't tell myself anymore what I meant by this 😅. I know how multi stage works
yes but people need a recent docker version.
Do you want to issue the PR for Opensuse so that we can see before? I created the branch suse_docker first to see image sizes on Dockerhub and here.
no prob at all
polarathene commentedon Mar 22, 2023
Only to build. Most users will just pull the image from DockerHub.
Building with HereDoc should work on older versions (see this blogpost from mid 2021, just needs an ENV prefix). We've been using the HereDoc syntax in
docker-mailserverimages since Oct 2022 and not had any issues from our users.If someone is building locally, they can probably get Docker Engine v23 (released early this year) which requires no ENV to opt-in. Likewise CI services like Github Actions keep that software up to date and it's less of a concern there.
Yeah I can give that a go. From what I've seen elsewhere, DockerHub images tend to report being less than 50% of the uncompressed size.
drwetter commentedon Mar 22, 2023
yes. But it'll confuse users who build locally. Just want to avoid questions and problems elsewhere. My systems here are using docker 20.10. Even Opensuse Tumbleweed (rolling distro) haven't made the step to v23
polarathene commentedon Mar 22, 2023
Not been a problem for us. Do you have users that build locally often that aren't contributing?
One option is just to inline a comment into the
Dockerfilesaying it builds on Docker Engine v23, and for earlier releases (past 2 years almost) useDOCKER_BUILDKIT=1 docker build ....This requirement will likely also mean
RUN --mountwill not be acceptable, as it also relied on BuildKit feature this way AFAIK (Announced in May 2022).drwetter commentedon Mar 22, 2023
I have no idea about what the consuming or randomly contributing users are using. Who has?
That explains why the RUN-mount thing didn't work for me either last week :-)
Please let's not settle on an edge bleeding too much. Robustness now. Looking forward in next dev branch things on the desktops might look different.
polarathene commentedon Mar 22, 2023
It's not exactly something that breaks the usability of the image. This doesn't affect being able to run the image pulled from DockerHub, only building locally.
Anyone doing that is probably capable of building it on a system that's had a Docker release in the past 1-2 years, and can use
DOCKER_BUILDKIT=1 docker build ...like mentioned. That's just a documentation issue, and a rather easy one to cover in theDockerfileif it's a concern.Docker also documents how to grab the latest (or earlier) releases for the bulk of distros users are running. You add an external repo from Docker, and now you've got access to the newer releases. Anyone running Docker Desktop (GUI app) uses BuildKit implicitly, just like you will with v23 of Docker Engine going forward.
As I stated before,
docker-mailserverwhich has a history of over 50 million pulls made this change in Sep 2022 and it's not been an issue to our users.I assume adding the ENV does though? We have that tucked away in a
Makefileso thatmake buildinternally prepends it to thedocker buildcommand.It was a bit frustrating to accommodate for, but I've refactored from those features
and will send the PR shortly(EDIT: Done).Good news is the final size remains unaffected, bad news is the cache benefit for local builds is lost, and arguably it's less maintenance friendly.