Repository navigation
BUG: error querying OCSP responder #2329
Description
Before you open an issue please check which version you are running and whether it is the latest in stable / dev branch
I am running version 3.0.8
Before you open an issue please whether this is a known problem by searching the issues
I couldn't find anything
Command line / docker command to reproduce
docker run --rm -ti drwetter/testssl.sh:3.0 --phone-out -S testssl.sh
Expected behavior
The following error message is not displayed
OCSP URI http://r3.o.lencr.org, error querying OCSP responder
Your system (please complete the following information):
- OS (outside of Docker):
Ubuntu 20.04.5 LTS - Platform (outside of Docker):
Linux 5.4.0-94-generic x86_64 - OpenSSL (inside Docker):
/home/testssl/bin/openssl.Linux.x86_64
Additional context
The error message does not show up when omitting the --phone-out option, but I guess this is only suppressing the faulty OCSP request and not actually solving the issue. See docker run --rm -ti drwetter/testssl.sh:3.0 -S testssl.sh
HTTP connections from inside the docker container to the OCSP URL seem to be working:
docker run --rm -ti --entrypoint /usr/bin/curl drwetter/testssl.sh:3.0 -v http://r3.o.lencr.org/
* Trying 2.16.186.10:80...
* Connected to r3.o.lencr.org (2.16.186.10) port 80 (#0)
> GET / HTTP/1.1
> Host: r3.o.lencr.org
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Content-Length: 0
< Cache-Control: max-age=19229
< Expires: Wed, 08 Mar 2023 13:10:10 GMT
< Date: Wed, 08 Mar 2023 07:49:41 GMT
< Connection: keep-alive
<
* Connection #0 to host r3.o.lencr.org left intact
Also no difference when using the latest DEV version:
docker pull drwetter/testssl.sh
Using default tag: latest
latest: Pulling from drwetter/testssl.sh
Digest: sha256:90e073332d15789047a22ccb3a65acaae7707c50a1877e8b054db5fe6289ee2b
Status: Image is up to date for drwetter/testssl.sh:latest
docker.io/drwetter/testssl.sh:latest
docker run --rm -ti drwetter/testssl.sh --phone-out -S testssl.sh
[..]
Certificate Revocation List --
OCSP URI http://r3.o.lencr.org, error querying OCSP responder
OCSP stapling offered, not revoked
OCSP must staple extension --
[..]
Issue is persistent when running the above commands on a local notebook as well as on a virtual server in a data center.
Activity
dcooper16 commentedon Mar 8, 2023
Hi @fogs,
Thanks for reporting. My environment is different, but I got the same result. My environment is:
I can reproduce the problem at the command line:
However, everything works if I use the built-in OpenSSL:
I found something at acmesh-official/acme.sh#4048 that suggests the issue is with the openssl.cnf that comes with the distribution.
dcooper16 commentedon Mar 8, 2023
It seems that my testing was incorrect, as my environment at the command line was incorrect. I can't reliably reproduce things at the moment. However, I believe that if my environment is set up the same as as testssl.sh sets it up the result is:
drwetter commentedon Mar 9, 2023
Hi David,
OPENSSL_CONF="" is missing, otherwise the system provided openssl.cnf will be used
drwetter commentedon Mar 9, 2023
... but I believe you found the problem, David.
For 3.1dev where we have openssl in the container it works using
docker run --rm -ti drwetter/testssl.sh --phone-out -S testssl.sh, whereas w/o openssl it does not.For 3.0 there's no openssl in the container. I added it later as we needed that for STARTTLS injection.
So it should probably suffice adding OPENSSL_CONF="" in front of that call you used when the supplied openssl is used -- outside the container $OPENSSL might not be equal to $HOME/bin/openssl.Linux.x86_64
drwetter commentedon Mar 9, 2023
not quite:
drwetter commentedon Mar 9, 2023
hmm... 🤔
b_sock.c:146is referring toBIO_gethostbynamewhich callsgethostbynamewhich comes from the libc. Not sure what a openssl.cnf can fix there. Educated guess: the musl libc is to blame. It works outside the containerdrwetter commentedon Mar 9, 2023
@polarathene : what's your take on this?
polarathene commentedon Mar 9, 2023
Not really my area of expertise, but as you mention it's probably due to
musl.apk?glibc? (the extent of my knowledge of this sort of problem 😛 )I am assuming it's never worked. I know that this can be a common problem of running binaries built with glibc (linked?) and copied over into an Alpine image to run.
Does not affect
apkinstalled OpenSSL packages, nor any change running without those and justrm /etc/ssl/openssl.cnfProbably not surprising output given maintainer comments above 😅
vs with OpenSSL 3.0 package
vs with OpenSSL 1.1.x package:
drwetter commentedon Mar 11, 2023
Thx, @polarathene .
Slept over it . The current supplied binary is statically linked. My understanding would be that the system call
gethostbynameis supplied by the binary then and musl libc isn't involved.A longer while back there were problems on Arch Linux with some DNS related system calls before. I compiled the Linux binaries on a legacy VM. That could be related. In any case that doesn't help to get this off the table immediately.
OTOH This seems hard to fix properly, unless we would check in testssl.sh whether we're in a container and use for this call e.g. /usr/bin/openssl1.1 . Which seems to me like an ugly hack.
Opinions?
drwetter commentedon Mar 11, 2023
If anybody wants to check whether it's the legacy platform which is to blame: https://github.com/drwetter/openssl-1.0.2.bad is your friend
drwetter commentedon Mar 13, 2023
Minor update...
It looks a bit like the static openssl we're providing needs glibc style nss libs. In an alpine 3.17 container:
So as one can see it looks at the name service switch, finds files and dns. It tries then to find the files library, then the dns lib. Nothing worked. So it ended up with return code 1.
Alpine doesn´t provide those libraries. The error is the same using 3.16 or edge.
Possible solutions
polarathene commentedon Mar 13, 2023
Either native openssl binaries work fine AFAIK. I don't think it's specific to a container in that sense, you should have the same issue on an Alpine host?
This is what came to mind, but I was of the understanding if the binary was built with static linking that it did not matter if glibc vs musl on the system. Alpine has some history of DNS issues apparently, and perhaps is related to your findings? 🤷♂️
Same question, if the openssl binary you supply is meant to be static why would this matter?
I have had to mess with NSS in the past on a linux system (glibc though) to configure it to properly handle mdns lookups with some Avahi setup I had and remember I sunk a bit of time troubleshooting that.
If the image just needs to install something extra or tweak config, that could be done, but AFAIK it's capable of handling DNS just fine, it's only an issue with the foreign binary you're providing it?
You can update the 3.0 Dockerfile to provide
openssl, but IIRC you still need to specify--openssl /path/to/binarytotestssl.shin this case, so in addition to that, add help / docs (or if you can detect the failure and inline the advice output perhaps) that state the feature may require--openssl.If you want it as a container only behaviour, then there are a few options that can be done there, some that don't require making any cahnges to
testssl.shscript, but I'm pretty sure this would still be an issue running 3.0 on host with musl instead of glibc? Not just in a container.I doubt anyone is extending the image, you could consider switching the base image? Wasn't that already done in the past for 3.0 or 3.1 before changing to Alpine?
Alternatively, you can publish both base images, this is often done with a tag like
3.0-debian. Then you can mention in DockerHub or anywhere else relevant that Debian is available as an alternative.Alpine provides the size benefit, but often at the expense of compatibility and performance (both in runtime speed and memory usage AFAIK). Debian is often a base image offered as an alternative (but it lags behind in package versions and much slower release cadence).
polarathene commentedon Mar 14, 2023
I think I got my answer about the static build not being able to prevent this 😅
Common advice is to build with static linking to
muslinstead ofglibcapparently, but for NSS, this has some implications about expected behaviour (for most uses oftestssl.shprobably not an issue though?).So technically I guess one other option is to go that route and distribute that binary.
I'm pretty sure the Docker ENTRYPOINT can just run
testssl.sh --openssl /usr/bin/opensslif you don't need to default to static build in the container 🤷♂️ (I totally understand why you'd rather avoid that implicit opt-in)drwetter commentedon Mar 14, 2023
At some point of time actually it's the best to get rid of that binary. I feel no so tempted to recompile with musl libc, bc of that and also bc I sense there will be one or the other obstacle when compiling.
drwetter commentedon Mar 14, 2023
probably. I have none though :-)
The NSS calls come probably from the static glibc but it expects a Gnu libc system . The static compile process doesn´t include those external libs. In my limited experience with libs it seem just normal that some system calls are coming from a lib and end up in another lib. And udring compile time one doesn´t know what the system is configure to use for DNS lookups.
See ;-) On my desktop system I have a couple of nss_mdns libs . Openssl could probably also use that when /etc/nsswitch.conf contains that. /usr/lib64/libnss_wins.so.2 on my system btw is provided by Samba. Oh, and btw, see
nss_ldap(5). Got the picture?a) for 3.0: probably not . For 3.1dev: not sure. Just pulled the bullseye-slim image. It´s 80MB. Was just trying to figure out how you ended up at 135 MB. But using your Dockerfile in #2299 I got
Error response from daemon: dockerfile parse error line 10: unknown instruction: APT-GETb) drwetter/testssl.sh@cbe38cc, for size reasons
drwetter commentedon Mar 14, 2023
I´ll sleep over it, see https://github.com/drwetter/testssl.sh/issues/2299#issuecomment-1468625898
polarathene commentedon Mar 15, 2023
You seem to have figured it out in the referenced issue, but just to clarify that is the base image weight 80MB, extra size comes from all the extra additions in the
Dockerfileafter that.Possibly due to your version of Docker?
If you have the release from earlier this year (v23) it should build it without errors like that I think. AFAIK it's due to the HereDoc syntax which I think may only be supported with BuildKit (should now be default in v23, but was opt-in for prior releases requiring an ENV
DOCKER_BUILDKIT=1ordocker buildx).Yeah, that's usually the motivator for using Alpine.
Until it becomes more trouble than it's worth due to compatibility issues like witnessed here, and potential degraded performance and higher memory usage / leaks (different allocator), and the DNS issues others cite.
Eventually if you experience those problems, you can try work around them or consider if the 100MB or so savings is actually a problem.
IIRC, the size in transit to pull is actually lower due to compression (DockerHub tags page usually reports the compressed size). Debian slim image has 80MB, but is only 30MB compressed. Often the concern for size is about network transfer more than disk, plus if any other images use that same base image that weight is no longer relevant.
drwetter commentedon Mar 16, 2023
I tried in the meantime to generate a libnss_dns.a to be included into openssl as a dirty hack but gave up as I couldn't get it to fly and it was ugly (too) in the first place
Add openssl, offer to query OCSP responder
drwetter commentedon Mar 19, 2023
In 3.0 this works for me with the recent commit when using within the container
testssl.sh -q -S --phone-out --openssl=/usr/bin/openssl testssl.shI hope this workaround is a solution which works for the time the 3.0 branch is still alive.
For 3.1dev: expect a solution this week.
drwetter commentedon Mar 20, 2023
Update: I missed yesterday night to commit also the link from openssl1.1 to openssl. --> see PR #2342 merged.