DNS errors in binaries built with muslrust when run in rootless Docker

Certain domains (for me it's github.com) always fail to resolve in rootless Docker when the image is built with muslrust:

main.rs

use std::net::ToSocketAddrs;

fn main() {
    let domains = ["google.com", "github.com"];
    for domain in domains
    {
        println!("Trying to resolve {domain} with to_socket_addrs()");
        format!("{domain}:443").to_socket_addrs().unwrap();
        println!("it worked");
    }
}

There are no dependencies installed.

Dockerfile

# syntax=docker/dockerfile:1
FROM clux/muslrust:1.71.0 as musl
#USER root
WORKDIR /app
COPY . .
ARG CARGO_INCREMENTAL=0
# trying to resolve URLs, just show errors, don't abort
# this *works* in rootless Docker in musl
RUN curl -sS -o /dev/null https://google.com; exit 0
# this *fails* in rootless Docker in musl
RUN curl -sS -o /dev/null https://github.com; exit 0
RUN cargo build --release --target x86_64-unknown-linux-musl
CMD ["/app/target/x86_64-unknown-linux-musl/release/mwedns"]

FROM rust:1.71.0-slim
WORKDIR /app
COPY . .
ARG CARGO_INCREMENTAL=0
# this *works* in rootless Docker with the official Rust image
RUN curl -sS -o /dev/null https://google.com; exit 0
# this *works* in rootless Docker with the official Rust image
RUN curl -sS -o /dev/null https://github.com; exit 0
RUN cargo build --release
COPY --link --from=musl /app/target/x86_64-unknown-linux-musl/release/mwedns /app/musl
# run the one compiled with the official Rust image first
CMD echo "official Rust" && /app/target/release/mwedns && echo "musl Rust" && /app/musl

You can test this MWE at https://github.com/KonradHoeffner/mwedns:

$ git clone [email protected]:KonradHoeffner/mwedns.git
$ cd mwedns
$ docker build . -t mwedns
[...]
$ docker run --rm mwedns

Output

official Rust
Trying to resolve google.com with to_socket_addrs()
it worked
Trying to resolve github.com with to_socket_addrs()
it worked
musl Rust
Trying to resolve google.com with to_socket_addrs()
it worked
Trying to resolve github.com with to_socket_addrs()
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Uncategorized, error: "failed to lookup address information: Name does not resolve" }', src/main.rs:8:51

I reported this first at algesten/ureq#645 but then found out it has nothing to do with ureq.

You can also verify that DNS lookup fails in rootless Docker with:

$ docker run --rm --entrypoint "/bin/bash" clux/muslrust:1.71.0 -c "curl -sS -o /dev/null https://github.com"
curl: (6) Could not resolve host: github.com

Relation to Alpine?

There is a known bug with the Alpine image reported at alpinelinux/docker-alpine#155.
However the things that differ here:

looking at the muslrust Dockerfile, it is based on Ubuntu Jammy, not on Alpine
the error occurs inside a muslrust container
inside an official Rust container, the errors occurs with a binary built within a muslrust container but not with a binary built within an official Rust container

This is almost certainly down to how musl does dns and not something we can solve here.

You are putting a musl built static binary inside of a rust container which is a debian based (glibc) distro, with dns resolution settings (resolv.conf / dnsmasq.conf) that is probably not ideal for what it was built for.

I would recommend following this post and seeing where that takes you. There's also a more kubernetes focused issue that may be useful at k3s-io/k3s#6132.

This issue is one of the few answers when searching for answers on this error. error: "failed to lookup address information: Name does not resolve"
It seems that MUSL does DNS different. I had several DNS servers in my resolv.conf file and it would arbitrarily check the public DNS servers. By limiting the list of servers in resolv.conf to only the local DNS server, everything worked again.

See also this discussion. https://musl.openwall.narkive.com/i7kiCD4c/resolv-conf-ordering

@KonradHoeffner

TL;DR:

I'm not able to reproduce the failure with either the rust example or curl 😕 Something else is amiss.
With builds using musl 1.2.4 or newer, your issue may presumably be resolved (assuming it's DNS-over-TCP related which that version of musl fixed). However even with the latest version of Rust (2 years since musl 1.2.4), it seems you can build a musl target without having musl on the build host, and instead that may be supplied via Rust instead (still musl 1.2.3, since May 2023).

You are putting a musl built static binary inside of a rust container which is a debian based (glibc) distro, with dns resolution settings (resolv.conf / dnsmasq.conf) that is probably not ideal for what it was built for.

A static musl build should generally be portable even to a distro that is using glibc. Dynamic linking not so much.

A static glibc build however isn't fully static in comparison. It has some calls for networking that still rely on glibc IIRC? An alternative to glibc for static builds when musl is not desired is Eyra (rust-based that hijacks the glibc target), but it's a bit spotty if you need native deps IIRC as not all of glibc ABI(?) is implemented.

I would recommend following this post and seeing where that takes you.

FWIW regarding the DNS over TCP concern at that issue, that was resolved a couple months after the blog post was published. With Alpine 3.18 / musl 1.2.4.

muslrust:1.85.0-stable (March 2025) is the first stable tag that upgraded from musl 1.2.2 to 1.2.4. The previous 1.84.0-stable tag was from Jan 2025, so it took a while to land in this image vs the Alpine 3.18 release (May 2023).

You can check the musl version like this:

$ docker run --rm -it docker.io/clux/muslrust:1.85.0-stable /lib/ld-musl-x86_64.so.1

musl libc (x86_64)
Version 1.2.4
Dynamic Program Loader
Usage: /lib/ld-musl-x86_64.so.1 [options] [--] pathname [args]

That said, I'm not familiar with this muslrust project, but at a glance it's using Ubuntu with musl provided via musl-gcc package to support building native deps for the musl target.

With Fedora, there's the musl-libc package (implicit via musl-gcc package), but they also release updates to packages until the release is EOL (approx 12 months, may be a little bit of lag between packages updating to newer releases):

Fedora 39 (Nov 2023 release) is the oldest release with 1.2.5 (March 2024 release).
Fedora 38 (April 2023 release) provides musl 1.2.3 (I'm not sure why Fedora 38 didn't pick up musl 1.2.4 🤷‍♂).

@KonradHoeffner I attempted your other (ureq) reproduction example, which needed to bump the muslrust image to 1.78.0-stable for the installation of cargo-chef v0.1.71. As shown above though, it should still reproduce that musl error? Yet I was not able to reproduce with Docker rootless 🤷‍♂

Even without chef involved, fails to reproduce:

# syntax=docker/dockerfile:1
FROM clux/muslrust:1.74.0-stable AS builder
WORKDIR /app
COPY . .
RUN cargo build --locked --release --target x86_64-unknown-linux-musl

FROM busybox:1.36.1 AS runtime
RUN addgroup -S myuser && adduser -S myuser -G myuser
COPY --link --from=builder /app/target/x86_64-unknown-linux-musl/release/mwedns /usr/local/bin/
USER myuser
WORKDIR /app
CMD ["/usr/local/bin/mwedns"]

Alternatively, here's a Fedora based equivalent of your smaller reproduction that you provided in this issue (single Dockerfile to build, nothing else needed):

# Fedora provides `musl 1.2.3` via `musl-gcc`
FROM fedora:38 AS builder
SHELL ["/bin/bash", "-eux", "-o", "pipefail", "-c"]
RUN <<HEREDOC
  dnf --setopt=install_weak_deps=0 install -y rustup musl-gcc
  rustup-init -y --profile minimal --default-toolchain stable --target x86_64-unknown-linux-musl
  dnf clean all
HEREDOC

# Create example project source files:
COPY <<HEREDOC /build/project/Cargo.toml
[package]
name = "example"
version = "0.1.0"
edition = "2024"

[dependencies]
HEREDOC

COPY <<HEREDOC /build/project/src/main.rs
use std::net::ToSocketAddrs;

fn main() {
    let domains = ["google.com", "github.com"];
    for domain in domains
    {
        println!("Trying to resolve {domain} with to_socket_addrs()");
        format!("{domain}:443").to_socket_addrs().unwrap();
        println!("it worked");
    }
}
HEREDOC

# Build the app:
WORKDIR /build/project
ENV PATH="/root/.cargo/bin:/root/.cargo/env:${PATH}"
RUN <<HEREDOC
  cargo build --release --target x86_64-unknown-linux-musl
  mv target/x86_64-unknown-linux-musl/release/example /usr/bin/example
HEREDOC

FROM scratch
COPY --link --from=builder /usr/bin/example /usr/bin/example
CMD ["/usr/bin/example"]

Likewise though, since I cannot reproduce the failure on my end, I am not sure how accurately that would reproduce the failure (it should based on the info provided). If it does simply switching from fedora:38 to fedora:39 ~~should resolve it, like with the other images mentioned that upgrade to newer musl release~~ (EDIT: Actually, that'll happily build without musl-gcc, just replace that package with gcc since there is no native deps to build with this smaller reproduction example, so system musl may not actually be used).

For the musl target, Rust will static link by default (unless using Alpine's cargo package which is patched to dynamic link by default) and provide it's own musl runtime to embed IIRC (due to RUSTFLAGS="-C link-self-contained=yes" default applied for the musl target).

So in that case it may depend on which version of musl Rust bundles, since my Fedora example doesn't pin an older Rust toolchain release via rustup, it seems like the actual musl would be 1.2.3 which should trigger the bug? 🤷‍♂
- Official rustc docs cite musl target as using musl 1.2.3
- Official Rust blog post (May 2023) announces Rust 1.71.0 onwards would be upgraded from musl 1.1.24 to 1.2.3.
NOTE: If you attempt to dynamic link (via RUSTFLAGS="-C target-feature=-crt-static"), that'll imply -lc that on a glibc distro will be resolved to glibc instead of musl, hence ldd (glibc) will reveal that despite the musl target, you did not link what you expected 😅 (I'm not sure how to go about that, might be better to build on alpine base instead)

UPDATE: Nothing to do with pre-built rust std lib either, no musl needed, only gcc for building that with the musl target on a glibc host:

rustup component add rust-src --toolchain nightly-x86_64-unknown-linux-gnu
cargo build --release --target x86_64-unknown-linux-musl -Z build-std

My test environment details:

I am running with Docker Rootless on a linux host.

# Docker 28.1.1 in rootless mode
# cgroups v2, kernel 6.14.2
# Fedora 42 (installed on Vultr VPS), running as non-root user

$ docker info | grep -E 'Context|Version'
 Version:    28.1.1
 Context:    rootless
    Version:  v0.23.0
    Version:  v2.35.1
 Server Version: 28.1.1
 Cgroup Version: 2
 Kernel Version: 6.14.2-300.fc42.x86_64

$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Fedora Linux 42 (Server Edition)"

# Running as the non-root user:
$ ps xo user,fname | grep dockerd
linuxuser   dockerd

After creating a new VPS on Vultr with Fedora 42, there is a default linuxuser (1000:1000) provisioned and I shell into the VPS via SSH to the root account:

# Install Docker:
dnf config-manager addrepo --from-repofile https://download.docker.com/linux/fedora/docker-ce.repo
dnf install -qy docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Proper user switching from root requires the `machinectl` command:
dnf install -qy systemd-container

# Switch to non-root user via `machinectl`,
# or you can set a password/key for the user and SSH in:
machinectl shell linuxuser@

# Install Docker rootless, and you're good to go from here:
dockerd-rootless-setuptool.sh install

Focusing on `curl` difference instead

inside an official Rust container, the errors occurs with a binary built within a muslrust container but not with a binary built within an official Rust container

This is an obvious one with difference of glibc vs the static musl. If you did install musl-gcc into this image the version is 1.2.2, so it should also produce the same problem as muslrust:1.71.0 did for you (also musl 1.2.2).

However if you had built with rust:1.71.0-alpine tag instead of the rust:1.71.0-slim (Debian), this shipped musl 1.2.4 (you can check /etc/os-release, that image is based on Alpine 3.18.2).

EDIT: As per my last comment, I'm probably mistaken. I think all AMD64 + ARM64 musl target builds since Rust 1.71.0 to present (1.88.0) are using musl 1.2.3? So it just might not be possible to resolve that.

Let's instead focus on the simpler curl example (although I'm also not able to reproduce this 😕)

Looking at that containers curl command:

$ docker run --rm -it clux/muslrust:1.71.0
$ which curl
/musl/bin/curl

# NOTE: `ldd` is just a shell script for aliasing a libc specific intrepreter,
# which handles dynamic loading of linked libs. Thus this is calling the glibc one:
$ ldd /musl/bin/curl
/musl/bin/curl: error while loading shared libraries: /lib/x86_64-linux-gnu/libc.so: invalid ELF header

$ /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --version
ld.so (Ubuntu GLIBC 2.35-0ubuntu3.1) stable release version 2.35.
Copyright (C) 2022 Free Software Foundation, Inc.

$ /lib/ld-musl-x86_64.so.1 --version
musl libc (x86_64)
Version 1.2.2
Dynamic Program Loader

# But it's linked for musl:
$ file /musl/bin/curl
/musl/bin/curl: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, with debug_info, not stripped

# This is the equivalent of what the `ldd` command does by default:
$ /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --list /musl/bin/curl
/musl/bin/curl: error while loading shared libraries: /lib/x86_64-linux-gnu/libc.so: invalid ELF header

# Now do the equivalent with musl dynamic loader instead, we see the linked deps:
$ /lib/ld-musl-x86_64.so.1 --list /musl/bin/curl

/lib/ld-musl-x86_64.so.1 (0x7f73ed7ee000)
libc.so => /lib/ld-musl-x86_64.so.1 (0x7f73ed7ee000)

So this curl is built with musl and we can see it's linked dynamically to use the musl 1.2.2 version available.

# Make the web request and return the HTTP status code:
$ docker run --rm -it clux/muslrust:1.71.0
$ curl -s -o /dev/null -w "%{http_code}\n" https://github.com
200

# Supposedly, this should fail too (musl 1.2.3):
$ docker run --rm alpine:3.17 ash -c 'apk add curl && curl -s -o /dev/null -w "%{http_code}\n" https://github.com'
# But this would be successful (musl 1.2.4):
$ docker run --rm alpine:3.18 ash -c 'apk add curl && curl -s -o /dev/null -w "%{http_code}\n" https://github.com'

Since curl won't fail for me either, there must be something else different with my environment vs what you had to reproduce. Unless since your original report something externally has changed? 🤔

github.com is still not serving IPv6 / AAAA DNS records:

$ docker run --rm -it alpine
$ nslookup github.com
Server:         10.0.2.3
Address:        10.0.2.3:53

Non-authoritative answer:

Non-authoritative answer:
Name:   github.com
Address: 4.237.22.38

# IPv4 only:
$ nslookup -type=A github.com
Server:         10.0.2.3
Address:        10.0.2.3:53

Non-authoritative answer:
Name:   github.com
Address: 4.237.22.38

# IPv6 only (no result):
$ nslookup -type=AAAA github.com
Server:         10.0.2.3
Address:        10.0.2.3:53

Non-authoritative answer:

I am running much newer Docker, kernel, etc for the host. So perhaps the issue is affected by that somewhat, or even networking environment.

I have experienced DNS issues in the past even with glibc on a system where the default DNS (old router) did not support TCP queries either (like musl prior to 1.2.4), a good example of where that'd fail would be dig TXT github.com. Since that was caused by the router lacking support rather than the client, it'd fail either even when querying TCP was supported by the client 😅

# Static build of doggo would fail on that system:
doggo TXT github.com

# But querying a DNS service that supported TCP queries would work:
doggo @1.1.1.1 TXT github.com

That was a bit more complicated network wise (Docker running via WSL2 on Windows, where the containers 127.0.0.11:53 provided the default DNS, which Docker managed internally, that'd proxy to the WSL2 or Docker Desktop managed DNS I think (or your custom DNS if --dns was set), followed by whatever was configured/inferred on the Windows host itself.

Thank you for the detailed investigation! After years of this not being fixed I had already given up and switched from Alpine/BusyBox to one of the new "distroless" glibc images where the overhead is not that large anymore but I will check if the problem still persists using the MWE from https://github.com/KonradHoeffner/mwedns/ on a server with the newest current rootless Docker version 28.1.1 installed after upgrading to the currently newest stable Rust/muslrust version of 1.86:

MWE Dockerfile

# syntax=docker/dockerfile:1
FROM clux/muslrust:1.86.0-stable AS musl
#USER root
WORKDIR /app
COPY . .
ARG CARGO_INCREMENTAL=0
# trying to resolve URLs, just show errors, don't abort
# this *works* in rootless Docker
RUN curl -sS -o /dev/null https://google.com; exit 0
# this *fails* in rootless Docker in musl
RUN curl -sS -o /dev/null https://github.com; exit 0
RUN cargo build --release --target x86_64-unknown-linux-musl
CMD ["/app/target/x86_64-unknown-linux-musl/release/mwedns"]

FROM rust:1.86.0-slim
WORKDIR /app
COPY . .
ARG CARGO_INCREMENTAL=0
# this *works* in rootless Docker with the official Rust image
RUN curl -sS -o /dev/null https://google.com; exit 0
# this *fails* in rootless Docker with the normal Rust image
RUN curl -sS -o /dev/null https://github.com; exit 0
RUN cargo build --release
COPY --link --from=musl /app/target/x86_64-unknown-linux-musl/release/mwedns /app/musl
# run the one compiled with the official Rust image first
CMD echo "official Rust" && /app/target/release/mwedns && echo "musl Rust" && /app/musl

Still failing with rootless Docker 28.1.1 and muslrust 1.86

$ docker run --rm -it docker.io/clux/muslrust:1.86.0-stable /lib/ld-musl-x86_64.so.1
musl libc (x86_64)
Version 1.2.4
[...]
$ docker build . -t mwedns
$ docker run mwedns
official Rust
Trying to resolve google.com with to_socket_addrs()
it worked
Trying to resolve github.com with to_socket_addrs()
it worked
musl Rust
Trying to resolve google.com with to_socket_addrs()
it worked
Trying to resolve github.com with to_socket_addrs()

thread 'main' panicked at src/main.rs:8:51:
called `Result::unwrap()` on an `Err` value: Custom { kind: Uncategorized, error: "failed to lookup address information: Name does not resolve" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Also failing with Fedora 38 and 39

docker run mwedns-fedora-38
official Rust
Trying to resolve google.com with to_socket_addrs()
it worked
Trying to resolve github.com with to_socket_addrs()
it worked
musl Rust
Trying to resolve google.com with to_socket_addrs()
it worked
Trying to resolve github.com with to_socket_addrs()

thread 'main' panicked at src/main.rs:8:51:
called `Result::unwrap()` on an `Err` value: Custom { kind: Uncategorized, error: "failed to lookup address information: Name does not resolve" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The same also happens with Fedora 39.

You are putting a musl built static binary inside of a rust container which is a debian based (glibc) distro, with dns resolution settings (resolv.conf / dnsmasq.conf) that is probably not ideal for what it was built for.

Good point, I used BusyBox in the past in the real world case, however I am using the Rust slim image to execute both the dynamic and the static MWE for simplicity.

dig/doggo

On the server itself dig TXT github.com works, but inside an Alpine container I get:

/ # doggo TXT github.com
NAME	TYPE	CLASS	TTL	ADDRESS	NAMESERVER 
/ # doggo @1.1.1.1 TXT github.com
ERROR[2025-04-29T09:36:14Z] error looking up DNS records                  error="read udp 172.17.0.2:59961->1.1.1.1:53: i/o timeout"
NAME	TYPE	CLASS	TTL	ADDRESS	NAMESERVER

Unfortunately I am a bit out of time right now but I really appreciate all the information, I will follow up and add the rest later.

Unfortunately I am a bit out of time right now but I really appreciate all the information, I will follow up and add the rest later.

No worries, thanks for the response! ❤

The fedora builds presumably would still have that problem if the musl-gcc package is swapped for gcc, so that there is no musl provided by Fedora. It'd purely be from Rust bundling musl 1.2.3 I think.

If you can find the time to confirm with the curl difference that would be good too:

# Supposedly, this should fail too (musl 1.2.3):
docker run --rm alpine:3.17 ash -c 'apk add curl && curl -s -o /dev/null -w "%{http_code}\n" https://github.com'

# But this would be successful (musl 1.2.4):
docker run --rm alpine:3.18 ash -c 'apk add curl && curl -s -o /dev/null -w "%{http_code}\n" https://github.com'

Although your last bit with doggo might suggest Alpine 3.18 fails too. You didn't provide much context for the Alpine container.

You compared dig on the host but show doggo as the command, was that installed from apk?
Which version of alpine did you use? The current latest? (cat /etc/os-release will show the version).

If the curl test fails on Alpine 3.18 or newer, then that is a bit interesting. I have no idea how to reproduce the failure on my end, while yours appears to not be DNS-over-TCP related at all (or if it is your network environment is doing something incompatible, but only with musl involved 🤔)

I had already given up and switched from Alpine/BusyBox to one of the new "distroless" glibc images where the overhead is not that large anymore

Yeah that's fair, I've done the same after too many hours lost to troubleshooting Alpine / musl specific bugs that don't occur with glibc images/builds.

If you can use Google distroless, that's pretty decent. Canonical chisel provides additional flexibility and can be just as competitive in size/performance while allowing for a wider range of customization package wise, but it's still fairly new so not as streamlined as I'd like.

Fedora and openSUSE Leap are great base images alternatives when you need something more familiar for customization when the earlier distroless focused options aren't viable. I've created Dockerfiles for a few projects where the size is better than Alpine offered and sometimes competitive with distroless equivalents. You just need to use the --installroot option of a package manager 👍

Here's an older example for the testssl project:

# syntax=docker.io/docker/dockerfile:1

ARG LEAP_VERSION=15.4
ARG INSTALL_ROOT=/rootfs

FROM opensuse/leap:${LEAP_VERSION} as builder
ARG CACHE_ZYPPER=/tmp/cache/zypper
ARG INSTALL_ROOT
# --mount is only necessary for persisting the zypper cache on the build host,
# Paired with --cache-dir below, RUN layer invalidation does not clear this cache.
# Not useful for CI, only local builds that retain the storage.
RUN --mount=type=cache,target="${CACHE_ZYPPER}",sharing=locked <<EOF
  # Provides $VERSION_ID
  source /etc/os-release
  ZYPPER_OPTIONS=(
    --releasever "${VERSION_ID}"
    --installroot "${INSTALL_ROOT}"
    --cache-dir "${CACHE_ZYPPER}"
  )

  # Sync package repos:
  zypper ${ZYPPER_OPTIONS[@]} --gpg-auto-import-keys refresh

  zypper ${ZYPPER_OPTIONS[@]} --non-interactive install \
    --download-in-advance --no-recommends \
    bash procps grep gawk sed coreutils busybox-util-linux busybox-vi ldns libidn2-0 socat openssl curl


  ## Cleanup (reclaim approx 13 MiB):
  # None of this content should be relevant to the container:
  rm -r "${INSTALL_ROOT}/usr/share/"{licenses,man,locale,doc,help,info}
  # Functionality that the container doesn't need:
  rm "${INSTALL_ROOT}/usr/share/misc/termcap"
  rm -r "${INSTALL_ROOT}/usr/lib/sysimage/rpm"
EOF


# Create a new image with the contents of $INSTALL_ROOT
FROM scratch
ARG INSTALL_ROOT
COPY --link --from=builder ${INSTALL_ROOT} /

Supposedly, this should fail too (musl 1.2.3):
docker run --rm alpine:3.17 ash -c 'apk add curl && curl -s -o /dev/null -w "%{http_code}\n" https://github.com'

This outputs HTTP Code 000 (I guess this means it cannot open a connection at all?)

But this would be successful (musl 1.2.4):
docker run --rm alpine:3.18 ash -c 'apk add curl && curl -s -o /dev/null -w "%{http_code}\n" https://github.com'

Alpine 3.18 also outputs HTTP code 000.
However Alpine 3.19, 3.20 and 3.21 successfully output code 200!

Although your last bit with doggo might suggest Alpine 3.18 fails too. You didn't provide much context for the Alpine container.
* You compared `dig` on the host but show `doggo` as the command, was that installed from `apk`?

* Which version of alpine did you use? The current latest? (_`cat /etc/os-release` will show the version_).

Yes, I am using the latest Alpine 3.21 with doggo from apk:

$ docker run --rm -it alpine:3.21
/ # apk add doggo
(1/1) Installing doggo (0.5.7-r10)
Executing busybox-1.37.0-r12.trigger
OK: 18 MiB in 16 packages
/ # doggo TXT github.com
NAME	TYPE	CLASS	TTL	ADDRESS	NAMESERVER 
/#

The host didn't have doggo installed but I now ran doggo using Docker on the server directly then it works:

docker run --rm ghcr.io/mr-karan/doggo:latest github.com
NAME       	TYPE	CLASS	TTL	ADDRESS     	NAMESERVER      
github.com.	A   	IN   	0s 	140.82.121.4	192.168.65.1:53	
github.com.	A   	IN   	0s 	140.82.121.4	192.168.65.1:53

Now I am wondering why Alpine 3.19 and newer succeed in the curl test but fail the other one.

In general, it seems like Alpine in any version fails to connect to github.com while busybox can starting with version 1.36:

~> docker run --rm busybox:1.35 ash -c 'wget -S  https://github.com 2>&1 | grep -e "bad" -e "HTTP" -e "connecting" -e "wget"'
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 80): 80
wget: error getting response: Connection reset by peer
~> docker run --rm busybox:1.36 ash -c 'wget -S  https://github.com 2>&1 | grep -e "bad" -e "HTTP" -e "connecting" -e "wget"'
wget: note: TLS certificate validation not implemented
  HTTP/1.1 200 OK
~> docker run --rm busybox:1.37 ash -c 'wget -S  https://github.com 2>&1 | grep -e "bad" -e "HTTP" -e "connecting" -e "wget"'
wget: note: TLS certificate validation not implemented
  HTTP/1.1 200 OK
~> docker run --rm alpine:3.17 ash -c 'wget -S  https://github.com 2>&1 | grep -e "bad" -e "HTTP" -e "connecting" -e "wget"'
wget: bad address 'github.com'
~> docker run --rm alpine:3.21 ash -c 'wget -S  https://github.com 2>&1 | grep -e "bad" -e "HTTP" -e "connecting" -e "wget"'
wget: bad address 'github.com'

The strange thing is that Alpine 3.21 includes busybox 1.37 which should work...

~> docker run --rm alpine:3.21 ash -c "wget 2>&1 | head -1"
BusyBox v1.37.0 (2025-01-17 18:12:01 UTC) multi-call binary.

I've created Dockerfiles for a few projects where the size is better than Alpine offered and sometimes competitive with distroless equivalents. You just need to use the

Wow, I didn't know you could achieve a size below Alpine! For our use case it's:

base	size
busybox using musl	16.1MB
chainguard/wolfi-base using glibc	26.6MB
debian:stable-slim using glibc	86.4MB

Now I am wondering why Alpine 3.19 and newer succeed in the curl test but fail the other one.

I have no clue, that's quite strange. As stated I can't reproduce the failure on my end, always works even on older Alpine.

Wow, I didn't know you could achieve a size below Alpine!

It's sometimes more effort than it's worth though 😓 (UPDATE: Turns out the Alpine image wasn't optimized well, it's approx half the size of the glibc images I came up with, but runtime performance is a tad worse)

I'm not a fan of chainguard/wolfi due to the lack of version pinning. Debian is a bit more of a pain to be competitive.

If you'd like some insights/advice regarding techniques or with Chisel based images, you may like this discussion thread as a reference. However for many projects you don't need the complexity and can often use Google Distroless if no extra packages are needed (otherwise chisel or fedora are pretty good for glibc).

I'm not a fan of chainguard/wolfi due to the lack of version pinning. Debian is a bit more of a pain to be competitive.

If you'd like some insights/advice regarding techniques or with Chisel based images, you may like this discussion thread as a reference. However for many projects you don't need the complexity and can often use Google Distroless if no extra packages are needed (otherwise chisel or fedora are pretty good for glibc).

Thanks for the link! I do not need any extra packages for the project which caused this issue because all I'm doing is wrapping a single executable binary compiled with Rust but I'm sure that will come up in the future in another project, so I will use that as a reference.

Navigation Menu

Repository navigation

DNS errors in binaries built with muslrust when run in rootless Docker #122

main.rs

Dockerfile

Output

Relation to Alpine?

Focusing on `curl` difference instead

MWE Dockerfile

Still failing with rootless Docker 28.1.1 and muslrust 1.86

Also failing with Fedora 38 and 39

dig/doggo

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

DNS errors in binaries built with muslrust when run in rootless Docker #122

Description

main.rs

Dockerfile

Output

Relation to Alpine?

Activity

clux commented on Aug 4, 2023

jhaand commented on Feb 3, 2025

polarathene commented on Apr 28, 2025

polarathene commented on Apr 28, 2025

Focusing on curl difference instead

KonradHoeffner commented on Apr 29, 2025

MWE Dockerfile

Still failing with rootless Docker 28.1.1 and muslrust 1.86

Also failing with Fedora 38 and 39

dig/doggo

polarathene commented on Apr 29, 2025

polarathene commented on Apr 29, 2025

KonradHoeffner commented on Apr 30, 2025

KonradHoeffner commented on Apr 30, 2025

polarathene commented on Apr 30, 2025

KonradHoeffner commented on May 2, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Focusing on `curl` difference instead