Skip to content

Run as non-root inside container #104

Closed

Description

@hoshsadiq

Would be nice to have caddy run as a non-root user inside the container. This is especially useful for rootfull containers (e.g. Docker), where the uid=0 inside the container is the same as uid=0 outside. Additionally, since the /data dir and the /config have been specified as a VOLUME, others are unable to extend the container and change the ownership of those directories. I'm currently using the below Dockerfile to run caddy as a non-root user. I've changed it slightly to suit my needs but much of it should be re-usable for this repo.

FROM caddy:2-builder AS builder

RUN caddy-builder github.com/caddy-dns/cloudflare

FROM caddy:2-alpine AS deps

# We cannot use FROM scratch because, despite adding cap_net_bind_service to the binary
# it still won't run. Presuming because libcap isn't available? Not sure.
FROM alpine:3.12

COPY --from=deps /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=deps /etc/mime.types /etc/nsswitch.conf /etc/

COPY --from=builder /usr/bin/caddy /caddy

RUN set -eux; \

    apk add --no-cache libcap; \
    setcap cap_net_bind_service=ep /caddy; \

    mkdir -p /config/caddy /data/caddy; \

    addgroup -g 101 -S www-data; \
    adduser -u 101 -D -S -G www-data www-data; \

    chown -R www-data:www-data /config /data

USER www-data

ENV XDG_CONFIG_HOME=/config XDG_DATA_HOME=/data

VOLUME /config /data

EXPOSE 80
EXPOSE 443

ENTRYPOINT ["/caddy"]
CMD ["run", "--config", "/Caddyfile", "--adapter", "caddyfile"]

One thing worth considering is that this might be not be an easy upgrade for many, indeed, it may be that we'd need a temporary stop gap that runs the container as root, changes ownership of files/folders, then drops privileges. Then after perhaps a couple of versions, this stop gap can be replaced fully with a non-root user without going through the trouble of dropping privileges.

Activity

  1. hairyhenderson commented on Aug 17, 2020

    @hairyhenderson
    Contributor

    Hi @hoshsadiq - my apologies for replying so late (been vacationing and otherwise distracted)!

    We actually originally did run as non-root by default, but simplicity we decided to drop that (see #24, and also #103 for some other related discussion).

    If your Dockerfile works for you, that's great. In most cases where users want to run as non-root, they also don't need to listen to :80/:443 in the container, so the setpcap magic isn't necessary at all.

    It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either.

    Finally, it's worth considering why you want to run as non-root. What attack vectors are you trying to avoid? Container escape vulnerabilities are pretty much the only real risk, but anyone running a modern Docker version is immune to many of them. It's also worth considering user namespace remapping as a mitigation. In my experience the main reason for running as non-root is to pass compliance checks - not a bad reason, but it's also worth recognizing that non-compliance does not automatically equal decreased security (and vice-versa).

    I will leave this open to discuss possibilities, but I think the starting point is probably to just provide some better documentation around running as non-root.

  2. mirekum commented on Sep 10, 2020

    @mirekum

    Hi, In the company I work for, I am not allowed to run any docker container as root. It would be nice to use the Caddy in the same way:
    es:
    container_name: es
    image: elasticsearch:6.8.8
    read_only: true
    user: "elasticsearch:elasticsearch"

  3. hoshsadiq commented on Sep 10, 2020

    @hoshsadiq
    Author

    @hairyhenderson sorry, I had forgotten about this issue.

    In most cases where users want to run as non-root, they also don't need to listen to :80/:443 in the container, so the setpcap magic isn't necessary at all.

    I'm aware of this. I didn't see an option to run caddy on a non-standard port, hence why I used setpcap. Maybe I was blind and/or I misunderstood. If you could let me know what that option is I'll gladly update my image.

    It's also worth noting that caddy is an official image, and as such needs to be similarly-shaped to other official images of the same type. At a quick glance, none of nginx, traefik, httpd, or haproxy support running as non-root out of the box either.

    While this is true, it's worth noting that nginx and httpd both have the option to delegate the actual workers (I don't know if it's the same with traefik and haproxy). One can indeed run as root, though it is discouraged in multiple places. In addition, for nginx, the configuration ignores the user/group options if the main process runs a non-root user.

    Caddy does not have an option to delegate workers (I'm presuming requests run in go routines so that wouldn't even make sense). It is also impossible to run it as a non-root user using docker's (and docker-compose's) user arguments because the volumes /data and /config are owned by root, so when caddy runs as non-root, it does not have the correct permissions to those directories to function correctly.

    it's worth considering why you want to run as non-root

    Definitely agree with this. I'm using caddy for a personal project, but I think running as non-root is a no-brainer for me considering it's such a trivial task that gives a lot with little effort. Though this is a personal preference, and isn't entirely based on any specific threat model, rather it's more of a precautionary issue.

    I think even if Caddy doesn't change the default, that would totally be okay, but making it easier for individual to run caddy in non-root mode, in my opinion is a must. As it stands, that is as simple as chowning the /data and /config directory as a non-root user. This was what initially prompted me to raise this as an issue. Due to docker's design, it is not possible to change the ownership (or the mode) of a VOLUME in a running container. It is only possible by explicitly doing so in another Dockerfile, changing the ownership/mode, and then re-assigning it as a VOLUME (as I've done in the my image).

  4. hairyhenderson commented on Sep 12, 2020

    @hairyhenderson
    Contributor

    @hoshsadiq thanks for following up!

    I didn't see an option to run caddy on a non-standard port

    That's really just a matter of configuration. In the Caddyfile syntax, there are two global options http_port and https_port to tell Caddy to listen to nonstandard ports. You'll still need to be listening at 80/443 outside the container in order to get automatic TLS from Let's Encrypt though.

    So a config like this should work for you (warning, I didn't have time to test this):

    {
      http_port 8080
      https_port 8443
    }
    
    mysite.com {
      root * /usr/share/caddy
      file_server
    }
    

    Then when you run the container, just map the ports appropriately:

    $ docker run -d -p 80:8080 -p 443:8443 ...

    As it stands, that is as simple as chowning the /data and /config directory as a non-root user

    Sure, but that's if you want to use /data and /config - you don't have to. You can set the XDG_DATA_HOME and XDG_CONFIG_HOME environment variables to different locations. (See https://caddyserver.com/docs/conventions#file-locations for details on what these are)

    The reason I used VOLUMEs is to prevent these files from being written to the container, and hence lost during a container restart. It's important for the the data directory in particular to persist, because it contains TLS certs from Let's Encrypt, and if those are lost you can very quickly get rate-limited by LE.

    So, if you choose to configure Caddy to use different data/config directories (which you'll have to if you're running as non-root), you'll need to make sure you use persisted volumes.

    Hope this helps! I'm definitely going to add some of this to the docs for the image on DockerHub, since most of this is definitely non-intuitive for people new to Caddy v2 😅

  5. westurner commented on Sep 13, 2020

    @westurner

    . It is also impossible to run it as a non-root user using docker's (and docker-compose's) user arguments because the volumes /data and /config are owned by root, so when caddy runs as non-root, it does not have the correct permissions to those directories to function correctly.

    The process should not have write access to the config.

    More about not running as root here: #24 (comment)

  6. francislavoie commented on Sep 13, 2020

    @francislavoie
    Member

    /config is actually a config cache directory, for when users are making use of Caddy's config API. If Caddy didn't write its current config to file, then the running config would be lost. In other words, any changes pushed to the config API would be lost. It also allows for the --resume option to work, which will make Caddy start up with the previously run configuration instead on start.

    https://caddyserver.com/docs/conventions#configuration-directory

    Don't worry, Caddy never writes to your Caddyfile (unless you use the caddy fmt command with the overwrite option, to have it re-indent your config).

  7. mholt commented on Sep 13, 2020

    @mholt
    Member

    If Caddy didn't write its current config to file, then the running config would be lost. In other words, any changes pushed to the config API would be lost.

    To clarify, the running config is stored in memory, so it would only be lost lost if the process or container was terminated. (Which, of course, happens all the time.) You're right, but I just wanted to make that distinction.

  8. hoshsadiq commented on Dec 20, 2020

    @hoshsadiq
    Author

    Sorry for the late reply - again.

    Sure, but that's if you want to use /data and /config - you don't have to. You can set the XDG_DATA_HOME and XDG_CONFIG_HOME environment variables to different locations

    Aah, that is fair. I did not think of this, I'll use that. Also thanks for the config options for running on non-standard port, I don't know why, but I just did not find the options prior to you mentioning it.

    I'm more than happy for this issue to be closed as my issue is resolved, however, I think it's still worth changing the default container to being back run as a non-root user. Considering root inside a container is root outside a container (for docker anyway), this can be destructive if Caddy is compromised. Security should not be compromised on such a level for a little convenience. This should be okay once rootless becomes the norm, but currently the majority users, and production workloads are still running Docker, where all containers are root-full.

    Anyway, if there is no wiggle room, feel free to close this issue. I suppose if that's the case, documentation is key.

  9. abjugard commented on Jan 17, 2023

    @abjugard
    Contributor

    Would it be an acceptable compromise for the official image to additionally install libcap and run setcap cap_net_bind_service=+eip /usr/bin/caddy so that users that want to use a more secure configuration can do so without having to build their own image? As it stands, the image isn't compatible with K8s securityContext runAsNonRoot + runAsUser options, because if I do that then it won't be able to bind port 80, which I personally think it should be able to do, in order to be compatible with e.g. host networking.

    I've provided a pull-request which mitigates this issue, by making it possible to at least use the image with a different PID, given the user configures it in a compatible manner.

  10. francislavoie commented on Jan 17, 2023

    @francislavoie
    Member

    Ah, yeah good idea @abjugard, that seems like a good approach.

  11. 1 remaining item

  12. hairyhenderson commented on Feb 8, 2023

    @hairyhenderson
    Contributor

    Fixed in #274 - this'll be available in the next Caddy release

  13. julsemaan commented on Dec 18, 2023

    @julsemaan

    Sorry for reviving a closed issue here but it seems there are capabilities to run this as non-root but although docs were called out in the issue description, I can't seem to find any docs related to how we can run this image as non-root

    Could you point me at the right location for the docs and I think it will help to have them linked here in the OG issue

    Appreciate the help!

  14. jjlin commented on Dec 18, 2023

    @jjlin

    @julsemaan In general, to run a container as non-root, you just pass -u/--user to docker run; see https://docs.docker.com/engine/reference/commandline/run/ for details.

    Some images (e.g., LinuxServer.io images) use environment variables like PUID/PGID to switch to another user/group during container startup, but there aren't any official standards behind that kind of behavior.

  15. polarathene commented on Sep 7, 2024

    @polarathene

    As it stands, the image isn't compatible with K8s securityContext runAsNonRoot + runAsUser options, because if I do that then it won't be able to bind port 80, which I personally think it should be able to do, in order to be compatible with e.g. host networking.

    FWIW, once containerd 2.0 is out k8s and others that leverage that dependency are likely to adopt the same default that Docker has had for a while where the need for the setcap will be unnecessary: kubernetes/kubernetes#102612 (comment)

    That said, I do not like the setcap approach as it enforces the capability even if you explicitly do not want it. I go into more detail about that issue here: #299 (comment)


    This is especially useful for rootfull containers (e.g. Docker), where the uid=0 inside the container is the same as uid=0 outside.

    Considering root inside a container is root outside a container (for docker anyway), this can be destructive if Caddy is compromised. Security should not be compromised on such a level for a little convenience.

    Just to clear up these two statements. Root in the container is not equivalent to root outside of the container. Yes they both share the same UID/GID, but the root user in the container is run in a namespace, there is isolation going on there, reduced capabilities is a good example of that.

    Regarding the statement about compromising security for convenience, that's technically what you get with setcap approach vs having the application request the capability at runtime if it's available.

    As the linked comment of mine notes, the setcap approach was being planned at one point to grant much more concerning capability to the binary which would not have been possible to opt-out of as a setcap workaround instead of at runtime.


    I ended up here looking for why/when the setcap got introduced. Only after I found this issue did I realize it wouldn't have been as tricky to figure out if I had noticed the main Dockerfile.tmpl file 😑

    For reference it was part of the 2.6.3 release onwards (would be nicer if the repo actually used git tags for releases since the commit would also be associated to the eventual tagged release).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions