Skip to content

Prometheus docker container shouldn't run as user nobody #3441

Closed

Description

@jcollie

With the update to 2.0, the Docker container was modified so that Prometheus runs as the user "nobody" (UID/GID 99/99) (#2859). While it's good that Prometheus is no longer running as root, it's not the best idea either to run as "nobody". There are many examples of Docker containers out there that allow changing the UID/GID at runtime that could be borrowed from. At a minimum a different static UID/GID should be used.

Activity

  1. grobie commented on Nov 8, 2017

    @grobie
    Member

    it's not the best idea either to run as "nobody"

    For what reasons?

  2. jcollie commented on Nov 8, 2017

    @jcollie
    Author
    • The nobody user/group was originally designed to be used by NFS to map the UID/GID 0/0 on remote mounts to something else so it's not really a "free" account that can be used by other services, especially ones that own files on disk.
    • There's also the fact that the nobody group is not always UID/GID 99 so that UID/GID may be used for other purposes on a system.
    • Really the most important reason is that using a fixed UID/GID (especially a low numbered one that is likely to be used for other purposes) makes it impossible to keep data separate from other services. That's why a lot of vendor-supplied Docker containers make it possible to change the UID/GID that the server runs as at runtime.
  3. grobie commented on Nov 8, 2017

    @grobie
    Member

    Sounds good to me. If it's the best practice to give the option to change the UID in order to have control over file owners, wanna send a PR for that?

  4. nrvnrvn commented on Nov 19, 2017

    @nrvnrvn

    It is a curious question.
    I've been reviewing security in our docker images recently and while wondering whether I should use a dedicated user or use something from the default set with the least privileges I came across this issue.

    So:

    • yes, historically there is a lot of legacy stuff that landed in /etc/passwd and others and even in the Linux Standard Base.
    • historically, a lot of daemons used to run under the nobody or daemon user.
    • but then they discovered that once more than one service starts sharing the same predefined user they find themselves in an unwanted security danger, thus no every service is trying to create and run on behalf of its own user in a shared linux namespace.

    BUT

    Containers are somewhat different beasts...

    See:

    Basically this instruction does not bind to the specific user but to the specific UID. And you always can easily override it in docker/kubernetes.

    $ docker run --user 1234:5678 --rm -ti busybox
    / $ id
    uid=1234 gid=5678
    / $ 

    So unless you f*cked up somewhere else I find it pretty safe to run a single process container under any uid:gid except root:root, just tell everyone the appropriate uid:gid (not the username:groupname !) to use for the mounted volumes.

    Any objections to the above are welcome :)

    P.S. From what I've seen so far from various vendors people follow both paths (either creating a separate user or using 65534[debian-based/alpine] or 99[centos-based])

  5. jcollie commented on Nov 20, 2017

    @jcollie
    Author

    Kubernetes does not (at least not yet) allow you to change the primary GID of a pod/container. There are some patches making their way through the process but AFAIK they are not part of any released version of Kubernetes. Also, "just" changing the UID/GID that the primary process runs as isn't always sufficient. Depending on the security model, changing the ownership of some files may be necessary as part of the startup process.

    In any case, choosing 99 as the default UID/GID isn't the best idea since it's likely that a low number may be in use by other things in the overall system. If we're going to stick with a fixed UID/GID it'd be better to use something semi-random like 9090.

  6. nrvnrvn commented on Nov 21, 2017

    @nrvnrvn

    Let's assume we have two containers running in the system (single or multi-host, no matter), with volumes attached and both of them use the same pair of uid:gids (say 5555:5555). Hence the attached volumes' files are recursively owned by 5555:5555.

    What are the possible implications in this case?

  7. 2color commented on Dec 5, 2017

    @2color

    With the update to 2.0, the Docker container was modified so that Prometheus runs as the user "nobody" (UID/GID 99/99)

    I think the correct UID would be 65534

    $ docker run -it --rm --entrypoint sh prom/prometheus:v2.0.0
    
    /prometheus $ id
    uid=65534(nobody) gid=65534(nogroup)
    

    UID 99 is the default nobody in Centos

  8. silentpete commented on Jan 9, 2018

    @silentpete
    Contributor

    In our docker compose, I was able to use ":user 1234:5678" concept as mentioned above along with our Ansible deploy play to set the directory permissions on anything volume mounted.

    Container runs perfectly

  9. nemo83 commented on Apr 13, 2018

    @nemo83

    @silentpete do you have an example of the docker-compose with the user uid/gid configuration? I can set the --user 1234:5678 with docker run, but not with compose.

  10. silentpete commented on Apr 13, 2018

    @silentpete
    Contributor

    Hello @nemo83 ,
    I'm currently using the following in my docker compose

    version: '2'
    services:
      prometheus:
        image: prom/prometheus:v2.2.1
        container_name: prometheus
        hostname: prometheus
        volumes:
          - $PWD/containers/prometheus/:/opt/prometheus/:ro
          # keep data
          - /data/prometheus/:/prometheus/data/:rw
          - /etc/localtime:/etc/localtime:ro
        command:
          - '--config.file=/opt/prometheus/prometheus.yml'
          # prometheus retention period
          - '--storage.tsdb.retention=30d'
        logging:
          driver: "json-file"
        mem_limit: 512m
        memswap_limit: 512m
        restart: unless-stopped
        user: 9090:9090
    

    Can see the user in the container with

    # docker exec prometheus ps -ef | grep prometheus
        1 9090       2:56 /bin/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.retention=30d
    

    which I copied out of: https://github.com/silentpete/pg-h.io/blob/master/docker-compose.yml

    Hope this helps,
    Peter

    PS, I created a 9090:9090 (user and group) on my host, then set the volume mounted in directory to have the correct permissions for the data being saved out of the container.

  11. jsravn commented on Nov 6, 2018

    @jsravn
    Contributor

    The main problem w/ running containers as non-root is K8s. How do you guys solve the fact that k8s persistent volumes are generally mounted as root-writeable only? fsGroup is not an option for me either - Openshift, for example, doesn't allow setting fsGroup (to 65543 in this case).

    It seems to me that the general trend is to let users specify their UIDs, either via remapping in docker, or runAsUser in k8s. The entire situation is sorta messy unfortunately. I'd suggest sticking with root for maximum compatibility.

  12. nrvnrvn commented on Nov 6, 2018

    @nrvnrvn

    Setting runAsUser and fsGroup to the same values has always worked for me in GKE. It works even with those images where no USER is defined in Dockerfile. Use any UID in range 1-2147483647.

    Can't say anything for openshift though. I hope they have got some solution for that anyway.

  13. jsravn commented on Nov 6, 2018

    @jsravn
    Contributor

    Another issue with fsGroup is that it operates at pod level only, so it forces all mounts to that gid which may not be ideal.

    Openshift manages UID/GID itself - it has an admission controller I believe which overrides runAsUser / fsGroup. But unfort. it will forbid your pod entirely if you set those values, so building a single pod definition to work across different k8s flavors isn't possible w/ non-root users (as far as I can tell).

  14. jcollie commented on Nov 6, 2018

    @jcollie
    Author

    The solution that I've seen/used is to use an init container. Perhaps not ideal but it works.

        spec:
          initContainers:
            - name: change-ownership
              image: busybox
              command:
                - sh
                - -c
                - "chown -Rc 991:991 /data"
              volumeMounts:
                - name: data
                  mountPath: /data
    
  15. 1 remaining item

  16. the-maldridge commented on Oct 31, 2020

    @the-maldridge

    I was just bit by this, Nomad doesn't have a way to set up the bind users like this so I am having to set up a bind volume manually, kind of defeating the point of using the container.

  17. ohritz commented on Jul 1, 2023

    @ohritz

    Ran into this issue this week when trying to run prometheus within OpenShift, was testing things out with open-telemetry and wanted to run prometheus as a deployment with the quay.io/prometheus/prometheus:v2.43.0 image.

    Setting a user with securityContext is not permitted, at least not anything lower than uid: 1001030000
    And the init container solution does not work either.

    As per OpenShift Guidlines https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines under the section Support Arbitrary User IDs

    All that would be required to not get permission denied is to add

    RUN chgrp -R 0 /prometheus && \
        chmod -R g+rwX /prometheus
    

    I built my own image with

    FROM quay.io/prometheus/prometheus:v2.43.0
    
    USER 0
    RUN chgrp -R 0 /prometheus && \
        chmod -R g+rwX /prometheus
    
    USER nobody
    

    and this worked.

    Would it be a alright to add it to the Dockerfile, I can make a PR if needed.

  18. marvinnitz18 commented on Jul 17, 2023

    @marvinnitz18

    @ohritz please open a merge request for that, i would support it

  19. marvinnitz18 commented on Aug 21, 2023

    @marvinnitz18

    i created one with your suggestions: #12728

  20. ohritz commented on Oct 13, 2023

    @ohritz

    @marvinnitz18 I'm sorry I never got to making a PR, I honestly forgot about it. I see you made you took over, thanks!

  21. marvinnitz18 commented on Oct 16, 2023

    @marvinnitz18

    @ohritz no worries, lets see if it can get merged

  22. elysweyr commented on Feb 21, 2024

    @elysweyr

    Hi,
    for those of you encountering this issue with the official prometheus image and need a fix right away, I got you covered.

    Red Hat itself is publishing a patched version of the prometheus image which is suitable for rootless execution on OpenShift.
    Image: registry.redhat.io/openshift4/ose-prometheus

    Here's a link to their latest image at the time this comment was published.

    On lines 26-28 you'll find their applied patch.

    RUN mkdir -p /prometheus && \
        chgrp -R 0 /etc/prometheus /prometheus && \
        chmod -R g=u /etc/prometheus /prometheus
    
  23. beorn7 commented on Feb 27, 2024

    @beorn7
    Contributor

    Hello from the bug scrub. #12728 is currently under review, so hopefully this will be addressed soon.

  24. naanlizard commented on Sep 22, 2024

    @naanlizard

    Perhaps we should rephrase this ticket as (and update the MR to) "Support docker --user flag" or something along those lines

    The exact user the dockerfile builds/runs as doesn't matter if we can set e.g. --user 1000:1000 when running it

  25. dgl commented on Feb 25, 2025

    @dgl
    Member

    Unfortunately the solution outlined by ohritz on Jul 2, 2023:

    As per OpenShift Guidlines https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines under the section Support Arbitrary User IDs

    Won't work here – that is OpenShift specific, whereas the Prometheus image needs to work under Docker, standard Kubernetes and anything else that can run OCI containers.

    We can confirm it won't work by simply running docker like this on the image:

    $ docker run --entrypoint /bin/id prom/prometheus:v3.2.0
    uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)
    

    As you can see that ends up with a group ID of nobody, not a gid of 0 as the OpenShift docs imply. (The reason is the USER instruction varies in behaviour, depending whether the given user exists, see the Warning under USER).

    This means the proposed PR will break users in some cases (as the ownership will change). We have a strong compatibility promise so can't change UID/GIDs like that.

    For Kubernetes the recommended approach is now to use fsGroup – if that approach doesn't work on OpenShift then it is likely it will continue to need its own special image (I haven't tested but OpenShift do support using fsGroup).

    For Docker there (still) aren't as good volume management options, so while we would prefer to defer to the container platform we possibly need to do something. It is possible to use --group-add with docker, which makes this work:

    $ docker run --entrypoint /bin/id --user 9090:9090 --group-add 65534 prom/prometheus:v3.2.0
    uid=9090 gid=9090 groups=9090,65534(nobody)
    

    Therefore simply doing chmod g+w /prometheus is enough to support running as any UID for Docker itself, while not changing any UID/GID or reducing permissions in the shipped image (it is tempting to add the group sticky bit, to preserve the group, but again this has the potential to break users) – I will open a PR doing just this.

    Note that Docker does not have the fsGroupChangePolicy mechanism that Kubernetes has, so this will only work for a clean volume (in order to change the UID/GID of a Prometheus running inside Docker you would have to chown the TSDB files manually).

  26. added a commit that references this issue on Feb 25, 2025
  27. locked as resolved and limited conversation to collaborators on Sep 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions