Support Unix socket for metrics address

@roidelapluie

Proposal

For current Prometheus, it seems that the metrics address to be pulled must be a valid TCP hostname. If Prometheus supports to pull from HTTP over Unix socket, each metrics server won't use a dedicated port to serve for Prometheus.

And I found the related comment:

If my idea does not work, we should still scrape using HTTP (maybe allow unix socket on linux) because changing this would change a lot of the semantics around scraping.
Originally posted by @roidelapluie in #9706 (comment)

This interests me as well. In my case I am tunneling exporters via SSH to the machine running Prometheus, so it makes more sense to use UNIX sockets than ports for each machine.

HTTP over UNIX socket should work fine and very likely most of the underlying connection code won't care about the difference. I will take a look next week and possibly begin drafting a pull.

@DaAwesomeP

Hi @DaAwesomeP, I'm curious about how exactly HTTP over Unix socket would work in Prometheus. Would it be Unix+HTTP in the URL? Additionally, how would HTTPS work in this case? It seems to raise a lot of questions. Maybe @rfratto can provide some insights as well. Thanks!

A UNIX socket can almost always be treated functionally the same as any other TCP socket, minus the hostname and port. It's very common to see web applications expose their HTTP endpoints as a UNIX socket that then get reverse-proxied from something like Nginx or Apache.

A URL like:

http://my-host.xyz:80/metrics

becomes

/path/to/socket.sock/metrics

But really this URL transformation is not accurate. In the case of the TCP socket, first the TCP layer opens the connection to my-host.xyz:80 and then the path /metrics occurs in the HTTP layer. Similarly for the UNIX socket first you open the connection to /path/to/socket.sock and then pass /metrics over the HTTP layer.

The handling of HTTP and HTTPS is 100% the same. The transport layer (TCP replaced with UNIX socket) is what changes.

I need to explore how exactly to do this in Go, but the Dialer just works with UNIX sockets in place of TCP: https://pkg.go.dev/net#example-Dialer-Unix

This sounds good. I have some services with a builtin exporter at /metrics (e.g. gitea and hedgedoc), where the whole service is listening to a UNIX socket proxy forwarded by nginx. As a workaround, I have setup nginx to listen on localhost/metrics/<service> and proxy forward http://unix:/run/<service>/<service>.sock:/metrics, but it would be nice to be able to skip nginx and just gather metrics directly from the socket, since it's all on the same host.

Here's the relevant nginx config for anyone interested

http {
  upstream hedgedoc {
    server unix:/run/hedgedoc/hedgedoc.sock;
  }
  
  upstream gitea {
    server unix:/run/gitea/gitea.sock;
  }
  
  # Metric endpoints
  server {
    listen 127.0.0.1:80;
    listen [::1]:80;
    location /metrics/gitea {
      proxy_pass http://gitea/metrics;
    }
    location /metrics/hedgedoc {
      proxy_pass http://hedgedoc/metrics;
    }
  }
  
  # Example of normal endpoint
  server {
    listen 0.0.0.0:443;
    listen [::0]:443;
    server_name hedgedoc.example.com;
    
    # Misc SSL config
    # ...
    
    location / {
      proxy_pass http://hedgedoc;
    }
  }
}

Did you try the systemd socket that we have in the latest releases? Le lun. 24 juil. 2023, 13:26, Øystein Tveit ***@***.***> a écrit :

…

This sounds good. I have some services with a builtin exporter at /metrics (e.g. gitea and hedgedoc), where the whole service is listening to a UNIX socket proxy forwarded by nginx. As a workaround, I have setup nginx to listen on localhost/metrics/<service> and proxy forward http://unix: /run/<service>/<service>.sock:/metrics, but it would be nice to be able to skip nginx and just gather metrics directly from the socket, since it's all on the same host. Here's the relevant nginx config for anyone interested http { upstream hedgedoc { server unix:/run/hedgedoc/hedgedoc.sock; } upstream gitea { server unix:/run/gitea/gitea.sock; } # Metric endpoints server { listen 127.0.0.1:80; listen [::1]:80; location /metrics/gitea { proxy_pass http://gitea/metrics; } location /metrics/hedgedoc { proxy_pass http://hedgedoc/metrics; } } # Example of normal endpoint server { listen 0.0.0.0:443; listen [::0]:443; server_name hedgedoc.example.com; # Misc SSL config # ... location / { proxy_pass http://hedgedoc; } } } — Reply to this email directly, view it on GitHub <#12024 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHHJX35DD4E4ZY4HWN3G3XRZLV7ANCNFSM6AAAAAAVI5MMVU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

I cannot find what you are referencing. I've grepped through release notes, docs, and the codebase, but I didn't see any relevant mentions of systemd nor sockets?

@roidelapluie

@roidelapluie it is currently possible for the exporter-toolkit to listen on a UNIX socket via a Systemd socket activation, but it is not currently possible for Prometheus to query these socket directly (unless I missed a commit/pull somewhere).

In my own installations I have also used socat to convert a UNIX socket to a TCP port listener, but that method has its own flaws/issues similar to using Nginx as a reverse proxy (accomplishes same goal).

As a work around, you can run the following docker container, and access metrics from inside the network or from the host exposed port:

docker run -d --restart=always \
	--network myNetwork \
	-v /path/to/socketfile.sock:/path/to/socketfile.sock \
	alpine/socat \
	tcp-listen:2375,fork,reuseaddr unix-connect:/path/to/socketfile.sock

or without docker:

socat tcp-listen:2375,fork,reuseaddr unix-connect:/path/to/socketfile.sock

test it with:

curl --unix-socket /path/to/socketfile.sock http://localhost/

Discussed at the bug-scrub:
We felt that something in target config would be needed to flag that a Unix socket should be used, and then the scraping code would be changed to interpret that config and use Unix sockets or TCP sockets as required.

I am using the Docker socat workaround in production. The caveat is if you need multiple connections to the socket. To handle this you must put socat in fork mode, which means that it won't restart/detect if a UNIX socket closes and reopens (i.e. providing service restarts or SSH tunnel reconnects). To solve this I additionally have a Systemd path unit watching the UNIX sockets with the PathModified flag that then restarts the socat containers when a UNIX socket is reopened. I never tried using Nginx for this (I should have thought of that!), so it might handle UNIX socket reconnection better. An advantage of many socat containers is that each has its own hostname (scales easily instead of many ports or having to adjust URL paths).

It was a lot to set up, but with this I am successfully and reliably able to reverse SSH tunnel exporters from another machine without polluting the port listeners on the Prometheus machine. Native UNIX socket support would simplify this a lot.

For implementation, it could be as simple as differentiating between a URL starting with http://, https://, and unix:// (or could just look for paths starting with a /). There would also need to be a way to manually provide a hostname (not required for the connection but for how it is represented in Prometheus). I solved this by naming my socat containers.

HTTP over unix socket would be pretty useful for stuff like HAProxy stats. I'd imagine syntax should look something like unix:///file/path.sock?path=/metrics" or 'unix+http:// if we want to differentiate vs raw TCP usage.

Other option would be something like unix://[/file/path.sock]/metrics.

I'm not sure how often passing other headers to unix socket would be required but simplistic solution could be just having header argument in url when there user can do whatever they need to do

apache uses this semantic:

ProxyPass unix:/var/run/test.sock|http://127.0.0.1/api/test

unix:/var/run/test.sock is real target address, http://127.0.0.1/api/test is a fake request address for proxied service environment

anubis: export metrics over TCP

metrics: init

chore(deps): update dependency prometheus/prometheus to v3.15.0

feat(container): update image quay.io/prometheus/prometheus (v3.14.0 …

Navigation Menu

Repository navigation

Support Unix socket for metrics address #12024

Proposal

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support Unix socket for metrics address #12024

Description

Proposal

Activity

DaAwesomeP commented on Mar 3, 2023

roidelapluie commented on Mar 6, 2023

DaAwesomeP commented on Mar 6, 2023

h7x4 commented on Jul 24, 2023

roidelapluie commented on Jul 24, 2023

h7x4 commented on Jul 24, 2023

DaAwesomeP commented on Jul 26, 2023

maxant commented on Nov 2, 2024

bboreham commented on Nov 5, 2024

DaAwesomeP commented on Nov 5, 2024

XANi commented on Nov 8, 2024

Shaeto commented on Nov 20, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions