[inputs.prometheus] 406 Not Acceptable returned from prometheus style metrics endpoint #6523
Description
Relevant telegraf.conf:
[global_tags]
org = "test_org"
country = "test_country"
dc = "test_dc"
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "5s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "s"
debug = true
quiet = false
# logfile = "/dev/null"
hostname = "test_host"
omit_hostname = false
[[outputs.file]]
files = [ "stdout" ]
data_format = "influx"
[[inputs.prometheus]]
urls = [ "http://10.6.0.249:15692/metrics" ]System info:
OS: CentOS 7
Telegraf 1.11.3 (git: HEAD d6024fd)
Steps to reproduce:
[1] Startup a RabbitMQ 3.8.0 server and enable the rabbitmq_prometheus plugin
[2] Configure telegraf prometheus input plugin to scrape the rabbitmq metrics endpoint
Expected behavior:
A successful scrape
Actual behavior:
The rabbitmq_prometheus plugin endpoint responds with a 406 Not Acceptable response.
A sample http trace is below:
Request (Telegraf Prometheus Input Plugin):
T 2019/10/14 12:25:02.854993 10.6.0.227:38254 -> 10.6.0.249:15692 [AP]
GET /metrics HTTP/1.1
Host: 10.6.0.249:15692
User-Agent: Go-http-client/1.1
Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;q=0.3
Accept-Encoding: gzip
Connection: close
Response (RMQ /metrics endpoint)
T 2019/10/14 12:25:02.856160 10.6.0.249:15692 -> 10.6.0.227:38254 [AP]
HTTP/1.1 406 Not Acceptable
connection: close
content-length: 0
date: Mon, 14 Oct 2019 11:25:02 GMT
server: Cowboy
The logs from the telegraf client are as follows:
2019-10-14T11:25:02Z E! [inputs.prometheus]: Error in plugin: http://10.6.0.249:15692/metrics returned HTTP status 406 Not Acceptable
2019-10-14T11:25:10Z D! [outputs.file] buffer fullness: 0 / 10000 metrics.
2019-10-14T11:25:14Z E! [inputs.prometheus]: Error in plugin: http://10.6.0.249:15692/metrics returned HTTP status 406 Not Acceptable
2019-10-14T11:25:20Z D! [outputs.file] buffer fullness: 0 / 10000 metrics.
2019-10-14T11:25:20Z E! [inputs.prometheus]: Error in plugin: http://10.6.0.249:15692/metrics returned HTTP status 406 Not Acceptable
2019-10-14T11:25:30Z D! [outputs.file] buffer fullness: 0 / 10000 metrics.
Additional info:
I initially opened up a bug on the rabbitmq-promethues github repository, however, this was subsequently closed stating that the [[inputs.prometheus]] plugin should support Prometheus v2.x exposition formats
Please could we check if the above is indeed supported in the prometheus inputs plugin.
If you require any additional information from me please dont hesitate to ask.