Skip to content

bug: some mistakes in prometheus plugin #6301

Description

@nic-6443

Issue description

The APISIX prometheus plugin uses the /nginx_status endpoint provided by nginx to get some internal metrics, which return the data like

Active connections: 291
server accepts handled requests
 16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

According to the nginx documentation:

The following status information is provided:

Active connections
The current number of active client connections including Waiting connections.
accepts
The total number of accepted client connections.
handled
The total number of handled connections. Generally, the parameter value is the same as accepts unless some resource limits have been reached (for example, the worker_connections limit).
requests
The total number of client requests.
Reading
The current number of connections where nginx is reading the request header.
Writing
The current number of connections where nginx is writing the response back to the client.
Waiting
The current number of idle client connections waiting for a request.

The third number in the third line is The total number of client requests, but APISIX now puts this metric in nginx_http_current_connections{label="total"}, which is obviously wrong.

metrics.connections = prometheus:gauge("nginx_http_current_connections",
"Number of HTTP connections",
{"state"})

local ngx_status_items = {"active", "accepted", "handled", "total",
"reading", "writing", "waiting"}
local label_values = {}
local function nginx_status()
local res = ngx_capture("/apisix/nginx_status")
if not res or res.status ~= 200 then
core.log.error("failed to fetch Nginx status")
return
end
-- Active connections: 2
-- server accepts handled requests
-- 26 26 84
-- Reading: 0 Writing: 1 Waiting: 1
local iterator, err = re_gmatch(res.body, [[(\d+)]], "jmo")
if not iterator then
core.log.error("failed to re.gmatch Nginx status: ", err)
return
end
for _, name in ipairs(ngx_status_items) do
local val = iterator()
if not val then
break
end
label_values[1] = name
metrics.connections:set(val[0], label_values)
end
end

Also the three metrics in the third line are all cumulative, should we use the counter type instead of the guage for those metrics? Which is more in line with the prometheus definition of metric types.

So I want to introduce two new metric:

  1. nginx_http_requests_total(counter type)
  2. nginx_http_connections_total(counter type) with a type label, which has two values: accepts, handled

If there are no problems, I can complete the fix.

Environment

  • apisix version (cmd: apisix version): 2.10.1

Steps to reproduce

Just enable prometheus plugin.

Actual result

/

Error log

/

Expected result

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions