Skip to content

bug: Prometheus metrics igress and egress is inaccurate #6555

Description

@qizhendong1

Issue description

There two routes, one priority is 0 route id = 1, anther is 30 route id = 3.

if request hit route 1, the prometheus metric igress and egress is accurate, but if request hit route 3, the igress and egress valuse is double of bandwidth.

I think that apisix records twice request_length and bytes_sent.

    metrics.bandwidth:inc(vars.request_length,
        gen_arr("ingress", route_id, service_id, consumer_name, balancer_ip))

    metrics.bandwidth:inc(vars.bytes_sent,
        gen_arr("egress", route_id, service_id, consumer_name, balancer_ip))

Environment

  • apisix version (cmd: apisix version): 2.7
  • OS (cmd: uname -a):
  • OpenResty / Nginx version (cmd: nginx -V or openresty -V):
  • etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
  • apisix-dashboard version, if have:
  • the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
  • luarocks version, if the issue is about installation (cmd: luarocks --version):

Steps to reproduce

route configuration

{
  "action": "get",
  "node": {
    "key": "/apisix/routes",
    "dir": true,
    "nodes": [
      {
        "key": "/apisix/routes/1",
        "modifiedIndex": 6332825,
        "value": {
          "status": 1,
          "methods": [
            "PUT",
            "GET",
            "POST",
            "DELETE",
            "PATCH",
            "HEAD",
            "OPTIONS",
            "CONNECT",
            "TRACE"
          ],
          "upstream": {
            "pass_host": "pass",
            "nodes": {
              "10.235.79.7:8079": 10
            },
            "scheme": "http",
            "hash_on": "vars",
            "type": "roundrobin"
          },
          "uri": "/*",
          "create_time": 1646098895,
          "id": "1",
          "priority": 0,
          "update_time": 1646098895,
          "plugins": {
            "prometheus": {
              "prefer_name": false
            }
          }
        },
        "createdIndex": 6332825
      },
      {
        "key": "/apisix/routes/3",
        "modifiedIndex": 4020996,
        "value": {
          "status": 1,
          "methods": [
            "PUT",
            "GET",
            "POST",
            "DELETE",
            "PATCH",
            "HEAD",
            "OPTIONS",
            "CONNECT",
            "TRACE"
          ],
          "upstream": {
            "hash_on": "vars",
            "type": "roundrobin",
            "nodes": {
              "10.235.79.7:9096": 10
            },
            "pass_host": "pass",
            "scheme": "http"
          },
          "uri": "/*",
          "priority": 20,
          "vars": [
            "OR",
            [
              "host",
              "~*",
              ".*(abc.cn)$"
            ],
            [
              "host",
              "==",
              "10.235.82.1"
            ]
          ],
          "id": "3",
          "update_time": 1643262492,
          "create_time": 1638353491
        },
        "createdIndex": 1682
      }
    ]
  }
}

Client write: 145MB/s + 133MB/s + 144MB/s + 1.15GB/s = 1.55GB/s, but apisix igress is 2.85GB/s

image

image

Actual result

The bandwidth = apisix igress or egress

Error log

no

Expected result

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions