Skip to content

bug: Multiple unhealthy HTTP increment if the size of passive.unhealthy.http_statuses is more than 1 #4103

Description

@Ben0625

Issue description

One request with unhealthy code returned will cause multiple unhealthy HTTP increment if the size of passive.unhealthy.http_statuses is more than 1.

Environment

  • apisix version: 2.0
  • OS: 3.10.0-693.el7.x86_64
  • OpenResty version: openresty/1.17.8.2
  • etcd version: 3.4.13
  • apisix-dashboard version: 2.0

Minimal test code / Steps to reproduce the issue

The route configuration in etcd is below:

{
	"id": "route_2050210843",
	"create_time": 1618455175,
	"update_time": 1619013202,
	"uris": ["/test1.do"],
	"name": "route-2050210843",
	"desc": "health_check_test",
	"methods": ["GET", "POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS"],
	"vars": [],
	"upstream": {
		"nodes": [{
			"host": "172.25.69.182",
			"port": 8080,
			"weight": 1
		}, {
			"host": "172.25.70.116",
			"port": 8080,
			"weight": 1
		}],
		"timeout": {
			"connect": 6,
			"read": 6,
			"send": 6
		},
		"type": "roundrobin",
		"checks": {
			"active": {
				"healthy": {
					"interval": 100,
					"successes": 10
				},
				"host": "172.25.69.182",
				"http_path": "/check.do",
				"req_headers": [""],
				"timeout": 3,
				"unhealthy": {
					"http_failures": 5,
					"interval": 3
				}
			},
			"passive": {
				"healthy": {
					"http_statuses": [200],
					"successes": 2
				},
				"unhealthy": {
					"http_failures": 5,
					"http_statuses": [500, 502, 504],
					"tcp_failures": 1
				}
			}
		}
	},
	"labels": {
		"serve": "2050210843"
	}
}
  1. Set a route with two upstreams and healthcheck active and passive configuration(passive.unhealthy.http_statuses is [500, 502, 504]). One upstream always return 500 and another always return 200.
  2. Send a request to trigger creating new checker. It will be passed to the normal upstream.
  3. Send a second request, it will be passed to the unnormal upstream.

What's the actual result?

One request with unnormal upstream will cause three times unhealthy HTTP increment.

2021/04/21 21:54:00 [warn] 73450#73450: *3017774 [lua] healthcheck.lua:1142: log(): [healthcheck] (upstream#/apisix/routes/route_2050210843) unhealthy HTTP increment (1/5) for '172.25.69.182(172.25.69.182:8080)' while logging request, client: 10.58.50.104, server: , request: "GET /test1.do HTTP/1.1", upstream: "http://172.25.69.182:8080/test1.do", host: "172.25.71.28:9080"

2021/04/21 21:54:00 [warn] 73450#73450: *3017774 [lua] healthcheck.lua:1142: log(): [healthcheck] (upstream#/apisix/routes/route_2050210843) unhealthy HTTP increment (2/5) for '172.25.69.182(172.25.69.182:8080)' while logging request, client: 10.58.50.104, server: , request: "GET /test1.do HTTP/1.1", upstream: "http://172.25.69.182:8080/test1.do", host: "172.25.71.28:9080"

2021/04/21 21:54:00 [warn] 73450#73450: *3017774 [lua] healthcheck.lua:1142: log(): [healthcheck] (upstream#/apisix/routes/route_2050210843) unhealthy HTTP increment (3/5) for '172.25.69.182(172.25.69.182:8080)' while logging request, client: 10.58.50.104, server: , request: "GET /test1.do HTTP/1.1", upstream: "http://172.25.69.182:8080/test1.do", host: "172.25.71.28:9080"

What's the expected result?

One request with unnormal upstream will cause only once unhealthy HTTP increment.
I cannot understand why it has to be double loop in function healcheck_passive of apisix/init.lua

for i, status in ipairs(http_statuses) do
        for i, status in ipairs(http_statuses) do
            if resp_status == status then
                checker:report_http_status(api_ctx.balancer_ip,
                                           port or api_ctx.balancer_port,
                                           host,
                                           resp_status)
            end
        end
    end

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions