Current Behavior
If the route contains upstream (inline or referenced upstream) nodes in domain name form, then when overriding the route, the old checker upon the upstream is not removed. But upstream which contains nodes all in IP form has no issue.
The bug is due to DNS parsed upstream changing its parent to a temporary route copy but not the original one from etcd. So when etcd watches new changes and fails to remove the associated checker.
|
route.dns_value = core.table.deepcopy(route).value |
Expected Behavior
The old checker should be removed.
Error Logs
No response
Steps to Reproduce
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{
"uri":"/get",
"upstream":{
"nodes":[
{
"host":"httpbin.org",
"port":80,
"weight":1
},
{
"host":"postman-echo.com",
"port":80,
"weight":1
}
],
"type":"roundrobin",
"checks":{
"active":{
"concurrency":10,
"healthy":{
"http_statuses":[
200,
302
],
"interval":1,
"successes":2
},
"http_path":"/get",
"timeout":1,
"type":"http",
"unhealthy":{
"http_failures":5,
"http_statuses":[
429,
404,
500,
501,
502,
503,
504,
505
],
"interval":1,
"tcp_failures":2,
"timeouts":3
}
}
}
}
}'
# trigger checker
curl http://127.0.0.1:9080/get -X GET
# override the route
# which should remove the above checker
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{
"uri":"/get",
"upstream":{
"nodes":[
{
"host":"httpbin.org",
"port":80,
"weight":1
},
{
"host":"postman-echo.com",
"port":80,
"weight":1
}
],
"type":"roundrobin",
"checks":{
"active":{
"concurrency":10,
"healthy":{
"http_statuses":[
200,
302
],
"interval":1,
"successes":2
},
"http_path":"/get",
"timeout":1,
"type":"http",
"unhealthy":{
"http_failures":5,
"http_statuses":[
429,
404,
500,
501,
502,
503,
504,
505
],
"interval":1,
"tcp_failures":2,
"timeouts":3
}
}
}
}
}'
# But checker still running
# logs
2023/02/24 14:49:42 [warn] 2384862#2384862: *65399 [lua] healthcheck.lua:1150: log(): [healthcheck] (upstream#/apisix/upstreams/1) unhealthy TIMEOUT increment (1/3) for '(52.1.93.201:80)', context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
2023/02/24 14:49:42 [warn] 2384862#2384862: *65446 [lua] healthcheck.lua:1150: log(): [healthcheck] (upstream#/apisix/upstreams/1) healthy SUCCESS increment (1/2) for '(52.1.93.201:80)', context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
2023/02/24 14:49:44 [warn] 2384862#2384862: *65479 [lua] healthcheck.lua:1150: log(): [healthcheck] (upstream#/apisix/upstreams/1) healthy SUCCESS increment (2/2) for '(52.1.93.201:80)', context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080Use inspect plugin to check if checker is removed:
dbg.set_hook("healthcheck.lua", 434, nil, function(info)
core.log.warn(debug.traceback("clear traceback", 3))
return false
end)For domain upstream, the breakpoint is not triggered.
But for IP upstreams, it's triggered.
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{
"uri":"/get",
"upstream":{
"nodes":[
{
"host":"34.224.50.110",
"port":80,
"weight":1
},
{
"host":"18.205.98.194",
"port":80,
"weight":1
}
],
"type":"roundrobin",
"checks":{
"active":{
"concurrency":10,
"healthy":{
"http_statuses":[
200,
302
],
"interval":1,
"successes":2
},
"http_path":"/get",
"timeout":1,
"type":"http",
"unhealthy":{
"http_failures":5,
"http_statuses":[
429,
404,
500,
501,
502,
503,
504,
505
],
"interval":1,
"tcp_failures":2,
"timeouts":3
}
}
}
}
}'
# breakpoint trigger logs
2023/02/24 14:52:44 [warn] 2385766#2385766: *44 [lua] [string "local dbg = require("apisix.inspect.dbg")..."]:240: clear traceback
stack traceback:
/opt/apisix/apisix/inspect/dbg.lua:75: in function </opt/apisix/apisix/inspect/dbg.lua:41>
/opt/apisix//deps/share/lua/5.1/resty/healthcheck.lua:434: in function </opt/apisix//deps/share/lua/5.1/resty/healthcheck.lua:432>
[C]: in function 'pcall'
/opt/apisix//deps/share/lua/5.1/resty/healthcheck.lua:249: in function 'run_fn_locked_target_list'
/opt/apisix//deps/share/lua/5.1/resty/healthcheck.lua:273: in function 'clear'
/opt/apisix/apisix/upstream.lua:85: in function 'f'
/opt/apisix/apisix/core/config_util.lua:104: in function 'fire_all_clean_handlers'
/opt/apisix/apisix/core/config_etcd.lua:464: in function 'sync_data'
/opt/apisix/apisix/core/config_etcd.lua:606: in function </opt/apisix/apisix/core/config_etcd.lua:596>
[C]: in function 'xpcall'
/opt/apisix/apisix/core/config_etcd.lua:596: in function </opt/apisix/apisix/core/config_etcd.lua:574>, context: ngx.timer
Environment
- APISIX version (run
apisix version): master
- Operating system (run
uname -a):
- OpenResty / Nginx version (run
openresty -V or nginx -V):
- etcd version, if relevant (run
curl http://127.0.0.1:9090/v1/server_info):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run
luarocks --version):
Current Behavior
If the route contains upstream (inline or referenced upstream) nodes in domain name form, then when overriding the route, the old checker upon the upstream is not removed. But upstream which contains nodes all in IP form has no issue.
The bug is due to DNS parsed upstream changing its parent to a temporary route copy but not the original one from etcd. So when etcd watches new changes and fails to remove the associated checker.
apisix/apisix/init.lua
Line 228 in 5c50ffd
Expected Behavior
The old checker should be removed.
Error Logs
No response
Steps to Reproduce
Use inspect plugin to check if checker is removed:
For domain upstream, the breakpoint is not triggered.
But for IP upstreams, it's triggered.
Environment
apisix version): masteruname -a):openresty -Vornginx -V):curl http://127.0.0.1:9090/v1/server_info):luarocks --version):