Description

Problem description
The customer has a domain name configured with health check (the upstream type is K8S service discovery). Now it is found that after the upstream service node is updated, the nodes in the health check are not updated, resulting in a large number of health check failure logs:

*content:2025/03/28 16:36:17 [error] 42#42: 309692373 [lua] healthcheck.lua:1150: log(): [healthcheck] (upstream#/common/ontest/internal/routes/570) failed to send http request to 'nil (10.24.64.115:14699)': connection timed out, context: ngx.timer, client: 172.24.128.181, server: 0.0.0.0:9443

Reproduction steps

1. Create a route and enable health check
2. Determine how to deploy multiple pods upstream
3. Access route triggers health check
4. Reduce the size of the upstream node to 1 pod (when the upstream node is 1, the health check related logic will no longer be entered, and because it is service discovery, there is no etcd change, so the health checker cannot be released https://github.com/apache/apisix/blob/release/3.10/apisix/upstream.lua#L352-L355)
5. Observing the logs, you will find that health checks will still be sent to all pods before scaling down (you need to set the log level to info or add a warn log after healthcheck attempts to connect to the node, otherwise healthcheck will not generate logs after timeout; the customer's IP and port are open, but the http request cannot be processed, so there are error level logs)

Solution

There are 3 possible solutions and in this PR I have decided to go with 1st one though reviewers can tell their opinion.

Fix can be any of the following

 diff orig fixed
1,4c1,2
<     if nodes_count > 1 then
<         local checker = fetch_healthchecker(up_conf)
<         api_ctx.up_checker = checker
<     end
---
>     local checker = fetch_healthchecker(up_conf)
>     api_ctx.up_checker = checker

OR

diff orig fixed2
1c1
<     if nodes_count > 1 then
---
>     if nodes_count > 0 then

OR

     if nodes_count > 1 then
         local checker = fetch_healthchecker(up_conf)
         api_ctx.up_checker = checker
     end    
>    if nodes_count == 1 and up_conf.parent and up_conf.parent.checker then
>        release_checker(up_conf.parent)
>    end

Conclusion

Solution 1 looks more general so we can go with it.

Logic of added test

• After first request the error from healthcheck will keep coming incrementing from 1/30 every second because an unreachable upstream is present in config.
• Then we wait 5 seconds and reset config to only reachable one and send another request.
• This SHOULD release previous healthchecker and the error logs will stop coming.
• This test fails without this change as the error logs will keep coming till 10/30

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change
• I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)