Background

1. Currently, lua-resty-etcd supports cluster mode, but the way of implementation is too simple: each connection changes to the next endpoint.

2. Using this mechanism, it can work well under normal circumstances, but once an api or an instance has a problem, the consequences will be unpredictable.

Issues with the current solution

1. In cluster mode, when an instance is down, there is no way to skip the down instance, and it will still be polled every time.

2. When all instances of a certain api fail (such as auth api), it will cause crazy retries, which may eventually overwhelm the ETCD cluster.

Suggested changes

1. Implement a health check mechanism, no active check is required, only passive check is required, that is, it is recorded when the connection fails.

2. There is no need to poll all instances, and only switch instances when the connection fails.

3. In a certain period of time, if there are n consecutive failures, the instance is considered unhealthy, and the instance will not be connected for a certain period of time in the future (the duration and times can be configured).

related issue: apache/apisix#2899

what do you think ?

Thanks.