thanks for running the tests and contributing ! Had a few questions -

1. should we also set net/ipv4/vs/expire_quiescent_template to 1 which should expire persistent connections to the real server with weight 0 (after the backend is down thanks to the net/ipv4/vs/expire_nodest_conn=1 setting ), but I see there is an open issue kube-proxy ipvs conn_reuse_mode setting causes errors with high load from single client kubernetes/kubernetes#81775 so would like to understand the negative implications of setting net/ipv4/vs/conn_reuse_mode to 0
   cc: @lbernail

2. will this setting work for most kernels ?

This fix resolved the same issue for me. My findings here: moby/moby#35082 (comment)

Happy to help in any way to move this forward!

@arkodg do you think this fix is likely to be introduced at this layer? Trying to understand if we should add scripting to our deployment that adds this after every docker stack deploy:

sudo nsenter --net=/var/run/docker/netns/{your_load_balancer} sysctl -w net.ipv4.vs.conn_reuse_mode=0
sudo nsenter --net=/var/run/docker/netns/{your_load_balancer} sysctl -w net.ipv4.vs.expire_nodest_conn=1

...or whether we should wait for a fix here. Any guidance would be much appreciated!

@geekdave
That's our current solution to this problem.

default_network_id=$(docker network ls | grep "myservice_default" | awk '{print $1}')
sudo nsenter --net=/var/run/docker/netns/lb_${default_network_id:0:9} sysctl -w net.ipv4.vs.conn_reuse_mode=0
sudo nsenter --net=/var/run/docker/netns/lb_${default_network_id:0:9} sysctl -w net.ipv4.vs.expire_nodest_conn=1

Once your stack is running, consecutive docker stack deploy is unnecessary since your load balancer sandbox will be re-used throughout the lifetime of your docker stack.

Thanks for this, @ahjumma ! We have some tooling that performs an initial stack deploy on fresh machines. I'm considering adding some scripting that waits in a loop until the ${service}_default network is created before running your commands. Does that seem like the right approach?

👋 In kube-proxy we set:

• net.ipv4.vs.conn_reuse_mode=0
• net.ipv4.vs.expire_nodest_conn=1
• net.ipv4.vs.expire_quiescent_template=1

expire_nodest_conn is required to avoid blackholing traffic to existing destinations after removing the RealServer (RST on next packet to backend for TCP, ICMP destination unreachable for UDP)

expire_quiescent_template is great when you set weight to 0 to avoid creating new connections to a real server with weight 0

conn_reuse_mode is great for performances: if you set it to 1, a new connection reusing the same 5 tuples will get its first packet dropped, which means the new connection will wait for 1s to retry.
However, it leads to some problems in kube-proxy because if ports are reused fast enough a realserver always has Active/InActive Connections and is never garbage collected (we currently asynchronously gc backends with weight 0 when they have no connections, but we are looking into ways to forcefully delete the backend after a while to avoid this situation)

thanks for running the tests and contributing ! Had a few questions -

1. should we also set net/ipv4/vs/expire_quiescent_template to 1 which should expire persistent connections to the real server with weight 0 (after the backend is down thanks to the net/ipv4/vs/expire_nodest_conn=1 setting ), but I see there is an open issue kubernetes/kubernetes#81775 so would like to understand the negative implications of setting net/ipv4/vs/conn_reuse_mode to 0
   cc: @lbernail
2. will this setting work for most kernels ?

Hi @arkodg.

It does seem net.ipv4.vs.expire_quiescent_template=1 would be also helpful.

Regarding your concern raised by possible problematic behaviours of net.ipv4.vs.conn_reuse_mode=0, net.ipv4.vs.expire_nodest_conn=1 is what allows by-passing the problem.
This has been acknowledged in kubernetes/kubernetes/issues/81775#issuecomment-542542167 you have referenced .

For the second question, what's the earliest linux kernel that Docker Swarm needs to support?

thanks @lbernail @ahjumma
can we also include net.ipv4.vs.expire_quiescent_template=1

Hi @arkodg ,

According to @lbernail's previous finding, kernel 2.6 is supported.

can you ptal as well @lbernail

will look into it soon.

I did the same steps but the ingress LB still low. Did I missed any steps
sudo nsenter --net=/var/run/docker/netns/lb_o7wjqxluj sysctl -w net.ipv4.vs.conn_reuse_mode = 0
sudo nsenter --net=/var/run/docker/netns/lb_o7wjqxluj sysctl -w net.ipv4.vs.expire_nodest_conn = 1
sudo nsenter --net=/var/run/docker/netns/lb_o7wjqxluj sysctl -w net.ipv4.vs.expire_quiescent_template = 1

Do I have to restart server!?
Or do I have to set same configs like above on the host deployed swarm!?