Constantly got error "Rejected a connection from server X since one is open already." #5456
Description
I've just upgraded my rethinkdb cluster v2.2.5 with 3 instances. There are 16 tables, ~17Gb data and this error show up continuously while all 3 instances are already connected:
Rejected a connection from server ff02cdcb-eeff-436f-9736-381dc64163dd since one is open already.
And there's heartbeat timeout error in the console logs:
Running on Linux 4.1.5-coreos x86_64
Loading data from directory /data
Listening for intracluster connections on port 29015
Connected to server "852c7df34d76_szm" ff02cdcb-eeff-436f-9736-381dc64163dd
Connected to server "9f214c8bca8d_e69" 0f9ba78a-5ed9-49b7-9d80-fe2c9571a15a
Connected to proxy 9056eeb9-aec4-41f5-bf10-7501506bfee9
Listening for client driver connections on port 28015
Listening for administrative HTTP connections on port 8080
Listening on addresses: 127.0.0.1, 172.17.0.17, ::1, fe80::42:acff:fe11:11%36
Server ready, "8fe0b3a66357_imu" f429ade8-5ffb-4ace-8491-14c07e62e5a6
Connected to proxy e43443f9-7626-40bc-9aa6-75d2fe2cbf45
Disconnected from server "9f214c8bca8d_e69" 0f9ba78a-5ed9-49b7-9d80-fe2c9571a15a
Connected to server "9f214c8bca8d_e69" 0f9ba78a-5ed9-49b7-9d80-fe2c9571a15a
error: Heartbeat timeout, killing connection to peer [::ffff:192.168.167.231]:45931
Disconnected from server "9f214c8bca8d_e69" 0f9ba78a-5ed9-49b7-9d80-fe2c9571a15a
Connected to server "9f214c8bca8d_e69" 0f9ba78a-5ed9-49b7-9d80-fe2c9571a15a
I'm using docker 1.7.1 in CoreOS v773.0.0 to run my rethinkdb cluster.
$ uname -a
Linux coreos-data-3 4.1.5-coreos #2 SMP Thu Aug 13 09:18:45 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz GenuineIntel GNU/Linux