-
Notifications
You must be signed in to change notification settings - Fork 643
Description
Describe the bug
We are using keyDB with opensips in a replicated cluster configuration. It works fine for a few days after which keydb-cli hangs.( no response or prompt after entering the command at linux CLI). Also remote requests fail to respond. Logging also stops working.
(the symptoms occur on both servers in the cluster).
To reproduce
Unknown. Seems to be load related. We have an identically configured but much less lightly loaded development instance which isnt impacted.
Expected behavior
We should be able to update keydb via keydb-cli or over network.
Additional information
- There are many FIFO Pipes open which I suspect is related to the issue.
[root@lxswat1002 c770190]# lsof |grep keydb | awk '{print $7}' |sort |uniq -c |sort -n |grep FIFO
122472 FIFO
- From logs before it ceased logging.
[root@lxswat1001 keydb]# grep "open files" /var/log/keydb/keydb-server.log-20230108 |grep "05 Jan"
2932708:2932723:S 05 Jan 2023 23:16:20.414 # Accepting client connection: accept: Too many open files
2932708:2932722:S 05 Jan 2023 23:16:20.414 # Accepting client connection: accept: Too many open files
2932708:2932723:S 05 Jan 2023 23:16:20.415 # Accepting client connection: accept: Too many open files
2932708:2932723:S 05 Jan 2023 23:16:20.415 # Accepting client connection: accept: Too many open files
config file attached.
log file attached
[keydb-conf.zip](https://github.com/Snapchat/KeyDB/files/10369785/keydb-conf.zip
keydb-server_log.zip