Skip to content

[BUG] keydb server becomes unresponsive #537

@brendanone

Description

@brendanone

Describe the bug
We are using keyDB with opensips in a replicated cluster configuration. It works fine for a few days after which keydb-cli hangs.( no response or prompt after entering the command at linux CLI). Also remote requests fail to respond. Logging also stops working.
(the symptoms occur on both servers in the cluster).

To reproduce
Unknown. Seems to be load related. We have an identically configured but much less lightly loaded development instance which isnt impacted.

Expected behavior
We should be able to update keydb via keydb-cli or over network.

Additional information

  • There are many FIFO Pipes open which I suspect is related to the issue.

[root@lxswat1002 c770190]# lsof |grep keydb | awk '{print $7}' |sort |uniq -c |sort -n |grep FIFO
122472 FIFO

  • From logs before it ceased logging.
    [root@lxswat1001 keydb]# grep "open files" /var/log/keydb/keydb-server.log-20230108 |grep "05 Jan"
    2932708:2932723:S 05 Jan 2023 23:16:20.414 # Accepting client connection: accept: Too many open files
    2932708:2932722:S 05 Jan 2023 23:16:20.414 # Accepting client connection: accept: Too many open files
    2932708:2932723:S 05 Jan 2023 23:16:20.415 # Accepting client connection: accept: Too many open files
    2932708:2932723:S 05 Jan 2023 23:16:20.415 # Accepting client connection: accept: Too many open files

config file attached.
log file attached

[keydb-conf.zip](https://github.com/Snapchat/KeyDB/files/10369785/keydb-conf.zip
keydb-server_log.zip

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions