Skip to content

Fix duplicate client_eof caused by connection timeout#88

Merged
wooparadog merged 1 commit intoeleme:masterfrom
doyoubi:fix/DoubleClienteof
Aug 29, 2016
Merged

Fix duplicate client_eof caused by connection timeout#88
wooparadog merged 1 commit intoeleme:masterfrom
doyoubi:fix/DoubleClienteof

Conversation

@doyoubi
Copy link
Copy Markdown
Contributor

@doyoubi doyoubi commented Aug 29, 2016

We have found the following logs in one of our running corvus:

2016-08-29 02:12:45,045 WARN [3348]: client '127.0.0.1:47576' timed out (timer.c:53)
2016-08-29 02:12:45,045 DEBUG [3348]: client eof (client.c:344)
2016-08-29 02:12:45,045 ERROR [3348]: event_deregister: 15 No such file or directory (event.c:88)
2016-08-29 02:12:45,045 ERROR [3348]: event_deregister: 16 No such file or directory (event.c:88)
2016-08-29 02:12:45,548 WARN [3348]: client '127.0.0.1:37176' timed out (timer.c:53)
2016-08-29 02:12:45,548 DEBUG [3348]: client eof (client.c:344)
2016-08-29 02:12:45,548 ERROR [3348]: event_deregister: 15 No such file or directory (event.c:88)
2016-08-29 02:12:45,548 ERROR [3348]: event_deregister: 16 No such file or directory (event.c:88)
2016-08-29 02:12:46,046 WARN [3348]: client '127.0.0.1:39411' timed out (timer.c:53)
2016-08-29 02:12:46,046 DEBUG [3348]: client eof (client.c:344)
2016-08-29 02:12:46,046 ERROR [3348]: event_deregister: 15 No such file or directory (event.c:88)
2016-08-29 02:12:46,046 ERROR [3348]: event_deregister: 16 No such file or directory (event.c:88)

The client eof messages are all followed by two event_deregister in the log.

How does it happen?

It's caused by duplicate call of client_eof issued by connection timeout check.
When a client connection is holding some unfinished cmds and encounters a client_eof (caused by receiving an illegal redis packet for example), the client connection is turned into an intermediate state. It set all its cmds to stale and wait for all its cmds to be closed and then finally close itself. However before its cmds are closed, the timeout check may find this client connection and call client_eof again. This will result in negative connected_clients(#85) and the error log above.

@wooparadog
Copy link
Copy Markdown
Contributor

wooparadog commented Aug 29, 2016

LGTM

@wooparadog wooparadog merged commit 367dae1 into eleme:master Aug 29, 2016
@doyoubi doyoubi mentioned this pull request Oct 31, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@doyoubi @wooparadog