Requests begin failing with IllegalStateExceptions #1645
Description
Bug Report
We have now had a few instances in the last few months where our service using Lettuce has started throwing exceptions on every request to Redis. The stack trace that occurs can be found below. We are using an AWS Elasticache Redis cluster, and this seems to happen when our instance starts encountering intermittent network issues as we see HTTP requests to other services failing around the same time. The HTTP request errors stop, but the Lettuce client never recovers until the service is restarted. In case it's relevant, the Lettuce requests that are failing are all asynchronous EVAL requests. Please let me know if you need any more information.
Current Behavior
Lettuce client gets in a bad state where all requests start to fail with an IllegalStateException.
Stack trace
java.lang.IllegalStateException: null
\tat io.lettuce.core.output.CommandOutput.set(CommandOutput.java:87)
\tat io.lettuce.core.protocol.RedisStateMachine.safeSet(RedisStateMachine.java:441)
\tat io.lettuce.core.protocol.RedisStateMachine.handleInteger(RedisStateMachine.java:215)
\tat io.lettuce.core.protocol.RedisStateMachine.access$300(RedisStateMachine.java:41)
\tat io.lettuce.core.protocol.RedisStateMachine$State$Type.lambda$static$2(RedisStateMachine.java:63)
\tat io.lettuce.core.protocol.RedisStateMachine$State$Type.handle(RedisStateMachine.java:76)
\tat io.lettuce.core.protocol.RedisStateMachine.doDecode(RedisStateMachine.java:166)
\tat io.lettuce.core.protocol.RedisStateMachine.decode(RedisStateMachine.java:142)
\tat io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:742)
\tat io.lettuce.core.protocol.CommandHandler.decode0(CommandHandler.java:706)
\tat io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:701)
\tat io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:622)
\tat io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:591)
\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533)
\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282)
\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329)
\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)
\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)
\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
\tat java.base/java.lang.Thread.run(Thread.java:834)Environment
- Lettuce version(s): 5.3.5.RELEASE
- Redis version: 6.0.5 (Also occurred on Redis 5.0.0)