Skip to content

Many Redis requests are timing out and throwing errors #3621

@nathanmarcos

Description

@nathanmarcos

Describe the bug
After fixing the latest Redis outage (#3509) on version 1.27.0, the application started throwing millions of mget error: Timeout Error: Request timed out.. It seems that this is caused when small changes in latency between Router and Redis requests last longer than the default timeout of 1ms.

To Reproduce
Steps to reproduce the behavior:

  1. Run Redis instance
  2. Run Router with APQ set up with Redis
  3. Submit request with persistedQuery hash in the extensions
  4. See errors on router's terminal:
2023-08-21T10:32:44.655911Z  ERROR [trace_id=84abf715700bdc7accbbe5bf7cac366a] mget error: Timeout Error: Request timed out.
2023-08-21T10:32:44.676982Z  ERROR [trace_id=c18fb3e62220025ab1858ee3295446d3] mget error: Timeout Error: Request timed out.
2023-08-21T10:32:44.678350Z  ERROR [trace_id=776705431a03a9d0adda5ac8eb32380c] mget error: Timeout Error: Request timed out.

Expected behavior
Errors should not happen.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions