Skip to content

ScaledJob: transient "redis: client is closed" error during scaler cache rebuild sets Ready=False #7574

Description

@nikmar13

Report

When a ScaledJob spec changes, KEDA rebuilds the scalers cache via HandleScalableObject. The active scale loop goroutine fires immediately after the rebuild and races against the client replacement, causing a
transient redis: client is closed error. This surfaces as scaler with id 0 not found. Len = 0 in the scale handler, which then sets Ready=False on the ScaledJob condition.

The error is self-healing within seconds, but the Ready=False condition is misleading - Redis is fully stable and the ScaledJob spec is valid.

Expected Behavior

When KEDA rebuilds the scalers cache due to a spec change, transient connection errors during the rebuild should not surface as Ready=False. The condition should either remain at its previous state or transition to
a neutral state until the new scaler client is confirmed healthy.

This is consistent with KEDA's own log message: "Error getting scaler metrics and activity, but continue" - indicating KEDA itself treats this as non-fatal, but the condition update does not reflect that.

Actual Behavior

Within ~30ms of a ScaledJob reconcile triggered by a generation change:

  1. HandleScalableObject closes the old Redis client and builds a new one
  2. The scale loop goroutine fires immediately and attempts to use the old/mid-replacement client
  3. redis: client is closed is returned by the redis scaler
  4. scale_handler logs "scaler with id 0 not found. Len = 0" and continues
  5. GetScalersCache in reconcileScaledJob returns an error
  6. Reconcile() sets Ready=False with reason ScaledJobCheckFailed
  7. The ScaledJob condition recovers on the next reconcile

The race window is deterministic and reproducible on every spec change - not random. Redis is stable throughout - no restarts, no failovers.

Steps to Reproduce the Problem

  1. Deploy a ScaledJob with a Redis list scaler targeting a stable Redis instance
  2. Apply any change to the ScaledJob spec (e.g. update an env var in jobTargetRef, change pollingInterval) to bump the Generation
  3. Observe KEDA operator logs immediately after the reconcile starts
  4. Within ~30ms you will see:
    • ERROR redis_scaler error getting list length {"error": "redis: client is closed"}
    • ERROR scale_handler Error getting scaler metrics and activity, but continue {"error": "scaler with id 0 not found. Len = 0"}
  5. Observe the ScaledJob status conditions - Ready will briefly be False
  6. On the next reconcile it recovers to True

Logs from KEDA operator

2026-03-30T10:23:53Z INFO  scaleexecutor  Scaling Jobs {"scaledJob.Name": "example-sj", "scaledJob.Namespace": "production", "Number of pending Jobs": 0}

2026-03-30T10:23:54Z INFO  Reconciling ScaledJob {"ScaledJob": {"name":"example-sj","namespace":"production"}, "reconcileID": "14d88900-a507-4eaf-87bc-99d6c9c6dae6"}

2026-03-30T10:23:54Z INFO  RolloutStrategy: immediate, No jobs owned by the previous version of the scaledJob

2026-03-30T10:23:54Z INFO  Initializing Scaling logic according to ScaledJob Specification

2026-03-30T10:23:54Z ERROR redis_scaler  error getting list length {"type": "ScaledJob", "namespace": "production", "name": "example-sj", "error": "redis: client is closed"}

2026-03-30T10:23:54Z ERROR scale_handler  Error getting scaler metrics and activity, but continue {"scaledJob.Name": "example-sj", "Scaler": "*scalers.redisScaler:",  "error": "scaler with id 0 not found. Len = 0"}

2026-03-30T10:23:54Z INFO  scaleexecutor  Scaling Jobs  {"scaledJob.Name": "example-sj", "scaledJob.Namespace": "production", "Number of running Jobs": 0}

KEDA Version

2.19.0

Kubernetes Version

1.34

Platform

Amazon Web Services

Scaler Details

Redis scaler (list length trigger)

Would you be open to contributing a fix?

Yes

Anything else?

Root cause trace through the source code:

In reconcileScaledJob (scaledjob_controller.go), the sequence is:

  1. GetScalersCache() - succeeds with existing cache
  2. requestScaleLoop()HandleScalableObject() - closes old Redis client, builds new one, starts scale loop goroutine
  3. Scale loop goroutine fires immediately and calls GetMetrics() on the mid-replacement client → redis: client is closed
  4. Back in Reconcile(), the error propagates and sets Ready=False

The same issue does NOT affect ScaledObject because it uses an HPA as an intermediary. The HPA buffers the scaler lifecycle, so client rebuilds never directly race against live metric queries. ScaledJob has no such buffer -
it queries scalers directly on every reconcile.

Suggested fix directions:

  • In Reconcile(), distinguish between "scaler permanently broken" vs "scaler transiently unavailable during rebuild" before setting Ready=False
  • Ensure HandleScalableObject completes new client initialization fully before the goroutine is allowed to call GetMetrics
  • Treat scaler with id 0 not found as non-fatal consistent with the existing "but continue" log behavior - don't let it propagate to Ready=False

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Ready To Ship

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions