Report
Hey there 👋 as the title suggests I'm seeing behaviour where the Jetstream scaler goes to maxReplicaCount when a consumer doesn't exist. I suspect this behaviour is a red-herring/symptom, and that the actual problem exists in the scalers getMaxMsgLag() function.
This line:
|
return s.stream.State.LastSequence |
If the consumer doesn't exist, getMaxMsgLag() returns the LastSequence. Which might describe the behaviour I'm seeing.
Expected Behavior
When the consumer name configured in the nats-jetstream trigger does not exist in the specified stream, the scaler should either:
- Return an error (so fallbacks can kick in if they are defined)
- or; treat the result as 0 - the consumer doesn't exist if there wasn't a failure in listing the consumers - 0 seems correct.
Actual Behavior
The scaler reports some large number which causes the computed lag to exceed any given threshold - which then scales the workload to max replicas.
Steps to Reproduce the Problem
- Create a NATS JetStream stream and have some messages sent through.
- Do not create the consumer named in the ScaledObject.
- Deploy a ScaledObject using the nats-jetstream scaler with a reasonable lagThreshold (1 for example). 3.
- Observe the deployment scaling up (even with no messages or activity).
Logs from KEDA operator
I haven't included any logs as the issue seems to be in the scaler code from a quick look. Happy to do so should more info be required 👍
KEDA Version
2.19.0
Kubernetes Version
Other
Platform
Amazon Web Services
Scaler Details
NATS Jetstream
Would you be open to contributing a fix?
Yes
Anything else?
I've included my suspicions for the root cause in the Report section - happy to attempt a fix, but would like guidance on the expected behaviours.
Report
Hey there 👋 as the title suggests I'm seeing behaviour where the Jetstream scaler goes to maxReplicaCount when a consumer doesn't exist. I suspect this behaviour is a red-herring/symptom, and that the actual problem exists in the scalers
getMaxMsgLag()function.This line:
keda/pkg/scalers/nats_jetstream_scaler.go
Line 430 in f282ce2
If the consumer doesn't exist,
getMaxMsgLag()returns theLastSequence. Which might describe the behaviour I'm seeing.Expected Behavior
When the consumer name configured in the
nats-jetstreamtrigger does not exist in the specified stream, the scaler should either:Actual Behavior
The scaler reports some large number which causes the computed lag to exceed any given threshold - which then scales the workload to max replicas.
Steps to Reproduce the Problem
Logs from KEDA operator
I haven't included any logs as the issue seems to be in the scaler code from a quick look. Happy to do so should more info be required 👍
KEDA Version
2.19.0
Kubernetes Version
Other
Platform
Amazon Web Services
Scaler Details
NATS Jetstream
Would you be open to contributing a fix?
Yes
Anything else?
I've included my suspicions for the root cause in the Report section - happy to attempt a fix, but would like guidance on the expected behaviours.