Report

Keda controller is constantly crashing after I deploy a new version of the service targeted by the scaled object.

It tends to work for a while but after deploying the service, no metrics can be queried. The Keda controller logs all spit a bunch or errors, but all of them are related to the GetMetricsfunction.

Expected Behavior

No crashes

Actual Behavior

Constant crashes in the keda controller

Steps to Reproduce the Problem

• Deploy scaled object. I can query the metrics.
• Redeploy the service that is the controlled by the scaled object.
• KEDA starts crashing. There are multiple bugs according to the logs, but all of them revolve around the GetMetrics function. Examples are "assignment to nil map", "out of index", "concurrent write to map". All of them are bugs in KEDA code so I don't think it's an issue with the scaled object config.

Logs from KEDA operator

2023-03-17T21:28:05Z    ERROR   scalehandler    Failed to patch ScaledObjects Status    {"error": "resourc>
github.com/kedacore/keda/v2/pkg/fallback.updateStatus
        /workspace/pkg/fallback/fallback.go:126
github.com/kedacore/keda/v2/pkg/fallback.GetMetricsWithFallback
        /workspace/pkg/fallback/fallback.go:58
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
        /workspace/pkg/scaling/scale_handler.go:446
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
        /workspace/pkg/metricsservice/server.go:45
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
        /workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
google.golang.org/grpc.(*Server).processUnaryRPC
        /workspace/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
        /workspace/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /workspace/vendor/google.golang.org/grpc/server.go:965
panic: runtime error: index out of range [29] with length 29

KEDA Version

2.9.2

Kubernetes Version

1.23

Platform

Amazon Web Services

Scaler Details

Datadog

Anything else?

The only thing weird about this scaler is that it has around 40 triggers. We are using this service to have a single interface to query the metrics provided by KEDA. I set the min/max replicas to 2. I even disabled autoscaling with 2 replicas, but that didn't help. But I don't think the scaledobject config is the issue because we can query the metrics for a little while.

Destroying keda and redeploying seemed to work for a while but it always breaks down around the time the service is deployed.