Report

We observed intermittent errors from the Datadog scaler in KEDA when querying metrics during short timeframes where no data points exist. This results in autoscaling failures.

Expected Behavior

No Errors should be observed. We debugged the issue and found the below

• Invalid metric name or query syntax: Metric name and syntax are correct.
• Wrong tags or filters: Tags and filters are valid.
• Querying a metric that doesn’t exist in your Datadog account: Metric exists and returns data in Datadog UI for the same timeframe.
• Time window issues (e.g., requesting data outside retention): We couldn't verify this

Actual Behavior

For a particular timeframe we are seeing the below pattern of error for our app

UnhandledError - apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc00299db40)}: rpc error: code = Unknown desc = error when getting metric values metric:datadogmetric@dev-app-master:profiles-parties-api-memory encountered error
Internal error - error getting metric value: error getting metric value: Internal error occurred: Datadog API call error (all queries), error: 422 Unprocessable Entity
_github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
/workspace/pkg/scaling/scale_handler.go:554
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
/workspace/pkg/metricsservice/server.go:48
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler.func1
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:118
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).Start.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func6
/workspace/vendor/github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/server.go:22
github.com/kedacore/keda/v2/pkg/metricsservice/api.MetricsService_GetMetrics_Handler
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:120
google.golang.org/grpc.(*Server).processUnaryRPC
/workspace/vendor/google.golang.org/grpc/server.go:1405
google.golang.org/grpc.(*Server).handleStream
/workspace/vendor/google.golang.org/grpc/server.go:1815
google.golang.org/grpc.(*Server).serveStreams.func2.1
/workspace/vendor/google.golang.org/grpc/server.go:1035

Internal error (error getting scale decision) - error getting metric value: error getting metric value: Internal error occurred: Datadog API call error (all queries), error: 422 Unprocessable Entity
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
/workspace/pkg/scaling/scale_handler.go:781
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
/workspace/pkg/scaling/scale_handler.go:634

Steps to Reproduce the Problem

1. This error occurs transiently but not consistent

Logs from KEDA operator

UnhandledError - apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc00299db40)}: rpc error: code = Unknown desc = error when getting metric values metric:datadogmetric@dev-app-master:profiles-parties-api-memory encountered error
Internal error - error getting metric value: error getting metric value: Internal error occurred: Datadog API call error (all queries), error: 422 Unprocessable Entity
_github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
/workspace/pkg/scaling/scale_handler.go:554
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
/workspace/pkg/metricsservice/server.go:48
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler.func1
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:118
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).Start.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func6
/workspace/vendor/github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/server.go:22
github.com/kedacore/keda/v2/pkg/metricsservice/api.MetricsService_GetMetrics_Handler
/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:120
google.golang.org/grpc.(*Server).processUnaryRPC
/workspace/vendor/google.golang.org/grpc/server.go:1405
google.golang.org/grpc.(*Server).handleStream
/workspace/vendor/google.golang.org/grpc/server.go:1815
google.golang.org/grpc.(*Server).serveStreams.func2.1
/workspace/vendor/google.golang.org/grpc/server.go:1035

Internal error (error getting scale decision) - error getting metric value: error getting metric value: Internal error occurred: Datadog API call error (all queries), error: 422 Unprocessable Entity
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
/workspace/pkg/scaling/scale_handler.go:781
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
/workspace/pkg/scaling/scale_handler.go:634

KEDA Version

2.17.1

Kubernetes Version

1.32

Platform

Amazon Web Services

Scaler Details

Datadog

Anything else?

No response