Report
version: v2.15.1
In some case, It can not connect success until restart keda-operator-metrics-apiserver.
|
func (c *GrpcClient) WaitForConnectionReady(ctx context.Context, logger logr.Logger) bool { |
E1118 11:13:07.474617 1 timeout.go:142] post-timeout activity - time-elapsed: 64.730857ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/devops-test-smalpha6gdsz6/s0-prometheus" result: <nil>
E1118 11:13:16.066779 1 provider.go:89] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E1118 11:13:16.066804 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server
E1118 11:13:16.066894 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E1118 11:13:16.067970 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1118 11:13:16.067986 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I1118 11:13:16.069077 1 trace.go:236] Trace[1074940346]: "List" accept:application/vnd.kubernetes.protobuf, */*,audit-id:2bd2fa6f-e720-4de6-a7af-543820c5b510,client:7.10.9.23,api-group:external.metrics.k8s.io,api-version:v1beta1,name:,subresource:,namespace:infra-al-cn-gdsz-6,protocol:HTTP/2.0,resource:s1-prometheus,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/infra-al-cn-gdsz-6/s1-prometheus,user-agent:kube-controller-manager/v1.30.1 (linux/amd64) kubernetes/2475f20/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (18-Nov-2025 11:12:16.016) (total time: 60052ms):
Trace[1074940346]: [1m0.052730689s] [1m0.052730689s] END
E1118 11:13:16.069147 1 timeout.go:142] post-timeout activity - time-elapsed: 56.53445ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/infra-al-cn-gdsz-6/s1-prometheus" result: <nil>
I1118 11:13:16.110988 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
I1118 11:13:23.915452 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
E1118 11:13:23.945562 1 provider.go:89] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E1118 11:13:23.945593 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server
E1118 11:13:23.945670 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E1118 11:13:23.946739 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1118 11:13:23.946751 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I1118 11:13:23.947837 1 trace.go:236] Trace[1184002668]: "List" accept:application/vnd.kubernetes.protobuf, */*,audit-id:df5ef0f1-a1db-44ac-b53c-1d15d64916b4,client:7.10.9.23,api-group:external.metrics.k8s.io,api-version:v1beta1,name:,subresource:,namespace:artifact-smprod,protocol:HTTP/2.0,resource:s0-prometheus,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/artifact-smprod/s0-prometheus,user-agent:kube-controller-manager/v1.30.1 (linux/amd64) kubernetes/2475f20/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (18-Nov-2025 11:12:23.897) (total time: 60050ms):
Trace[1184002668]: [1m0.05073097s] [1m0.05073097s] END
Expected Behavior
connect success.
Actual Behavior
throw exception "timeout while waiting to establish gRPC connection to KEDA Metrics Service server". It be ok when restart keda-operator-metrics-apiserver.
Steps to Reproduce the Problem
I can not reproduce the problem.
Logs from KEDA operator
keda-operator-metrics-apiserver logs:
E1118 11:13:07.474617 1 timeout.go:142] post-timeout activity - time-elapsed: 64.730857ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/devops-test-smalpha6gdsz6/s0-prometheus" result:
E1118 11:13:16.066779 1 provider.go:89] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E1118 11:13:16.066804 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server
E1118 11:13:16.066894 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E1118 11:13:16.067970 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1118 11:13:16.067986 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I1118 11:13:16.069077 1 trace.go:236] Trace[1074940346]: "List" accept:application/vnd.kubernetes.protobuf, /,audit-id:2bd2fa6f-e720-4de6-a7af-543820c5b510,client:7.10.9.23,api-group:external.metrics.k8s.io,api-version:v1beta1,name:,subresource:,namespace:infra-al-cn-gdsz-6,protocol:HTTP/2.0,resource:s1-prometheus,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/infra-al-cn-gdsz-6/s1-prometheus,user-agent:kube-controller-manager/v1.30.1 (linux/amd64) kubernetes/2475f20/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (18-Nov-2025 11:12:16.016) (total time: 60052ms):
Trace[1074940346]: [1m0.052730689s] [1m0.052730689s] END
E1118 11:13:16.069147 1 timeout.go:142] post-timeout activity - time-elapsed: 56.53445ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/infra-al-cn-gdsz-6/s1-prometheus" result:
I1118 11:13:16.110988 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
I1118 11:13:23.915452 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
E1118 11:13:23.945562 1 provider.go:89] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E1118 11:13:23.945593 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server
E1118 11:13:23.945670 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E1118 11:13:23.946739 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1118 11:13:23.946751 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I1118 11:13:23.947837 1 trace.go:236] Trace[1184002668]: "List" accept:application/vnd.kubernetes.protobuf, /,audit-id:df5ef0f1-a1db-44ac-b53c-1d15d64916b4,client:7.10.9.23,api-group:external.metrics.k8s.io,api-version:v1beta1,name:,subresource:,namespace:artifact-smprod,protocol:HTTP/2.0,resource:s0-prometheus,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/artifact-smprod/s0-prometheus,user-agent:kube-controller-manager/v1.30.1 (linux/amd64) kubernetes/2475f20/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (18-Nov-2025 11:12:23.897) (total time: 60050ms):
Trace[1184002668]: [1m0.05073097s] [1m0.05073097s] END
KEDA Version
2.15.1
Kubernetes Version
None
Platform
Alibaba Cloud
Scaler Details
prometheus
Anything else?
No response
Report
version: v2.15.1
In some case, It can not connect success until restart keda-operator-metrics-apiserver.
keda/pkg/metricsservice/client.go
Line 218 in 55875b1
Expected Behavior
connect success.
Actual Behavior
throw exception "timeout while waiting to establish gRPC connection to KEDA Metrics Service server". It be ok when restart keda-operator-metrics-apiserver.
Steps to Reproduce the Problem
I can not reproduce the problem.
Logs from KEDA operator
keda-operator-metrics-apiserver logs:
E1118 11:13:07.474617 1 timeout.go:142] post-timeout activity - time-elapsed: 64.730857ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/devops-test-smalpha6gdsz6/s0-prometheus" result:
E1118 11:13:16.066779 1 provider.go:89] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E1118 11:13:16.066804 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server
E1118 11:13:16.066894 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E1118 11:13:16.067970 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1118 11:13:16.067986 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I1118 11:13:16.069077 1 trace.go:236] Trace[1074940346]: "List" accept:application/vnd.kubernetes.protobuf, /,audit-id:2bd2fa6f-e720-4de6-a7af-543820c5b510,client:7.10.9.23,api-group:external.metrics.k8s.io,api-version:v1beta1,name:,subresource:,namespace:infra-al-cn-gdsz-6,protocol:HTTP/2.0,resource:s1-prometheus,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/infra-al-cn-gdsz-6/s1-prometheus,user-agent:kube-controller-manager/v1.30.1 (linux/amd64) kubernetes/2475f20/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (18-Nov-2025 11:12:16.016) (total time: 60052ms):
Trace[1074940346]: [1m0.052730689s] [1m0.052730689s] END
E1118 11:13:16.069147 1 timeout.go:142] post-timeout activity - time-elapsed: 56.53445ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/infra-al-cn-gdsz-6/s1-prometheus" result:
I1118 11:13:16.110988 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
I1118 11:13:23.915452 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
E1118 11:13:23.945562 1 provider.go:89] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E1118 11:13:23.945593 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server
E1118 11:13:23.945670 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E1118 11:13:23.946739 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1118 11:13:23.946751 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I1118 11:13:23.947837 1 trace.go:236] Trace[1184002668]: "List" accept:application/vnd.kubernetes.protobuf, /,audit-id:df5ef0f1-a1db-44ac-b53c-1d15d64916b4,client:7.10.9.23,api-group:external.metrics.k8s.io,api-version:v1beta1,name:,subresource:,namespace:artifact-smprod,protocol:HTTP/2.0,resource:s0-prometheus,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/artifact-smprod/s0-prometheus,user-agent:kube-controller-manager/v1.30.1 (linux/amd64) kubernetes/2475f20/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (18-Nov-2025 11:12:23.897) (total time: 60050ms):
Trace[1184002668]: [1m0.05073097s] [1m0.05073097s] END
KEDA Version
2.15.1
Kubernetes Version
None
Platform
Alibaba Cloud
Scaler Details
prometheus
Anything else?
No response