[Bug]: Google Cloud Monitoring drops workqueue exponential histograms with INVALID_ARGUMENT (num_finite_buckets = 0) #1472

What happened?

When running Substrate on Google Kubernetes Engine (GKE) with the Managed OpenTelemetry addon enabled, ate-controller exports workqueue metrics through controller-runtime via prombridge.NewMetricProducer.

controller-runtime enables Prometheus native exponential histograms by default on workqueue_work_duration_seconds and workqueue_queue_duration_seconds. When a controller queue is idle or has not recorded any finite duration samples (such as egressmitmtrust), the OTLP exporter serializes an exponential histogram with count: 0 and empty bucket arrays.

Every 60-second metric push tick, the GKE Managed OTel Collector (otelcol-google exporter) attempts to export these points to Google Cloud Monitoring. The Cloud Monitoring backend rejects the metric points with an INVALID_ARGUMENT gRPC error:

Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.

The collector drops the data points and logs a Partial success response warning every minute.

Expected Behavior

ate-controller metrics should be successfully ingested by Google Cloud Monitoring without data points being dropped or producing error spam in the collector logs.

Steps to Reproduce

1. Deploy Agent Substrate on a GKE cluster with the Managed OpenTelemetry addon enabled:

   ./hack/install-ate.sh --deploy-ate-system

2. Check that ate-controller is running and exporting OTLP telemetry to opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317.
3. Inspect the logs of the Managed OTel collector:

   kubectl logs -n gke-managed-otel -l app.kubernetes.io/name=opentelemetry-collector --tail=100 | grep "num_finite_buckets"

4. Observe the INVALID_ARGUMENT error and dropped data points every 60 seconds.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

main

Kubernetes Version & Environment

GKE 1.36.3

Host OS & Architecture

Linux x86_64

Relevant Logs and Diagnostic Output

{
  "level": "warn",
  "ts": "2026-09-04T14:05:16.912Z",
  "caller": "otlpexporter/otlp.go:125",
  "msg": "Partial success response",
  "resource": {
    "service.instance.id": "8ed03c03-6bad-47d7-b51f-19a633ddc8a5",
    "service.name": "otelcol-google",
    "service.version": "0.153.0-gke.2"
  },
  "otelcol.component.id": "otlp_grpc",
  "otelcol.component.kind": "exporter",
  "otelcol.signal": "metrics",
  "message": "{\"errors\":[{\"code\":\"INVALID_ARGUMENT\",\"error_message\":\"Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.\",\"num_data_points\":2,\"example_resource_attributes\":{\"k8s.namespace.name\":\"ate-system\",\"k8s.pod.name\":\"ate-controller-5888878bbb-pr985\",\"k8s.pod.uid\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.instance.id\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.name\":\"atecontroller\",\"telemetry.sdk.language\":\"go\",\"telemetry.sdk.name\":\"opentelemetry\",\"telemetry.sdk.version\":\"1.44.0\",\"k8s.pod.start_time\":\"2026-09-03T22:07:01Z\",\"k8s.replicaset.name\":\"ate-controller-5888878bbb\",\"k8s.deployment.name\":\"ate-controller\",\"k8s.node.name\":\"gke-otel-standard-cluste-default-pool-33a08656-6pa6\",\"gcp.project_id\":\"jeffluoo-gke-dev\",\"cloud.provider\":\"gcp\",\"cloud.platform\":\"gcp_kubernetes_engine\",\"cloud.account.id\":\"jeffluoo-gke-dev\",\"k8s.cluster.name\":\"otel-standard-cluster-2\",\"cloud.availability_zone\":\"us-central1-a\"},\"example_metric_name\":\"workqueue_work_duration_seconds\",\"example_metric_attributes\":{\"controller\":\"egressmitmtrust\",\"name\":\"egressmitmtrust\",\"top_level_controller_type\":\"Deployment\",\"top_level_controller_name\":\"ate-controller\"}}]}",
  "dropped_data_points": 2
}

Additional Context

No response

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.