Skip to content

Prometheus metric value does not handle NaN result #7475

Description

@tcp13equals2

Report

See https://github.com/kedacore/keda/blob/main/pkg/scalers/prometheus_scaler.go#L285C2-L292C3

An Inf result from the prometheus query is explicitly handled. A NaN result from prometheus is not handled.

The impact is that a NaN is returned from the prometheus metrics query and the resulting keda scaling metrics value is evaluated to a large negative number (depending on the architecture).

On an X86_64 it will result in a keda metrics value of -9223372036854776

Expected Behavior

It would seem logical to also test if the prometheus result is a NaN - not just if it is an Inf number.

The same handling of the NaN as the Inf should apply. If IgnoreNullValues=true then return 0, else return an error.

This test case illustrates the expected behavior. It can be added to the prometheus_scaler_overflow_test.go file.

func TestPrometheusScalerOverflowIssue(t *testing.T) {
	tests := []struct {
		name                  string
		prometheusValue       string
		expectError           bool
		ignoreNullValues      bool
		expectedRecordedValue float64
		description           string
	}{
		{
			name:                  "Negative infinity with ignoreNullValues=true",
			prometheusValue:       "-Inf",
			ignoreNullValues:      true,
			expectedRecordedValue: 0.0,
			description:           "Infinity is caught and 0 returned",
		},
		{
			name:                  "Infinity with ignoreNullValues=true",
			prometheusValue:       "Inf",
			ignoreNullValues:      true,
			expectedRecordedValue: 0.0,
			description:           "Infinity is caught and 0 returned",
		},
		{
			name:                  "NaN with ignoreNullValues=true",
			prometheusValue:       "NaN",
			ignoreNullValues:      true,
			expectedRecordedValue: 0.0,
			description:           "NaN is caught and 0 returned",
		},
		{
			name:             "Negative infinity with ignoreNullValues=false",
			prometheusValue:  "-Inf",
			expectError:      true,
			ignoreNullValues: false,
			description:      "Infinity is caught and error returned",
		},
		{
			name:             "Infinity with ignoreNullValues=false",
			prometheusValue:  "Inf",
			expectError:      true,
			ignoreNullValues: false,
			description:      "Infinity is caught and error returned",
		},
		{
			name:             "NaN with ignoreNullValues=false",
			prometheusValue:  "NaN",
			expectError:      true,
			ignoreNullValues: false,
			description:      "NaN is caught and error returned",
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			// Create a mock Prometheus server
			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
				response := map[string]interface{}{
					"status": "success",
					"data": map[string]interface{}{
						"resultType": "vector",
						"result": []map[string]interface{}{
							{
								"metric": map[string]string{},
								"value":  []interface{}{1234567890, tt.prometheusValue},
							},
						},
					},
				}
				json.NewEncoder(w).Encode(response)
			}))
			defer server.Close()

			// Create scaler config
			config := &scalersconfig.ScalerConfig{
				TriggerMetadata: map[string]string{
					"serverAddress":    server.URL,
					"query":            "test_metric",
					"threshold":        "100",
					"ignoreNullValues": fmt.Sprintf("%v", tt.ignoreNullValues),
				},
				TriggerIndex: 0,
			}

			// Create the scaler
			scaler, err := NewPrometheusScaler(config)
			require.NoError(t, err)
			defer scaler.Close(context.Background())

			promScaler := scaler.(*prometheusScaler)

			// Execute the query
			val, err := promScaler.ExecutePromQuery(context.Background())
			if tt.expectError {
				assert.Error(t, err)
				return
			}
			require.NoError(t, err)

			// Generate the metric using GenerateMetricInMili
			metric := GenerateMetricInMili("test_metric", val)

			// Convert back to float64 as it would be recorded in Prometheus metrics
			recordedValue := metric.Value.AsApproximateFloat64()

			assert.Equal(t, tt.expectedRecordedValue, recordedValue)
		})
	}
}

Actual Behavior

Currently a NaN can be returned from the ExecutePromQuery() func.

Steps to Reproduce the Problem

  1. See above unit test example.
  2. Setup a PromQL scaling trigger which returns a NaN result.
  3. Inspect the resulting keda_scaler_metrics_value and it will be a large negative number (assuming KEDA is running on a X86_64 node)

Logs from KEDA operator

example

KEDA Version

2.15.1

Kubernetes Version

1.33

Platform

Amazon Web Services

Scaler Details

No response

Would you be open to contributing a fix?

Yes

Anything else?

No response

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Ready To Ship

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions