feat: expose Prometheus metrics for EC2 endpoint discovery by puertomontt · Pull Request #14318 · kgateway-dev/kgateway

puertomontt · 2026-06-26T15:10:36Z

Description

Implements the remaining open requirement (FR-10) from GG-1509: expose Prometheus metrics for the EC2 backend endpoint-discovery loop added in #13961. Operators previously had only structured logs and no quantitative signal for EC2 discovery health.

Four per-Backend metrics are recorded by the EC2 EDS polling loop, following the codebase's kgateway_<subsystem>_<name> convention:

kgateway_ec2_discovery_poll_total (counter) — total discovery refresh attempts per Backend, labeled namespace, name, result (success/error), and reason. The reason values match the EndpointsDiscovered status-condition reasons (none, NoMatchingInstances, CredentialError, AuthorizationError, DiscoveryError). A successful poll that matches zero instances is recorded as result=success, reason=NoMatchingInstances (not an error). A failed poll that still serves carried-forward endpoints (Degraded) records the underlying classification reason, not Degraded, so the counter always attributes a concrete cause.
kgateway_ec2_discovery_endpoints_active (gauge) — active endpoint count, set after each successful poll. Left unchanged on a failed poll so it retains the last successful value (NFR-3 graceful degradation). Labels: namespace, name.
kgateway_ec2_discovery_error_state (gauge) — 1 when the Backend's most recent discovery outcome was a failure, 0 on success. No reason label by design; reason-specific diagnosis is via the counter and the Backend status condition. Labels: namespace, name. This also covers the case where a secret-auth Backend's secret cannot be resolved (it never enters the poll loop but is set to 1 from the status path), so missing/typo'd secretRef misconfigurations are visible to alerting.
kgateway_ec2_discovery_poll_duration_seconds (native histogram) — wall-clock duration of the AWS DescribeInstances round trip per poll, attributed to every Backend in the credential scope. Both successful and failed polls are observed so slow timeouts are visible; configured as a native histogram (bucket factor 1.1) with classic buckets retained as a fallback, spanning a fast in-region listing up to the 30s refresh timeout. Labels: namespace, name.

Metrics are recorded per Backend even though discovery batches AWS DescribeInstances calls by credential scope. Series for deleted Backends are removed via a delete handler on the backends collection so stale series don't linger indefinitely. This required adding a DeletePartialMatch method to the metrics Counter/Gauge/Histogram interfaces (no prior per-series deletion capability existed). The delete handler is registered unconditionally (not via metrics.RegisterEvents) so cleanup stays symmetric with the per-poll recording, which is guarded dynamically by metrics.Active(); it cleans up by object-source identity, which is always populated and is a no-op for non-EC2 backends.

Change Type

/kind feature

Changelog

EC2 backends now expose Prometheus metrics for endpoint discovery: `kgateway_ec2_discovery_poll_total`, `kgateway_ec2_discovery_endpoints_active`, `kgateway_ec2_discovery_error_state`, and `kgateway_ec2_discovery_poll_duration_seconds`.

Additional Notes

New unit tests cover successful polls, no-matching-instances, error-with-carry-forward (degraded), poll-duration recording, unresolved-secret error state, and deletion cleanup. Metric names pass promlint.
Includes a small cleanup removing an unreachable missing-secret guard in ec2ConfigFromBackend (translation already fails closed for unresolved secrets).
No central metric registry/doc enumerates metrics, so no doc update was needed; devel/architecture/metrics.md documents only patterns.

Add per-Backend Prometheus metrics for the EC2 EDS polling loop (FR-10): - kgateway_ec2_discovery_poll_total (counter) partitioned by result and reason, matching the EndpointsDiscovered status condition reasons - kgateway_ec2_discovery_endpoints_active (gauge), set to the matched instance count after each successful poll and retained on failure (NFR-3 graceful degradation) - kgateway_ec2_discovery_error_state (gauge), 1 when the last poll failed Metrics are recorded per Backend even though discovery batches AWS DescribeInstances calls by credential scope. Series for deleted Backends are removed via a backends-collection delete handler so stale gauges don't linger; this required adding DeletePartialMatch to the metrics Counter/Gauge interfaces. Signed-off-by: omar <[email protected]>

Copilot

Pull request overview

This PR adds Prometheus instrumentation for the EC2 backend endpoint-discovery polling loop so operators can monitor discovery health and current endpoint counts quantitatively (in addition to existing logs/status).

Changes:

Introduces three per-Backend EC2 discovery metrics (poll counter, active endpoint gauge, error-state gauge) and records them on successful/failed polls.
Adds per-series cleanup on Backend deletion by extending the metrics Counter/Gauge interfaces with partial-label deletion.
Adds unit tests covering success, no-matching-instances, error-with-carry-forward, and metric cleanup behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pkg/metrics/metrics.go	Extends Counter/Gauge with `DeletePartialMatch` and implements partial-label deletion for Prometheus vec-backed metrics.
pkg/kgateway/extensions2/plugins/backend/metrics.go	Defines and records EC2 discovery metrics (poll total, active endpoints, error state) and adds deletion cleanup helpers.
pkg/kgateway/extensions2/plugins/backend/metrics_test.go	Adds unit tests validating metric recording and deletion cleanup behavior.
pkg/kgateway/extensions2/plugins/backend/ec2.go	Wires EC2 poll success/error metric recording into the discovery loop and registers a delete handler to remove per-Backend series.

+	gathered := metricstest.MustGatherMetrics(t)
+	// Only the surviving Backend's gauge series remain; the deleted Backend leaves
+	// no stale per-Backend gauge behind.
+	gathered.AssertMetricsLabels(endpointsActiveMetric, [][]metrics.Label{
+		backendIdentityLabels("default", "backend-b"),
+	})
+	gathered.AssertMetricsLabels(errorStateMetric, [][]metrics.Label{
+		backendIdentityLabels("default", "backend-b"),
+	})
+}


Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

krisztianfekete · 2026-06-29T10:44:15Z

+	reason := ec2PollReasonNone
+	if endpointCount == 0 {
+		reason = string(kgateway.BackendReasonNoMatchingInstances)
+	}


Is this correct?

currently yes that matches the status condition

Wouldn't this give none while the EndpointsDiscovered is Discovered (BackendReasonDiscovered)?

krisztianfekete · 2026-06-29T10:47:33Z

Does this cover metrics as well?

this log is unreachable, removed

My question was more around also reflecting these scenarios in metrics, as this situation can likely happen (?)

I see what you mean now, addressed.

krisztianfekete · 2026-06-29T10:49:08Z

+// EC2 endpoint discovery metrics. The exposed names are
+// kgateway_ec2_discovery_poll_total, kgateway_ec2_discovery_endpoints_active, and
+// kgateway_ec2_discovery_error_state, following the kgateway_<subsystem>_<name>
+// convention used elsewhere in the codebase.


Can we add a kgateway_ec2_discovery_poll_duration_seconds Histogram as well?

Add kgateway_ec2_discovery_poll_duration_seconds, a per-Backend histogram of the AWS DescribeInstances round-trip time for each discovery poll (labels namespace, name). Both successful and failed polls are observed so slow timeouts are visible. It is configured as a native histogram (bucket factor 1.1), matching the other duration histograms in the codebase, with classic buckets retained as a fallback for scrapers that lack native support, spanning a fast in-region listing up to the 30s refresh timeout. The duration is attributed to every Backend sharing a credential scope, consistent with poll_total being recorded per Backend. Adds DeletePartialMatch to the metrics Histogram interface so the new series is cleaned up alongside the gauges/counter on Backend deletion. Also simplifies the deletion handler to clean up by object-source identity unconditionally: the identity is always populated, every Backend is unique by namespace/name, and deleteEc2DiscoveryMetrics is a no-op for non-EC2 backends, so this is safe and avoids depending on the deleted Obj's type. Signed-off-by: omar <[email protected]>

An EC2 backend whose secret-auth credential cannot be resolved never builds an ec2Ir: buildTranslateFunc's loadAWSSecret returns an error (the secret lookup fails closed), so the EC2 branch breaks before setting awsIr. By the time ec2ConfigFromBackend dereferences ec2Ir, a non-nil ec2Ir therefore implies the secret resolved, making the ec2Ir.secret == nil guard (and its debug log) dead code. Unresolved-secret backends are already surfaced as CredentialError via discoveryStatusForBackend/ec2UnresolvedSecretCredential. Replace the guard with a comment explaining the invariant. Signed-off-by: omar <[email protected]>

A secret-auth EC2 backend whose secret cannot be resolved never enters the discovery poll loop (translation fails closed), so it previously recorded no metrics at all even though it reports a CredentialError status condition. This left the most common misconfiguration (a missing or typo'd secretRef) invisible to error_state alerting. Set error_state=1 (and endpoints_active=0) for these backends from discoveryStatusForBackend, where the CredentialError condition is already computed. The gauge Set is idempotent under KRT recompute and is overwritten once the secret resolves and the poll loop takes over; the delete handler clears it on backend removal. poll_total and poll_duration_seconds stay poll-scoped, since no poll actually occurs. Signed-off-by: omar <[email protected]>

krisztianfekete · 2026-07-03T10:41:38Z

+			NativeHistogramBucketFactor:     1.1,
+			NativeHistogramMaxBucketNumber:  100,
+			NativeHistogramMinResetDuration: time.Hour,


Nice, thanks!

krisztianfekete · 2026-07-03T10:43:12Z

+	reason := ec2PollReasonNone
+	if endpointCount == 0 {
+		reason = string(kgateway.BackendReasonNoMatchingInstances)
+	}


Wouldn't this give none while the EndpointsDiscovered is Discovered (BackendReasonDiscovered)?

krisztianfekete · 2026-07-03T10:45:14Z

+			NativeHistogramMaxBucketNumber:  100,
+			NativeHistogramMinResetDuration: time.Hour,
+		},
+		[]string{ec2MetricNamespaceLabel, ec2MetricNameLabel},


What do you think about adding result as a label? Cardinality should be low, while users would have the option to filter out fast rejections.

Add a result (success/error) label to kgateway_ec2_discovery_poll_duration_seconds so operators can isolate successful-poll latency from failures, whose latency is bimodal (fast credential/authorization rejections vs. slow timeouts). Only result is added, not reason: reason belongs on the counter, and native histograms allocate per label-set. Signed-off-by: omar <[email protected]>

recordEc2PollSuccess recorded reason=none when endpoints resolved, but the EndpointsDiscovered status condition uses reason=Discovered for the same outcome. This contradicted the poll_total contract that reason values match the status condition. Use BackendReasonDiscovered so poll_total{reason="Discovered"} joins cleanly to the condition, and drop the now-unused ec2PollReasonNone constant. Signed-off-by: omar <[email protected]>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

+// EC2 endpoint discovery metrics. The exposed names are
+// kgateway_ec2_discovery_poll_total, kgateway_ec2_discovery_endpoints_active, and
+// kgateway_ec2_discovery_error_state, following the kgateway_<subsystem>_<name>
+// convention used elsewhere in the codebase.


Copilot AI review requested due to automatic review settings June 26, 2026 15:10

Copilot started reviewing on behalf of puertomontt June 26, 2026 15:11 View session

gateway-bot added do-not-merge/description-invalid kind/feature Categorizes issue or PR as related to a new feature. release-note labels Jun 26, 2026

Copilot AI reviewed Jun 26, 2026

View reviewed changes

gateway-bot removed the do-not-merge/description-invalid label Jun 26, 2026

puertomontt requested a review from Copilot June 28, 2026 18:13

Copilot started reviewing on behalf of puertomontt June 28, 2026 18:14 View session

Copilot AI reviewed Jun 28, 2026

View reviewed changes

Comment thread pkg/kgateway/extensions2/plugins/backend/metrics.go

Comment thread pkg/kgateway/extensions2/plugins/backend/ec2.go

krisztianfekete reviewed Jun 29, 2026

View reviewed changes

gateway-bot added the do-not-merge/description-invalid label Jun 30, 2026

puertomontt force-pushed the ec2-metrics branch from 8345582 to 84eca8a Compare June 30, 2026 12:10

gateway-bot removed the do-not-merge/description-invalid label Jun 30, 2026

gateway-bot added do-not-merge/description-invalid and removed do-not-merge/description-invalid labels Jun 30, 2026

krisztianfekete reviewed Jul 3, 2026

View reviewed changes

puertomontt added 2 commits July 6, 2026 09:40

puertomontt requested a review from Copilot July 13, 2026 16:31

Copilot started reviewing on behalf of puertomontt July 13, 2026 16:35 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

davidjumani approved these changes Jul 13, 2026

View reviewed changes

davidjumani added this pull request to the merge queue Jul 13, 2026

Merged via the queue into kgateway-dev:main with commit 9fe0632 Jul 13, 2026
32 checks passed

puertomontt deleted the ec2-metrics branch July 15, 2026 11:08

Uh oh!

Conversation

puertomontt commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Change Type

Changelog

Additional Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

puertomontt commented Jun 26, 2026 •

edited

Loading