feat: reflect EC2 backend discovery failures in Backend status conditions#14173
Merged
Conversation
Add an EndpointsDiscovered status condition to the Backend resource so
operators get a Kubernetes-native signal for EC2 runtime discovery
outcomes (gloo-gateway#2283, FR-8/NFR-2).
The EC2 EDS controller now records a per-backend discovery status after
every refresh cycle and exposes it via a DiscoveryStatus KRT collection:
- success with >=1 endpoint -> True/Discovered ("N endpoints active")
- success with 0 matches -> False/NoMatchingInstances (describes filters)
- unresolved/malformed creds -> False/CredentialError
- AWS auth/authz rejection -> False/AuthorizationError
- transient/other AWS error -> False/DiscoveryError
Failures preserve carried-forward endpoints (NFR-3); the condition is
recomputed on every poll because the discovery status participates in
state equality. Backends with an unresolved secretRef are surfaced
synchronously even though they are filtered out before discovery runs.
Conditions are merged into the single backendStatusReport writer
alongside Accepted, and a managed condition absent from a fresh report
is now dropped (instead of preserved) so a backend that stops doing
discovery does not retain stale state.
Fixes solo-io/gloo-gateway#2283
Signed-off-by: omar <[email protected]>
gateway-bot
added
do-not-merge/description-invalid
kind/feature
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a Kubernetes-native signal for runtime AWS EC2 backend discovery by introducing an EndpointsDiscovered Backend status condition, plumbing plugin-contributed conditions into the core backend status writer, and ensuring stale kgateway-managed Backend conditions are dropped when no longer produced.
Changes:
- Introduces
BackendConditionEndpointsDiscoveredplus standardized reason enums for discovery outcomes. - Adds EC2 discovery status production (including error classification) and wires it into Backend status reporting via a plugin “extra conditions” collection.
- Updates Backend status building to drop stale kgateway-owned condition types that are absent from the latest report, and adds/extends unit tests for merging and stale-condition behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
api/v1alpha1/kgateway/backend_types.go |
Defines EndpointsDiscovered condition type and reason constants for discovery outcomes. |
pkg/kgateway/extensions2/plugins/backend/ec2.go |
Tracks per-backend EC2 discovery status and exposes it as a KRT collection for status reporting. |
pkg/kgateway/extensions2/plugins/backend/ec2_test.go |
Adds unit tests for discovery status generation and error classification. |
pkg/kgateway/extensions2/plugins/backend/plugin.go |
Wires EC2 discovery status collection into the backend plugin’s ExtraConditions. |
pkg/pluginsdk/ir/backend.go |
Adds ir.BackendObjectStatus for plugin-contributed Backend conditions and equality semantics. |
pkg/pluginsdk/types.go |
Extends BackendPlugin to optionally expose ExtraConditions and includes it in sync gating. |
pkg/kgateway/proxy_syncer/status.go |
Merges plugin-contributed Backend conditions into the single Backend status report writer. |
pkg/kgateway/proxy_syncer/proxy_syncer.go |
Fetches plugin ExtraConditions and passes them into backend status report generation. |
pkg/kgateway/proxy_syncer/backend_status_test.go |
Adds coverage validating extra-condition merging behavior. |
pkg/reports/status.go |
Drops stale kgateway-owned Backend condition types that are absent from a fresh report. |
pkg/reports/observed_generation_test.go |
Adds a regression test ensuring stale managed Backend conditions are dropped while foreign ones are preserved. |
Comment on lines
305
to
+311
| if source.secret != nil { | ||
| derived, err := deriveStaticSecret(source.secret) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("invalid aws secret: %w", err) | ||
| // Malformed credential data is a credential problem, not an AWS-side | ||
| // rejection; classify it so the Backend reports CredentialError. The | ||
| // wrapped error never includes secret values (see deriveStaticSecret). | ||
| return nil, &ec2CredentialError{err: fmt.Errorf("invalid aws secret: %w", err)} |
When an EC2 discovery poll fails but endpoints carried forward from a previous successful poll are still being served, report the EndpointsDiscovered condition with reason Degraded instead of the raw failure reason. This lets operators tell a degraded-but-serving backend apart from one that is hard down (no endpoints), e.g. to alert on the two cases differently. The underlying failure cause is preserved in the condition message, which now also states how many endpoints are still served (or that none are available). Also fix TestDiscoveryStatusForBackendReportsCredentialErrorForUnresolvedSecret, which built a backend with no auth set and so never exercised the secret-credential branch it claimed to, falling through to a nil-trigger panic. It now configures Secret auth with an unresolved secret. Signed-off-by: omar <[email protected]>
andy-fong
approved these changes
Jun 25, 2026
puertomontt
enabled auto-merge
June 25, 2026 09:02
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jun 25, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The EC2 backend's EDS polling loop runs asynchronously after translation, so credential and discovery failures (bad credentials, IAM denial, AWS API errors, zero matching instances) produced only a log line and never surfaced on the
Backendresource. Operators had no Kubernetes-native signal that discovery was broken.This PR adds an
EndpointsDiscoveredstatus condition that is written back to theBackendafter every refresh cycle.What changed
api/v1alpha1/kgateway/backend_types.go):EndpointsDiscoveredwith reasonsDiscovered,NoMatchingInstances,CredentialError,AuthorizationError,DiscoveryError.ec2.go): each refresh records a per-backendec2DiscoveryStatus(part of state equality, so the condition is recomputed on every poll — including transient failures that preserve carried-forward endpoints per NFR-3). A newDiscoveryStatusKRT collection exposes it. Errors are classified: locally-detected credential problems →CredentialError; AWS auth/authz codes →AuthorizationError; everything else →DiscoveryError. Backends with an unresolvedsecretRef(filtered out before discovery runs) reportCredentialErrorsynchronously.ir.BackendObjectStatuscarries plugin-contributed conditions;BackendPlugin.ExtraConditionsexposes the collection;GenerateBackendStatusReportmerges them alongsideAcceptedso a single writer owns all Backend conditions.pkg/reports/status.go): a kgateway-managed condition absent from a fresh report is now dropped rather than preserved forever, so a backend that stops doing discovery (or is no longer an EC2 backend) doesn't retain stale state.Condition mapping
TrueDiscoveredN endpoints activeFalseNoMatchingInstancesFalseCredentialErrorFalseAuthorizationErrorFalseDiscoveryErrorNote on classification
The issue lists "invalid keys" under
CredentialError, but AWS rejects well-formed-but-wrong keys server-side (AuthFailure/InvalidClientTokenId), which this PR maps toAuthorizationError.CredentialErrorcovers locally-detectable problems (unresolved secret, malformed secret data);AuthorizationErrorcovers AWS-side rejections — consistent with the table's "AWS error message" vs. "description without exposing secret values" split. Happy to fold those AWS codes intoCredentialErrorif preferred.Change Type
/kind feature
Changelog
Additional Notes
Discovery against live AWS is not exercised by e2e here; coverage is via unit tests for the success/no-match/auth-failure/credential-error paths, the report merge, and stale-condition dropping.