Description

EC2 endpoint discovery resolves instances on a periodic refresh tick (default 30s) and caches the result per backend. Three eventual-consistency gaps follow from the cached state only updating on that tick:

1. Spec changes served stale endpoints. When a backend's ec2 config changed (port, address type, filters, region), KRT recomputed the endpoints immediately but served the addresses and port resolved under the old config until the next tick — a port change routed traffic to the wrong port for up to a full refresh interval.
2. New backends served nothing for up to a tick. A newly created EC2 backend had no cached state and waited out the refresh interval before getting endpoints.
3. Credential rotation + transient AWS failure wiped endpoints. The carry-forward that preserves endpoints across a failed listing required full config equality, including the secret resourceVersion and role ARN. Any secret update (even a no-op annotation change) reset the carried state, so a concurrent listing failure dropped healthy endpoints.

Changes

• ec2BackendStateKey gains endpointSemanticsEqual, comparing only the fields that determine which endpoints a listing yields (region, port, address type, filters) and deliberately excluding credential fields (role ARN, secret), which only affect authorization to list.
• endpointsForBackend now compares the cached state's config key against the backend's current config:
  • cached state resolved under different endpoint semantics is discarded rather than served (fixes 1);
  • a credential-only mismatch keeps serving the cached endpoints — they are still the right targets — while discovery re-lists under the new credentials.
• A single-slot refreshCh lets the KRT transform request an immediate on-demand refresh whenever state is missing (fixes 2) or stale (shrinks the staleness window in 1 from the refresh interval to roughly one DescribeInstances round-trip). Requests coalesce in the buffer; there is no re-trigger loop because every refresh — including a failed one — rewrites the state key to the current config.
• The carry-forward in computeState now uses endpointSemanticsEqual, so a credential-only change followed by a failed re-list keeps the previously resolved endpoints (fixes 3). Semantic changes still clear them.

Behavior note

For a semantic spec change, this trades "briefly wrong endpoints" for "briefly no endpoints" until the on-demand refresh lands. For a port change, serving the old port was actively wrong; the no-endpoint window is now typically a single AWS API round-trip rather than the refresh interval.

Change Type

/kind fix

Changelog

Fixed EC2 backend discovery serving endpoints resolved under an outdated config (e.g. the old port) for up to a refresh interval after a Backend spec change. Spec changes and newly created EC2 backends now trigger an immediate discovery refresh, and a credential rotation combined with a transient AWS API failure no longer drops healthy endpoints.

Additional Notes

Unit-test coverage: refresh requested when state is missing, stale state discarded after a port change, cached endpoints served across a credential-only change, and carry-forward preserved across a credential-only change with a failing lister. The existing test asserting that a full config change clears endpoints on failure is unchanged.