fix: stamp Gateway observedGeneration from the report, not the live object#14302
Merged
puertomontt merged 3 commits intoJun 24, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes an intermittent Gateway status race where observedGeneration could “freeze” due to skew between the translation (KRT) cache that produces reports and the controller-runtime cache that the status syncer reads from. The change aligns observedGeneration stamping with the report generation that triggered the sync.
Changes:
- Update
BuildGWStatusto stamp conditionObservedGenerationfromgwReport.observedGeneration(the translated/report generation) rather than the liveGateway.Generation. - Replace the prior Gateway observedGeneration unit test with coverage for both cache-skew directions (report ahead of live, and report behind live).
- Add an explicit
go test -timeoutfor conformance Makefile targets to fail with actionable output instead of timing out mid-suite.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/reports/status.go | Stamp Gateway + listener condition ObservedGeneration from the report’s generation to prevent cache-skew freezes. |
| pkg/reports/observed_generation_test.go | Update tests to assert report-driven observedGeneration semantics under both skew scenarios. |
| Makefile | Set explicit conformance go test timeout via CONFORMANCE_GO_TEST_ARGS to avoid default-timeout panics. |
…bject BuildGWStatus stamped each condition's ObservedGeneration from the live Gateway read by the status syncer (s.mgr.GetClient(), the controller-runtime cache), while the report that triggers the sync is produced by translation from a separate cache (the istio KRT cache). The status syncer is driven only by report changes, so when those two caches skew - translation observes a new generation and enqueues a report, but the syncer's live read still returns the old generation - the sync fires once, stamps the stale generation, and nothing re-triggers it. observedGeneration then freezes at the old value. This was masked before kgateway-dev#14295: report-map equality was pointer-identity, so status re-synced on every translation tick and a later tick eventually stamped the current generation. kgateway-dev#14295 made report equality a deep value comparison, so status now only re-syncs on a real report diff, exposing the freeze. It surfaced as the GatewayObservedGenerationBump conformance test hanging for its full 300s budget (and pushing the suite past the test timeout) instead of passing in ~3s. Source the stamp from the report's observedGeneration so the value that triggers the sync and the value that is published always agree. This is also more correct: observedGeneration should reflect the generation that was actually translated, not whatever the live cache happens to show. Signed-off-by: omar <[email protected]>
puertomontt
force-pushed
the
fix/gateway-observed-generation-cache-skew
branch
from
June 24, 2026 14:38
6209333 to
0fa70aa
Compare
BuildRouteStatus stamped each parentRef condition's ObservedGeneration from the live route re-read by the syncer (s.mgr.GetClient()), not from the report that triggered the sync. This is the same cross-cache freeze as the Gateway case: the route is re-read from a different cache than translation used, so a skew can stamp a stale generation that never re-syncs. Source the stamp from routeReport.observedGeneration so the trigger and the published value stay consistent. BuildPolicyStatus already stamps from the report's observed generation, so no change is needed there. Signed-off-by: omar <[email protected]>
BuildBackendStatus and BuildListenerSetStatus stamped each condition's ObservedGeneration from the live object re-read by their syncers (s.mgr.GetClient()), not from the report that triggered the sync. This is the same cross-cache freeze fixed for Gateway and Route status: the object is re-read from a different cache than translation used, so a skew can stamp a stale generation that never re-syncs. Source the stamp from the report's observedGeneration (report.observedGeneration for backends, lsReport.observedGeneration for listener sets) so the sync trigger and the published value stay consistent. With this, all report builders (Gateway, Route, Backend, ListenerSet, and the already-correct Policy) are uniformly report-driven. Add skew regression tests covering both directions for each builder. Signed-off-by: omar <[email protected]>
andy-fong
approved these changes
Jun 24, 2026
chandler-solo
pushed a commit
to chandler-solo/kgateway
that referenced
this pull request
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
BuildGWStatus(and the other report builders) stamped each condition'sObservedGenerationfrom the live object re-read by the status syncer (s.mgr.GetClient(), the controller-runtime cache), while the report that triggers the sync is produced by translation from a separate cache (the istio KRT cache backingGatewayIndex.Gateways).The status syncer is driven only by report changes. When those two caches skew — translation observes a new generation and enqueues a report, but the syncer's live read still returns the old generation — the sync fires once, stamps the stale generation, and nothing re-triggers it.
observedGenerationthen freezes at the old value.This was masked before #14295: report-map equality was pointer-identity (always unequal), so status re-synced on every translation tick and a later tick eventually stamped the current generation. #14295 made report equality a deep value comparison, so status now only re-syncs on a real report diff — exposing the freeze.
It surfaced as the
GatewayObservedGenerationBumpconformance test hanging for its full 300s budget (and pushing the suite past the Go test timeout, producing apanic: test timed out) instead of passing in ~3s. The failure is intermittent because it depends on which informer wins the cache race at sync time.What changed
Source the
ObservedGenerationstamp from the report's generation instead of the live object, across all report builders, so the value that triggers the sync and the value that is published always agree:BuildGWStatus— gateway and listener conditions (gwReport.observedGeneration).BuildRouteStatus— parentRef conditions (routeReport.observedGeneration).BuildBackendStatus—report.observedGeneration.BuildListenerSetStatus— listener and listener-set conditions (lsReport.observedGeneration).BuildPolicyStatus— already report-driven; no change.This is also more correct:
observedGenerationshould reflect the generation that was actually translated, not whatever the live cache happens to show.pkg/reports/observed_generation_test.goreplaces the prior "FromCurrentObject" tests with skew regression tests covering both directions (report ahead of a stale live read, and report behind the live read) for Gateway, Route, Backend, and ListenerSet.Notes for reviewers
*RefreshesObservedGenerationFromCurrentObjecttests (added in Implement BackendTLSPolicy core conformance #13801). Report-driven stamping is the correct semantics: because the syncer is triggered only by report changes, stamping from the live read is exactly what enables the freeze.Change Type
/kind fix
Changelog