Skip to content

fix: stamp Gateway observedGeneration from the report, not the live object#14302

Merged
puertomontt merged 3 commits into
kgateway-dev:mainfrom
puertomontt:fix/gateway-observed-generation-cache-skew
Jun 24, 2026
Merged

fix: stamp Gateway observedGeneration from the report, not the live object#14302
puertomontt merged 3 commits into
kgateway-dev:mainfrom
puertomontt:fix/gateway-observed-generation-cache-skew

Conversation

@puertomontt

@puertomontt puertomontt commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Description

BuildGWStatus (and the other report builders) stamped each condition's ObservedGeneration from the live object re-read by the status syncer (s.mgr.GetClient(), the controller-runtime cache), while the report that triggers the sync is produced by translation from a separate cache (the istio KRT cache backing GatewayIndex.Gateways).

The status syncer is driven only by report changes. When those two caches skew — translation observes a new generation and enqueues a report, but the syncer's live read still returns the old generation — the sync fires once, stamps the stale generation, and nothing re-triggers it. observedGeneration then freezes at the old value.

This was masked before #14295: report-map equality was pointer-identity (always unequal), so status re-synced on every translation tick and a later tick eventually stamped the current generation. #14295 made report equality a deep value comparison, so status now only re-syncs on a real report diff — exposing the freeze.

It surfaced as the GatewayObservedGenerationBump conformance test hanging for its full 300s budget (and pushing the suite past the Go test timeout, producing a panic: test timed out) instead of passing in ~3s. The failure is intermittent because it depends on which informer wins the cache race at sync time.

What changed

Source the ObservedGeneration stamp from the report's generation instead of the live object, across all report builders, so the value that triggers the sync and the value that is published always agree:

  • BuildGWStatus — gateway and listener conditions (gwReport.observedGeneration).
  • BuildRouteStatus — parentRef conditions (routeReport.observedGeneration).
  • BuildBackendStatusreport.observedGeneration.
  • BuildListenerSetStatus — listener and listener-set conditions (lsReport.observedGeneration).
  • BuildPolicyStatus — already report-driven; no change.

This is also more correct: observedGeneration should reflect the generation that was actually translated, not whatever the live cache happens to show.

pkg/reports/observed_generation_test.go replaces the prior "FromCurrentObject" tests with skew regression tests covering both directions (report ahead of a stale live read, and report behind the live read) for Gateway, Route, Backend, and ListenerSet.

Notes for reviewers

  • This reverses the intent of the *RefreshesObservedGenerationFromCurrentObject tests (added in Implement BackendTLSPolicy core conformance #13801). Report-driven stamping is the correct semantics: because the syncer is triggered only by report changes, stamping from the live read is exactly what enables the freeze.
  • agentgateway's equivalent control plane already stamps from the report and builds status inside its translation collection (no separate cross-cache syncer read), which is why it doesn't exhibit this bug — this change brings kgateway's stamping in line.

Change Type

/kind fix

Changelog

Fixed a bug where a Gateway, Route, Backend, or ListenerSet status observedGeneration could intermittently freeze at a stale value after a spec change, due to a skew between the translation cache and the status syncer's cache.

Copilot AI review requested due to automatic review settings June 24, 2026 14:34
@gateway-bot gateway-bot added do-not-merge/description-invalid kind/fix Categorizes issue or PR as related to a bug. release-note labels Jun 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes an intermittent Gateway status race where observedGeneration could “freeze” due to skew between the translation (KRT) cache that produces reports and the controller-runtime cache that the status syncer reads from. The change aligns observedGeneration stamping with the report generation that triggered the sync.

Changes:

  • Update BuildGWStatus to stamp condition ObservedGeneration from gwReport.observedGeneration (the translated/report generation) rather than the live Gateway.Generation.
  • Replace the prior Gateway observedGeneration unit test with coverage for both cache-skew directions (report ahead of live, and report behind live).
  • Add an explicit go test -timeout for conformance Makefile targets to fail with actionable output instead of timing out mid-suite.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pkg/reports/status.go Stamp Gateway + listener condition ObservedGeneration from the report’s generation to prevent cache-skew freezes.
pkg/reports/observed_generation_test.go Update tests to assert report-driven observedGeneration semantics under both skew scenarios.
Makefile Set explicit conformance go test timeout via CONFORMANCE_GO_TEST_ARGS to avoid default-timeout panics.

…bject

BuildGWStatus stamped each condition's ObservedGeneration from the live
Gateway read by the status syncer (s.mgr.GetClient(), the controller-runtime
cache), while the report that triggers the sync is produced by translation
from a separate cache (the istio KRT cache). The status syncer is driven only
by report changes, so when those two caches skew - translation observes a new
generation and enqueues a report, but the syncer's live read still returns the
old generation - the sync fires once, stamps the stale generation, and nothing
re-triggers it. observedGeneration then freezes at the old value.

This was masked before kgateway-dev#14295: report-map equality was pointer-identity, so
status re-synced on every translation tick and a later tick eventually stamped
the current generation. kgateway-dev#14295 made report equality a deep value comparison,
so status now only re-syncs on a real report diff, exposing the freeze. It
surfaced as the GatewayObservedGenerationBump conformance test hanging for its
full 300s budget (and pushing the suite past the test timeout) instead of
passing in ~3s.

Source the stamp from the report's observedGeneration so the value that
triggers the sync and the value that is published always agree. This is also
more correct: observedGeneration should reflect the generation that was
actually translated, not whatever the live cache happens to show.

Signed-off-by: omar <[email protected]>
@puertomontt
puertomontt force-pushed the fix/gateway-observed-generation-cache-skew branch from 6209333 to 0fa70aa Compare June 24, 2026 14:38
BuildRouteStatus stamped each parentRef condition's ObservedGeneration from the
live route re-read by the syncer (s.mgr.GetClient()), not from the report that
triggered the sync. This is the same cross-cache freeze as the Gateway case:
the route is re-read from a different cache than translation used, so a skew can
stamp a stale generation that never re-syncs.

Source the stamp from routeReport.observedGeneration so the trigger and the
published value stay consistent.

BuildPolicyStatus already stamps from the report's observed generation, so no
change is needed there.

Signed-off-by: omar <[email protected]>
BuildBackendStatus and BuildListenerSetStatus stamped each condition's
ObservedGeneration from the live object re-read by their syncers
(s.mgr.GetClient()), not from the report that triggered the sync. This is the
same cross-cache freeze fixed for Gateway and Route status: the object is
re-read from a different cache than translation used, so a skew can stamp a
stale generation that never re-syncs.

Source the stamp from the report's observedGeneration (report.observedGeneration
for backends, lsReport.observedGeneration for listener sets) so the sync trigger
and the published value stay consistent. With this, all report builders
(Gateway, Route, Backend, ListenerSet, and the already-correct Policy) are
uniformly report-driven.

Add skew regression tests covering both directions for each builder.

Signed-off-by: omar <[email protected]>
@puertomontt
puertomontt added this pull request to the merge queue Jun 24, 2026
Merged via the queue into kgateway-dev:main with commit 2155322 Jun 24, 2026
37 of 40 checks passed
@puertomontt
puertomontt deleted the fix/gateway-observed-generation-cache-skew branch June 24, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/fix Categorizes issue or PR as related to a bug. release-note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants