feat(backendconfigpolicy): add zone-aware routing by MayorFaj · Pull Request #13978 · kgateway-dev/kgateway

MayorFaj · 2026-05-04T23:39:12Z

Description

Adds zone-aware routing support to BackendConfigPolicy load balancer settings.

This introduces a new loadBalancer.zoneAware.preferLocal API that configures Envoy's native zone-aware load balancing for supported load balancer policies, including:

prefer-local zone-aware routing
configurable minimum endpoint threshold
configurable routing percentage through routingEnabled
optional force mode through force.minEndpointsInZoneThreshold, which maps to Envoy forceLocalZone

The implementation also wires proxy locality into Envoy bootstrap through KGATEWAY_NODE_REGION, KGATEWAY_NODE_ZONE, and KGATEWAY_NODE_SUBZONE, and configures node.cluster, cluster_manager.local_cluster_name, and a local static cluster so Envoy can make native zone-aware and force-local routing decisions.

In addition, BackendConfigPolicy zone-aware settings now take precedence over Service trafficDistribution, and endpoint processing uses the backend's resolved AttachedPolicies, which preserves hostname and alias attachment semantics for ServiceEntry-style backends.

Zone-aware routing requires proxy locality env vars to be present on the Envoy pod.

Fixes #13303

Note; users must configure locality env vars

Change Type

/kind feature

Changelog

Added BackendConfigPolicy zone-aware routing with native Envoy prefer-local and force-local support, including bootstrap locality wiring for Envoy proxies.

Copilot

Pull request overview

Adds zone-aware routing support to BackendConfigPolicy load balancer settings, including an optional “force local zone” mode implemented via per-client endpoint prioritization, and wires proxy node locality into the Envoy bootstrap to enable Envoy’s zone-aware LB behavior. It also adjusts endpoint processing to rely on the backend’s already-resolved AttachedPolicies to preserve attachment semantics for ServiceEntry/hostname-alias backends.

Changes:

Introduces loadBalancer.zoneAware.preferLocal (and optional force) API + CRD schema, and translates it into Envoy zone_aware_lb_config (falling back to legacy lb_policy/lb_config when required).
Adds per-client endpoint processing for BackendConfigPolicy zone-aware force mode, using resolved policy attachments instead of re-matching by endpoint resource names.
Populates bootstrap.node.locality from KGATEWAY_NODE_REGION/ZONE/SUBZONE via envoyinit downward API.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/pluginsdk/ir/model.go	Adds `AttachedPolicies` to `EndpointsForBackend` and propagates it through constructors/copies.
pkg/kgateway/translator/irtranslator/backend.go	Ensures inline endpoint inputs carry the backend’s resolved `AttachedPolicies`.
pkg/kgateway/translator/gateway/testutils/outputs/backendconfigpolicy/zone-aware-prefer-local.yaml	Adds golden output coverage for prefer-local zone-aware LB config.
pkg/kgateway/translator/gateway/testutils/outputs/backendconfigpolicy/zone-aware-force.yaml	Adds golden output coverage for force-mode + legacy lb_policy output.
pkg/kgateway/translator/gateway/testutils/inputs/backendconfigpolicy/zone-aware-prefer-local.yaml	Adds gateway translator input fixture for prefer-local.
pkg/kgateway/translator/gateway/testutils/inputs/backendconfigpolicy/zone-aware-force.yaml	Adds gateway translator input fixture for force mode.
pkg/kgateway/translator/gateway/gateway_translator_test.go	Registers new translator golden test cases for zone-aware BCP scenarios.
pkg/kgateway/proxy_syncer/effective_endpoints.go	Copies backend-resolved `AttachedPolicies` into final endpoint IR for per-client translation.
pkg/kgateway/extensions2/plugins/kubernetes/k8s.go	Maps Service `topology-mode: Auto` annotation to prefer-same-zone traffic distribution.
pkg/kgateway/extensions2/plugins/backendconfigpolicy/plugin.go	Adds BackendConfigPolicy endpoint plugin for zone-aware “force” endpoint prioritization.
pkg/kgateway/extensions2/plugins/backendconfigpolicy/plugin_test.go	Adds unit tests for force-mode priority assignment and policy-selection behavior.
pkg/kgateway/extensions2/plugins/backendconfigpolicy/lb.go	Adds zone-aware LB translation, legacy lb_policy conversion, and IR equality updates.
pkg/kgateway/extensions2/plugins/backendconfigpolicy/lb_test.go	Adds translation/equality tests for zone-aware config and force-mode IR.
internal/envoyinit/pkg/downward/transform.go	Initializes `bootstrap.Node` if nil and sets `Node.Locality` from env-derived downward API values.
internal/envoyinit/pkg/downward/transform_test.go	Adds tests verifying locality is set and node is initialized when needed.
internal/envoyinit/pkg/downward/template_test.go	Extends mock downward API with node locality getters.
internal/envoyinit/pkg/downward/interface.go	Extends the downward API interface with `NodeZone/Region/Subzone`.
internal/envoyinit/pkg/downward/downward.go	Reads `KGATEWAY_NODE_REGION/ZONE/SUBZONE` into the downward injectable implementation.
install/helm/kgateway-crds/templates/gateway.kgateway.dev_backendconfigpolicies.yaml	Updates CRD schema/validations/docs for the new zone-aware load balancer API.
api/v1alpha1/kgateway/zz_generated.deepcopy.go	Adds deep-copies for new zone-aware API types and `LoadBalancer.ZoneAware`.
api/v1alpha1/kgateway/backend_config_policy_types.go	Defines new zone-aware API types + validation constraints on `LoadBalancer`.

alexliu541

Had a problem when testing locally with 3 zones. I believe its due to Envoy requires the local cluster and upstream cluster to have the same number of localities, and a static local cluster only has 1 locality thus not working.

Move BackendConfigPolicy zone-aware load balancing to Envoy typed locality LB config and restore deterministic force-local behavior across all endpoint paths. - emit zone-aware and forceLocalZone through typed LB policies - remove the legacy cluster-level zone-aware fallback - run BackendConfigPolicy endpoint plugins last in a shared ordered irtranslator endpoint phase - apply the same ordering to inline endpoint backends so force mode overrides DestinationRule priority consistently - update envoyinit Transform to inject node locality in the production IO path and preserve typed configs - rebuild envoy-wrapper when internal envoyinit sources change - refresh tests, goldens, CRD descriptions, and KinD validation Signed-off-by: MayorFaj <[email protected]>

Signed-off-by: MayorFaj <[email protected]>

…tadata removal Signed-off-by: MayorFaj <[email protected]>

Signed-off-by: MayorFaj <[email protected]>

alexliu541

a few minor comments, rest looks good and works on local testing. For these nits, feel free to skip them, I can handle them when I add e2e tests for zone aware feature.

Thanks for the update!

Signed-off-by: MayorFaj <[email protected]>

… xdsDumper to use it Signed-off-by: MayorFaj <[email protected]>

…yaml Signed-off-by: MayorFaj <[email protected]>

…t tests Signed-off-by: MayorFaj <[email protected]>

alexliu541

LGTM, thanks for the PR!

andy-fong · 2026-06-17T23:40:15Z

@MayorFaj sorry for the delay in this. Looks like merging with main brought in some new translator tests that do not have the new locality_weighted_lb_config empty settings:

2026-06-17T20:20:37.8010609Z     test.go:330: 
2026-06-17T20:20:37.8011012Z         	Error Trace:	/home/runner/work/kgateway/kgateway/test/translator/test.go:330
2026-06-17T20:20:37.8011310Z         	Error:      	Should be empty, but was   []*clusterv3.Cluster{
2026-06-17T20:20:37.8011784Z         	            	  	Inverse(protocmp.Transform, protocmp.Message{
2026-06-17T20:20:37.8012340Z         	            	  		"@type":              s"envoy.config.cluster.v3.Cluster",
2026-06-17T20:20:37.8012871Z         	            	+ 		"common_lb_config":   s"{locality_weighted_lb_config:{}}",
2026-06-17T20:20:37.8013954Z         	            	  		"connect_timeout":    protocmp.Message{"@type": s"google.protobuf.Duration", "seconds": int64(5)},
2026-06-17T20:20:37.8016741Z         	            	  		"eds_cluster_config": protocmp.Message{"@type": s"envoy.config.cluster.v3.Cluster.EdsClusterConfig", "eds_config": protocmp.Message{"@type": s"envoy.config.core.v3.ConfigSource", "ads": protocmp.Message{"@type": s"envoy.config.core.v3.AggregatedConfigSource"}, "resource_api_version": s"V3"}},
2026-06-17T20:20:37.8017074Z         	            	  		... // 3 identical entries
2026-06-17T20:20:37.8017250Z         	            	  	}),
2026-06-17T20:20:37.8019472Z         	            	  	Inverse(protocmp.Transform, protocmp.Message{"@type": s"envoy.config.cluster.v3.Cluster", "connect_timeout": protocmp.Message{"@type": s"google.protobuf.Duration", "seconds": int64(5)}, "name": string("test-backend-plugin_default_example-svc_80")}),
2026-06-17T20:20:37.8019694Z         	            	  }
2026-06-17T20:20:37.8019968Z         	Test:       	TestBasic/ListenerPolicy_with_localReplies

Once you merge with main again an regenerate the golden files, I will approve and address any follow up with @alexliu541 e2e PR to avoid keep fixing new tests.

… main Signed-off-by: MayorFaj <[email protected]>

Signed-off-by: MayorFaj <[email protected]>

Copilot AI review requested due to automatic review settings May 4, 2026 23:39

MayorFaj requested a review from a team as a code owner May 4, 2026 23:39

Copilot started reviewing on behalf of MayorFaj May 4, 2026 23:39 View session

gateway-bot added kind/feature Categorizes issue or PR as related to a new feature. release-note labels May 4, 2026

MayorFaj changed the title ~~feat: dd BackendConfigPolicy zone-aware routing~~ feat: add BackendConfigPolicy zone-aware routing May 4, 2026

Copilot AI reviewed May 4, 2026

View reviewed changes

Comment thread pkg/kgateway/extensions2/plugins/backendconfigpolicy/lb.go Outdated

Comment thread pkg/kgateway/extensions2/plugins/backendconfigpolicy/plugin.go Outdated

Comment thread internal/envoyinit/pkg/downward/transform.go Outdated

MayorFaj force-pushed the feature/backendconfigpolicy-zone-aware-routing branch 2 times, most recently from de5de34 to 1a605c5 Compare May 5, 2026 14:43

puertomontt requested a review from NomadXD May 14, 2026 15:52

NomadXD reviewed May 15, 2026

View reviewed changes

Comment thread pkg/kgateway/extensions2/plugins/kubernetes/k8s.go Outdated

Comment thread pkg/kgateway/extensions2/plugins/backendconfigpolicy/plugin.go Outdated

MayorFaj requested a review from NomadXD May 18, 2026 12:06

andy-fong self-assigned this May 18, 2026

alexliu541 reviewed May 21, 2026

View reviewed changes

Comment thread api/v1alpha1/kgateway/backend_config_policy_types.go Outdated

Comment thread pkg/kgateway/extensions2/plugins/backendconfigpolicy/lb.go Outdated

Comment thread pkg/kgateway/extensions2/plugins/backendconfigpolicy/plugin.go Outdated

alexliu541 reviewed May 28, 2026

View reviewed changes

Comment thread pkg/kgateway/helm/envoy/templates/configmap.yaml Outdated

Comment thread pkg/kgateway/translator/irtranslator/endpoint_plugins.go

MayorFaj added 7 commits June 4, 2026 22:13

fix: drop topology-mode annotation handling

4527bc2

Signed-off-by: MayorFaj <[email protected]>

fix: enforce zone-aware force via endpoint priority

21ad1d4

Signed-off-by: MayorFaj <[email protected]>

fix: use native Envoy zone-aware routing for BackendConfigPolicy

45d963c

Signed-off-by: MayorFaj <[email protected]>

replace stale UniquelyConnectedClient references

2d1ef0e

Signed-off-by: MayorFaj <[email protected]>

Add zone-aware routing guide and update related tests

d056f15

Signed-off-by: MayorFaj <[email protected]>

test: refresh zone-aware BackendConfigPolicy goldens after cluster me…

c0af68a

…tadata removal Signed-off-by: MayorFaj <[email protected]>

MayorFaj force-pushed the feature/backendconfigpolicy-zone-aware-routing branch from d4b85f6 to c0af68a Compare June 4, 2026 21:25

MayorFaj and others added 3 commits June 5, 2026 00:16

Refactor gateway cluster configuration from STATIC to EDS

d2a2448

Signed-off-by: MayorFaj <[email protected]>

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

85967f2

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

34f22a0

MayorFaj requested a review from alexliu541 June 9, 2026 00:04

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

b832e1f

alexliu541 reviewed Jun 10, 2026

View reviewed changes

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

6288485

feat: enhance zone-aware routing configuration and testing

86043bd

Signed-off-by: MayorFaj <[email protected]>

MayorFaj requested a review from alexliu541 June 10, 2026 12:22

MayorFaj and others added 3 commits June 10, 2026 13:42

fix: reorder import statements in transform_test.go for consistency

8a9dc53

Signed-off-by: MayorFaj <[email protected]>

implement LocalClusterName function for zone-aware routing and update…

a8c9c52

… xdsDumper to use it Signed-off-by: MayorFaj <[email protected]>

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

396b44a

MayorFaj closed this Jun 10, 2026

MayorFaj reopened this Jun 10, 2026

MayorFaj and others added 4 commits June 10, 2026 21:32

fix: add localityWeightedLbConfig to commonLbConfig in hash-policies.…

5db3c6f

…yaml Signed-off-by: MayorFaj <[email protected]>

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

fb94e28

Implement default locality configuration for EDS clusters and add uni…

e2a918e

…t tests Signed-off-by: MayorFaj <[email protected]>

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

1d3f4fb

alexliu541 approved these changes Jun 11, 2026

View reviewed changes

davidjumani reviewed Jun 12, 2026

View reviewed changes

Comment thread pkg/kgateway/extensions2/plugins/backendconfigpolicy/lb.go

MayorFaj requested a review from davidjumani June 12, 2026 15:34

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

e233ff0

Merge branch 'main' into feature/backendconfigpolicy-zone-aware-routing

a91f706

MayorFaj changed the title ~~feat: add BackendConfigPolicy zone-aware routing~~ feat(backendconfigpolicy): add zone-aware routing Jun 18, 2026

MayorFaj added 2 commits June 18, 2026 06:37

test: refresh translator goldens for locality config after merge with…

3453db7

… main Signed-off-by: MayorFaj <[email protected]>

test: refresh waypoint goldens after merge with main

bf0efe5

Signed-off-by: MayorFaj <[email protected]>

andy-fong approved these changes Jun 18, 2026

View reviewed changes

andy-fong added this pull request to the merge queue Jun 18, 2026

Merged via the queue into kgateway-dev:main with commit bc561b0 Jun 18, 2026
36 of 38 checks passed

ayushi-work mentioned this pull request Jun 18, 2026

fix: resolve FrontendTLS CA cert refs in Gateway namespace, not ListenerSet #14232

Merged

4 tasks

alexliu541 mentioned this pull request Jun 23, 2026

Rework and clean up zone aware routing and add e2e test #14288

Merged

babakzarrinbal mentioned this pull request Jul 3, 2026

bug: per-client xDS snapshot deferral permanently starves new/scaled proxies that reference an always-empty cluster (ExternalName / scale-to-zero) #14352

Closed

chandler-solo mentioned this pull request Jul 7, 2026

v2.3.0 upstream connect failures bug #14184

Open

davidjumani mentioned this pull request Jul 17, 2026

fix: Set initial_fetch_timeout to resuce startup time in envoy #14433

Merged

Uh oh!

Conversation

MayorFaj commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Change Type

Changelog

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexliu541 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alexliu541 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexliu541 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andy-fong commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

MayorFaj commented May 4, 2026 •

edited

Loading