TL;DR

Gateway goes Programmed=False when its LB Service loses its IP (correct), but can become permanently stuck and fail to self-recover to True if the IP is restored during a controller restart or cache warmup window — regression in v1.6.0. Root cause: PR #7451 switched MergeConditions to reflect.DeepEqual including LastTransitionTime. Since PR #7268 moved LTT stamping to the Mutator, translator conditions always have LTT=zero — DeepEqual against a K8s-stored condition (real timestamp) always returns false, causing False to be unconditionally re-written on every stale-cache reconcile, overwriting correctly computed True writes. Fix: restore conditionChanged() comparing only Status, Reason, Message, ObservedGeneration. Verified locally against v1.7.1 source. Not reproducible on single-node local clusters (cache too fast) — requires a real multi-node cluster with a LoadBalancer controller.

Description:

When the LoadBalancer Service backing an Envoy Gateway transiently loses its status.loadBalancer.ingress IP (e.g. during a node drain, LB controller re-election, or controller pod restart), the Gateway correctly transitions to Programmed=False with reason AddressNotAssigned.

However, once the IP is successfully restored to the LoadBalancer Service, the Gateway can become permanently stuck and fail to self-recover if the IP is restored during a controller restart or cache warmup window — even though the underlying LoadBalancer Service has a valid IP and the Envoy data plane is routing traffic normally.

Impact:
While the Envoy data plane continues routing traffic without interruption, tools that rely on Gateway.status.addresses — such as external-dns with --source=gateway-httproute --policy=sync — read the empty addresses array and delete all DNS records for every HTTPRoute attached to that Gateway, causing a complete DNS routing outage.

Expected behavior: When the LoadBalancer Service's status.loadBalancer.ingress is restored, the Gateway's Programmed condition should automatically transition back to True upon the next reconcile triggered by the Service MODIFIED watch event.

Actual behavior: The Gateway can become permanently stuck at Programmed=False if the IP is restored during a controller restart or cache warmup window. When stuck, the only recovery methods are:

• Restarting the envoy-gateway controller pod
• Manually annotating the LoadBalancer Service to force a fresh watch event (see workaround below)

Note: This behavior was not present in Envoy Gateway v1.3.0, where the Gateway would recover automatically within ~1 second of the IP being restored. This is a confirmed regression introduced in v1.6.0 via PR #7451.

Repro steps:

Environment requirements

Important: This bug requires a real multi-node Kubernetes cluster with a LoadBalancer controller (MetalLB, AWS NLB, cloud provider, etc.). It cannot be reliably reproduced on a single-node local cluster (e.g. k3s/kind/minikube) because on a single node the informer cache updates sub-millisecond — the race window that causes permanent stuck state never materialises. The root cause is confirmed by code analysis and a local patched build, but the production symptom requires realistic network latency between the API server, LB controller, and EG controller.

Tested on:

• Multi-node Kubernetes cluster, v1.30.x and v1.34.x
• Envoy Gateway v1.7.1 and v1.7.3
• MetalLB v0.14.x L2 mode and cloud provider NLB
• Gateway API v1.2.x

Step 1 — Observe the bug (no setup required on affected clusters)

After any controller restart on a cluster running EG v1.6.x–v1.8.0-rc.1 with a Gateway backed by a LoadBalancer Service, the Gateway may be found stuck:

kubectl get gateway -A
# NAMESPACE    NAME          CLASS         ADDRESS      PROGRAMMED
# my-ns        my-gateway    my-class                   False       ← stuck
# my-ns        my-gateway2   my-class2     10.0.0.1     True

# The backing LoadBalancer Service has a valid IP:
kubectl get svc envoy-<gateway>-<hash> -n <namespace> \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
# 10.0.0.2   ← IP present, gateway still False

The gateway remains stuck indefinitely without intervention.

Step 2 — Confirm recovery only happens via Service watch event

# Touch the Service — forces a MODIFIED watch event — gateway recovers in ~1s
kubectl annotate svc envoy-<gateway>-<hash> -n <namespace> \
  touch="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

kubectl get gateway <name> -n <namespace>
# PROGRAMMED: True   ← recovered immediately

This confirms the controller is healthy and the Service IP is valid — the only problem is that the Gateway status is not updated unless a Service watch event fires.

Step 3 — Reproduce the stuck state

# 1. Restart the controller (simulates rolling update, leader re-election, node drain)
kubectl rollout restart deployment/envoy-gateway -n <eg-namespace>

# 2. During the cache warmup window (~30–120s after the new pod starts),
#    trigger a reconcile via any non-Service event. In practice this is triggered
#    by GitOps tools (Flux/Argo re-applying resources), certgen Jobs completing,
#    or any HTTPRoute/Secret update in the namespace. Manually:
kubectl annotate gateway <name> -n <namespace> \
  touch="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

# 3. Observe: Gateway flips to Programmed=False
#    The LoadBalancer Service still has its IP — traffic is unaffected.
#    The gateway can become permanently stuck and fail to self-recover.
kubectl get gateway <name> -n <namespace>
# ADDRESS: <empty>   PROGRAMMED: False   ← stuck

kubectl get svc envoy-<gateway>-<hash> -n <namespace>
# EXTERNAL-IP: 10.x.x.x   ← IP present

Recovery workaround

kubectl annotate svc envoy-<gateway>-<hash> -n <namespace> \
  touch="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite
# Gateway recovers to Programmed=True within ~5 seconds

Root cause

The regression was introduced by PR #7451 (chore: remove last transition time comparison as no longer set), which changed MergeConditions in internal/gatewayapi/status/conditions.go from a meaningful-field comparison to reflect.DeepEqual.

v1.3.0 — working:

func conditionChanged(a, b metav1.Condition) bool {
    return a.Status != b.Status || a.Reason != b.Reason || a.Message != b.Message
}

v1.7.1/v1.7.3 — broken:

if !reflect.DeepEqual(conditions[j], updates[i]) {  // includes LastTransitionTime

PR #7451 was a follow-up cleanup to PR #7268, which moved LastTransitionTime stamping from the translator layer to the Mutator (immediately before the Kubernetes API write). After that change, conditions produced by the translator always carry LTT=zero.

The problem is that MergeConditions does not compare two translator-produced conditions — it compares a newly produced condition (LTT=zero) against the Kubernetes-stored condition (LTT=<previous write timestamp>). reflect.DeepEqual therefore always returns false even when Status, Reason, and Message are identical, causing the condition to be unconditionally re-written on every reconcile cycle.

The consequence: when the informer cache is stale (empty LB ingress) and a non-Service reconcile fires, False is computed and written. Even if a correct True was written moments before, the next stale-cache reconcile re-writes False because reflect.DeepEqual({False, LTT=zero}, {True, LTT=<stored>}) always returns false and triggers an update. The gateway oscillates or gets permanently stuck depending on which reconcile fires last.

The deduplication benefit from PR #7268 operates correctly at the watchable layer (comparing two translator outputs, both LTT=zero). The reflect.DeepEqual change inside MergeConditions was applied to the wrong comparison point.

Proposed fix

Restore the conditionChanged helper that ignores LastTransitionTime. This does not regress the performance improvement from PR #7268/#7451 — that deduplication operates at a different layer.

// internal/gatewayapi/status/conditions.go

func MergeConditions(conditions []metav1.Condition, updates ...metav1.Condition) []metav1.Condition {
    var additions []metav1.Condition
    for i := range updates {
        updates[i].Message = truncateConditionMessage(updates[i].Message)
        add := true
        for j := range conditions {
            if conditions[j].Type == updates[i].Type {
                add = false
                if conditionChanged(conditions[j], updates[i]) {
                    conditions[j].Status = updates[i].Status
                    conditions[j].Reason = updates[i].Reason
                    conditions[j].Message = updates[i].Message
                    conditions[j].ObservedGeneration = updates[i].ObservedGeneration
                    break
                }
            }
        }
        if add {
            additions = append(additions, updates[i])
        }
    }
    conditions = append(conditions, additions...)
    return conditions
}

// conditionChanged compares only meaningful fields, intentionally ignoring
// LastTransitionTime which is always zero in translator-produced conditions.
func conditionChanged(a, b metav1.Condition) bool {
    return a.Status != b.Status || a.Reason != b.Reason ||
        a.Message != b.Message || a.ObservedGeneration != b.ObservedGeneration
}

This fix was verified by building a patched binary from the v1.7.1 source. With the fix applied, the Gateway correctly self-recovers to Programmed=True within ~1s of the LB IP being restored, matching v1.3.0 behaviour. Without the fix, the gateway remains stuck.

Environment:

• Envoy Gateway version: v1.6.x, v1.7.1, v1.7.3, v1.8.0-rc.1 (confirmed working in v1.3.0)
• Kubernetes version: v1.30.x and v1.34.x (multi-node)
• Load balancer: MetalLB v0.14.x (L2 mode) and cloud provider NLB
• Gateway API version: v1.2.x
• Regression introduced by: PR chore: remove last transition time comparison as no longer set #7451, commit 137d5286, cherry-picked into v1.6.0 via PR Cherry pick/v1.6.0 #7482

Logs:

Multiple gateways observed stuck simultaneously — Services have valid IPs throughout

Two gateways on the same cluster stuck at the same time. Both had valid LB IPs throughout. Neither recovered without manual intervention.

NAMESPACE    NAME         CLASS        ADDRESS    PROGRAMMED   AGE
my-ns-1      gateway-a    class-a                 False        ← stuck
my-ns-1      gateway-b    class-b      10.0.0.1   True
my-ns-2      gateway-c    class-c                 False        ← stuck
my-ns-2      gateway-d    class-d      10.0.0.2   True

Programmed=False condition

{
  "lastTransitionTime": "2026-05-09T23:02:36Z",
  "message": "No addresses have been assigned to the Gateway",
  "observedGeneration": 1,
  "reason": "AddressNotAssigned",
  "status": "False",
  "type": "Programmed"
}

Backing LoadBalancer Services at the same time — IPs present, data plane unaffected

kubectl get svc envoy-<gateway-a>-<hash> -n my-ns-1 \
  -o jsonpath='{.status.loadBalancer}'
# {"ingress":[{"ip":"10.0.0.3","ipMode":"VIP"}]}

kubectl get svc envoy-<gateway-c>-<hash> -n my-ns-2 \
  -o jsonpath='{.status.loadBalancer}'
# {"ingress":[{"ip":"10.0.0.4","ipMode":"VIP"}]}

Both Services have valid IPs. Traffic routing is completely unaffected.

Recovery — annotating the Service recovers the Gateway in ~1 second

kubectl annotate svc envoy-<gateway-a>-<hash> -n my-ns-1 \
  touch="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

# t+1s: Programmed=True  ← immediate recovery

Controller logs — silent failure, no errors or warnings

The controller reconciles successfully and logs no warnings or errors when writing Programmed=False:

2026-05-09T23:02:35Z  INFO  provider  kubernetes/controller.go:318      reconciling gateways       {"runner": "provider"}
2026-05-09T23:02:36Z  INFO  provider  kubernetes/status_updater.go:143  received a status update   {"namespace": "my-ns-1", "name": "gateway-a", "kind": "Gateway"}

Programmed=False is written with no indication anything is wrong.