E2E test flake: TestK8sResolverEndpointSliceUpdates asserts on the first update instead of convergence #1001

Symptom

TestK8sResolverEndpointSliceUpdates intermittently fails at resolver_test.go:215:

updated state.Addresses = [{Addr: "10.0.0.1:443", ServerName: "", }],
                    want [{Addr: "10.0.0.1:443", ServerName: "", } {Addr: "10.0.0.2:443", ServerName: "", }]

Hit rate is high: 8 of the 30 most recent failed pr-workflow runs (~27%). All eight verified individually — each fails on this test at resolver_test.go:215:

• https://github.com/agent-substrate/substrate/actions/runs/32045558292
• https://github.com/agent-substrate/substrate/actions/runs/32045171791
• https://github.com/agent-substrate/substrate/actions/runs/31987832602
• https://github.com/agent-substrate/substrate/actions/runs/31978191442
• https://github.com/agent-substrate/substrate/actions/runs/31937546256
• https://github.com/agent-substrate/substrate/actions/runs/31916613395
• https://github.com/agent-substrate/substrate/actions/runs/31851248168
• https://github.com/agent-substrate/substrate/actions/runs/31849526035

Cause

Build registers the informer event handlers and also starts a goroutine that reports once the cache has synced:

go func() {
    if cache.WaitForCacheSync(ctx.Done(), inf.Informer().HasSynced) {
        r.updateState()
    }
}()

That extra report is deliberate and correct: without it, a service with no EndpointSlices at all would never fire AddFunc, and the resolver would never call UpdateState — leaving gRPC waiting instead of learning the answer is an empty set.

The catch is how it waits. WaitForCacheSync polls with syncedPollPeriod = 100 * time.Millisecond (vendor/k8s.io/client-go/tools/cache/shared_informer.go:369) via PollImmediateUntil. The immediate check normally finds the cache not yet synced, so the goroutine fires at roughly 100ms after Build — exactly once, then exits.

So there is a race between that 100ms timer and how fast the test gets from its first select to creating the second EndpointSlice:

The second select takes whatever is next in the channel, so on a slow runner it takes that stale report and asserts against it.

Measured timeline, with a 150ms sleep inserted between the first select and the Create to stand in for CI scheduling delay:

[  0.5ms]  update #1: [10.0.0.1]              <- AddFunc for slice1
           test's first select takes it
           ... 150ms stall ...
[101.1ms]  update #2: [10.0.0.1]              <- the WaitForCacheSync goroutine
           test creates slice2
[151.4ms]  update #3: [10.0.0.1, 10.0.0.2]

Without the stall, Build-to-Create is 0.6ms locally — far ahead of the timer — which is why this never reproduces on a developer machine and only surfaces on loaded runners.

Note the resolver is not at fault. It never promised the first update after a change is the final one; it promises eventual convergence, and a duplicate report costs gRPC nothing. The test asserted on the wrong contract.

Verification

-count=N proves nothing here — the unfixed test passes locally at any count because the window is never hit. A/B with the delay as the only variable: