Skip to content

CI: Data race in (*cesManagerIdentity).updateCESInCache() #17914

@ti-mo

Description

@ti-mo

This tripped the race detector on two separate tests:

Zips:

2021-11-16T16:34:47.847587948Z level=debug msg="Generated CES" CESName=ces-8mh2bjfxv-fp9hs subsys=ces-controller
2021-11-16T16:34:47.847596970Z level=debug msg="Queueing CEP in the CES" CEPName=testclient-lr2zs CESName=ces-8mh2bjfxv-fp9hs subsys=ces-controller totalCEPCount=0
2021-11-16T16:34:47.863964821Z ==================
2021-11-16T16:34:47.863993853Z WARNING: DATA RACE
2021-11-16T16:34:47.863998587Z Write at 0x00c00040ade0 by goroutine 48:
2021-11-16T16:34:47.864002094Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*cesManagerIdentity).updateCESInCache()
2021-11-16T16:34:47.864005638Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/manager.go:589 +0x2c4
2021-11-16T16:34:47.864009057Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*reconciler).reconcileCESCreate()
2021-11-16T16:34:47.864022638Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/reconciler.go:60 +0x44a
2021-11-16T16:34:47.864026550Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).syncCES()
2021-11-16T16:34:47.864029809Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:306 +0x49a
2021-11-16T16:34:47.864033190Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).processNextWorkItem()
2021-11-16T16:34:47.864036482Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:248 +0x175
2021-11-16T16:34:47.864039416Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).worker()
2021-11-16T16:34:47.864042465Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:237 +0x39
2021-11-16T16:34:47.864045542Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).worker-fm()
2021-11-16T16:34:47.864048781Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:236 +0x1d
2021-11-16T16:34:47.864051970Z   k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
2021-11-16T16:34:47.864055123Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x81
2021-11-16T16:34:47.864058305Z   k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
2021-11-16T16:34:47.864061380Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xce
2021-11-16T16:34:47.864064568Z   k8s.io/apimachinery/pkg/util/wait.JitterUntil()
2021-11-16T16:34:47.864067628Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x104
2021-11-16T16:34:47.864070955Z   k8s.io/apimachinery/pkg/util/wait.Until()
2021-11-16T16:34:47.864073979Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x48
2021-11-16T16:34:47.864077019Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).Run·dwrap·15()
2021-11-16T16:34:47.864082667Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:177 +0x58
2021-11-16T16:34:47.864086384Z 
2021-11-16T16:34:47.864089253Z Previous read at 0x00c00040ade0 by goroutine 99:
2021-11-16T16:34:47.864092319Z   k8s.io/apimachinery/pkg/apis/meta/v1.(*ObjectMeta).GetName()
2021-11-16T16:34:47.864095571Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/meta.go:139 +0x350
2021-11-16T16:34:47.864098946Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*cesManagerIdentity).InsertCEPInCache()
2021-11-16T16:34:47.864102253Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/manager.go:574 +0x325
2021-11-16T16:34:47.864105458Z   github.com/cilium/cilium/operator/watchers.endpointUpdated()
2021-11-16T16:34:47.864113394Z       /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:211 +0x129
2021-11-16T16:34:47.864117184Z   github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit.func1.1()
2021-11-16T16:34:47.864120558Z       /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:86 +0x44
2021-11-16T16:34:47.864123802Z   k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd()
2021-11-16T16:34:47.864127037Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:231 +0x79
2021-11-16T16:34:47.864130395Z   k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd()
2021-11-16T16:34:47.864133786Z       <autogenerated>:1 +0x33
2021-11-16T16:34:47.864137702Z   github.com/cilium/cilium/pkg/k8s/informer.NewInformerWithStore.func1()
2021-11-16T16:34:47.864140818Z       /go/src/github.com/cilium/cilium/pkg/k8s/informer/informer.go:108 +0x2d8
2021-11-16T16:34:47.864143818Z   k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop()
2021-11-16T16:34:47.864146911Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:539 +0x4fb
2021-11-16T16:34:47.864150632Z   k8s.io/client-go/tools/cache.(*controller).processLoop()
2021-11-16T16:34:47.864154020Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:183 +0x61
2021-11-16T16:34:47.864157134Z   k8s.io/client-go/tools/cache.(*controller).processLoop-fm()
2021-11-16T16:34:47.864159997Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:181 +0x39
2021-11-16T16:34:47.864163130Z   k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
2021-11-16T16:34:47.864166322Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x81
2021-11-16T16:34:47.864169736Z   k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
2021-11-16T16:34:47.864172714Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xce
2021-11-16T16:34:47.864175926Z   k8s.io/apimachinery/pkg/util/wait.JitterUntil()
2021-11-16T16:34:47.864180068Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x104
2021-11-16T16:34:47.864183555Z   k8s.io/apimachinery/pkg/util/wait.Until()
2021-11-16T16:34:47.864186502Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x616
2021-11-16T16:34:47.864189644Z   k8s.io/client-go/tools/cache.(*controller).Run()
2021-11-16T16:34:47.864192862Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:154 +0x5cd
2021-11-16T16:34:47.864195859Z   github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit.func1·dwrap·1()
2021-11-16T16:34:47.864199012Z       /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:116 +0x59
2021-11-16T16:34:47.864202206Z 
2021-11-16T16:34:47.864205184Z Goroutine 48 (running) created at:
2021-11-16T16:34:47.864212584Z   github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).Run()
2021-11-16T16:34:47.864216129Z       /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:177 +0x4be
2021-11-16T16:34:47.864219432Z   main.onOperatorStartLeading·dwrap·13()
2021-11-16T16:34:47.864222626Z       /go/src/github.com/cilium/cilium/operator/main.go:366 +0x64
2021-11-16T16:34:47.864225842Z 
2021-11-16T16:34:47.864228898Z Goroutine 99 (running) created at:
2021-11-16T16:34:47.864231932Z   github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit.func1()
2021-11-16T16:34:47.864235198Z       /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:116 +0x59a
2021-11-16T16:34:47.864238492Z   sync.(*Once).doSlow()
2021-11-16T16:34:47.864241528Z       /usr/local/go/src/sync/once.go:68 +0x127
2021-11-16T16:34:47.864244774Z   sync.(*Once).Do()
2021-11-16T16:34:47.864247811Z       /usr/local/go/src/sync/once.go:59 +0x46
2021-11-16T16:34:47.864250660Z   github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit()
2021-11-16T16:34:47.864253765Z       /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:76 +0x84
2021-11-16T16:34:47.864256895Z   github.com/cilium/cilium/operator/watchers.CiliumEndpointsSliceInit()
2021-11-16T16:34:47.864260051Z       /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:58 +0x3ba
2021-11-16T16:34:47.864263325Z   main.onOperatorStartLeading()
2021-11-16T16:34:47.864266212Z       /go/src/github.com/cilium/cilium/operator/main.go:364 +0x311
2021-11-16T16:34:47.864269238Z   k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run·dwrap·4()
2021-11-16T16:34:47.864272315Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x58
2021-11-16T16:34:47.864275348Z ==================
2021-11-16T16:34:47.884582616Z level=debug msg="Marking identity alive" identity=5103 subsys=identity-heartbeat
2021-11-16T16:34:47.931849205Z level=debug msg="Generated CES" CESName=ces-x9xxv5ybq-hsdcg subsys=ces-controller
2021-11-16T16:34:47.931865624Z level=debug msg="Queueing CEP in the CES" CEPName=graceful-term-client CESName=ces-x9xxv5ybq-hsdcg subsys=ces-controller totalCEPCount=0

@krishgobinath

Metadata

Metadata

Assignees

Labels

area/CIContinuous Integration testing issue or flakearea/operatorImpacts the cilium-operator componentkind/bugThis is a bug in the Cilium logic.kind/bug/race-detectorThis is a bug identified by concurrency tests.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions