-
Notifications
You must be signed in to change notification settings - Fork 3.7k
CI: Data race in (*cesManagerIdentity).updateCESInCache() #17914
Copy link
Copy link
Closed
Labels
area/CIContinuous Integration testing issue or flakeContinuous Integration testing issue or flakearea/operatorImpacts the cilium-operator componentImpacts the cilium-operator componentkind/bugThis is a bug in the Cilium logic.This is a bug in the Cilium logic.kind/bug/race-detectorThis is a bug identified by concurrency tests.This is a bug identified by concurrency tests.
Description
This tripped the race detector on two separate tests:
- https://jenkins.cilium.io/view/Race%20Detection/job/cilium-master-k8s-1.21-kernel-4.19-race-detection/217/
- https://jenkins.cilium.io/view/Race%20Detection/job/cilium-master-k8s-1.22-kernel-4.9-race-detection/209/
Zips:
- 5c2ec42f_K8sServicesTest_Checks_graceful_termination_of_service_endpoints_Checks_client_terminates_gracefully_on_service_endpoint_deletion.zip
- cd2bb6ab_K8sLRPTests_Checks_local_redirect_policy_LRP_connectivity.zip
2021-11-16T16:34:47.847587948Z level=debug msg="Generated CES" CESName=ces-8mh2bjfxv-fp9hs subsys=ces-controller
2021-11-16T16:34:47.847596970Z level=debug msg="Queueing CEP in the CES" CEPName=testclient-lr2zs CESName=ces-8mh2bjfxv-fp9hs subsys=ces-controller totalCEPCount=0
2021-11-16T16:34:47.863964821Z ==================
2021-11-16T16:34:47.863993853Z WARNING: DATA RACE
2021-11-16T16:34:47.863998587Z Write at 0x00c00040ade0 by goroutine 48:
2021-11-16T16:34:47.864002094Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*cesManagerIdentity).updateCESInCache()
2021-11-16T16:34:47.864005638Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/manager.go:589 +0x2c4
2021-11-16T16:34:47.864009057Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*reconciler).reconcileCESCreate()
2021-11-16T16:34:47.864022638Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/reconciler.go:60 +0x44a
2021-11-16T16:34:47.864026550Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).syncCES()
2021-11-16T16:34:47.864029809Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:306 +0x49a
2021-11-16T16:34:47.864033190Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).processNextWorkItem()
2021-11-16T16:34:47.864036482Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:248 +0x175
2021-11-16T16:34:47.864039416Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).worker()
2021-11-16T16:34:47.864042465Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:237 +0x39
2021-11-16T16:34:47.864045542Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).worker-fm()
2021-11-16T16:34:47.864048781Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:236 +0x1d
2021-11-16T16:34:47.864051970Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
2021-11-16T16:34:47.864055123Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x81
2021-11-16T16:34:47.864058305Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
2021-11-16T16:34:47.864061380Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xce
2021-11-16T16:34:47.864064568Z k8s.io/apimachinery/pkg/util/wait.JitterUntil()
2021-11-16T16:34:47.864067628Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x104
2021-11-16T16:34:47.864070955Z k8s.io/apimachinery/pkg/util/wait.Until()
2021-11-16T16:34:47.864073979Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x48
2021-11-16T16:34:47.864077019Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).Run·dwrap·15()
2021-11-16T16:34:47.864082667Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:177 +0x58
2021-11-16T16:34:47.864086384Z
2021-11-16T16:34:47.864089253Z Previous read at 0x00c00040ade0 by goroutine 99:
2021-11-16T16:34:47.864092319Z k8s.io/apimachinery/pkg/apis/meta/v1.(*ObjectMeta).GetName()
2021-11-16T16:34:47.864095571Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/meta.go:139 +0x350
2021-11-16T16:34:47.864098946Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*cesManagerIdentity).InsertCEPInCache()
2021-11-16T16:34:47.864102253Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/manager.go:574 +0x325
2021-11-16T16:34:47.864105458Z github.com/cilium/cilium/operator/watchers.endpointUpdated()
2021-11-16T16:34:47.864113394Z /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:211 +0x129
2021-11-16T16:34:47.864117184Z github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit.func1.1()
2021-11-16T16:34:47.864120558Z /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:86 +0x44
2021-11-16T16:34:47.864123802Z k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd()
2021-11-16T16:34:47.864127037Z /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:231 +0x79
2021-11-16T16:34:47.864130395Z k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd()
2021-11-16T16:34:47.864133786Z <autogenerated>:1 +0x33
2021-11-16T16:34:47.864137702Z github.com/cilium/cilium/pkg/k8s/informer.NewInformerWithStore.func1()
2021-11-16T16:34:47.864140818Z /go/src/github.com/cilium/cilium/pkg/k8s/informer/informer.go:108 +0x2d8
2021-11-16T16:34:47.864143818Z k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop()
2021-11-16T16:34:47.864146911Z /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:539 +0x4fb
2021-11-16T16:34:47.864150632Z k8s.io/client-go/tools/cache.(*controller).processLoop()
2021-11-16T16:34:47.864154020Z /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:183 +0x61
2021-11-16T16:34:47.864157134Z k8s.io/client-go/tools/cache.(*controller).processLoop-fm()
2021-11-16T16:34:47.864159997Z /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:181 +0x39
2021-11-16T16:34:47.864163130Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
2021-11-16T16:34:47.864166322Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x81
2021-11-16T16:34:47.864169736Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
2021-11-16T16:34:47.864172714Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xce
2021-11-16T16:34:47.864175926Z k8s.io/apimachinery/pkg/util/wait.JitterUntil()
2021-11-16T16:34:47.864180068Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x104
2021-11-16T16:34:47.864183555Z k8s.io/apimachinery/pkg/util/wait.Until()
2021-11-16T16:34:47.864186502Z /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x616
2021-11-16T16:34:47.864189644Z k8s.io/client-go/tools/cache.(*controller).Run()
2021-11-16T16:34:47.864192862Z /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:154 +0x5cd
2021-11-16T16:34:47.864195859Z github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit.func1·dwrap·1()
2021-11-16T16:34:47.864199012Z /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:116 +0x59
2021-11-16T16:34:47.864202206Z
2021-11-16T16:34:47.864205184Z Goroutine 48 (running) created at:
2021-11-16T16:34:47.864212584Z github.com/cilium/cilium/operator/pkg/ciliumendpointslice.(*CiliumEndpointSliceController).Run()
2021-11-16T16:34:47.864216129Z /go/src/github.com/cilium/cilium/operator/pkg/ciliumendpointslice/endpointslice.go:177 +0x4be
2021-11-16T16:34:47.864219432Z main.onOperatorStartLeading·dwrap·13()
2021-11-16T16:34:47.864222626Z /go/src/github.com/cilium/cilium/operator/main.go:366 +0x64
2021-11-16T16:34:47.864225842Z
2021-11-16T16:34:47.864228898Z Goroutine 99 (running) created at:
2021-11-16T16:34:47.864231932Z github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit.func1()
2021-11-16T16:34:47.864235198Z /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:116 +0x59a
2021-11-16T16:34:47.864238492Z sync.(*Once).doSlow()
2021-11-16T16:34:47.864241528Z /usr/local/go/src/sync/once.go:68 +0x127
2021-11-16T16:34:47.864244774Z sync.(*Once).Do()
2021-11-16T16:34:47.864247811Z /usr/local/go/src/sync/once.go:59 +0x46
2021-11-16T16:34:47.864250660Z github.com/cilium/cilium/operator/watchers.CiliumEndpointsInit()
2021-11-16T16:34:47.864253765Z /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:76 +0x84
2021-11-16T16:34:47.864256895Z github.com/cilium/cilium/operator/watchers.CiliumEndpointsSliceInit()
2021-11-16T16:34:47.864260051Z /go/src/github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:58 +0x3ba
2021-11-16T16:34:47.864263325Z main.onOperatorStartLeading()
2021-11-16T16:34:47.864266212Z /go/src/github.com/cilium/cilium/operator/main.go:364 +0x311
2021-11-16T16:34:47.864269238Z k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run·dwrap·4()
2021-11-16T16:34:47.864272315Z /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x58
2021-11-16T16:34:47.864275348Z ==================
2021-11-16T16:34:47.884582616Z level=debug msg="Marking identity alive" identity=5103 subsys=identity-heartbeat
2021-11-16T16:34:47.931849205Z level=debug msg="Generated CES" CESName=ces-x9xxv5ybq-hsdcg subsys=ces-controller
2021-11-16T16:34:47.931865624Z level=debug msg="Queueing CEP in the CES" CEPName=graceful-term-client CESName=ces-x9xxv5ybq-hsdcg subsys=ces-controller totalCEPCount=0
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area/CIContinuous Integration testing issue or flakeContinuous Integration testing issue or flakearea/operatorImpacts the cilium-operator componentImpacts the cilium-operator componentkind/bugThis is a bug in the Cilium logic.This is a bug in the Cilium logic.kind/bug/race-detectorThis is a bug identified by concurrency tests.This is a bug identified by concurrency tests.