K8sDatapathLRPTests Checks local redirect policy
LRP connectivity
/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:515
22:30:47 STEP: Running BeforeAll block for EntireTestsuite K8sDatapathLRPTests
22:30:48 STEP: Ensuring the namespace kube-system exists
22:30:48 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
22:30:48 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
22:30:49 STEP: Running BeforeAll block for EntireTestsuite K8sDatapathLRPTests Checks local redirect policy
22:30:49 STEP: Installing Cilium
22:30:56 STEP: Waiting for Cilium to become ready
22:31:25 STEP: Validating if Kubernetes DNS is deployed
22:31:25 STEP: Checking if deployment is ready
22:31:26 STEP: Checking if kube-dns service is plumbed correctly
22:31:26 STEP: Checking if pods have identity
22:31:26 STEP: Checking if DNS can resolve
22:31:30 STEP: Kubernetes DNS is not ready: %!s(<nil>)
22:31:30 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
22:31:31 STEP: Waiting for Kubernetes DNS to become operational
22:31:31 STEP: Checking if deployment is ready
22:31:32 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:32 STEP: Checking if deployment is ready
22:31:33 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:33 STEP: Checking if deployment is ready
22:31:34 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:34 STEP: Checking if deployment is ready
22:31:35 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:35 STEP: Checking if deployment is ready
22:31:36 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:36 STEP: Checking if deployment is ready
22:31:37 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:37 STEP: Checking if deployment is ready
22:31:38 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:38 STEP: Checking if deployment is ready
22:31:39 STEP: Kubernetes DNS is not ready yet: only 0 of 2 replicas are available
22:31:39 STEP: Checking if deployment is ready
22:31:40 STEP: Kubernetes DNS is not ready yet: only 1 of 2 replicas are available
22:31:40 STEP: Checking if deployment is ready
22:31:41 STEP: Checking if kube-dns service is plumbed correctly
22:31:41 STEP: Checking if pods have identity
22:31:41 STEP: Checking if DNS can resolve
22:31:42 STEP: Validating Cilium Installation
22:31:42 STEP: Performing Cilium controllers preflight check
22:31:42 STEP: Performing Cilium health check
22:31:42 STEP: Checking whether host EP regenerated
22:31:42 STEP: Performing Cilium status preflight check
22:31:44 STEP: Performing Cilium service preflight check
22:31:44 STEP: Performing K8s service preflight check
22:31:45 STEP: Waiting for cilium-operator to be ready
22:31:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
22:31:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
22:31:46 STEP: WaitforPods(namespace="default", filter="-l role=frontend")
22:31:50 STEP: WaitforPods(namespace="default", filter="-l role=frontend") => <nil>
22:31:50 STEP: WaitforPods(namespace="default", filter="-l role=backend")
22:31:51 STEP: WaitforPods(namespace="default", filter="-l role=backend") => <nil>
22:31:51 STEP: WaitforPods(namespace="default", filter="-l role=lrpAddrBackend")
22:31:51 STEP: WaitforPods(namespace="default", filter="-l role=lrpAddrBackend") => <nil>
22:31:53 STEP: Checking traffic goes to local backend
Pod Ingress Egress
grafana-6f4755f98c-vlfvn
prometheus-67fdcf4796-z6d56
k8s1-backend
k8s1-client
k8s2-backend
k8s2-client
coredns-76f75df574-8pvx8
coredns-76f75df574-x8ls2
Cilium agent 'cilium-64t8x': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 29 Failed 0
Cilium agent 'cilium-kq54f': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 51 Failed 0
</Checks>
• Failure in Spec Teardown (AfterEach) [136.897 seconds]
K8sDatapathLRPTests
/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:461
Checks local redirect policy
/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:461
LRP connectivity [AfterEach]
/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:515
Found 6 k8s-app=cilium logs matching list of errors that must be investigated:
2024-03-04T22:31:37.194947916Z level=warning msg="Regeneration of endpoint failed" bpfCompilation=14.631117962s bpfLoadProg=515.994526ms bpfWaitForELF=14.631543195s bpfWriteELF=14.744104ms ciliumEndpointName=kube-system/coredns-76f75df574-85k62 containerID=24fea10bf3 containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3173 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=686 ipv4=10.0.0.237 ipv6="fd02::d5" k8sPodName=kube-system/coredns-76f75df574-85k62 mapSync="121.004µs" policyCalculation=1.724529ms prepareBuild=10.524056ms proxyConfiguration="10.248µs" proxyPolicyCalculation="774.945µs" proxyWaitForAck="4.448µs" reason="syncing state to host" subsys=endpoint total=15.210727697s waitingForCTClean=552ns waitingForLock="45.164µs"
2024-03-04T22:31:37.195980470Z level=error msg="endpoint regeneration failed" ciliumEndpointName=kube-system/coredns-76f75df574-85k62 containerID=24fea10bf3 containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3173 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=686 ipv4=10.0.0.237 ipv6="fd02::d5" k8sPodName=kube-system/coredns-76f75df574-85k62 subsys=endpoint
2024-03-04T22:31:37.196506781Z level=warning msg="could not set degraded status on subreporter \"9-datapath-regenerate\"" error="reporter 9-datapath-regenerate has been stopped" subsys=hive
2024-03-04T22:31:37.381[298](https://github.com/cilium/cilium/actions/runs/8147153559/job/22269541305#step:16:300)791Z level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=704.95865ms bpfWaitForELF=14.621677359s bpfWriteELF=11.414126ms ciliumEndpointName=kube-system/coredns-76f75df574-qklfc containerID=907fa13179 containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=1037 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=686 ipv4=10.0.0.58 ipv6="fd02::f7" k8sPodName=kube-system/coredns-76f75df574-qklfc mapSync="91.58µs" policyCalculation=2.392209ms prepareBuild=21.725394ms proxyConfiguration="9.698µs" proxyPolicyCalculation="496.[304](https://github.com/cilium/cilium/actions/runs/8147153559/job/22269541305#step:16:306)µs" proxyWaitForAck="5.731µs" reason="syncing state to host" subsys=endpoint total=15.39751845s waitingForCTClean=461ns waitingForLock="1.642µs"
2024-03-04T22:31:37.381371136Z level=error msg="endpoint regeneration failed" ciliumEndpointName=kube-system/coredns-76f75df574-qklfc containerID=907fa1[317](https://github.com/cilium/cilium/actions/runs/8147153559/job/22269541305#step:16:319)9 containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=1037 error="Error while configuring proxy redirects: context cancelled before waiting for proxy updates: context canceled" identity=686 ipv4=10.0.0.58 ipv6="fd02::f7" k8sPodName=kube-system/coredns-76f75df574-qklfc subsys=endpoint
2024-03-04T22:31:37.381373971Z level=warning msg="could not set degraded status on subreporter \"10-datapath-regenerate\"" error="reporter 10-datapath-regenerate has been stopped" subsys=hive
/home/runner/work/cilium/cilium/test/ginkgo-ext/scopes.go:413
Looks like there's a race condition where endpoint regeneration was canceled somewhere during the test and this generated error/warning logs.
CI failure
PR failure: https://github.com/cilium/cilium/actions/runs/8147153559/job/22269541305
cilium-sysdumps (20).zip
Looks like there's a race condition where endpoint regeneration was canceled somewhere during the test and this generated error/warning logs.