Skip to content

CI: Cilium Cluster Mesh upgrade (ci-clustermesh) - check-log-errors - Error while reloading endpoint BPF program #39855

@pippolo84

Description

@pippolo84

https://github.com/cilium/cilium/actions/runs/15324471667/job/43116069181

[=] [cilium-test-1] Test [check-log-errors] [123/123]
.........................
  [-] Scenario [check-log-errors/no-errors-in-logs]
  ...
  [.] Action [check-log-errors/no-errors-in-logs:kind-cluster1/kube-system/cilium-lcnfc (cilium-agent)]
  ❌ Found 6 logs in kind-cluster1/kube-system/cilium-lcnfc (cilium-agent) matching list of errors that must be investigated:
time=2025-05-29T13:21:21Z level=error source=/go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:609 msg="Error while reloading endpoint BPF program" k8sPodName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b ipv6=fd00:10:242::7a82 endpointID=2695 datapathPolicyRevision=0 ipv4=10.242.0.130 desiredPolicyRevision=2 identity=51353 containerInterface=eth0 containerID=0fc29ed3ea ciliumEndpointName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b subsys=endpoint error="retrieving device lxc5639a951e2f9: Link not found" (7 occurrences)
time=2025-05-29T13:21:21Z level=warn source=/go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:607 msg="Regeneration of endpoint failed" k8sPodName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b ipv6=fd00:10:242::7a82 endpointID=2695 datapathPolicyRevision=0 ipv4=10.242.0.130 desiredPolicyRevision=2 identity=51353 containerInterface=eth0 containerID=0fc29ed3ea ciliumEndpointName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b subsys=endpoint reason="retrying regeneration" policyCalculation=35.507µs total=132.84[233](https://github.com/cilium/cilium/actions/runs/15324471667/job/43116069181#step:40:234)2ms waitingForCTClean=350ns proxyPolicyCalculation=0s mapSync=0s bpfCompilation=0s bpfLoadProg=130.460262ms waitingForLock=1.864µs proxyConfiguration=0s selectorPolicyCalculation=0s endpointPolicyCalculation=0s proxyWaitForAck=0s prepareBuild=52.027µs bpfWaitForELF=1.195123ms waitingForPolicyRepository=311ns bpfWaitForELF=1.195123ms bpfLoadProg=130.460262ms bpfCompilation=0s error="retrieving device lxc5639a951e2f9: Link not found" (7 occurrences)
time=2025-05-29T13:21:21Z level=error source=/go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:803 msg="endpoint regeneration failed" k8sPodName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b ipv6=fd00:10:242::7a82 endpointID=2695 datapathPolicyRevision=0 ipv4=10.242.0.130 desiredPolicyRevision=2 identity=51353 containerInterface=eth0 containerID=0fc29ed3ea ciliumEndpointName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b subsys=endpoint error="retrieving device lxc5639a951e2f9: Link not found" (7 occurrences)
time=2025-05-29T13:21:12Z level=error source=/go/src/github.com/cilium/cilium/pkg/endpointmanager/endpointsynchronizer.go:309 msg="Cannot update CEP" k8sPodName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b ipv6=fd00:10:242::7a82 endpointID=2695 datapathPolicyRevision=0 ipv4=10.242.0.130 desiredPolicyRevision=2 identity=51353 containerInterface=eth0 containerID=0fc29ed3ea ciliumEndpointName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b subsys=endpointsynchronizer controller="sync-to-k8s-ciliumendpoint (2695)" error="ciliumendpoints.cilium.io \"clustermesh-apiserver-7c5c4b5b54-nvq2b\" not found" (1 occurrences)
time=2025-05-29T13:21:22Z level=warn source=/go/src/github.com/cilium/cilium/pkg/endpoint/watchdog/ep-bpfprog-watchdog.go:102 msg="Unable to assert if endpoint BPF programs need to be reloaded" module=agent.controlplane.ep-bpf-prog-watchdog endpoint=lxc5639a951e2f9 endpointID=2695 ciliumEndpointName=kube-system/clustermesh-apiserver-7c5c4b5b54-nvq2b error="retrieving device lxc5639a951e2f9: Link not found" (1 occurrences)
time=2025-05-29T13:21:22Z level=error source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:203 msg="Timer job errored" module=agent.controlplane.ep-bpf-prog-watchdog name=ep-bpf-prog-watchdog func="watchdog.registerEndpointBPFProgWatchdog.func1 (.../endpoint/watchdog/ep-bpfprog-watchdog.go:71)" error="failed to assert if endpoint BPF programs need to be reloaded: retrieving device lxc5639a951e2f9: Link not found" (1 occurrences)

📋 Test Report [cilium-test-1]
❌ 1/8 tests failed (1/167 actions), 115 tests skipped, 3 scenarios skipped:
Test [check-log-errors]:
  ❌ check-log-errors/no-errors-in-logs:pkg/endpoint:kind-cluster1/kube-system/cilium-lcnfc (cilium-agent)
    ⛑️ The following owners are responsible for reliability of the testsuite: 
        - @cilium/endpoint (no-errors-in-logs:pkg/endpoint)
        - @cilium/sig-clustermesh (.github/workflows/tests-clustermesh-upgrade.yaml)
        - @cilium/ci-structure (.github/workflows/tests-clustermesh-upgrade.yaml)
[cilium-test-1] 1 tests failed

Logs: logs_39311580068.zip
Features Tested: features-tested.zip
Sysdump:

Metadata

Metadata

Assignees

Labels

affects/v1.18This issue affects v1.18 brancharea/CIContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!

Type

No type

Projects

Status

Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions