Is there an existing issue for this?
What happened?
Started Cilium with Spire enabled. The Spire server came up and said it's healthy, but no no entries got created for Cilium endpoints. Restarting the Cilium Operator got it working (but this shouldn't really be necessary)
Cilium Version
cilium version
cilium-cli: v0.15.10 compiled with go1.21.2 on linux/arm64
cilium image (default): v1.14.2
cilium image (stable): v1.14.3
cilium image (running): 1.14.2
Kernel Version
uname -a
Linux lima-ubuntu-kind 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:29:36 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Kubernetes Version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.24.0
Sysdump
Sorry, I didn't grab a sysdump before restarting the Cilium Operator
Relevant log output
Slightly suspicious “no identity issued” log from one of the Spire agents:
$ k logs -n cilium-spire spire-agent-v4jlx
Defaulted container "spire-agent" out of: spire-agent, init (init)
time="2023-10-17T07:46:11Z" level=error msg="no identity issued" method=SubscribeToX509SVIDs service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api
time="2023-10-17T07:46:11Z" level=error msg="no identity issued" method=SubscribeToX509Bundles service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api
lots of suspicious looking things in cilium-operator logs - a few samples:
level=error msg="Failed to watch the Workload API: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/spire/sockets/agent/agent.sock: connect: no such file or directory\"" subsys=spire-client
...
level=error msg="Unable to connect to SPIRE server, attempt 133" error="failed to create X509 source: context deadline exceeded" subsys=spire-client
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=13385 subsys=auth-identity
...
level=error msg="Failed to watch the Workload API: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/spire/sockets/agent/agent.sock: connect: no such file or directory\"" subsys=spire-client
level=error msg="Unable to connect to SPIRE server, attempt 134" error="failed to create X509 source: context deadline exceeded" subsys=spire-client
...
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=45106 subsys=auth-identity
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=20849 subsys=auth-identity
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=14734 subsys=auth-identity
...
level=error msg="Deleting unused identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity="&{{ } {2052 abae3759-a107-4e33-842d-a0e673214d73 52648 1 2023-10-17 07:44:26 +0000 UTC <nil> <nil> map[app.kubernetes.io/name:tiefighter class:tiefighter io.cilium.k8s.policy.cluster:kind-kind io.cilium.k8s.policy.serviceaccount:default io.kubernetes.pod.namespace:farfaraway org:empire] map[io.cilium.heartbeat:2023-10-17T08:23:45.096303441Z] [] [] [{cilium-agent Update cilium.io/v2 2023-10-17 07:44:26 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:app.kubernetes.io/name\":{},\"f:class\":{},\"f:io.cilium.k8s.policy.cluster\":{},\"f:io.cilium.k8s.policy.serviceaccount\":{},\"f:io.kubernetes.pod.namespace\":{},\"f:org\":{}}},\"f:security-labels\":{\".\":{},\"f:k8s:app.kubernetes.io/name\":{},\"f:k8s:class\":{},\"f:k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name\":{},\"f:k8s:io.cilium.k8s.policy.cluster\":{},\"f:k8s:io.cilium.k8s.policy.serviceaccount\":{},\"f:k8s:io.kubernetes.pod.namespace\":{},\"f:k8s:org\":{}}} } {cilium-operator-generic Update cilium.io/v2 2023-10-17 08:23:45 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:io.cilium.heartbeat\":{}}}} }]} map[k8s:app.kubernetes.io/name:tiefighter k8s:class:tiefighter k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name:farfaraway k8s:io.cilium.k8s.policy.cluster:kind-kind k8s:io.cilium.k8s.policy.serviceaccount:default k8s:io.kubernetes.pod.namespace:farfaraway k8s:org:empire]}" subsys=identity-heartbeat
Anything else?
No response
Code of Conduct
Is there an existing issue for this?
What happened?
Started Cilium with Spire enabled. The Spire server came up and said it's healthy, but no no entries got created for Cilium endpoints. Restarting the Cilium Operator got it working (but this shouldn't really be necessary)
Cilium Version
Kernel Version
Kubernetes Version
Sysdump
Sorry, I didn't grab a sysdump before restarting the Cilium Operator
Relevant log output
Anything else?
No response
Code of Conduct