Skip to content

Cilium Operator failing to connect to SPIRE server  #28694

@lizrice

Description

@lizrice

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Started Cilium with Spire enabled. The Spire server came up and said it's healthy, but no no entries got created for Cilium endpoints. Restarting the Cilium Operator got it working (but this shouldn't really be necessary)

Cilium Version

cilium version 
cilium-cli: v0.15.10 compiled with go1.21.2 on linux/arm64
cilium image (default): v1.14.2
cilium image (stable): v1.14.3
cilium image (running): 1.14.2

Kernel Version

uname -a
Linux lima-ubuntu-kind 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:29:36 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Kubernetes Version

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.24.0

Sysdump

Sorry, I didn't grab a sysdump before restarting the Cilium Operator

Relevant log output

Slightly suspicious “no identity issued” log from one of the Spire agents:

$ k logs -n cilium-spire spire-agent-v4jlx
Defaulted container "spire-agent" out of: spire-agent, init (init)
time="2023-10-17T07:46:11Z" level=error msg="no identity issued" method=SubscribeToX509SVIDs service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api
time="2023-10-17T07:46:11Z" level=error msg="no identity issued" method=SubscribeToX509Bundles service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api

lots of suspicious looking things in cilium-operator logs - a few samples:

level=error msg="Failed to watch the Workload API: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/spire/sockets/agent/agent.sock: connect: no such file or directory\"" subsys=spire-client
...
level=error msg="Unable to connect to SPIRE server, attempt 133" error="failed to create X509 source: context deadline exceeded" subsys=spire-client
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=13385 subsys=auth-identity
...
level=error msg="Failed to watch the Workload API: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/spire/sockets/agent/agent.sock: connect: no such file or directory\"" subsys=spire-client
level=error msg="Unable to connect to SPIRE server, attempt 134" error="failed to create X509 source: context deadline exceeded" subsys=spire-client
...
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=45106 subsys=auth-identity
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=20849 subsys=auth-identity
level=info msg="Upsert identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity=14734 subsys=auth-identity
...
level=error msg="Deleting unused identity" error="unable to connect to SPIRE server spire-server.cilium-spire.svc:8081" identity="&{{ } {2052    abae3759-a107-4e33-842d-a0e673214d73 52648 1 2023-10-17 07:44:26 +0000 UTC <nil> <nil> map[app.kubernetes.io/name:tiefighter class:tiefighter io.cilium.k8s.policy.cluster:kind-kind io.cilium.k8s.policy.serviceaccount:default io.kubernetes.pod.namespace:farfaraway org:empire] map[io.cilium.heartbeat:2023-10-17T08:23:45.096303441Z] [] [] [{cilium-agent Update cilium.io/v2 2023-10-17 07:44:26 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:app.kubernetes.io/name\":{},\"f:class\":{},\"f:io.cilium.k8s.policy.cluster\":{},\"f:io.cilium.k8s.policy.serviceaccount\":{},\"f:io.kubernetes.pod.namespace\":{},\"f:org\":{}}},\"f:security-labels\":{\".\":{},\"f:k8s:app.kubernetes.io/name\":{},\"f:k8s:class\":{},\"f:k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name\":{},\"f:k8s:io.cilium.k8s.policy.cluster\":{},\"f:k8s:io.cilium.k8s.policy.serviceaccount\":{},\"f:k8s:io.kubernetes.pod.namespace\":{},\"f:k8s:org\":{}}} } {cilium-operator-generic Update cilium.io/v2 2023-10-17 08:23:45 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:io.cilium.heartbeat\":{}}}} }]} map[k8s:app.kubernetes.io/name:tiefighter k8s:class:tiefighter k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name:farfaraway k8s:io.cilium.k8s.policy.cluster:kind-kind k8s:io.cilium.k8s.policy.serviceaccount:default k8s:io.kubernetes.pod.namespace:farfaraway k8s:org:empire]}" subsys=identity-heartbeat

Anything else?

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions