Skip to content

Avoid using multiple endpoints for etcd cluster #856

Closed

Description

@kvaps

In the sample configuration you recommend exposing etcd over multiple endpoints:

endpoints:
- etcd-0.etcd.kamaji-system.svc.cluster.local:2379
- etcd-1.etcd.kamaji-system.svc.cluster.local:2379
- etcd-2.etcd.kamaji-system.svc.cluster.local:2379

I like the approach, but it clashes with normal Kubernetes behaviour. If one endpoint becomes unavailable- say a node fails and its etcd-replica pod is deleted—the replacement pod can stay in the Pending state without ever getting an IP address.

Because of that, CoreDNS never publishes an A record for the pod in the headless service, and the API server floods the logs with errors (around three per second in my case):

W0627 10:54:32.114258       1 logging.go:55] [core] [Channel #262 SubChannel #265]grpc: addrConn.createTransport failed to connect to {Addr: "etcd-2.etcd-headless.tenant-root.svc:2379", ServerName: "etcd-2.etcd-headless.tenant-root.svc:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup etcd-2.etcd-headless.tenant-root.svc on 10.96.0.10:53: no such host"
W0627 10:54:33.877005       1 logging.go:55] [core] [Channel #316 SubChannel #319]grpc: addrConn.createTransport failed to connect to {Addr: "etcd-2.etcd-headless.tenant-root.svc:2379", ServerName: "etcd-2.etcd-headless.tenant-root.svc:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup etcd-2.etcd-headless.tenant-root.svc on 10.96.0.10:53: no such host"
W0627 10:54:34.184994       1 logging.go:55] [core] [Channel #449 SubChannel #452]grpc: addrConn.createTransport failed to connect to {Addr: "etcd-2.etcd-headless.tenant-root.svc:2379", ServerName: "etcd-2.etcd-headless.tenant-root.svc:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup etcd-2.etcd-headless.tenant-root.svc on 10.96.0.10:53: no such host"

As a workaround in Cozystack we decided to drop the headless service and switch to a regular ones for accessing etcd.

Activity

  1. prometherion commented on Jun 27, 2025

    @prometherion
    Member

    As a workaround in Cozystack we decided to drop the headless service and switch to a regular ones for accessing etcd

    I'm a bit confused by this sentence: what do you mean, exactly?

    Found the reference.

    So far, anybody had any issue with the etcd: meaning we always thrived to keep etcd up and running. We could document this in our official documentation to shed more light on this specific "bug" report: @bsctl what do you think?

    TL;DR; I'd keep the current endpoints, and enhance documentation just in case.

  2. self-assigned this
    on Jun 30, 2025
  3. bsctl commented on Jul 20, 2025

    @bsctl
    Member

    @kvaps we addressed this with kamaji-etcd so you can switch between headless service (default) to a regular ClusterIP service. The sample manifest in Kamaji can stay with the headless service as it is just, an example of supported configuration.

    apiVersion: kamaji.clastix.io/v1alpha1
    kind: DataStore
    metadata:
      name: sample-etcd
    spec:
      driver: etcd
      endpoints:
        - sample-etcd-client.default.svc.cluster.local:2379
      tlsConfig:
        certificateAuthority:
          certificate:
            secretReference:
              keyPath: ca.crt
              name: sample-etcd-certs
              namespace: default
          privateKey:
            secretReference:
              keyPath: ca.key
              name: sample-etcd-certs
              namespace: default
        clientCertificate:
          certificate:
            secretReference:
              keyPath: tls.crt
              name: sample-etcd-root-client-certs
              namespace: default
          privateKey:
            secretReference:
              keyPath: tls.key
              name: sample-etcd-root-client-certs
              namespace: default

    I think we can close the issue. Please, let me know if you have other concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions