Repository navigation
Closed
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation
Description
In the sample configuration you recommend exposing etcd over multiple endpoints:
kamaji/config/samples/kamaji_v1alpha1_datastore_etcd.yaml
Lines 7 to 10 in d40996d
| endpoints: | |
| - etcd-0.etcd.kamaji-system.svc.cluster.local:2379 | |
| - etcd-1.etcd.kamaji-system.svc.cluster.local:2379 | |
| - etcd-2.etcd.kamaji-system.svc.cluster.local:2379 |
I like the approach, but it clashes with normal Kubernetes behaviour. If one endpoint becomes unavailable- say a node fails and its etcd-replica pod is deleted—the replacement pod can stay in the Pending state without ever getting an IP address.
Because of that, CoreDNS never publishes an A record for the pod in the headless service, and the API server floods the logs with errors (around three per second in my case):
W0627 10:54:32.114258 1 logging.go:55] [core] [Channel #262 SubChannel #265]grpc: addrConn.createTransport failed to connect to {Addr: "etcd-2.etcd-headless.tenant-root.svc:2379", ServerName: "etcd-2.etcd-headless.tenant-root.svc:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup etcd-2.etcd-headless.tenant-root.svc on 10.96.0.10:53: no such host"
W0627 10:54:33.877005 1 logging.go:55] [core] [Channel #316 SubChannel #319]grpc: addrConn.createTransport failed to connect to {Addr: "etcd-2.etcd-headless.tenant-root.svc:2379", ServerName: "etcd-2.etcd-headless.tenant-root.svc:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup etcd-2.etcd-headless.tenant-root.svc on 10.96.0.10:53: no such host"
W0627 10:54:34.184994 1 logging.go:55] [core] [Channel #449 SubChannel #452]grpc: addrConn.createTransport failed to connect to {Addr: "etcd-2.etcd-headless.tenant-root.svc:2379", ServerName: "etcd-2.etcd-headless.tenant-root.svc:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup etcd-2.etcd-headless.tenant-root.svc on 10.96.0.10:53: no such host"
As a workaround in Cozystack we decided to drop the headless service and switch to a regular ones for accessing etcd.
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation
Activity
prometherion commentedon Jun 27, 2025
I'm a bit confused by this sentence: what do you mean, exactly?Found the reference.
So far, anybody had any issue with the etcd: meaning we always thrived to keep etcd up and running. We could document this in our official documentation to shed more light on this specific "bug" report: @bsctl what do you think?
TL;DR; I'd keep the current endpoints, and enhance documentation just in case.
[etcd] Update etcd application (fix resources and headless services) (#…
bsctl commentedon Jul 20, 2025
@kvaps we addressed this with kamaji-etcd so you can switch between headless service (default) to a regular ClusterIP service. The sample manifest in Kamaji can stay with the headless service as it is just, an example of supported configuration.
I think we can close the issue. Please, let me know if you have other concerns.