Skip to content

Commit 6a32349

Browse files
authored
docs(networking): publish Kubernetes API endpoint via external-dns with kuberture (#539)
2 parents dbd5a1d + 9597dc6 commit 6a32349

1 file changed

Lines changed: 157 additions & 0 deletions

File tree

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
---
2+
title: "Publishing the Kubernetes API endpoint via external-dns"
3+
linkTitle: "Kubernetes API DNS publication"
4+
description: "Translate the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS, with the bundled kuberture component."
5+
weight: 26
6+
---
7+
8+
## What this page covers
9+
10+
How to publish the cluster's own Kubernetes API endpoint as a DNS record using `external-dns`, why the obvious approach (point `external-dns` directly at the `default/kubernetes` EndpointSlice) does not work, the cozystack-bundled bridge (`kuberture`, see [`lexfrei/kuberture`](https://github.com/lexfrei/kuberture)), how a single `kuberture` install can serve more than one `external-dns` instance simultaneously, and the disable path.
11+
12+
## Why external-dns cannot read EndpointSlice directly
13+
14+
`external-dns` reads DNS-relevant state from Kubernetes sources — `service`, `ingress`, `gateway-httproute`, and a handful of others. It does **not** support `EndpointSlice` as a source. The control-plane EndpointSlice that Kubernetes maintains at `default/kubernetes` is the canonical, always-up-to-date list of node addresses serving the API; it is also the only first-class object that carries that information without operator-supplied labels or selectors. Operators who want their cluster's API endpoint published to public DNS through `external-dns` (cert-manager `dns01` challenges, internal automation reaching the API by FQDN, multi-cluster service discovery, …) hit this gap immediately.
15+
16+
The workarounds usually proposed — a hand-maintained `Service` of type `ExternalName`, a hand-maintained headless `Service` with manually-pinned `Endpoints`, an out-of-band script that polls the API and updates a record — all share the same flaw: they stop matching reality as soon as a control-plane node is added, removed, or renumbered. The EndpointSlice already tracks this state correctly. `kuberture` bridges the two.
17+
18+
## How kuberture fixes it
19+
20+
`kuberture` is a small in-cluster controller that watches `EndpointSlice` and writes annotated headless `Service` objects. The flow is:
21+
22+
```text
23+
EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS
24+
```
25+
26+
For each operator-configured output, `kuberture` creates a headless Service (`spec.clusterIP: None`) in its own namespace (`cozy-kuberture`) and stamps three annotations on it. Each key is the operator-supplied `annotationPrefix` followed by a fixed suffix:
27+
28+
| Full annotation key | Source |
29+
| --- | --- |
30+
| `<annotationPrefix>hostname` | the operator-supplied hostname(s) for this output |
31+
| `<annotationPrefix>target` | comma-joined IP addresses resolved from the EndpointSlice or from the Nodes it points at |
32+
| `<annotationPrefix>ttl` | the operator-supplied `recordTTL`, falling back to the controller default when omitted |
33+
34+
With the default prefix that resolves to `external-dns.alpha.kubernetes.io/hostname`, `external-dns.alpha.kubernetes.io/target`, `external-dns.alpha.kubernetes.io/ttl` — the keys the platform `external-dns` reads out of the box. `external-dns` reads the Service, sees the annotations, and creates the DNS record. The Service has no selector and no manually-managed Endpoints — it exists solely to carry annotations for `external-dns` consumption.
35+
36+
The controller does **not** write `EndpointSlice` or `Endpoints` objects (it only has read permission on `EndpointSlice`), and it does **not** touch the upstream `default/kubernetes` EndpointSlice. It is strictly additive: a separate Service per output, in its own namespace, with no side effects on existing cluster state.
37+
38+
## Enabling on the host
39+
40+
`kuberture` is an optional system package, opt-in via `bundles.enabledPackages`. The package itself ships no usable default beyond enabling `ServiceMonitor` for the platform Prometheus — the deployment fails fast if `config.outputs` is empty. The operator must declare at least one output, because the only thing `kuberture` cannot infer is the DNS hostname the operator actually wants to publish.
41+
42+
Single output, consumed by the platform-level `external-dns`:
43+
44+
```yaml
45+
apiVersion: cozystack.io/v1alpha1
46+
kind: Package
47+
metadata:
48+
name: cozystack.cozystack-platform
49+
spec:
50+
variant: isp-full
51+
components:
52+
platform:
53+
values:
54+
bundles:
55+
enabledPackages:
56+
- cozystack.external-dns
57+
- cozystack.kuberture
58+
---
59+
apiVersion: cozystack.io/v1alpha1
60+
kind: Package
61+
metadata:
62+
name: cozystack.kuberture
63+
spec:
64+
variant: default
65+
components:
66+
kuberture:
67+
values:
68+
kuberture:
69+
config:
70+
outputs:
71+
- name: api
72+
hostname:
73+
- api.k8s.example.com
74+
serviceName: kuberture-api
75+
addressSource: endpointslice
76+
recordTTL: 60
77+
```
78+
79+
`annotationPrefix` is omitted here, so the controller uses the default `external-dns.alpha.kubernetes.io/` — the prefix the platform `external-dns` watches out of the box. With both packages enabled, the `external-dns` HelmRelease picks up the `kuberture-api` Service in `cozy-kuberture` and creates the `api.k8s.example.com` record in the configured provider.
80+
81+
The `cozystack.external-dns` Package is the cluster-wide system instance (it runs with `namespaced: false` and watches Services across the cluster). `cozystack.external-dns-application` is the tenant-namespaced variant and will not pick up Services outside its tenant namespace — pair `kuberture` with `cozystack.external-dns`, not `cozystack.external-dns-application`, on the host control plane.
82+
83+
## Routing to multiple external-dns instances
84+
85+
A single `kuberture` install can address any number of `external-dns` instances by varying `annotationPrefix` per output. Each `external-dns` instance is started with `--annotation-prefix=<your-prefix>/` so it rebuilds every `hostname`/`target`/`ttl` annotation key under that prefix and ignores everything else; `kuberture` stamps the matching prefix on each Service. This is the upstream-documented [Split Horizon DNS pattern](https://kubernetes-sigs.github.io/external-dns/v0.20.0/docs/advanced/split-horizon/) — distinct from `--annotation-filter`, which is a Kubernetes label-selector that filters *which Services* an instance considers, not which prefix it reads data from.
86+
87+
```yaml
88+
apiVersion: cozystack.io/v1alpha1
89+
kind: Package
90+
metadata:
91+
name: cozystack.kuberture
92+
spec:
93+
variant: default
94+
components:
95+
kuberture:
96+
values:
97+
kuberture:
98+
config:
99+
outputs:
100+
- name: public
101+
hostname:
102+
- api.k8s.example.com
103+
serviceName: kuberture-public
104+
addressSource: endpointslice
105+
- name: internal
106+
hostname:
107+
- api-internal.k8s.example.internal
108+
annotationPrefix: internal-dns.example.com/
109+
serviceName: kuberture-internal
110+
addressSource: endpointslice
111+
recordTTL: 300
112+
```
113+
114+
This renders two headless Services in `cozy-kuberture`:
115+
116+
- `kuberture-public` carries the default `external-dns.alpha.kubernetes.io/*` annotations and is consumed by the platform `external-dns`.
117+
- `kuberture-internal` carries `internal-dns.example.com/*` annotations and is invisible to the platform `external-dns` (which is reading the default prefix). A second `external-dns` instance started with `--annotation-prefix=internal-dns.example.com/` consumes it: the prefix flag tells that instance to look up `hostname`/`target`/`ttl` under `internal-dns.example.com/*`, so it sees `kuberture-internal` and is blind to `kuberture-public`.
118+
119+
Each Service carries **only** its own prefix's annotations — there is no cross-pollution between outputs.
120+
121+
`annotationPrefix` accepts two forms: omit the field to inherit the controller default `external-dns.alpha.kubernetes.io/`, or set it to a non-empty string ending in `/`. The empty string `""` is rejected by the chart's values schema; omission is the only way to fall back to the default prefix.
122+
123+
## Address resolution strategies
124+
125+
`addressSource` selects where each output gets its target IPs:
126+
127+
| `addressSource` | What `kuberture` publishes |
128+
| --- | --- |
129+
| `endpointslice` (default) | The addresses listed in the `default/kubernetes` EndpointSlice verbatim. Use this when the EndpointSlice IPs are the addresses you want in DNS. |
130+
| `node-internal` | The `InternalIP` of each Node hosting an EndpointSlice endpoint. Use this when the EndpointSlice carries pod-network IPs but external-dns must publish node-internal addresses. |
131+
| `node-external` | The `ExternalIP` of each Node. Use this for cloud-managed clusters where Nodes have public addresses on a different interface than the API listens on. |
132+
| `node-public` | A node's public IP looked up through a public-IP discovery hook (provider-specific). |
133+
134+
`addressType` (`IPv4` / `IPv6`) further filters the resolved set; default is `IPv4`.
135+
136+
## Disable path
137+
138+
Removing `cozystack.kuberture` from `bundles.enabledPackages` stops the platform Helm chart from emitting the `Package` CR. As with every optional system package, the existing `Package` CR is **not** removed automatically: the platform helper stamps `helm.sh/resource-policy: keep` on each emitted Package, so Helm/Flux leaves it in place when it stops being rendered. To fully remove `kuberture`:
139+
140+
```bash
141+
kubectl delete package.cozystack.io cozystack.kuberture
142+
```
143+
144+
The cascade removes the HelmRelease in `cozy-kuberture`, which removes the Deployment, the operator-supplied output Services, and (eventually) the namespace. `external-dns` records previously published from those Services follow `external-dns`'s own delete policy (`policy: upsert-only` is the cozystack default, which means records are **not** retracted on Service deletion; switch to `policy: sync` ahead of time if records must be cleaned up automatically).
145+
146+
## Supply-chain notes
147+
148+
The chart is pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and the controller image from `ghcr.io/lexfrei/kuberture`. Both live in the maintainer's personal namespace and are intentionally not mirrored under `ghcr.io/cozystack/*`. Air-gapped operators must mirror both the chart and the controller image into their internal registry and override `kuberture.image.repository` in the Package values. Chart version and OCI manifest digest are pinned in the cozystack package Makefile; the controller image tag and image digest are pinned in the package's `values.yaml`. Pins advance in lockstep with each upstream release.
149+
150+
See the package README at [`packages/system/kuberture/`](https://github.com/cozystack/cozystack/tree/main/packages/system/kuberture) for the full values surface, the bump procedure, and the `NetworkPolicy` default-off choice.
151+
152+
## See also
153+
154+
- [Enabling and disabling components]({{% ref "/docs/next/operations/configuration/components#enabling-and-disabling-components" %}}) — the `bundles.enabledPackages` / `bundles.disabledPackages` mechanism used here.
155+
- [Platform Package reference]({{% ref "/docs/next/operations/configuration/platform-package" %}}) — `bundles.enabledPackages` field reference.
156+
- [`lexfrei/kuberture`](https://github.com/lexfrei/kuberture) — upstream controller source and chart, configuration reference, and roadmap.
157+
- [`packages/system/kuberture/`](https://github.com/cozystack/cozystack/tree/main/packages/system/kuberture) — cozystack package: values shim, helm-unittest fixtures, end-to-end smoke test.

0 commit comments

Comments
 (0)