|
| 1 | +--- |
| 2 | +title: "Publishing the Kubernetes API endpoint via external-dns" |
| 3 | +linkTitle: "Kubernetes API DNS publication" |
| 4 | +description: "Translate the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS, with the bundled kuberture component." |
| 5 | +weight: 26 |
| 6 | +--- |
| 7 | + |
| 8 | +## What this page covers |
| 9 | + |
| 10 | +How to publish the cluster's own Kubernetes API endpoint as a DNS record using `external-dns`, why the obvious approach (point `external-dns` directly at the `default/kubernetes` EndpointSlice) does not work, the cozystack-bundled bridge (`kuberture`, see [`lexfrei/kuberture`](https://github.com/lexfrei/kuberture)), how a single `kuberture` install can serve more than one `external-dns` instance simultaneously, and the disable path. |
| 11 | + |
| 12 | +## Why external-dns cannot read EndpointSlice directly |
| 13 | + |
| 14 | +`external-dns` reads DNS-relevant state from Kubernetes sources — `service`, `ingress`, `gateway-httproute`, and a handful of others. It does **not** support `EndpointSlice` as a source. The control-plane EndpointSlice that Kubernetes maintains at `default/kubernetes` is the canonical, always-up-to-date list of node addresses serving the API; it is also the only first-class object that carries that information without operator-supplied labels or selectors. Operators who want their cluster's API endpoint published to public DNS through `external-dns` (cert-manager `dns01` challenges, internal automation reaching the API by FQDN, multi-cluster service discovery, …) hit this gap immediately. |
| 15 | + |
| 16 | +The workarounds usually proposed — a hand-maintained `Service` of type `ExternalName`, a hand-maintained headless `Service` with manually-pinned `Endpoints`, an out-of-band script that polls the API and updates a record — all share the same flaw: they stop matching reality as soon as a control-plane node is added, removed, or renumbered. The EndpointSlice already tracks this state correctly. `kuberture` bridges the two. |
| 17 | + |
| 18 | +## How kuberture fixes it |
| 19 | + |
| 20 | +`kuberture` is a small in-cluster controller that watches `EndpointSlice` and writes annotated headless `Service` objects. The flow is: |
| 21 | + |
| 22 | +```text |
| 23 | +EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS |
| 24 | +``` |
| 25 | + |
| 26 | +For each operator-configured output, `kuberture` creates a headless Service (`spec.clusterIP: None`) in its own namespace (`cozy-kuberture`) and stamps three annotations on it. Each key is the operator-supplied `annotationPrefix` followed by a fixed suffix: |
| 27 | + |
| 28 | +| Full annotation key | Source | |
| 29 | +| --- | --- | |
| 30 | +| `<annotationPrefix>hostname` | the operator-supplied hostname(s) for this output | |
| 31 | +| `<annotationPrefix>target` | comma-joined IP addresses resolved from the EndpointSlice or from the Nodes it points at | |
| 32 | +| `<annotationPrefix>ttl` | the operator-supplied `recordTTL`, falling back to the controller default when omitted | |
| 33 | + |
| 34 | +With the default prefix that resolves to `external-dns.alpha.kubernetes.io/hostname`, `external-dns.alpha.kubernetes.io/target`, `external-dns.alpha.kubernetes.io/ttl` — the keys the platform `external-dns` reads out of the box. `external-dns` reads the Service, sees the annotations, and creates the DNS record. The Service has no selector and no manually-managed Endpoints — it exists solely to carry annotations for `external-dns` consumption. |
| 35 | + |
| 36 | +The controller does **not** write `EndpointSlice` or `Endpoints` objects (it only has read permission on `EndpointSlice`), and it does **not** touch the upstream `default/kubernetes` EndpointSlice. It is strictly additive: a separate Service per output, in its own namespace, with no side effects on existing cluster state. |
| 37 | + |
| 38 | +## Enabling on the host |
| 39 | + |
| 40 | +`kuberture` is an optional system package, opt-in via `bundles.enabledPackages`. The package itself ships no usable default beyond enabling `ServiceMonitor` for the platform Prometheus — the deployment fails fast if `config.outputs` is empty. The operator must declare at least one output, because the only thing `kuberture` cannot infer is the DNS hostname the operator actually wants to publish. |
| 41 | + |
| 42 | +Single output, consumed by the platform-level `external-dns`: |
| 43 | + |
| 44 | +```yaml |
| 45 | +apiVersion: cozystack.io/v1alpha1 |
| 46 | +kind: Package |
| 47 | +metadata: |
| 48 | + name: cozystack.cozystack-platform |
| 49 | +spec: |
| 50 | + variant: isp-full |
| 51 | + components: |
| 52 | + platform: |
| 53 | + values: |
| 54 | + bundles: |
| 55 | + enabledPackages: |
| 56 | + - cozystack.external-dns |
| 57 | + - cozystack.kuberture |
| 58 | +--- |
| 59 | +apiVersion: cozystack.io/v1alpha1 |
| 60 | +kind: Package |
| 61 | +metadata: |
| 62 | + name: cozystack.kuberture |
| 63 | +spec: |
| 64 | + variant: default |
| 65 | + components: |
| 66 | + kuberture: |
| 67 | + values: |
| 68 | + kuberture: |
| 69 | + config: |
| 70 | + outputs: |
| 71 | + - name: api |
| 72 | + hostname: |
| 73 | + - api.k8s.example.com |
| 74 | + serviceName: kuberture-api |
| 75 | + addressSource: endpointslice |
| 76 | + recordTTL: 60 |
| 77 | +``` |
| 78 | +
|
| 79 | +`annotationPrefix` is omitted here, so the controller uses the default `external-dns.alpha.kubernetes.io/` — the prefix the platform `external-dns` watches out of the box. With both packages enabled, the `external-dns` HelmRelease picks up the `kuberture-api` Service in `cozy-kuberture` and creates the `api.k8s.example.com` record in the configured provider. |
| 80 | + |
| 81 | +The `cozystack.external-dns` Package is the cluster-wide system instance (it runs with `namespaced: false` and watches Services across the cluster). `cozystack.external-dns-application` is the tenant-namespaced variant and will not pick up Services outside its tenant namespace — pair `kuberture` with `cozystack.external-dns`, not `cozystack.external-dns-application`, on the host control plane. |
| 82 | + |
| 83 | +## Routing to multiple external-dns instances |
| 84 | + |
| 85 | +A single `kuberture` install can address any number of `external-dns` instances by varying `annotationPrefix` per output. Each `external-dns` instance is started with `--annotation-prefix=<your-prefix>/` so it rebuilds every `hostname`/`target`/`ttl` annotation key under that prefix and ignores everything else; `kuberture` stamps the matching prefix on each Service. This is the upstream-documented [Split Horizon DNS pattern](https://kubernetes-sigs.github.io/external-dns/v0.20.0/docs/advanced/split-horizon/) — distinct from `--annotation-filter`, which is a Kubernetes label-selector that filters *which Services* an instance considers, not which prefix it reads data from. |
| 86 | + |
| 87 | +```yaml |
| 88 | +apiVersion: cozystack.io/v1alpha1 |
| 89 | +kind: Package |
| 90 | +metadata: |
| 91 | + name: cozystack.kuberture |
| 92 | +spec: |
| 93 | + variant: default |
| 94 | + components: |
| 95 | + kuberture: |
| 96 | + values: |
| 97 | + kuberture: |
| 98 | + config: |
| 99 | + outputs: |
| 100 | + - name: public |
| 101 | + hostname: |
| 102 | + - api.k8s.example.com |
| 103 | + serviceName: kuberture-public |
| 104 | + addressSource: endpointslice |
| 105 | + - name: internal |
| 106 | + hostname: |
| 107 | + - api-internal.k8s.example.internal |
| 108 | + annotationPrefix: internal-dns.example.com/ |
| 109 | + serviceName: kuberture-internal |
| 110 | + addressSource: endpointslice |
| 111 | + recordTTL: 300 |
| 112 | +``` |
| 113 | + |
| 114 | +This renders two headless Services in `cozy-kuberture`: |
| 115 | + |
| 116 | +- `kuberture-public` carries the default `external-dns.alpha.kubernetes.io/*` annotations and is consumed by the platform `external-dns`. |
| 117 | +- `kuberture-internal` carries `internal-dns.example.com/*` annotations and is invisible to the platform `external-dns` (which is reading the default prefix). A second `external-dns` instance started with `--annotation-prefix=internal-dns.example.com/` consumes it: the prefix flag tells that instance to look up `hostname`/`target`/`ttl` under `internal-dns.example.com/*`, so it sees `kuberture-internal` and is blind to `kuberture-public`. |
| 118 | + |
| 119 | +Each Service carries **only** its own prefix's annotations — there is no cross-pollution between outputs. |
| 120 | + |
| 121 | +`annotationPrefix` accepts two forms: omit the field to inherit the controller default `external-dns.alpha.kubernetes.io/`, or set it to a non-empty string ending in `/`. The empty string `""` is rejected by the chart's values schema; omission is the only way to fall back to the default prefix. |
| 122 | + |
| 123 | +## Address resolution strategies |
| 124 | + |
| 125 | +`addressSource` selects where each output gets its target IPs: |
| 126 | + |
| 127 | +| `addressSource` | What `kuberture` publishes | |
| 128 | +| --- | --- | |
| 129 | +| `endpointslice` (default) | The addresses listed in the `default/kubernetes` EndpointSlice verbatim. Use this when the EndpointSlice IPs are the addresses you want in DNS. | |
| 130 | +| `node-internal` | The `InternalIP` of each Node hosting an EndpointSlice endpoint. Use this when the EndpointSlice carries pod-network IPs but external-dns must publish node-internal addresses. | |
| 131 | +| `node-external` | The `ExternalIP` of each Node. Use this for cloud-managed clusters where Nodes have public addresses on a different interface than the API listens on. | |
| 132 | +| `node-public` | A node's public IP looked up through a public-IP discovery hook (provider-specific). | |
| 133 | + |
| 134 | +`addressType` (`IPv4` / `IPv6`) further filters the resolved set; default is `IPv4`. |
| 135 | + |
| 136 | +## Disable path |
| 137 | + |
| 138 | +Removing `cozystack.kuberture` from `bundles.enabledPackages` stops the platform Helm chart from emitting the `Package` CR. As with every optional system package, the existing `Package` CR is **not** removed automatically: the platform helper stamps `helm.sh/resource-policy: keep` on each emitted Package, so Helm/Flux leaves it in place when it stops being rendered. To fully remove `kuberture`: |
| 139 | + |
| 140 | +```bash |
| 141 | +kubectl delete package.cozystack.io cozystack.kuberture |
| 142 | +``` |
| 143 | + |
| 144 | +The cascade removes the HelmRelease in `cozy-kuberture`, which removes the Deployment, the operator-supplied output Services, and (eventually) the namespace. `external-dns` records previously published from those Services follow `external-dns`'s own delete policy (`policy: upsert-only` is the cozystack default, which means records are **not** retracted on Service deletion; switch to `policy: sync` ahead of time if records must be cleaned up automatically). |
| 145 | + |
| 146 | +## Supply-chain notes |
| 147 | + |
| 148 | +The chart is pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and the controller image from `ghcr.io/lexfrei/kuberture`. Both live in the maintainer's personal namespace and are intentionally not mirrored under `ghcr.io/cozystack/*`. Air-gapped operators must mirror both the chart and the controller image into their internal registry and override `kuberture.image.repository` in the Package values. Chart version and OCI manifest digest are pinned in the cozystack package Makefile; the controller image tag and image digest are pinned in the package's `values.yaml`. Pins advance in lockstep with each upstream release. |
| 149 | + |
| 150 | +See the package README at [`packages/system/kuberture/`](https://github.com/cozystack/cozystack/tree/main/packages/system/kuberture) for the full values surface, the bump procedure, and the `NetworkPolicy` default-off choice. |
| 151 | + |
| 152 | +## See also |
| 153 | + |
| 154 | +- [Enabling and disabling components]({{% ref "/docs/next/operations/configuration/components#enabling-and-disabling-components" %}}) — the `bundles.enabledPackages` / `bundles.disabledPackages` mechanism used here. |
| 155 | +- [Platform Package reference]({{% ref "/docs/next/operations/configuration/platform-package" %}}) — `bundles.enabledPackages` field reference. |
| 156 | +- [`lexfrei/kuberture`](https://github.com/lexfrei/kuberture) — upstream controller source and chart, configuration reference, and roadmap. |
| 157 | +- [`packages/system/kuberture/`](https://github.com/cozystack/cozystack/tree/main/packages/system/kuberture) — cozystack package: values shim, helm-unittest fixtures, end-to-end smoke test. |
0 commit comments