Repository navigation
Commit 4a2dea2
authored
feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages (#4087)
## What this PR does
**Series behind #3481** — three separable pull requests:
1. **#4087 — Cluster API providers** (capmox + in-cluster IPAM)
2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes`
3. #4089 — the Proxmox CSI driver and cloud-controller-manager
#4087 and #4088 share no files and can be reviewed in parallel. #4089
builds on #4088.
---
This is the provisioning layer #3481 asks to specify first: the part
that
creates the VMs. Neither of the other two is needed to review it.
### What it adds
Two Cluster API providers, packaged the way the existing KubeVirt
provider is —
vendored components, `gzip -9 -n` for a reproducible artifact, a `make
components` target that regenerates it from the upstream release and
checks
the downloads against the SHA-256 sums the Makefile pins:
| Package | Version | Contract |
|---|---|---|
| `capi-providers-infraprovider-proxmox` | capmox v0.7.7 | v1beta1 |
| `capi-providers-ipam-in-cluster` | CAIP v1.0.3 | v1beta1 |
capmox v0.7.7 rather than the current v0.9.x on purpose: from v0.8 it
declares
contract `v1beta2`, which the CAPI core shipped here (v1.10.1) does not
serve.
Both join the `iaas` bundle as **opt-in** packages, where the KubeVirt
infrastructure provider is unconditional. The asymmetry is deliberate:
capmox
drives a hypervisor outside the cluster and needs a Secret holding a
Proxmox API
token, which capi-operator reads before it installs anything — with no
Secret the
provider is never deployed at all. Enabling it by default would ask
every
Cozystack install that has no Proxmox to supply hypervisor credentials.
```yaml
bundles:
enabledPackages:
- cozystack.capi-provider-infra-proxmox # brings IPAM with it
- cozystack.capi-provider-ipam-in-cluster # also enableable on its own
```
Enabling capmox emits the IPAM provider automatically — capmox allocates
worker
addresses from an `InClusterIPPool` and has no DHCP mode, so on its own
it never
gets a `ProxmoxCluster` to `Ready`. IPAM stays independently enableable,
since
assigning addresses is a generic Cluster API concern.
Naming capmox in `enabledPackages` while disabling IPAM stops the
render. The
cluster would catch that combination on its own — the capmox
`PackageSource`
depends on the IPAM `Package`, so with that `Package` absent capmox goes
`Ready=False` with `DependenciesNotReady` and no `HelmRelease` is
created — but
it would not say why. Failing the render names the two conflicting list
entries
at the moment someone writes them, instead of leaving a condition to
decode.
### Why the earlier attempt did not produce a VM
#1927's blocker — "ProxmoxMachine does not produce a VM" — was not a
capmox
incompatibility. It was four independent causes, each of which only
became
visible once the previous one was cleared:
1. **`PROXMOX_URL` included `/api2/json`.** capmox appends that itself,
so every
call went to `/api2/json/api2/json/…` and returned 501.
2. **No IPAM provider.** `no matches for kind "InClusterIPPool"`; the
`ProxmoxCluster` never reached `Ready` and every `ProxmoxMachine` sat in
`WaitingForClusterInfrastructure`.
3. **`templateSelector.matchTags` is set equality, not containment**
(`slices.Equal` after sort). A template carrying four tags is not
selected by
naming one of them: `found 0 VM templates with tags`.
4. **capmox waits for the qemu-guest-agent, which Talos does not ship.**
`error waiting for agent: timed out` — cleared with
`checks.skipQemuGuestAgent` and `skipCloudInitStatus`.
None of these are in the provider. They are packaging and configuration,
which
is why the earlier work could look complete and still never boot a
machine.
### Evidence
Run on Proxmox VE 9.2.11 with Talos v1.13.6 and Cozystack's CAPI stack
(core v1.10.1, CABPT v0.6.12, Kamaji control-plane provider v0.19.0):
```
$ kubectl get proxmoxmachine,machine -n tenant-proxmox
proxmoxmachine/kubernetes-px-md0-… Ready=True VMProvisioned=True
machine/kubernetes-px-md0-… Running proxmox://0ef88cc1-6aea-4563-af64-344d8060d5c0
$ qm list
106 kubernetes-px-md0-jmzds-h9npx running
```
The machine boots Talos from the nocloud ISO, reads the machineconfig
capmox
writes, and joins. That is the layer #69 called a hard blocker.
### What this PR does not claim
- One Proxmox node, one tenant cluster, manual runs — there is no e2e
coverage
for Proxmox in CI, and this PR does not add any.
- It does not make Proxmox a supported platform. It makes the
provisioning layer
reviewable, which #3481 names as the precondition for that conversation.
- It does not touch `capi-providers-core`. The same drift check, pointed
at that
package, shows that the `startupProbe` from #2946 never reached the
shipped
blob; that fix changes what every new install runs and is #4483, with
its
own release note.
- Nothing here exposes `ProxmoxCluster.spec.credentialsRef` to a tenant,
and
nothing should: `capmox-manager-role` grants get/list/watch/patch on
Secrets
cluster-wide and that field takes a namespace. Whatever renders a
`ProxmoxCluster` for a tenant in the rest of the series has to keep the
field
out of tenant hands.
### Testing
`make unit-tests` — exit 0. New suites:
`packages/core/platform/tests/sources_capi_provider_proxmox_test.yaml`
and
`bundles_proxmox_wiring_test.yaml` (both bundle paths, the
enable/disable
precedence, and the refusal above).
Every negated `containsDocument` in the wiring suite carries `any:
true`;
without it the negation passes whether or not the document is in the
render.
Three mutations, each reverted afterwards — emitting the capmox
`Package`
unconditionally, emitting the IPAM `Package` unconditionally, and
dropping the
`disabledPackages` check from `$proxmoxEnabled` — each turn red the
tests named
after the property they break, and the unmutated chart stays green (44
suites,
198 tests).
`hack/capi-provider-components_test.bats` covers both new provider
packages:
the blob and the YAML are the same bytes; the Makefile version, the
ConfigMap
name and `spec.version` are one value — `applied-spec-hash` covers the
provider
spec and the config Secret but not the ConfigMap, so a bump that
regenerates the
files while `spec.version` stays behind installs the old provider
silently; and
the vendored files match the SHA-256 sums the Makefile pins, so the pin
holds
even when nobody runs `make components`.
The render was compared against `main` for five configurations
(defaults,
`isp-full`, `isp-full-generic` with paas, an unrelated
`enabledPackages`, and
`disabledPackages` naming IPAM). No line is removed in any of them and
the
`Package` count is unchanged; the only additions are the two
`PackageSource`
objects, which the chart renders for every package, optional or not.
### Screenshots
Not applicable — no UI change.
### Downstream repositories
I walked the trigger map in `docs/agents/contributing.md` against this
diff, file by file. What it matches is below. **No box is ticked**,
because every follow-up it points at depends on a decision that is not
mine to make: #3481 asks whether Proxmox becomes a supported platform at
all, and writing its documentation or its Terraform schema before that
is answered would be work built on an assumption. I have not filed
speculative PRs or issues in those repositories for the same reason. If
the direction is accepted, I open them; a human should decide which.
**cozystack/website — matches, no follow-up yet.**
- `packages/core/platform/values.yaml` changed.
`content/en/docs/next/operations/configuration/platform-package.md` is a
hand-written table of the `spec.components.platform.values.*` keys; this
PR does not add a key, but it documents two new values that
`bundles.enabledPackages` now accepts, and the table is where a reader
would look for them.
- Two platform components are added
(`capi-providers-infraprovider-proxmox`,
`capi-providers-ipam-in-cluster`) → `guides/platform-stack/_index.md`,
`operations/configuration/licenses.md`.
- The `iaas` bundle gains two optional packages →
`operations/configuration/variants.md`.
**Everything else: no match.** Nothing under `packages/apps/` or
`packages/extra/` (terraform-provider, website app lists), no
`packages/core/installer/values.yaml` or `networking.*`/`publishing.*`
key the Ansible role sets, no `hack/` file or package-layout change
(ccp, external-apps-example), no `ApplicationDefinition` semantics, no
telemetry metric, no node prerequisites (talm, ansible), no
commit-convention change (community).
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(cluster-api): add optional Cluster API packages for Proxmox — the capmox infrastructure provider and the in-cluster IPAM provider it requires. Neither is deployed unless named in `bundles.enabledPackages`, since capmox drives a hypervisor outside the cluster and does nothing without Proxmox API credentials. Enabling the Proxmox provider brings the IPAM provider with it.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added optional Proxmox infrastructure provider support for IaaS
clusters.
- Added in-cluster IP address management, usable independently or
alongside Proxmox.
- Proxmox deployments require a pre-created credentials Secret and
automatically include in-cluster IPAM.
- Added validation for incompatible Proxmox and IPAM settings.
- **Tests**
- Added coverage for provider dependencies, package enablement,
disabling behavior, invalid configurations, and packaged provider
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->24 files changed
Lines changed: 5710 additions & 0 deletions
File tree
- hack
- packages
- core/platform
- sources
- templates/bundles
- tests
- system
- capi-providers-infraprovider-proxmox
- files
- templates
- capi-providers-ipam-in-cluster
- files
- templates
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
218 | 218 | | |
219 | 219 | | |
220 | 220 | | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
221 | 259 | | |
222 | 260 | | |
223 | 261 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
0 commit comments