Skip to content

Commit 6580d2e

Browse files
authored
feat(kubernetes): storage and cloud-controller integration for Proxmox-backed tenants (#4089)
## What this PR does **Series behind #3481** — three separable pull requests: 1. #4087 — the Cluster API providers (capmox + in-cluster IPAM) 2. #4088 — the `substrate` switch in `kubernetes` and `kubernetes-nodes` 3. #4089 — the Proxmox CSI driver and cloud-controller-manager #4088 has merged, so this diff is the eight commits on top of it: `proxmox.pool`, `csi-proxmox`, `ccm-proxmox`, `delete.yaml`, the tests, the e2e suite, one README sentence on token privileges, and the reconcile Job's replace-on-immutable step for existing clusters. The `substrate` value this gates on is on `main`. ### One chain, not two components Storage and the cloud-controller-manager are one chain, not two components: ``` kubelet --cloud-provider=external → taint node.cloudprovider.kubernetes.io/uninitialized → CCM applies topology.kubernetes.io/region|zone + providerID → CSI node plugin starts → PVC binds ``` The CSI node plugin reads region and zone off its own Node and exits fatally without them: ``` F main.go:126] Failed to get region or zone for node: …, region: , zone: , see documentation about topology labels ``` Those labels come from the cloud-controller-manager, and the CCM ignores a Node until the kubelet's `--cloud-provider=external` has tainted it. Shipping the CSI driver alone gives a tenant a StorageClass, a healthy controller, and a node plugin in CrashLoopBackOff. The CSI driver has no equivalent knob for this. Its `features.provider` accepts `default` and `capmox` and validates neither, so an unrecognised value falls through to the default path — the key is therefore not set at all. The cloud-controller-manager is the opposite case: `capmox` is a real mode there and is set, because the default one writes a Node providerID of `proxmox://<node>/<vmid>` while capmox has already written `proxmox://<SMBIOS UUID>` on the Machine. So the CCM has no `enabled` switch and its credentials reference is required: on this substrate, a cluster without a CCM is a cluster where every worker keeps the `uninitialized` taint and nothing schedules. A chart that cannot name the Secret should stop rather than render that. The CSI release depends on cilium only; the CCM is a Deployment of this chart, not a HelmRelease, so nothing in the tenant can depend on it — the node plugin waits on the labels the CCM applies, which is the ordering that matters. ### Packaging | Package | Chart | App | |---|---|---| | `packages/system/proxmox-csi` | proxmox-csi-plugin 0.5.12 | v0.20.0 | The vendored chart is the tenant-side half only. It **drops the upstream `values.talos.yaml` preset**, which pins the workloads onto `node-role.kubernetes.io/control-plane` and `node.cloudprovider.kubernetes.io/platform: nocloud`: a Kamaji tenant has no control-plane nodes at all — its control plane is a Deployment in the management cluster — so that preset leaves the components unschedulable. It is published as a component of the existing `kubevirt` variant of `kubernetes-application` rather than in a new variant: a second variant would rename the artifacts of all eighteen tenant addons, and exactly one component differs between the substrates. The cloud-controller-manager is not a vendored chart. It is a Deployment the `kubernetes` chart renders itself (`templates/ccm-proxmox/controller.yaml`), because it runs here, in the management cluster, and there is nothing of it to install in the tenant. Credentials never reach the tenant. Each controller names a Secret in the tenant's namespace of this cluster (`proxmox.csi.credentialsSecretName`, `proxmox.ccm.credentialsSecretName`), an init container composes the config file from it into a memory-backed emptyDir at start-up, and the `kubernetes` release stores only the Secret's name. They are separate knobs from the capmox credentials, and from each other, so an operator can scope them: attaching disks is a different blast radius from creating VMs, and the CCM only reads inventory. `proxmox.insecure` defaults to `false`; a stand whose Proxmox serves a self-signed certificate sets it to `true`. `proxmox.csi.enabled: false` switches off the controller as well as the tenant release, and needs no CSI token. ### Where the hypervisor token lives, and why it decides the shape Both controllers run in the tenant's namespace of **this** cluster and drive the tenant over its admin kubeconfig. Only the CSI node plugin goes into the tenant, and it mounts no credentials. Installing them into the tenant would hand the token to the tenant. The resource definition lists the admin kubeconfig in `secrets.include`, so a tenant is cluster-admin of its own cluster, and the vendored chart writes the driver's `config.yaml` — token included — wherever it is installed. The privileges the driver documents are granted on `/`, so such a token reaches every VM and every volume on the Proxmox cluster. The KubeVirt path already has the shape that works, in `templates/csi/deploy.yaml`, and the upstream components support it: the CSI controller's `--kubeconfig` is documented for running out of cluster. Three consequences of the split are handled rather than left to be discovered: the vendored chart has no switch for its controller, so the tenant release sets `replicaCount: 0`; its Secret template is gated on a non-empty `config.clusters`, so an empty list keeps the credential out; and `CSIStorageCapacity` is published by the provisioner sidecar, which now owns those objects through a Pod that does not exist in the tenant, so `CSIDriver.storageCapacity` is patched to false by a kustomize postRenderer — the chart hardcodes it, and left true the scheduler refuses every Pod with "node(s) did not have enough free storage". Volumes are owned per cluster. The driver names them `vm-<controllerVmID>-pvc-…` and owns every volume as VMID 9999 unless `controllerVmID` is set, which leaves different clusters' volumes indistinguishable on shared storage. Each cluster sets its own, derived from its tenant namespace and release name, so two tenants' clusters with the same name still get different owners. **This does not make one tenant unable to reach another's volumes.** A tenant can still create a static PV naming a foreign `volumeHandle`, and the controller will attach it, detach it or delete it: the driver checks neither the VMID a node ID names nor the volume a handle names against the cluster it serves. Closing that needs a per-tenant Proxmox scope, and the charts can now produce one: `proxmox.pool` on the `kubernetes-nodes` chart puts every worker VM into a Proxmox resource pool as capmox clones it, replacements included, so one ACL on `/pool/<name>` follows every VM without being granted again. The README states the scope where an operator configuring the chart reads: a storage of the tenant's own (`Datastore.Allocate` on a storage reaches every volume on it), `VM.Audit`/`VM.Config.Disk` on that pool for the CSI controller, `VM.Audit` and `VM.GuestAgent.Audit` on it for the CCM — a NotReady Node whose VM the token cannot see is reported as gone and deleted — and what that scope still leaves open: the tenant's own disks, which a cluster-admin of the tenant could reach anyway. ### Deleting a cluster frees its disks The pre-delete hook deletes the CSI driver in Step 2, and a volume is freed only by the driver answering DeleteVolume, so a Step 1b runs first: it deletes the tenant's claims (`--wait=false`; a claim in use carries `kubernetes.io/pvc-protection`, so it clears once nothing mounts it), then the Pods holding each claim, then waits for the PVs rather than the PVCs. The Pods are found by listing every Pod with the claims it mounts and matching in the shell: a JSONPath filter on the claim name fails for a whole namespace as soon as one Pod there mounts two claims, and would find no Pod at all. The claims go first because a claim being deleted is one the scheduler places no new Pod on, so a Pod that a StatefulSet brings back cannot mount it again. Without the step the disks stayed on the hypervisor with the tenant's data in them and no PV object left to name their owner. A tenant apiserver that does not answer is told apart from a tenant with no claims: the first goes through the hook's backstop and shows in its summary, the second is the clean case. The probe is bounded with `timeout`, not `--request-timeout`, which the repo's hook check refuses because a non-zero value diverts an in-cluster kubectl call to localhost:8080. On the KubeVirt path two things change. The pre-delete hook's Step 5 checks that the DataVolume CRD exists before deleting DataVolumes, so a hook that runs where CDI is not installed no longer fails after every earlier step has run; the hook runs only at deletion. The worker pool's talos-reconcile Job changes at upgrade: its Role's TalosConfigTemplate rights are narrowed to the pool's own template and gain a named `delete`, and its script gains the replace step, which moves the Job's content-hash name. Every existing worker pool therefore gets a new reconcile Job when it upgrades. Where a pool's live TalosConfigTemplate still matches the render, that Job changes nothing; where it has drifted, the Job now deletes and recreates it instead of failing. Machines already running keep the bootstrap data they joined with; Machines created afterwards boot from the current template. ### Existing clusters: the TalosConfigTemplate CABPT will not let us patch CABPT's validating webhook rejects any change to `TalosConfigTemplate.spec` (`TalosConfigTemplate.Spec is immutable`). The kubelet flag this PR adds changes that spec, so on a cluster created before it the reconcile Job's `kubectl apply` fails with `is immutable` on every run, and a pool whose `ProxmoxMachineTemplate` also changed (say, `proxmox.pool` set) rolls its workers off the old template: they join without `--cloud-provider=external`, the CCM leaves them unmanaged, and the CSI node plugin exits on the missing zone label. The last commit makes the Job replace the object when apply reports the spec immutable: delete, apply again, log both. Existing Machines keep their bootstrap data; new ones are built from the replaced template. Pinned by `talos_tct_immutable_test.yaml`, and the RBAC the Job runs with covers exactly that delete (`resourceNames` of its own template). ### Evidence ``` $ KUBECONFIG=<tenant> kubectl get node -o custom-columns=… kubernetes-px3-md0-rp4s5-rgr48 prox-home prox-home proxmox://b903f905-7b4f-4048-b1b6-3e8f58530d8f $ kubectl -n tenant-proxmox3 get deploy kubernetes-px3-pcsi-controller 1/1 kubernetes-px3-pccm 1/1 $ KUBECONFIG=<tenant> kubectl get pods -n csi-proxmox csi-plugin-node 3/3 Running (the controller Deployment has 0 replicas) $ KUBECONFIG=<tenant> kubectl get secret -A | grep -c token_secret 0 $ KUBECONFIG=<tenant> kubectl get pvc csi-probe csi-probe Bound pvc-… 1Gi RWO proxmox ``` That the volume is real, not merely `Bound` in the API, and that it is this tenant's: ``` $ qm config <vmid> scsi1: main-pool:vm-798497-pvc-…,size=1G ``` Deleting the PVC frees the disk from the storage; the e2e probe checks that too. ### Known limitation The CCM logs every five minutes on PVE 9: ``` error get ha-groups 500 cannot index groups: ha groups have been migrated to rules ``` Proxmox 9 changed the HA-groups API. Node initialization is unaffected — labels and providerID are applied — but the CCM image v0.15.0's HA-groups lookup is not compatible with PVE 9. ### Testing `make unit-tests` — exit 0. `apps/kubernetes` 28 suites / 338 tests; `apps/kubernetes-nodes` 33 suites / 182 tests / 12 snapshots. Each of the eight commits passes both suites on its own. What the new asserts pin, each shown to go red on the mutation it exists for: the tenant release carries no credentials and no `valuesFrom`; the controllers run here with the Secret mounted and the tenant as their target; `csi.enabled: false` renders no CSI controller and needs no CSI token; the kubelet runs with `cloud-provider: external` on proxmox and without it on kubevirt; Step 1b and the cleanup Role's admin-kubeconfig entry render on proxmox and not on kubevirt, and Step 1b deletes the claims before the Pods; `proxmox.pool` reaches the `ProxmoxMachineTemplate` and an empty one renders no `pool`; each tenant gets its own volume owner id; the CCM runs in capmox mode; the CCM tolerates exactly two taints. `proxmox.ccm`, `proxmox.csi` and `proxmox.insecure` are optional in the schema — each has a default, and the render enforces what the substrate needs — so a `Kubernetes` object written against `main`'s `proxmox` block stays valid; the API review gate reports no sizeable change at any of the commits. The e2e suite ships `.disabled` and no CI lane runs it, so the substrate has no automated coverage until a runner with the `proxmox` label exists. What is in the tree is runnable: the fixture is rendered by `run-proxmox.sh` with explicit defaults (GNU `envsubst` does not expand `${VAR:-default}`, so the defaults live in the script and the render refuses a fixture with a variable left in it — pinned by `hack/proxmox-e2e-fixture_test.bats`), the runner reads the `.conf` kubeconfig a runner outside the management cluster can reach, the StorageClass and the PVC probe read the same storage variable, the probe checks the disk is gone after the claim is deleted, `keep_tenant` is honoured by the suite's own teardown, the workflow fails on an unset stand variable instead of falling back to a hardcoded address, and its schedule runs only where `vars.PROXMOX_NIGHTLY` is set, since the job needs a self-hosted runner with the `proxmox` label. The fixture takes the stand's resource pool through `COZY_PVE_POOL`. ### Screenshots Not applicable — no UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against this diff, file by file. What it matches is below. **No box is ticked**, because every follow-up it points at depends on a decision that is not mine to make: #3481 asks whether Proxmox becomes a supported platform at all, and writing its documentation or its Terraform schema before that is answered would be work built on an assumption. I have not filed speculative PRs or issues in those repositories for the same reason. If the direction is accepted, I open them; a human should decide which. **cozystack/terraform-provider-cozystack — matches, no follow-up yet.** - `packages/apps/kubernetes/values.schema.json` gains `proxmox.ccm.credentialsSecretName`, `proxmox.csi` (`enabled`, `credentialsSecretName`, `storageClasses[]`) and `proxmox.insecure`, with defaults in `values.yaml`. Same coupling as the substrate PR: hand-written schema, model and expand/flatten, and defaults the provider restates and sends itself. **cozystack/website — unclear, deliberately left unticked.** This PR adds one package under `packages/system/`, not under `packages/apps/` or `packages/extra/`, so the app-list trigger does not fire. Whether `proxmox-csi` counts as a "platform component" for `guides/platform-stack/_index.md` is a judgement about that page's scope: it is not part of the platform stack — it is an addon installed *into* a tenant cluster, like the KubeVirt CSI driver, which has no entry there either. I read that as no, but I am not confident enough to decide it for you, so the box stays empty. The README section on token privileges and per-tenant scope is the kind of text `operations/` pages carry; that follows the decision #3481 asks for. **Everything else: no match.** No app package added or renamed; no version enum, `kind`, `plural` or `release.prefix` change; `packages/core/platform/values.yaml` untouched; `packages/core/platform/sources/kubernetes-application.yaml` gains one component but is not a renamed `*-rd` file (ccp); `hack/` gains `e2e-chainsaw/_lib/run-proxmox.sh`, a suite directory shipped `.disabled`, and `proxmox-e2e-fixture_test.bats`, which runs with the other bats files; no `ApplicationDefinition` semantics or `release.chartRef.kind` enum change (external-apps-example); no node prerequisites; no telemetry metric; no commit-convention change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubernetes): add the Proxmox CSI driver and cloud-controller-manager for tenant clusters on the `proxmox` substrate, giving them PersistentVolumes backed by Proxmox storage. The cloud-controller-manager is not optional there: worker kubelets run with `--cloud-provider=external`, so without it every node keeps the `uninitialized` taint and the CSI node plugin cannot start. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## New Features - Added Proxmox VE as an infrastructure option alongside KubeVirt for Kubernetes clusters and worker pools. - Added configuration for Proxmox networking, DNS, VM templates, storage, node placement, credentials, IPv4 allocation, and storage classes. - Added Proxmox cloud-controller-manager and CSI integrations. - Added validation for required Proxmox settings and unsupported combinations. ## Bug Fixes - Improved recovery when Talos configuration templates cannot be updated because they are immutable. ## Documentation - Documented Proxmox configuration, requirements, and supported parameters. ## Tests - Added coverage for Proxmox rendering, sizing, integrations, validation, and KubeVirt compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ### Verified on a live stand Run against three throwaway tenants on Proxmox VE 9.2, each built and destroyed: - every Secret in the tenant cluster scanned for the token — none found, while the PVC round-trip works; - a PVC bound, mounted, written and read back, its disk visible on the hypervisor as `vm-798497-pvc-…` rather than `vm-9999-…`; - `helm uninstall` with the volume still mounted by a running Pod, after which the disk is gone from the storage with nothing done by hand. What that run pinned: the CCM's capmox mode, capacity-aware scheduling off with `CSIDriver.storageCapacity` patched, and the teardown Role's admin-kubeconfig entry. A second run, on the head of this PR, used a pool per tenant and two privilege-separated tokens holding exactly the README's rows, on a throwaway tenant and then on the stand's long-lived cluster upgraded in place: - the worker VM lands in the pool as capmox clones it, and so does its replacement after the Machine is deleted — the CCM initialises the new node and the CSI attaches a disk to it with no ACL touched; - no `401` or `403` from the hypervisor in the CSI controller or the CCM over the whole run, once the roles were granted to the user as well as the token; - the PVC round-trip on both clusters, the disk gone from the storage after the claim; `helm uninstall` with a claim held by a running Pod deletes the claim before the Pod and waits for the volume to be released; - the in-place upgrade replaced the immutable template on the first apply conflict and the re-created worker came up with `--cloud-provider=external`.
2 parents 7d44372 + e5dc086 commit 6580d2e

69 files changed

Lines changed: 3791 additions & 73 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/e2e-proxmox.yaml‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
name: E2E (proxmox)
2+
3+
# The Proxmox substrate's own lane, separate from every other e2e workflow for
4+
# one reason that cannot be engineered away: it needs a Proxmox hypervisor, and
5+
# the oracle-vm-* runners the other lanes use will never have one.
6+
#
7+
# Deliberately NOT triggered by pull_request. A self-hosted runner executes
8+
# whatever the workflow checks out, and this one holds credentials to a
9+
# hypervisor and a live cluster; letting an arbitrary fork PR start it would
10+
# hand both to its author. Manual dispatch and a schedule are the whole trigger
11+
# surface, and both come from someone who already has write access.
12+
on:
13+
workflow_dispatch:
14+
inputs:
15+
keep_tenant:
16+
description: "Leave the tenant in place after the run, for debugging"
17+
type: boolean
18+
default: false
19+
schedule:
20+
# Nightly, offset from the upstream lanes so a shared hypervisor is not
21+
# asked to do two things at once.
22+
- cron: "40 2 * * *"
23+
24+
permissions:
25+
contents: read
26+
27+
concurrency:
28+
# One hypervisor, one tenant namespace, fixed VM ids — two runs would fight
29+
# over all three. Never cancel in progress: a killed run leaks VMs.
30+
group: e2e-proxmox
31+
cancel-in-progress: false
32+
33+
jobs:
34+
e2e:
35+
name: Proxmox substrate
36+
# The job needs a self-hosted runner carrying the `proxmox` label. On a
37+
# repository without one a scheduled run would sit in the queue until GitHub
38+
# cancels it, so the schedule fires only where the stand is declared;
39+
# manual dispatch always runs.
40+
if: github.event_name == 'workflow_dispatch' || vars.PROXMOX_NIGHTLY == 'true'
41+
runs-on: [self-hosted, proxmox]
42+
timeout-minutes: 90
43+
env:
44+
COZY_PROXMOX_NS: tenant-e2eproxmox
45+
COZY_PROXMOX_CLUSTER: k8s-e2e
46+
# No fallbacks: an unset variable fails the first step below instead of
47+
# sending a key to a guessed address or probing a storage the stand lacks.
48+
COZY_PROXMOX_STORAGE: ${{ vars.PROXMOX_STORAGE }}
49+
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH }}
50+
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
51+
COZY_KEEP_TENANT: ${{ inputs.keep_tenant == true }}
52+
steps:
53+
- name: Checkout
54+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
55+
with:
56+
persist-credentials: false
57+
58+
- name: Require the stand's variables
59+
env:
60+
PVE_URL: ${{ vars.PROXMOX_URL }}
61+
PVE_REGION: ${{ vars.PROXMOX_REGION }}
62+
run: |
63+
set -eu
64+
for v in COZY_PROXMOX_STORAGE COZY_PVE_SSH PVE_URL PVE_REGION; do
65+
eval "val=\${$v:-}"
66+
[ -n "$val" ] || { echo "$v is unset: set the PROXMOX_* repository variables for this stand" >&2; exit 1; }
67+
done
68+
69+
- name: Stage credentials
70+
env:
71+
KUBECONFIG_B64: ${{ secrets.COZY_KUBECONFIG }}
72+
PVE_SSH_KEY: ${{ secrets.PROXMOX_SSH_KEY }}
73+
PVE_TOKEN_ID: ${{ secrets.PROXMOX_TOKEN_ID }}
74+
PVE_TOKEN_SECRET: ${{ secrets.PROXMOX_TOKEN_SECRET }}
75+
PVE_URL: ${{ vars.PROXMOX_URL }}
76+
PVE_REGION: ${{ vars.PROXMOX_REGION }}
77+
run: |
78+
set -eu
79+
umask 077
80+
printf '%s' "$KUBECONFIG_B64" | base64 -d > "$GITHUB_WORKSPACE/.kubeconfig"
81+
printf '%s' "$PVE_SSH_KEY" > "$COZY_PVE_SSH_KEY"
82+
echo "KUBECONFIG=$GITHUB_WORKSPACE/.kubeconfig" >> "$GITHUB_ENV"
83+
84+
# The credentials the management-side CSI controller and CCM read. A
85+
# Secret rather than values so no token is ever written into a
86+
# rendered HelmRelease.
87+
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" create namespace "$COZY_PROXMOX_NS" \
88+
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
89+
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" -n "$COZY_PROXMOX_NS" \
90+
create secret generic proxmox-credentials \
91+
--from-literal=url="$PVE_URL" \
92+
--from-literal=token_id="$PVE_TOKEN_ID" \
93+
--from-literal=token_secret="$PVE_TOKEN_SECRET" \
94+
--from-literal=region="$PVE_REGION" \
95+
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
96+
97+
- name: Enable the suite
98+
# It ships disabled so the normal lanes cannot select it; this is the
99+
# one place that is allowed to turn it on.
100+
run: mv hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled
101+
hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml
102+
103+
- name: Run the Proxmox suite
104+
run: |
105+
chainsaw test \
106+
--config hack/e2e-chainsaw/.chainsaw.yaml \
107+
--test-dir hack/e2e-chainsaw/kubernetes-proxmox
108+
109+
- name: Tear down
110+
# always(), because the whole point of the teardown is the failing run:
111+
# a leaked tenant leaves VMs on the hypervisor and the next run inherits
112+
# them.
113+
if: always() && !inputs.keep_tenant
114+
run: |
115+
set -eu
116+
kubectl delete kubernetesnodes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
117+
kubectl delete kubernetes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
118+
kubectl delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found
119+
120+
- name: Remove staged credentials
121+
if: always()
122+
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY"

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 40 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 52 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/agents/e2e-testing.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,16 @@ The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per a
1010

1111
Every convention below follows from that finding: **fail fast, fail loud, make the failure legible.** A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each `assert` waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.
1212

13+
## The Proxmox lane
14+
15+
`hack/e2e-chainsaw/kubernetes-proxmox/` covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the `oracle-vm-*` runners have none, so it ships as `chainsaw-test.yaml.disabled`: suite discovery is `find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml`, which registers it nowhere and makes it unselectable rather than merely unselected. `.github/workflows/e2e-proxmox.yaml` drops the suffix on a self-hosted runner that has one.
16+
17+
Two properties of that workflow are security boundaries rather than preferences. It is never triggered by `pull_request`, because a self-hosted runner executes what the workflow checks out and this one stages a hypervisor token and a live kubeconfig — a fork PR trigger would hand both to its author. And its concurrency group never cancels in progress, because a killed run leaks VMs on the hypervisor rather than merely losing a result.
18+
19+
It does not install Cozystack. Every other lane builds a sandbox per run; this one asserts against an existing cluster and an existing hypervisor, which is a weaker guarantee and is stated in the suite header rather than left to be discovered.
20+
21+
The imperative half lives in `_lib/run-proxmox.sh` and is deliberately not a port of `_lib/run-kubernetes.sh`: that script's diagnostics are virt-launcher Pods, VirtualMachineInstances and guest consoles, none of which exist on an external hypervisor.
22+
1323
## Conventions
1424

1525
### 1. No retries on deterministic steps; retry only pure infrastructure

0 commit comments

Comments
 (0)