Skip to content

Commit 1f35c26

Browse files
authored
fix(lineage, kubernetes): resolve CAPK node VMs to their cluster (#5124)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The virt-launcher pods of Kubernetes worker nodes are never marked as managed by the lineage webhook, so anything that relies on the webhook having resolved them misses them. The visible symptom: a SecurityGroup attached to a `Kubernetes` application labels the control-plane pods but never the worker nodes, because the securitygroup-controller caches only pods carrying `internal.cozystack.io/managed-by-cozystack=true`. **Root cause.** The webhook is not skipped for these pods (they do not carry the managed-by label, so the `objectSelector` lets them through); it runs and fails to resolve their owner chain. The walk goes Pod → VirtualMachineInstance → VirtualMachine and stops there: the Cluster API KubeVirt provider creates the VirtualMachine with no ownerReference and links it to its KubevirtMachine through the `capk.cluster.x-k8s.io/kubevirt-machine-{name,namespace}` labels only. With no ownerReference, no HelmRelease kind and no `helm.toolkit.fluxcd.io/name` label on the VM, `WalkOwnershipGraph` returns no ancestor, `computeLabels` returns nothing and the pod is admitted unmanaged. The `apps.cozystack.io/application.{group,kind,name}` labels the pods do carry come from the `KubevirtMachineTemplate` in `packages/apps/kubernetes-nodes/templates/nodegroup.yaml`, not from the webhook. **Measured on a v1.6.5 cluster.** Every Kubernetes cluster on it showed the same shape (three clusters across two tenants). For one worker node: the pod carries the three lineage labels but not `managed-by-cozystack`; the pod is owned by its VMI and the VMI by its VM; the VM has no ownerReferences, and its labels include `capk.cluster.x-k8s.io/kubevirt-machine-name` and `capk.cluster.x-k8s.io/kubevirt-machine-namespace` pointing at a KubevirtMachine in the same namespace. From there ownerReferences run KubevirtMachine → Machine → MachineSet → MachineDeployment → Cluster, and the CAPI Cluster carries `helm.toolkit.fluxcd.io/name: kubernetes-<name>`. The bridge is one hop. Setting `managed-by-cozystack=true` by hand on a worker's virt-launcher pod made the securitygroup-controller stamp the membership label within ten seconds. **Fix.** `WalkOwnershipGraph` now follows a VirtualMachine without ownerReferences to the KubevirtMachine named by its CAPK labels, and keeps walking from there. Node pods then resolve to the parent `Kubernetes` application, which is the attribution the machine template already hard-codes, and they get `managed-by-cozystack` like every other managed pod. The bridge is only taken when the label names the VM's own namespace: CAPK can target an external infrastructure cluster, and the walk must never attribute an object across namespaces. The webhook's ClusterRole already grants `get` on every resource, so no RBAC change is needed. **Why at the source rather than in the consumer.** Relaxing the securitygroup-controller's pod cache to `apps.cozystack.io/application.kind` Exists would also have fixed the symptom, but it would leave every other consumer of the managed-by label blind to node pods, and it would rest membership on labels the webhook never vouched for. Stamping `managed-by-cozystack` from the machine template was ruled out too: the `objectSelector` would then skip these pods entirely, and they would lose the scheduling-class injection the webhook performs on every pod. **Blast radius.** Only objects below a CAPK VM change: the VMI, the virt-launcher pods and the PVCs of the VM's `dataVolumeTemplates` now get the lineage labels of their Kubernetes application. None of them match a resource selector of the `Kubernetes` ApplicationDefinition, so they get `tenantresource=false`. Nothing selects PVCs on `application.kind=Kubernetes` (the Velero strategy selects `VMDisk`). The `-userdata` cloud-init Secret CAPK creates is not affected either way: at the pinned provider commit, `reconcileKubevirtBootstrapSecret` creates it with the bootstrap Secret's labels and no ownerReferences, so the walk reaches neither the VM nor the KubevirtMachine from it, before or after this change. Existing node pods pick the labels up on their next admission (recreation, live migration, or any update). **Tests.** `pkg/lineage/capk_test.go` builds the observed chain in a fake dynamic client and checks that a worker's virt-launcher pod resolves to `Kubernetes/cluster-a`, that a VM pointing at a KubevirtMachine in another namespace is not followed, that a VM without CAPK labels still has no ancestor, and, in a table of negative cases, that the CAPK labels are ignored on any object other than a `kubevirt.io/v1` VirtualMachine (another kind of the `kubevirt.io` group, a VirtualMachineInstance, a VirtualMachine of another group or version). Weakening the kind and API version guard to match on either field alone fails all four cases. With the bridge disabled, the first test fails. No chainsaw case: the only suites that bring up worker VMs are `kubernetes-latest` and `kubernetes-previous`, whose shared runner script is already over its time budget, and the unit test covers the walk exactly as it runs in the webhook. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(lineage, kubernetes): the virt-launcher pods of Kubernetes worker nodes are now resolved to their Kubernetes application and marked as managed by Cozystack, so a SecurityGroup attached to a Kubernetes cluster now applies to its worker nodes, not only to its control plane. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Lineage results now connect supported virtual machines to their associated machine and Kubernetes application when the required association labels are present. * **Bug Fixes** * Association tracing is limited to supported virtual machine resources and machines in the same namespace. Unlabelled standalone virtual machines and machines in another namespace do not gain ancestors through this association. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents da7f2c2 + fd9245f commit 1f35c26

2 files changed

Lines changed: 167 additions & 0 deletions

File tree

‎pkg/lineage/capk_test.go‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
package lineage
2+
3+
import (
4+
"context"
5+
"testing"
6+
7+
helmv2 "github.com/fluxcd/helm-controller/api/v2"
8+
"k8s.io/apimachinery/pkg/api/meta"
9+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
10+
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
11+
"k8s.io/apimachinery/pkg/runtime"
12+
"k8s.io/apimachinery/pkg/runtime/schema"
13+
dynamicfake "k8s.io/client-go/dynamic/fake"
14+
)
15+
16+
const capkNamespace = "tenant-demo"
17+
18+
// kubernetesAppMapper maps the cluster's HelmRelease to its Kubernetes
19+
// application, as the webhook's ApplicationDefinition mapper does.
20+
type kubernetesAppMapper struct{}
21+
22+
func (kubernetesAppMapper) Map(*helmv2.HelmRelease) (string, string, string, error) {
23+
return "apps.cozystack.io/v1alpha1", "Kubernetes", "kubernetes-", nil
24+
}
25+
26+
func capkObject(apiVersion, kind, name string, labels map[string]string, owner *unstructured.Unstructured) *unstructured.Unstructured {
27+
o := &unstructured.Unstructured{}
28+
o.SetAPIVersion(apiVersion)
29+
o.SetKind(kind)
30+
o.SetNamespace(capkNamespace)
31+
o.SetName(name)
32+
o.SetLabels(labels)
33+
if owner != nil {
34+
o.SetOwnerReferences([]metav1.OwnerReference{{APIVersion: owner.GetAPIVersion(), Kind: owner.GetKind(), Name: owner.GetName()}})
35+
}
36+
return o
37+
}
38+
39+
// capkNodeChain builds the objects behind the virt-launcher pod of a
40+
// Kubernetes worker node, as observed on a live cluster: the pod is owned by
41+
// its VMI and the VMI by its VM, but CAPK leaves the VM without an
42+
// ownerReference and points at its KubevirtMachine through labels only. From
43+
// the KubevirtMachine up, ownerReferences run to the CAPI Cluster, which the
44+
// cluster's HelmRelease labels.
45+
func capkNodeChain(vmLabels map[string]string) (*unstructured.Unstructured, []runtime.Object, meta.RESTMapper) {
46+
app := capkObject("apps.cozystack.io/v1alpha1", "Kubernetes", "cluster-a", nil, nil)
47+
hr := capkObject(HRAPIVersion, HRKind, "kubernetes-cluster-a", nil, nil)
48+
cluster := capkObject("cluster.x-k8s.io/v1beta1", "Cluster", "kubernetes-cluster-a", map[string]string{HRLabel: "kubernetes-cluster-a"}, nil)
49+
md := capkObject("cluster.x-k8s.io/v1beta1", "MachineDeployment", "kubernetes-cluster-a-md0", nil, cluster)
50+
ms := capkObject("cluster.x-k8s.io/v1beta1", "MachineSet", "kubernetes-cluster-a-md0-x", nil, md)
51+
machine := capkObject("cluster.x-k8s.io/v1beta1", "Machine", "kubernetes-cluster-a-md0-x-y", nil, ms)
52+
kvm := capkObject(KubevirtMachineAPIVersion, KubevirtMachineKind, "kubernetes-cluster-a-md0-x-y", nil, machine)
53+
vm := capkObject(VMAPIVersion, VMKind, "kubernetes-cluster-a-md0-x-y", vmLabels, nil)
54+
vmi := capkObject("kubevirt.io/v1", "VirtualMachineInstance", "kubernetes-cluster-a-md0-x-y", nil, vm)
55+
pod := capkObject("v1", "Pod", "virt-launcher-kubernetes-cluster-a-md0-x-y-abcde", nil, vmi)
56+
57+
objs := []runtime.Object{app, hr, cluster, md, ms, machine, kvm, vm, vmi}
58+
mapper := meta.NewDefaultRESTMapper(nil)
59+
for _, o := range objs {
60+
mapper.Add(o.GetObjectKind().GroupVersionKind(), meta.RESTScopeNamespace)
61+
}
62+
mapper.Add(schema.GroupVersionKind{Version: "v1", Kind: "Pod"}, meta.RESTScopeNamespace)
63+
return pod, objs, mapper
64+
}
65+
66+
func walkNodePod(t *testing.T, vmLabels map[string]string) []ObjectID {
67+
t.Helper()
68+
pod, objs, mapper := capkNodeChain(vmLabels)
69+
dyn := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), objs...)
70+
return WalkOwnershipGraph(context.Background(), dyn, mapper, kubernetesAppMapper{}, pod)
71+
}
72+
73+
func TestWalkResolvesCAPKNodeVMToKubernetesApp(t *testing.T) {
74+
got := walkNodePod(t, map[string]string{
75+
KubevirtMachineNameLabel: "kubernetes-cluster-a-md0-x-y",
76+
KubevirtMachineNamespaceLabel: capkNamespace,
77+
})
78+
want := ObjectID{APIVersion: "apps.cozystack.io/v1alpha1", Kind: "Kubernetes", Namespace: capkNamespace, Name: "cluster-a"}
79+
if len(got) != 1 || got[0] != want {
80+
t.Fatalf("WalkOwnershipGraph(node pod) = %+v, want [%+v]", got, want)
81+
}
82+
}
83+
84+
func TestWalkFollowsCAPKLabelsOnlyFromKubeVirtVMs(t *testing.T) {
85+
// The CAPK labels name a valid KubevirtMachine in the same namespace in
86+
// every case: only the kind and API group of the labelled object differ, so
87+
// each case fails solely on the guard that restricts the bridge to
88+
// kubevirt.io/v1 VirtualMachines.
89+
capkLabels := map[string]string{
90+
KubevirtMachineNameLabel: "kubernetes-cluster-a-md0-x-y",
91+
KubevirtMachineNamespaceLabel: capkNamespace,
92+
}
93+
for _, tc := range []struct {
94+
name, apiVersion, kind string
95+
}{
96+
{name: "other kind in the kubevirt.io group", apiVersion: VMAPIVersion, kind: "Cluster"},
97+
{name: "VirtualMachineInstance", apiVersion: VMAPIVersion, kind: "VirtualMachineInstance"},
98+
{name: "VirtualMachine of another group", apiVersion: "example.com/v1", kind: VMKind},
99+
{name: "VirtualMachine of another kubevirt.io version", apiVersion: "kubevirt.io/v1alpha3", kind: VMKind},
100+
} {
101+
t.Run(tc.name, func(t *testing.T) {
102+
_, objs, mapper := capkNodeChain(nil)
103+
impostor := capkObject(tc.apiVersion, tc.kind, "impostor", capkLabels, nil)
104+
dyn := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), objs...)
105+
if got := WalkOwnershipGraph(context.Background(), dyn, mapper, kubernetesAppMapper{}, impostor); len(got) != 0 {
106+
t.Fatalf("WalkOwnershipGraph(%s %s) = %+v, want no ancestors", tc.apiVersion, tc.kind, got)
107+
}
108+
})
109+
}
110+
}
111+
112+
func TestWalkIgnoresKubevirtMachineOfAnotherNamespace(t *testing.T) {
113+
// A VM pointing at a KubevirtMachine outside its own namespace (an external
114+
// infrastructure cluster, or a forged label) must not be resolved through it.
115+
got := walkNodePod(t, map[string]string{
116+
KubevirtMachineNameLabel: "kubernetes-cluster-a-md0-x-y",
117+
KubevirtMachineNamespaceLabel: "tenant-other",
118+
})
119+
if len(got) != 0 {
120+
t.Fatalf("WalkOwnershipGraph(node pod) = %+v, want no ancestors", got)
121+
}
122+
}
123+
124+
func TestWalkStopsAtUnlabelledVM(t *testing.T) {
125+
// A standalone VM with no CAPK labels and no ownerReference has no ancestors.
126+
if got := walkNodePod(t, nil); len(got) != 0 {
127+
t.Fatalf("WalkOwnershipGraph(pod of standalone VM) = %+v, want no ancestors", got)
128+
}
129+
}

‎pkg/lineage/lineage.go‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,16 @@ const (
2020
HRAPIVersion = "helm.toolkit.fluxcd.io/v2"
2121
HRKind = "HelmRelease"
2222
HRLabel = "helm.toolkit.fluxcd.io/name"
23+
24+
// KubeVirt VirtualMachines created by the Cluster API KubeVirt provider
25+
// (CAPK) for cluster nodes carry no ownerReference: CAPK links them to
26+
// their KubevirtMachine through these labels only.
27+
VMAPIVersion = "kubevirt.io/v1"
28+
VMKind = "VirtualMachine"
29+
KubevirtMachineAPIVersion = "infrastructure.cluster.x-k8s.io/v1alpha1"
30+
KubevirtMachineKind = "KubevirtMachine"
31+
KubevirtMachineNameLabel = "capk.cluster.x-k8s.io/kubevirt-machine-name"
32+
KubevirtMachineNamespaceLabel = "capk.cluster.x-k8s.io/kubevirt-machine-namespace"
2333
)
2434

2535
// AppMapper maps HelmRelease to application metadata.
@@ -91,6 +101,18 @@ func WalkOwnershipGraph(
91101
return
92102
}
93103

104+
// A node VM of a Kubernetes cluster has no ownerReference, so without this
105+
// step the walk would stop at it and never reach the cluster's application.
106+
if name, ok := kubevirtMachineName(obj); ok {
107+
ownerObj, err := getUnstructuredObject(ctx, client, mapper, state.cache, KubevirtMachineAPIVersion, KubevirtMachineKind, obj.GetNamespace(), name)
108+
if err != nil {
109+
l.Error(err, "couldn't get KubevirtMachine of VirtualMachine", "namespace", obj.GetNamespace(), "virtualMachine", obj.GetName(), "kubevirtMachine", name)
110+
return
111+
}
112+
out = append(out, WalkOwnershipGraph(ctx, client, mapper, appMapper, ownerObj, state)...)
113+
return
114+
}
115+
94116
// I want "if err1 != nil go to next block, if err2 != nil, go to next block, etc semantics",
95117
// like an early return from a function, but if all checks succeed, I don't want to do the rest
96118
// of the function, so it's a `for { if err { break } if othererr { break } if allgood { return }
@@ -138,6 +160,22 @@ func WalkOwnershipGraph(
138160
return
139161
}
140162

163+
// kubevirtMachineName returns the KubevirtMachine a CAPK-created
164+
// VirtualMachine belongs to. It only follows a KubevirtMachine in the VM's own
165+
// namespace: CAPK can also run against an external infrastructure cluster,
166+
// where the KubevirtMachine lives elsewhere and is not ours to resolve.
167+
func kubevirtMachineName(obj *unstructured.Unstructured) (string, bool) {
168+
if obj.GetAPIVersion() != VMAPIVersion || obj.GetKind() != VMKind {
169+
return "", false
170+
}
171+
labels := obj.GetLabels()
172+
name := labels[KubevirtMachineNameLabel]
173+
if name == "" || labels[KubevirtMachineNamespaceLabel] != obj.GetNamespace() {
174+
return "", false
175+
}
176+
return name, true
177+
}
178+
141179
func parseWalkMemory(memory []interface{}) (*walkState, error) {
142180
switch len(memory) {
143181
case 0:

0 commit comments

Comments
 (0)