Skip to content

Commit c8e651d

Browse files
committed
fix(engine): keep replace-merged fields whole in node files
Before a node file is laid over the rendered config, the engine prunes from its body what the render already carries. Where Talos replaces instead of merging, that prune is wrong: the body is cut down to the operator's delta and the replace writes only the delta. The engine knew the `merge:"replace"` fields of v1alpha1 and NetworkRuleConfig only. From Talos v1.14 the charts write KubeNetworkConfig, and on cozystack with OIDC KubeAuthenticationConfig, into every node file. Adding a pod subnet there dropped the rendered one; adding an OIDC audience dropped the anonymous health-check access. A body BGPInstanceConfig or CRIBaseRuntimeSpecConfig over a rendered one lost entries the same way. Typed documents now root their prune paths at their kind, since one field name is replace-tagged on one kind and merged on another (KubeAdmissionControlConfig's configuration merges), and the table carries every replace-tagged field of the typed documents. Component extraArgs, in v1alpha1 and in the typed documents, merge key by key and replace the whole value of each key. A list-valued flag such as feature-gates extended in the node file lost its rendered entries; the prune now drops only keys the body restates exactly. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 06642ce commit c8e651d

5 files changed

Lines changed: 343 additions & 27 deletions

File tree

‎docs/configuration/node-files.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ A node file can carry Talos config below its modeline (for example, a custom `ho
7373

7474
## Idempotent applies
7575

76-
Repeated `talm apply` runs against an already-configured node do not duplicate entries. Before the strategic merge runs, the engine prunes from the body every primitive-list entry the rendered template already carries (e.g. certSANs, nameservers, validSubnets). For object arrays the upstream patcher merges by identity (machine.network.interfaces by `interface:` or `deviceSelector:`, vlans by `vlanId:`, apiServer admissionControl by `name:`), the prune descends into matched pairs and dedupes the inner primitive lists too — so re-applying after `talm template -I` does not double interface addresses, vlan addresses, or admission-control exemption namespaces. For object arrays without an upstream identity merge (extraVolumes, kernel.modules, wireguard.peers, ...), body items that deep-equal a rendered counterpart are dropped, covering the dominant full-restate case. Fields tagged `merge:"replace"` upstream are passed through verbatim — pruning them would let the upstream replace silently drop the rendered entries on a partial edit. This covers v1alpha1 root paths `cluster.network.podSubnets`, `cluster.network.serviceSubnets`, `cluster.apiServer.auditPolicy`, and the typed `NetworkRuleConfig` paths `ingress` and `portSelector.ports`.
76+
Repeated `talm apply` runs against an already-configured node do not duplicate entries. Before the strategic merge runs, the engine prunes from the body every primitive-list entry the rendered template already carries (e.g. certSANs, nameservers, validSubnets). For object arrays the upstream patcher merges by identity (machine.network.interfaces by `interface:` or `deviceSelector:`, vlans by `vlanId:`, apiServer admissionControl by `name:`), the prune descends into matched pairs and dedupes the inner primitive lists too — so re-applying after `talm template -I` does not double interface addresses, vlan addresses, or admission-control exemption namespaces. For object arrays without an upstream identity merge (extraVolumes, kernel.modules, wireguard.peers, ...), body items that deep-equal a rendered counterpart are dropped, covering the dominant full-restate case. Fields tagged `merge:"replace"` upstream are passed through verbatim — pruning them would let the upstream replace silently drop the rendered entries on a partial edit. This covers v1alpha1 root paths `cluster.network.podSubnets`, `cluster.network.serviceSubnets`, `cluster.apiServer.auditPolicy`, the typed `NetworkRuleConfig` paths `ingress` and `portSelector.ports`, `advertise`, `importRoutes` and `neighbors` of `BGPInstanceConfig`, `overrides` of `CRIBaseRuntimeSpecConfig`, and from Talos v1.14 `podSubnets` and `serviceSubnets` of `KubeNetworkConfig`, `pod` of `KubeStaticPodConfig` and `configuration` of `KubeAuthenticationConfig`, `KubeAuditPolicyConfig` and `KubeCredentialProviderConfig`. The typed paths are matched per kind: `configuration` of `KubeAdmissionControlConfig` merges and is pruned as usual. Component `extraArgs` (kubelet, apiserver, controller-manager, scheduler, proxy, etcd, in v1alpha1 and in the typed documents) merge key by key and replace the whole value of each key the body sets, so the prune drops only keys the body restates exactly; a list-valued flag such as `feature-gates` edited in the node file keeps the rendered entries the body lists.
7777

7878
## What `talm template` does not do
7979

‎docs/manual-test-plan.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -750,7 +750,9 @@ Node files from an earlier contract, on a node running Talos v1.14: in a project
750750

751751
On a node running Talos v1.14 (any preset): `talm apply --dry-run -f nodes/<node>.yaml`, then apply. Expected: the drift preview lists `Kube*` documents rather than `cluster.*` fields, the node accepts the config, and `talm get kubeletconfig` / `talm get staticpods` show the kubelet and control-plane args from values. On cozystack, `kubectl get ds --namespace kube-system` lists neither `kube-flannel` nor `kube-proxy`, and control-plane nodes carry no `node-role.kubernetes.io/control-plane:NoSchedule` taint.
752752

753-
Regression anchor: `TestContract_ShippedChartsRenderValidConfig` loads both presets through the engine on v1.12, v1.13 and v1.14, at defaults and with every Kubernetes knob set, and validates the full render and the applied node file in machinery; `TestContract_ApiServerAuthFlagsRefusedFromV114` pins the refused flags; `TestApplyTemplatesPerNode_RefusesNodeFileWithSupersededFields` pins the apply-time check on an old node file; `TestContract_ChartKubeDocumentMergesIntoBundle`, `TestContract_ChartDeleteSelectorRemovesBundleDocument` and `TestContract_NonFullRenderKeepsChartKubeDocumentsVerbatim` pin the merge.
753+
Partial edits of replace-tagged typed fields, on a node running Talos v1.14. A list-valued flag needs a rendered list to edit, and `extraApiServerArgs` takes scalars only, so first, on generic, set `extraApiServerArgs: { max-requests-inflight: "400" }` in `values.yaml`, add `feature-gates: [A=true]`, indented two spaces, on the line after the `extraApiServerArgs` include under `kind: KubeAPIServerConfig` in the project's `templates/_helpers.tpl`, and re-render the node file with `talm template -I`. Then, in a v1.14 node file, add a second entry to `podSubnets` of the `KubeNetworkConfig` document (keep the rendered one); on cozystack with `oidcIssuerUrl` set, add a second audience to the `KubeAuthenticationConfig` jwt entry; and in the generic node file change the flag to `feature-gates: [A=true, B=true]`. Run `talm apply --dry-run -f nodes/<node>.yaml`. Expected: the config sent keeps the rendered pod subnet next to the added one, and `KubeAuthenticationConfig` still carries the `anonymous` block with `/livez`, `/readyz` and `/healthz` next to both audiences, and the flag keeps both entries.
754+
755+
Regression anchor: `TestContract_ShippedChartsRenderValidConfig` loads both presets through the engine on v1.12, v1.13 and v1.14, at defaults and with every Kubernetes knob set, and validates the full render and the applied node file in machinery; `TestContract_ApiServerAuthFlagsRefusedFromV114` pins the refused flags; `TestApplyTemplatesPerNode_RefusesNodeFileWithSupersededFields` pins the apply-time check on an old node file; `TestContract_ChartKubeDocumentMergesIntoBundle`, `TestContract_ChartDeleteSelectorRemovesBundleDocument` and `TestContract_NonFullRenderKeepsChartKubeDocumentsVerbatim` pin the merge; the `KubeNetworkConfig` and `KubeAuthenticationConfig` cases of `TestMergeFileAsPatch`, `TestPruneIdenticalKeys_ReplacePathsAreKeyedByKind` and `TestPruneIdenticalKeys_KeepsReplaceTaggedNetworkAndCRIFields` (the same rule for `BGPInstanceConfig` and `CRIBaseRuntimeSpecConfig` body documents) pin the partial edits.
754756

755757
## C. Apply (auth path)
756758

‎pkg/engine/engine.go‎

Lines changed: 104 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -956,14 +956,20 @@ func pruneBodyIdentitiesAgainstRendered(body, rendered []byte) ([]byte, bool, er
956956
// body is the zero value, in which case rendered survives. Each entry
957957
// mirrors a struct field tagged `merge:"replace"` in the upstream
958958
// machinery types (collected from
959-
// pkg/machinery/config/types/v1alpha1/v1alpha1_types.go and
960-
// pkg/machinery/config/types/network/rule_config.go):
959+
// pkg/machinery/config/types/v1alpha1/v1alpha1_types.go and the typed
960+
// documents under pkg/machinery/config/types/{network,k8s,cri}):
961961
//
962962
// - cluster/network/podSubnets — v1alpha1 `PodSubnet []string ... merge:"replace"`
963963
// - cluster/network/serviceSubnets — v1alpha1 `ServiceSubnet []string ... merge:"replace"`
964964
// - cluster/apiServer/auditPolicy — v1alpha1 `AuditPolicyConfig Unstructured ... merge:"replace"`
965-
// - ingress — typed NetworkRuleConfig `Ingress IngressConfig ... merge:"replace"`
966-
// - portSelector/ports — typed NetworkRuleConfig `Ports PortRanges ... merge:"replace"`
965+
// - NetworkRuleConfig/ingress — `Ingress IngressConfig ... merge:"replace"`
966+
// - NetworkRuleConfig/portSelector/ports — `Ports PortRanges ... merge:"replace"`
967+
// - BGPInstanceConfig/advertise, BGPInstanceConfig/importRoutes,
968+
// BGPInstanceConfig/neighbors
969+
// - CRIBaseRuntimeSpecConfig/overrides
970+
// - KubeNetworkConfig/podSubnets, KubeNetworkConfig/serviceSubnets
971+
// - KubeAuthenticationConfig/configuration, KubeAuditPolicyConfig/configuration,
972+
// KubeCredentialProviderConfig/configuration, KubeStaticPodConfig/pod
967973
//
968974
// At these paths the prune must NOT subtract rendered-side entries from
969975
// body's primitive list, recurse into body's map, or descend into body's
@@ -974,13 +980,13 @@ func pruneBodyIdentitiesAgainstRendered(body, rendered []byte) ([]byte, bool, er
974980
// original; a partial-edit on auditPolicy that adds a rule loses the
975981
// rendered apiVersion/kind/other rules.
976982
//
977-
// Paths are walked relative to the document root that owns the field,
978-
// so the typed NetworkRuleConfig entries appear without an apiVersion/
979-
// kind prefix — pruneBodyIdentitiesAgainstRendered pairs body and
980-
// rendered docs by identity tuple before calling pruneIdenticalKeys, so
981-
// each walk sees only its own document's keys. The bare paths above do
982-
// not collide with anything in the v1alpha1 root (no `ingress` or
983-
// `portSelector` keys exist there), so a flat lookup is sufficient.
983+
// Paths are walked relative to the document that owns the field:
984+
// pruneBodyIdentitiesAgainstRendered pairs body and rendered docs by
985+
// identity tuple before calling pruneIdenticalKeys, so each walk sees
986+
// only its own document's keys. A typed document's paths start with its
987+
// kind, because the same field is replace-tagged on one kind and merged
988+
// on another (KubeAdmissionControlConfig's `configuration` merges); the
989+
// v1alpha1 document has no kind, so its paths start at the root.
984990
//
985991
// The deep-equal short-circuit at the top of pruneIdenticalKeysAt is
986992
// still safe for replace paths: when body byte-equals rendered, deleting
@@ -993,8 +999,88 @@ var replaceSemanticPaths = map[string]struct{}{
993999
"cluster/network/podSubnets": {},
9941000
"cluster/network/serviceSubnets": {},
9951001
"cluster/apiServer/auditPolicy": {},
996-
"ingress": {},
997-
"portSelector/ports": {},
1002+
1003+
"NetworkRuleConfig/ingress": {},
1004+
"NetworkRuleConfig/portSelector/ports": {},
1005+
"BGPInstanceConfig/advertise": {},
1006+
"BGPInstanceConfig/importRoutes": {},
1007+
"BGPInstanceConfig/neighbors": {},
1008+
"CRIBaseRuntimeSpecConfig/overrides": {},
1009+
1010+
"KubeNetworkConfig/podSubnets": {},
1011+
"KubeNetworkConfig/serviceSubnets": {},
1012+
"KubeAuthenticationConfig/configuration": {},
1013+
"KubeAuditPolicyConfig/configuration": {},
1014+
"KubeCredentialProviderConfig/configuration": {},
1015+
"KubeStaticPodConfig/pod": {},
1016+
}
1017+
1018+
// keyReplaceMapPaths lists the maps Talos merges key by key, replacing the
1019+
// whole value of every key the patch sets: the component extraArgs, typed
1020+
// meta.Args, whose Merge is a maps.Copy. At these paths the prune drops a
1021+
// key only when body restates it exactly; a key whose value differs goes
1022+
// to the patcher whole, or a list-valued flag edited in the node file
1023+
// would be cut to the operator's delta and replace the rendered entries.
1024+
//
1025+
//nolint:gochecknoglobals // immutable lookup table consulted by pruneIdenticalKeysAt.
1026+
var keyReplaceMapPaths = map[string]struct{}{
1027+
"machine/kubelet/extraArgs": {},
1028+
"cluster/apiServer/extraArgs": {},
1029+
"cluster/controllerManager/extraArgs": {},
1030+
"cluster/proxy/extraArgs": {},
1031+
"cluster/scheduler/extraArgs": {},
1032+
"cluster/etcd/extraArgs": {},
1033+
1034+
"KubeletConfig/extraArgs": {},
1035+
"KubeAPIServerConfig/extraArgs": {},
1036+
"KubeControllerManagerConfig/extraArgs": {},
1037+
"KubeSchedulerConfig/extraArgs": {},
1038+
"KubeProxyConfig/extraArgs": {},
1039+
}
1040+
1041+
// pruneAtReplacePath handles body[key] when Talos replaces rather than merges
1042+
// at childPath, and reports whether it did. Under `merge:"replace"` the value
1043+
// is left whole: any prune leaks rendered-side entries out of the final
1044+
// config (see replaceSemanticPaths). Under a per-key replace only exactly
1045+
// restated keys go (see keyReplaceMapPaths).
1046+
func pruneAtReplacePath(body map[string]any, key string, bodyV, renderedV any, childPath string) bool {
1047+
if _, replace := replaceSemanticPaths[childPath]; replace {
1048+
return true
1049+
}
1050+
1051+
if _, perKey := keyReplaceMapPaths[childPath]; perKey {
1052+
pruneEqualMapKeys(body, key, bodyV, renderedV)
1053+
1054+
return true
1055+
}
1056+
1057+
return false
1058+
}
1059+
1060+
// pruneEqualMapKeys drops from body[key] the entries rendered carries with
1061+
// the same value, and body[key] itself once nothing is left in it.
1062+
func pruneEqualMapKeys(body map[string]any, key string, bodyV, renderedV any) {
1063+
bodySub, ok := bodyV.(map[string]any)
1064+
if !ok {
1065+
return
1066+
}
1067+
1068+
renderedSub, ok := renderedV.(map[string]any)
1069+
if !ok {
1070+
return
1071+
}
1072+
1073+
before := len(bodySub)
1074+
1075+
for k, v := range bodySub {
1076+
if rv, exists := renderedSub[k]; exists && reflect.DeepEqual(v, rv) {
1077+
delete(bodySub, k)
1078+
}
1079+
}
1080+
1081+
if before > 0 && len(bodySub) == 0 {
1082+
delete(body, key)
1083+
}
9981084
}
9991085

10001086
// objectArrayMergeKeys lists the identity fields Talos's upstream
@@ -1085,7 +1171,10 @@ var objectArrayMergeKeys = map[string][]string{
10851171
// threads the YAML path so the object-array descent can look up the
10861172
// identity key for the current location.
10871173
func pruneIdenticalKeys(body, rendered map[string]any) {
1088-
pruneIdenticalKeysAt(body, rendered, "")
1174+
// A typed document's paths are rooted at its kind: the same field name
1175+
// can be replace-tagged on one kind and merged on another.
1176+
kind, _ := body[k8sKeyKind].(string)
1177+
pruneIdenticalKeysAt(body, rendered, kind)
10891178
}
10901179

10911180
// pruneIdenticalKeysAt is pruneIdenticalKeys's recursive workhorse.
@@ -1112,10 +1201,7 @@ func pruneIdenticalKeysAt(body, rendered map[string]any, yamlPath string) {
11121201
}
11131202

11141203
childPath := joinYAMLPath(yamlPath, key)
1115-
if _, replace := replaceSemanticPaths[childPath]; replace {
1116-
// Upstream `merge:"replace"` overwrites rendered with body
1117-
// verbatim. Any prune at this path leaks rendered-side
1118-
// entries out of the final config — see replaceSemanticPaths.
1204+
if pruneAtReplacePath(body, key, bodyV, renderedV, childPath) {
11191205
continue
11201206
}
11211207

‎pkg/engine/engine_test.go‎

Lines changed: 71 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import (
2323
"os"
2424
"os/exec"
2525
"path/filepath"
26+
"reflect"
2627
"regexp"
2728
"strconv"
2829
"strings"
@@ -1205,15 +1206,25 @@ func TestObjectArrayMergeKeysMatchesUpstreamMergerSurface(t *testing.T) {
12051206
// - cluster/network/podSubnets — v1alpha1 PodSubnet
12061207
// - cluster/network/serviceSubnets — v1alpha1 ServiceSubnet
12071208
// - cluster/apiServer/auditPolicy — v1alpha1 AuditPolicyConfig
1208-
// - ingress — typed NetworkRuleConfig Ingress
1209-
// - portSelector/ports — typed NetworkRuleConfig Ports
1209+
// - NetworkRuleConfig/ingress, NetworkRuleConfig/portSelector/ports
1210+
// - the replace-tagged fields of the typed Kubernetes documents
12101211
func TestReplaceSemanticPathsMatchesUpstreamReplaceTags(t *testing.T) {
12111212
expected := map[string]struct{}{
1212-
"cluster/network/podSubnets": {},
1213-
"cluster/network/serviceSubnets": {},
1214-
"cluster/apiServer/auditPolicy": {},
1215-
"ingress": {},
1216-
"portSelector/ports": {},
1213+
"cluster/network/podSubnets": {},
1214+
"cluster/network/serviceSubnets": {},
1215+
"cluster/apiServer/auditPolicy": {},
1216+
"NetworkRuleConfig/ingress": {},
1217+
"NetworkRuleConfig/portSelector/ports": {},
1218+
"BGPInstanceConfig/advertise": {},
1219+
"BGPInstanceConfig/importRoutes": {},
1220+
"BGPInstanceConfig/neighbors": {},
1221+
"CRIBaseRuntimeSpecConfig/overrides": {},
1222+
"KubeNetworkConfig/podSubnets": {},
1223+
"KubeNetworkConfig/serviceSubnets": {},
1224+
"KubeAuthenticationConfig/configuration": {},
1225+
"KubeAuditPolicyConfig/configuration": {},
1226+
"KubeCredentialProviderConfig/configuration": {},
1227+
"KubeStaticPodConfig/pod": {},
12171228
}
12181229
if len(replaceSemanticPaths) != len(expected) {
12191230
t.Fatalf("table size drift: have %d entries, want %d (re-grep upstream for `merge:\"replace\"` and reconcile)\nhave: %#v\nwant: %#v",
@@ -1226,6 +1237,59 @@ func TestReplaceSemanticPathsMatchesUpstreamReplaceTags(t *testing.T) {
12261237
}
12271238
}
12281239

1240+
// The same field is replace-tagged on one kind and merged on another, so
1241+
// the prune has to tell them apart by kind: KubeAuthenticationConfig's
1242+
// configuration reaches the patcher whole, KubeAdmissionControlConfig's is
1243+
// reduced to the operator's delta.
1244+
func TestPruneIdenticalKeys_ReplacePathsAreKeyedByKind(t *testing.T) {
1245+
doc := func(kind string, keys ...string) map[string]any {
1246+
configuration := map[string]any{}
1247+
for _, k := range keys {
1248+
configuration[k] = k
1249+
}
1250+
1251+
return map[string]any{k8sKeyAPIVersion: "v1alpha1", k8sKeyKind: kind, "configuration": configuration}
1252+
}
1253+
1254+
for kind, wantKept := range map[string]int{"KubeAuthenticationConfig": 2, "KubeAdmissionControlConfig": 1} {
1255+
body := doc(kind, "rendered", "added")
1256+
pruneIdenticalKeys(body, doc(kind, "rendered"))
1257+
1258+
configuration, _ := body["configuration"].(map[string]any)
1259+
if len(configuration) != wantKept {
1260+
t.Errorf("%s: configuration keeps %d keys after the prune, want %d: %v", kind, len(configuration), wantKept, configuration)
1261+
}
1262+
}
1263+
}
1264+
1265+
// A partial edit of a replace-tagged list or map in the BGP and CRI
1266+
// documents has to reach the patcher whole: pruned to the operator's delta,
1267+
// the replace would drop what the render carries.
1268+
func TestPruneIdenticalKeys_KeepsReplaceTaggedNetworkAndCRIFields(t *testing.T) {
1269+
for _, tc := range []struct {
1270+
kind, field string
1271+
rendered, ok any
1272+
}{
1273+
{"BGPInstanceConfig", "advertise", []any{"dummy0"}, []any{"dummy0", "dummy1"}},
1274+
{"BGPInstanceConfig", "importRoutes", []any{map[string]any{"instance": "a"}}, []any{map[string]any{"instance": "a"}, map[string]any{"instance": "b"}}},
1275+
{"BGPInstanceConfig", "neighbors", []any{map[string]any{"address": "192.0.2.1"}}, []any{map[string]any{"address": "192.0.2.1"}, map[string]any{"address": "192.0.2.2"}}},
1276+
{"CRIBaseRuntimeSpecConfig", "overrides", map[string]any{"process": "a"}, map[string]any{"process": "a", "hooks": "b"}},
1277+
} {
1278+
t.Run(tc.kind+"/"+tc.field, func(t *testing.T) {
1279+
doc := func(v any) map[string]any {
1280+
return map[string]any{k8sKeyAPIVersion: "v1alpha1", k8sKeyKind: tc.kind, "name": "fabric", tc.field: v}
1281+
}
1282+
1283+
body := doc(deepClone(tc.ok))
1284+
pruneIdenticalKeys(body, doc(tc.rendered))
1285+
1286+
if !reflect.DeepEqual(body[tc.field], tc.ok) {
1287+
t.Errorf("%s was pruned to %v, want it whole: %v", tc.field, body[tc.field], tc.ok)
1288+
}
1289+
})
1290+
}
1291+
}
1292+
12291293
// TestJoinYAMLPath pins joinYAMLPath's contract: the document root
12301294
// (empty parent) joins without a leading separator so the path lookup
12311295
// in objectArrayMergeKeys matches the documented form ("machine/...");

0 commit comments

Comments
 (0)