Skip to content

Commit 8c1a78d

Browse files
authored
feat(operator): default system namespace memory via LimitRange (#3494)
## What this PR does System components now get a default container memory limit, so their pod cgroups carry a `memory.max` and drop out of the Talos userspace OOM handler's victim set. `cozystack-operator` maintains a LimitRange named `cozystack-system-defaults` in every system namespace it reconciles. The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current` and drops everything that scores zero. Only a memory limit sets `memory.max`, so only a limit takes a pod out of the candidate set. A request just moves a pod from BestEffort to Burstable, which under the default `strictCgroupClassOrdering: true` still means killed second. `PriorityClass` does nothing here, because the handler reads cgroupfs and never talks to the apiserver. Victim selection is also unrelated to the trigger, so a tenant workload thrashing against its own limit can drive node memory pressure while a system component with no limit is what gets killed. An audit of a running cluster found around 70 workloads across ~30 packages with at least one container missing a memory limit. Per-chart resources would not cover that set, and would not keep covering it as packages are added. So the LimitRange is written by `reconcileNamespaces`, which already creates these namespaces and already tells system from tenant. It also covers init containers, which #3495 can't reach: one of them without a limit leaves the pod cgroup without `memory.max`, whatever the main containers declare. Tenant namespaces are skipped. Where the tenant chart's own `tenant-range-limits` exists (only when the operator configured `resourceQuotas`), a second LimitRange next to it would leave the result to the order LimitRanger iterates them. Where it doesn't exist, a tenant namespace is not the place for the platform to impose a default. ### Why the default is 32Gi The handler grants immunity for `memory.max` existing at all, whatever its value. So the ceiling doesn't need to be fitted, and 32Gi is high enough that nothing plausibly reaches it. A ceiling low enough to bind would turn a rare pressure-driven kill into a deterministic one for any component whose real usage nobody measured. This repo already has one such case, recorded in the `vertical-pod-autoscaler` values, where a 110Mi limit OOM-killed the updater and failed HelmReleases with it. Scheduling is unaffected, since pods are scheduled against requests and the paired default request is 32Mi. An operator who wants the ceiling to bind can lower it, per component in that component's chart or fleet-wide with the knob, once real usage is measured. ### When the default is withheld A default below a container's own request makes that pod impossible to admit, so the operator checks before it applies. It scans Deployments, StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init containers included, for a container that requests more than the configured limit and declares no limit of its own. If it finds one, it withdraws the managed LimitRange and records a hold in the `operator.cozystack.io/system-memory-state` annotation on the Namespace. The hold does not clear by itself when the blocker disappears, because an empty scan looks the same as a pod between deletion and recreation. Lowering a previously applied limit also creates a hold, even with no visible blocker, because LimitRanger writes the defaulted limit onto admitted pods and the scan can no longer tell which containers declare one. A cluster administrator releases the hold by copying the `<hold>:<target>` token from the state into the `operator.cozystack.io/system-memory-ack` annotation. The token is used once and only for that hold and target. If a blocker is still visible, the token is rejected and the hold stays. Package reconciles are scheduled at least every five minutes, so an acknowledgement takes effect on a quiet cluster too. The reason and the token also appear in the operator log under `withholding the default container memory limit`. ### Namespace another LimitRange already governs If a system namespace already has a LimitRange that says anything about memory, at the Container or the Pod scope, the operator writes nothing there and withdraws what it wrote before. A second default leaves the effective value to iteration order. A foreign `max` below this default, a `min` above the default request or a `maxLimitRequestRatio` rejects pods outright, which takes out a whole namespace. At the Pod scope the collision can't be decided in advance, because it depends on how many containers a future pod has. A LimitRange that bounds only cpu or storage coexists normally. A same-named LimitRange without the operator's managed-by label is treated as foreign and left alone. This matters most in `kube-system`, which is in scope because `cozystack-scheduler` installs there, and where a distribution's or an administrator's own LimitRange is likeliest to sit. ### Disabling Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables the feature. On the next reconcile the operator removes every LimitRange it owns cluster-wide, found by the managed-by label. While the feature is on, it also removes its LimitRange from namespaces no active Package targets any more. The safety state on the Namespace stays in both cases. ### Docs `docs/operations/system-memory-limits.md` landed in #3496 ahead of this code, so main describes a `4Gi` default and installer values that do nothing yet. The page is corrected to the 32Gi default and now describes withholding, the acknowledgement, and the foreign LimitRange rule. ### Not in this PR linstor policy. #3824 extends the controller's existing exemption from limits to the satellite, and a linstor limit here would contradict it, so linstor stays with that PR. The scan reads six workload kinds through `APIReader`, uncached, on every reconcile of every system namespace. That cost is tracked in #3977, since the system namespace set isn't known when the cache is built and the right fix depends on a reconcile-rate measurement nobody has taken. ### Verification `go vet` and `go test` pass for `cmd/cozystack-operator` and `internal/operator`, and `helm unittest` passes on the installer chart for the knobs unset, set, and the limit at `0`. The reconciler tests cover withholding on an oversized request, the hold surviving the blocker's removal, the hold on a lowered limit, acknowledgement and token rejection, every memory field of a foreign LimitRange at both scopes, ownership by label, the disabled sweep, and a deleting Package not creating policy. The withhold design has not been run on a live cluster yet. An earlier design of this PR was, and that run does not carry over. ### Screenshots Not a UI change. ### Downstream repositories I walked the trigger map against the diff. `packages/core/installer/values.yaml` gains two keys, but the `ansible-cozystack` trigger is for a required key or a rename of one the role passes, and these are new, optional and default to `""`, so the role is unaffected. Nothing else in the map is reached by the Go changes, the operator template or the page. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed. ```
2 parents 84e8fcd + 9927c29 commit 8c1a78d

10 files changed

Lines changed: 3329 additions & 13 deletions

File tree

‎cmd/cozystack-operator/main.go‎

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ import (
3636
sourcewatcherv1beta1 "github.com/fluxcd/source-watcher/api/v2/v1beta1"
3737
corev1 "k8s.io/api/core/v1"
3838
apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
39+
"k8s.io/apimachinery/pkg/api/resource"
3940
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
4041
"k8s.io/apimachinery/pkg/fields"
4142
"k8s.io/apimachinery/pkg/labels"
@@ -98,6 +99,8 @@ func main() {
9899
var platformSourceName string
99100
var platformSourceRef string
100101
var platformSourceSecret string
102+
var systemNamespaceMemoryLimit string
103+
var systemNamespaceMemoryRequest string
101104

102105
flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
103106
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
@@ -145,6 +148,14 @@ func main() {
145148
flag.StringVar(&cozyValuesSecretName, "cozy-values-secret-name", "cozystack-values", "The name of the secret containing cluster-wide configuration values.")
146149
flag.StringVar(&cozyValuesSecretNamespace, "cozy-values-secret-namespace", "cozy-system", "The namespace of the secret containing cluster-wide configuration values.")
147150
flag.StringVar(&cozyValuesNamespaceSelector, "cozy-values-namespace-selector", "cozystack.io/system=true", "The label selector for namespaces where the cluster-wide configuration values must be replicated.")
151+
flag.StringVar(&systemNamespaceMemoryLimit, "system-namespace-memory-limit", DefaultSystemNamespaceMemoryLimit,
152+
"Default container memory limit in eligible system namespaces, including kube-system. "+
153+
"Lowering or an oversized request creates a durable hold requiring an administrator's "+
154+
"exact-value acknowledgement. Empty or 0 disables and removes managed LimitRanges; "+
155+
"safety state is retained. See docs/operations/system-memory-limits.md.")
156+
flag.StringVar(&systemNamespaceMemoryRequest, "system-namespace-memory-request", DefaultSystemNamespaceMemoryRequest,
157+
"Default memory request paired with the system namespace limit. Keep small; 0 is supported. "+
158+
"An empty installer value selects the operator default. See docs/operations/system-memory-limits.md.")
148159

149160
opts := zap.Options{
150161
Development: true,
@@ -171,6 +182,13 @@ func main() {
171182
os.Exit(1)
172183
}
173184

185+
systemNSMemoryLimit, systemNSMemoryRequest, err := parseSystemNamespaceMemory(systemNamespaceMemoryLimit, systemNamespaceMemoryRequest)
186+
if err != nil {
187+
setupLog.Error(err, "invalid system namespace memory flags",
188+
"limit", systemNamespaceMemoryLimit, "request", systemNamespaceMemoryRequest)
189+
os.Exit(1)
190+
}
191+
174192
config := ctrl.GetConfigOrDie()
175193

176194
// Create a direct client (without cache) for pre-start operations
@@ -307,13 +325,20 @@ func main() {
307325

308326
// Setup Package reconciler
309327
if err := (&operator.PackageReconciler{
310-
Client: mgr.GetClient(),
328+
Client: mgr.GetClient(),
329+
// Non-cached, so the per-namespace pod and workload scan guarding the
330+
// system defaults LimitRange does not start cluster-wide informers in an
331+
// operator whose whole purpose here is to bound memory.
332+
APIReader: mgr.GetAPIReader(),
311333
Scheme: mgr.GetScheme(),
312334
HelmReleaseInterval: hrIntervalDuration,
313335
HelmReleaseRetryInterval: hrRetryIntervalDuration,
314336
HelmReleaseInstallTimeout: hrInstallTimeoutDuration,
315337
HelmReleaseUpgradeTimeout: hrUpgradeTimeoutDuration,
316338
HelmReleaseMaxHistory: helmReleaseMaxHistory,
339+
340+
SystemNamespaceMemoryLimit: systemNSMemoryLimit,
341+
SystemNamespaceMemoryRequest: systemNSMemoryRequest,
317342
}).SetupWithManager(mgr); err != nil {
318343
setupLog.Error(err, "unable to create controller", "controller", "Package")
319344
os.Exit(1)
@@ -454,6 +479,59 @@ func installPlatformSourceResource(ctx context.Context, k8sClient client.Client,
454479
return nil
455480
}
456481

482+
// The shipped defaults for the two system-namespace memory knobs. Constants keep flag
483+
// registration and the parser test inputs on the same values; the test compares the parsed
484+
// result with literal contract values so changing a default cannot stay green by construction.
485+
const (
486+
DefaultSystemNamespaceMemoryLimit = "32Gi"
487+
DefaultSystemNamespaceMemoryRequest = "32Mi"
488+
)
489+
490+
// parseSystemNamespaceMemory turns the --system-namespace-memory-limit and
491+
// --system-namespace-memory-request flag values into the quantities the Package reconciler
492+
// takes, rejecting the combinations that would fail later and further away.
493+
//
494+
// Empty is how both flags are disabled and parses to the zero quantity; for the limit that
495+
// means "no LimitRange, and remove any this operator created", which is the documented
496+
// off switch. A negative quantity parses fine as a Quantity and is meaningless as memory,
497+
// so it is caught here rather than being written into a LimitRange the API server rejects.
498+
//
499+
// The pair check is the one that matters most. A LimitRange whose defaultRequest exceeds
500+
// its default is rejected by the API server, and because the reconciler applies one per
501+
// system namespace on every Package reconcile, a bad pair would wedge namespace
502+
// reconciliation for every system package with an error nothing connects back to a flag.
503+
// A request of 0 against a non-zero limit passes deliberately: it is the supported way to
504+
// take the memory.max without reserving anything at schedule time, not a broken value.
505+
// Both zero is the disabled case and never reaches the comparison.
506+
func parseSystemNamespaceMemory(rawLimit, rawRequest string) (limit, request resource.Quantity, err error) {
507+
parse := func(flagName, raw string) (resource.Quantity, error) {
508+
if raw == "" {
509+
return resource.Quantity{}, nil
510+
}
511+
q, err := resource.ParseQuantity(raw)
512+
if err != nil {
513+
return resource.Quantity{}, fmt.Errorf("%s: %w", flagName, err)
514+
}
515+
if q.Sign() < 0 {
516+
return resource.Quantity{}, fmt.Errorf("%s must not be negative, got %s", flagName, raw)
517+
}
518+
return q, nil
519+
}
520+
521+
if limit, err = parse("--system-namespace-memory-limit", rawLimit); err != nil {
522+
return resource.Quantity{}, resource.Quantity{}, err
523+
}
524+
if request, err = parse("--system-namespace-memory-request", rawRequest); err != nil {
525+
return resource.Quantity{}, resource.Quantity{}, err
526+
}
527+
if !limit.IsZero() && request.Cmp(limit) > 0 {
528+
return resource.Quantity{}, resource.Quantity{}, fmt.Errorf(
529+
"--system-namespace-memory-request (%s) must not exceed --system-namespace-memory-limit (%s)",
530+
request.String(), limit.String())
531+
}
532+
return limit, request, nil
533+
}
534+
457535
// parsePlatformSourceURL parses the source URL and returns the source type and repository URL.
458536
// Supports formats:
459537
// - oci://registry.example.com/repo

‎cmd/cozystack-operator/main_test.go‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import (
2323

2424
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
2525
"github.com/cozystack/cozystack/internal/marketplace/tapconst"
26+
"k8s.io/apimachinery/pkg/api/resource"
2627
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
2728
"k8s.io/apimachinery/pkg/runtime"
2829
"sigs.k8s.io/controller-runtime/pkg/client"
@@ -650,3 +651,118 @@ func TestGenerateGitRepository_WithSecret(t *testing.T) {
650651
t.Errorf("expected SecretRef.Name %q, got %q", "my-git-creds", obj.Spec.SecretRef.Name)
651652
}
652653
}
654+
655+
// The system namespace memory flags are the only ones on this binary whose bad values do
656+
// not surface here at all: a limit and a request that parse but contradict each other
657+
// produce a LimitRange the API server rejects, once per system namespace, on every Package
658+
// reconcile, with nothing in the error pointing back at a flag. Each branch below is a way
659+
// that has happened or could.
660+
func TestParseSystemNamespaceMemory(t *testing.T) {
661+
tests := []struct {
662+
name string
663+
rawLimit string
664+
rawRequest string
665+
wantErr bool
666+
wantLimit string
667+
wantRequest string
668+
}{
669+
{
670+
// Inputs follow the constants registered as flag defaults; literal
671+
// expectations pin the shipped contract so changing either constant
672+
// requires an intentional test update.
673+
name: "defaults",
674+
rawLimit: DefaultSystemNamespaceMemoryLimit,
675+
rawRequest: DefaultSystemNamespaceMemoryRequest,
676+
wantLimit: "32Gi",
677+
wantRequest: "32Mi",
678+
},
679+
{
680+
// Empty is the documented off switch, not a missing value: the chart
681+
// leaves both keys empty unless an operator sets them.
682+
name: "empty disables both",
683+
rawLimit: "",
684+
rawRequest: "",
685+
wantLimit: "0",
686+
wantRequest: "0",
687+
},
688+
{
689+
// An explicit 0 limit is the other way to disable, and it must not
690+
// trip the pair check against a request that is still set.
691+
name: "zero limit disables and ignores the request",
692+
rawLimit: "0",
693+
rawRequest: "32Mi",
694+
wantLimit: "0",
695+
wantRequest: "32Mi",
696+
},
697+
{
698+
// Supported on purpose: the container still gets memory.max, the
699+
// scheduler reserves nothing. See the flag help.
700+
name: "zero request against a real limit is accepted",
701+
rawLimit: "4Gi",
702+
rawRequest: "0",
703+
wantLimit: "4Gi",
704+
wantRequest: "0",
705+
},
706+
{
707+
name: "malformed limit",
708+
rawLimit: "4GB",
709+
wantErr: true,
710+
},
711+
{
712+
name: "malformed request",
713+
rawLimit: "4Gi",
714+
rawRequest: "thirty-two",
715+
wantErr: true,
716+
},
717+
{
718+
// Parses as a Quantity and is meaningless as memory.
719+
name: "negative limit",
720+
rawLimit: "-1Gi",
721+
wantErr: true,
722+
},
723+
{
724+
name: "negative request",
725+
rawLimit: "4Gi",
726+
rawRequest: "-32Mi",
727+
wantErr: true,
728+
},
729+
{
730+
// The one the API server would otherwise reject per namespace, per
731+
// reconcile, forever.
732+
name: "request above limit",
733+
rawLimit: "1Gi",
734+
rawRequest: "2Gi",
735+
wantErr: true,
736+
},
737+
{
738+
// Equal is the boundary and is legal: defaultRequest may equal default.
739+
name: "request equal to limit",
740+
rawLimit: "1Gi",
741+
rawRequest: "1Gi",
742+
wantLimit: "1Gi",
743+
wantRequest: "1Gi",
744+
},
745+
}
746+
747+
for _, tt := range tests {
748+
t.Run(tt.name, func(t *testing.T) {
749+
limit, request, err := parseSystemNamespaceMemory(tt.rawLimit, tt.rawRequest)
750+
if tt.wantErr {
751+
if err == nil {
752+
t.Fatalf("parseSystemNamespaceMemory(%q, %q) = %s/%s, want an error",
753+
tt.rawLimit, tt.rawRequest, limit.String(), request.String())
754+
}
755+
return
756+
}
757+
if err != nil {
758+
t.Fatalf("parseSystemNamespaceMemory(%q, %q): %v", tt.rawLimit, tt.rawRequest, err)
759+
}
760+
if want := resource.MustParse(tt.wantLimit); limit.Cmp(want) != 0 {
761+
t.Errorf("limit = %s, want %s", limit.String(), want.String())
762+
}
763+
if want := resource.MustParse(tt.wantRequest); request.Cmp(want) != 0 {
764+
t.Errorf("request = %s, want %s", request.String(), want.String())
765+
}
766+
})
767+
}
768+
}

0 commit comments

Comments
 (0)