Skip to content

Commit 93aac35

Browse files
authored
feat(fluxcd): add flux-shard-operator for tenant helm-controller sharding (#2821)
## What this PR does Implements the tenant Flux sharding design from #2817: a new `flux-shard-operator` system package that spreads tenant HelmReleases across multiple helm-controller shards, so a noisy tenant (e.g. a HelmRelease stuck in infinite remediation) cannot degrade the others. - **Placement controller** owns the tenant→shard assignment: the unit of placement is the tenant (all of its HelmReleases share one shard label), distribution is greedy least-loaded weighted by HelmRelease count, with pinned tenants, threshold-gated rebalance, paced moves, and never moving deleting tenants. Watches are metadata-only, so the controller does not decode the helm-controller status-patch firehose. Assignments are recorded on tenant namespaces; HelmRelease labels remain the source of truth on restarts. - **Shard runtime** is provisioned by the operator: `shardCount` helm-controller Deployments cloned from the flux-aio `flux` Deployment and sanitised, so the image and feature-gates stay version-synced automatically. flux-aio itself is unchanged and keeps owning everything without a shard key. - **CREATE-only mutating webhook** (`failurePolicy: Ignore`) stamps the tenant's shard onto every HelmRelease at admission, so each one is born on the correct shard regardless of creation path; any miss degrades gracefully to the placement controller. - The hand-rolled `flux-tenants` Deployment is retired: the operator drains it (backfill `tenants` → `shard<i>`) and deletes it once empty; migration 44 removes a leftover only under the same guard. Ships with `shardCount: auto` (the default): the operator sizes the shard count from the tenant HelmRelease count — `K = clamp(ceil(H/100), 1, min(16, T))` with hysteresis (scale up past 120 HR/shard, down under 60) — so small clusters stay at one shard (≈ today's behaviour) and large fleets shard out automatically. An integer value pins the count explicitly. One deviation from the design doc: the webhook/controller scope is the `sharding.fluxcd.io/key` label rather than `internal.cozystack.io/tenantmodule` — every tenant HelmRelease is born with `key=tenants` (stamped by ApplicationDefinition `release.labels` and chart templates), so the webhook rewrites it and the placement controller continuously relabels stragglers; `tenantmodule` covers only the 6 tenant chart modules. ### Screenshots Not a UI change. ### Release note ```release-note feat(fluxcd): tenant HelmReleases are now reconciled by operator-managed helm-controller shards; the flux-shard-operator balances tenants across `shardCount` instances and retires the legacy flux-tenants deployment ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced **flux-shard-operator** to shard helm-controller workloads with deterministic tenant placement, optional rebalance controls, and autosized shard count. * Added a **CREATE-only mutating webhook** to stamp the correct shard label on tenant HelmReleases. * **Bug Fixes** * Application updates now preserve existing **flux-shard** label assignments to prevent unwanted shard re-stamping. * **Chores** * Added platform packaging for the new component, including an image build step. * Removed the legacy tenants deployment and added a migration to retire it safely. * **Tests** * Expanded unit and chart tests for placement, provisioning, webhook behavior, and configuration parsing. * **Documentation** * Added operator documentation describing configuration and metrics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 6329ffe + dc94272 commit 93aac35

38 files changed

Lines changed: 3238 additions & 112 deletions

File tree

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ build: build-deps
2020
make -C packages/system/backup-controller image
2121
make -C packages/system/backupstrategy-controller image
2222
make -C packages/system/lineage-controller-webhook image
23+
make -C packages/system/flux-shard-operator image
2324
make -C packages/system/cilium image
2425
make -C packages/system/linstor image
2526
make -C packages/system/linstor-gui image

‎cmd/flux-shard-operator/main.go‎

Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
/*
2+
Copyright 2026.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// flux-shard-operator spreads tenant HelmReleases across multiple
18+
// helm-controller shards so one noisy tenant cannot degrade the others. It
19+
// provisions the shard Deployments (cloned from flux-aio), owns the
20+
// tenant->shard placement and stamps the shard label on new HelmReleases at
21+
// admission time.
22+
package main
23+
24+
import (
25+
"crypto/tls"
26+
"flag"
27+
"os"
28+
29+
// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
30+
// to ensure that exec-entrypoint and run can make use of them.
31+
_ "k8s.io/client-go/plugin/pkg/client/auth"
32+
33+
appsv1 "k8s.io/api/apps/v1"
34+
corev1 "k8s.io/api/core/v1"
35+
"k8s.io/apimachinery/pkg/api/resource"
36+
"k8s.io/apimachinery/pkg/labels"
37+
"k8s.io/apimachinery/pkg/runtime"
38+
"k8s.io/apimachinery/pkg/selection"
39+
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
40+
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
41+
ctrl "sigs.k8s.io/controller-runtime"
42+
"sigs.k8s.io/controller-runtime/pkg/cache"
43+
"sigs.k8s.io/controller-runtime/pkg/client"
44+
"sigs.k8s.io/controller-runtime/pkg/healthz"
45+
"sigs.k8s.io/controller-runtime/pkg/log/zap"
46+
"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
47+
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
48+
"sigs.k8s.io/controller-runtime/pkg/webhook"
49+
50+
fso "github.com/cozystack/cozystack/internal/fluxshardoperator"
51+
)
52+
53+
var (
54+
scheme = runtime.NewScheme()
55+
setupLog = ctrl.Log.WithName("setup")
56+
)
57+
58+
func init() {
59+
utilruntime.Must(clientgoscheme.AddToScheme(scheme))
60+
}
61+
62+
func main() {
63+
var metricsAddr string
64+
var enableLeaderElection bool
65+
var probeAddr string
66+
var secureMetrics bool
67+
var enableHTTP2 bool
68+
var fluxNamespace string
69+
var shardCount string
70+
var shardConcurrent int
71+
var rebalanceThreshold float64
72+
var pinnedTenants string
73+
var shardCPURequest, shardCPULimit, shardMemoryRequest, shardMemoryLimit string
74+
var tlsOpts []func(*tls.Config)
75+
76+
flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
77+
"Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
78+
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
79+
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
80+
"Enable leader election for controller manager. "+
81+
"Enabling this will ensure there is only one active controller manager.")
82+
flag.BoolVar(&secureMetrics, "metrics-secure", true,
83+
"If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.")
84+
flag.BoolVar(&enableHTTP2, "enable-http2", false,
85+
"If set, HTTP/2 will be enabled for the metrics and webhook servers")
86+
flag.StringVar(&fluxNamespace, "flux-namespace", "cozy-fluxcd",
87+
"Namespace of the flux-aio Deployment and the shard Deployments.")
88+
flag.StringVar(&shardCount, "shard-count", fso.ShardCountAuto,
89+
"Number of helm-controller shards to provision and distribute tenants over: "+
90+
"\"auto\" (default) sizes from the tenant HelmRelease count with hysteresis, "+
91+
"or a positive integer pins it explicitly.")
92+
flag.IntVar(&shardConcurrent, "shard-concurrent", 5,
93+
"Value of --concurrent for each shard helm-controller.")
94+
flag.Float64Var(&rebalanceThreshold, "rebalance-threshold", 0.25,
95+
"Load spread ratio (maxLoad-minLoad)/avgLoad above which tenants are rebalanced between shards.")
96+
flag.StringVar(&pinnedTenants, "pinned-tenants", "",
97+
"Comma-separated tenant pins, e.g. tenant-bigone=shard3,tenant-other=shard0.")
98+
flag.StringVar(&shardCPURequest, "shard-cpu-request", "",
99+
"CPU request for shard helm-controllers; empty inherits the flux-aio value.")
100+
flag.StringVar(&shardCPULimit, "shard-cpu-limit", "",
101+
"CPU limit for shard helm-controllers; empty inherits the flux-aio value.")
102+
flag.StringVar(&shardMemoryRequest, "shard-memory-request", "",
103+
"Memory request for shard helm-controllers; empty inherits the flux-aio value.")
104+
flag.StringVar(&shardMemoryLimit, "shard-memory-limit", "",
105+
"Memory limit for shard helm-controllers; empty inherits the flux-aio value.")
106+
opts := zap.Options{
107+
Development: false,
108+
}
109+
opts.BindFlags(flag.CommandLine)
110+
flag.Parse()
111+
112+
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
113+
114+
pinned, err := fso.ParsePinnedTenants(pinnedTenants)
115+
if err != nil {
116+
setupLog.Error(err, "invalid --pinned-tenants")
117+
os.Exit(1)
118+
}
119+
explicitShards, autoShards, err := fso.ParseShardCount(shardCount)
120+
if err != nil {
121+
setupLog.Error(err, "invalid --shard-count")
122+
os.Exit(1)
123+
}
124+
shardResources, err := parseShardResources(shardCPURequest, shardCPULimit, shardMemoryRequest, shardMemoryLimit)
125+
if err != nil {
126+
setupLog.Error(err, "invalid shard resources")
127+
os.Exit(1)
128+
}
129+
cfg := &fso.Config{
130+
FluxNamespace: fluxNamespace,
131+
ShardCount: explicitShards,
132+
AutoShardCount: autoShards,
133+
ShardConcurrent: shardConcurrent,
134+
RebalanceThreshold: rebalanceThreshold,
135+
PinnedTenants: pinned,
136+
ShardResources: shardResources,
137+
}
138+
139+
// if the enable-http2 flag is false (the default), http/2 should be disabled
140+
// due to its vulnerabilities. More specifically, disabling http/2 will
141+
// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
142+
// Rapid Reset CVEs. For more information see:
143+
// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
144+
// - https://github.com/advisories/GHSA-4374-p667-p6c8
145+
disableHTTP2 := func(c *tls.Config) {
146+
setupLog.Info("disabling http/2")
147+
c.NextProtos = []string{"http/1.1"}
148+
}
149+
150+
if !enableHTTP2 {
151+
tlsOpts = append(tlsOpts, disableHTTP2)
152+
}
153+
154+
webhookServer := webhook.NewServer(webhook.Options{
155+
TLSOpts: tlsOpts,
156+
})
157+
158+
metricsServerOptions := metricsserver.Options{
159+
BindAddress: metricsAddr,
160+
SecureServing: secureMetrics,
161+
TLSOpts: tlsOpts,
162+
}
163+
164+
if secureMetrics {
165+
// FilterProvider is used to protect the metrics endpoint with authn/authz.
166+
metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
167+
}
168+
169+
shardKeyExists, err := labels.NewRequirement(fso.ShardKeyLabel, selection.Exists, nil)
170+
if err != nil {
171+
setupLog.Error(err, "building HelmRelease label selector")
172+
os.Exit(1)
173+
}
174+
175+
restConfig := ctrl.GetConfigOrDie()
176+
restConfig.QPS = 50.0 // Increased from default 5.0
177+
restConfig.Burst = 100 // Increased from default 10
178+
179+
mgr, err := ctrl.NewManager(restConfig, ctrl.Options{
180+
Scheme: scheme,
181+
Metrics: metricsServerOptions,
182+
WebhookServer: webhookServer,
183+
HealthProbeBindAddress: probeAddr,
184+
LeaderElection: enableLeaderElection,
185+
LeaderElectionID: "flux-shard-operator.cozystack.io",
186+
Cache: cache.Options{
187+
ByObject: map[client.Object]cache.ByObject{
188+
// Metadata-only stubs: the operator never decodes HelmRelease
189+
// specs/statuses, and the cache only holds tenant HRs (the
190+
// shard key label exists on every one of them).
191+
fso.HelmReleaseMeta(): {Label: labels.NewSelector().Add(*shardKeyExists)},
192+
fso.NamespaceMeta(): {},
193+
// Full Deployments, but only from the flux namespace.
194+
&appsv1.Deployment{}: {Namespaces: map[string]cache.Config{fluxNamespace: {}}},
195+
},
196+
},
197+
})
198+
if err != nil {
199+
setupLog.Error(err, "unable to start manager")
200+
os.Exit(1)
201+
}
202+
203+
placement := &fso.PlacementReconciler{Client: mgr.GetClient(), Config: cfg}
204+
if err := placement.SetupWithManager(mgr); err != nil {
205+
setupLog.Error(err, "unable to setup controller", "controller", "Placement")
206+
os.Exit(1)
207+
}
208+
shardSet := &fso.ShardSetReconciler{Client: mgr.GetClient(), Config: cfg}
209+
if err := shardSet.SetupWithManager(mgr); err != nil {
210+
setupLog.Error(err, "unable to setup controller", "controller", "ShardSet")
211+
os.Exit(1)
212+
}
213+
shardWebhook := &fso.ShardWebhook{Reader: mgr.GetClient()}
214+
if err := shardWebhook.SetupWithManager(mgr); err != nil {
215+
setupLog.Error(err, "unable to setup webhook", "webhook", "ShardWebhook")
216+
os.Exit(1)
217+
}
218+
219+
if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
220+
setupLog.Error(err, "unable to set up health check")
221+
os.Exit(1)
222+
}
223+
if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
224+
setupLog.Error(err, "unable to set up ready check")
225+
os.Exit(1)
226+
}
227+
228+
setupLog.Info("starting manager",
229+
"fluxNamespace", fluxNamespace, "shardCount", shardCount, "shardConcurrent", shardConcurrent)
230+
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
231+
setupLog.Error(err, "problem running manager")
232+
os.Exit(1)
233+
}
234+
}
235+
236+
// parseShardResources builds the shard helm-controller resource requirements
237+
// from flags; empty flags inherit the corresponding flux-aio values.
238+
func parseShardResources(cpuReq, cpuLim, memReq, memLim string) (corev1.ResourceRequirements, error) {
239+
res := corev1.ResourceRequirements{}
240+
set := func(list *corev1.ResourceList, name corev1.ResourceName, value string) error {
241+
if value == "" {
242+
return nil
243+
}
244+
q, err := resource.ParseQuantity(value)
245+
if err != nil {
246+
return err
247+
}
248+
if *list == nil {
249+
*list = corev1.ResourceList{}
250+
}
251+
(*list)[name] = q
252+
return nil
253+
}
254+
if err := set(&res.Requests, corev1.ResourceCPU, cpuReq); err != nil {
255+
return res, err
256+
}
257+
if err := set(&res.Limits, corev1.ResourceCPU, cpuLim); err != nil {
258+
return res, err
259+
}
260+
if err := set(&res.Requests, corev1.ResourceMemory, memReq); err != nil {
261+
return res, err
262+
}
263+
if err := set(&res.Limits, corev1.ResourceMemory, memLim); err != nil {
264+
return res, err
265+
}
266+
return res, nil
267+
}

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ require (
2121
github.com/spf13/cobra v1.9.1
2222
github.com/vmware-tanzu/velero v1.17.1
2323
go.uber.org/zap v1.27.0
24+
gomodules.xyz/jsonpatch/v2 v2.4.0
2425
k8s.io/api v0.34.1
2526
k8s.io/apiextensions-apiserver v0.34.1
2627
k8s.io/apimachinery v0.34.2
@@ -115,7 +116,6 @@ require (
115116
golang.org/x/text v0.37.0 // indirect
116117
golang.org/x/time v0.12.0 // indirect
117118
golang.org/x/tools v0.44.0 // indirect
118-
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
119119
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
120120
google.golang.org/genproto/googleapis/rpc v0.0.0-20260406210006-6f92a3bedf2d // indirect
121121
google.golang.org/grpc v1.80.0 // indirect

‎internal/fluxinstall/manifests/fluxcd-tenants.yaml‎

Lines changed: 0 additions & 102 deletions
This file was deleted.

0 commit comments

Comments
 (0)