Skip to content

Commit 07301c5

Browse files
authored
fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names (#5083)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Fixes #5014. A bucket's `Workload` was named `"bucket-" + <BucketClaim name>`, and a Bucket application names its claim after its release, which already starts with `bucket-`. A bucket an operator knows as `cozy-backups` therefore appeared as `bucket-bucket-cozy-backups`, and that string is the `workload` of the bucket's consumer in every billing usage report. - **Naming.** The `bucket-` prefix is now added only when the claim name does not already carry it. A Bucket application's claim `bucket-cozy-backups` gives the `Workload` `bucket-cozy-backups`; a claim without the prefix, such as `my-bucket`, still gives `bucket-my-bucket`. In the tree today every monitored BucketClaim comes from the Bucket application (Harbor's registry claim carries no labels its monitors select, so it has no `Workload` on either side of this change). - **Migration of existing Workloads.** Nothing garbage-collects bucket `Workload`s by name: they are owned by their BucketClaim and live as long as it does. Without a migration the old `bucket-bucket-…` object would stay next to the new one and the bucket would be metered twice. When the new `Workload` is reconciled, the controller looks up the legacy name and, only if the same BucketClaim (matched by UID) is its sole owner, uses its recorded sizes as the starting point and then deletes it, with UID and resourceVersion preconditions so a `Workload` that took over the name in the meantime survives. If the legacy `Workload` has other owners as well, it is another bucket's renamed `Workload` that a rolled-back controller attached this claim to: the controller takes neither its sizes nor the object and only drops its own owner reference. A legacy `Workload` with no owners left (a claim deleted with `--cascade=orphan` and recreated) that still carries this monitor's label is deleted the same way, without taking its sizes. Sizes already recorded at the new name win over the legacy ones only when the claim already owned that `Workload`. Carrying the sizes over matters because the controller deliberately keeps the last known size when the object-store metrics are unavailable; a fresh `Workload` would otherwise report no size until the next successful poll, which billing reads as an empty bucket. - **Never taking over another claim's Workload.** With Bucket applications `x` and `bucket-x` in one namespace, the legacy name of the first (`bucket-bucket-x`) is the new name of the second. The controller never takes over a `Workload` that has owners when the claim is not one of them (in practice another BucketClaim). While the new name is held that way, it keeps metering the bucket under its legacy name, without migrating, and emits a `BucketWorkloadNameTaken` warning on the monitor. For Bucket applications the other claim renames its `Workload` away on its own reconcile and the migration completes on the next pass. Only when the legacy name is held as well — which for a claim without the prefix, whose two names coincide, means a claim `x` outside the Bucket application next to the application `x` — is the bucket skipped, and the warning says it is not metered. Without this, whichever claim reconciled first decided whether the second bucket ended up with the first one's size or with no size at all. The same rule keeps two claims that map to one name (`x` and `bucket-x`) from sharing a `Workload`. **Rollback.** The migration is one-way. A controller rolled back to a previous version recreates `bucket-bucket-<name>` next to the renamed `bucket-<name>`, which its monitor keeps alive, so the bucket is metered twice until the `bucket-<name>` Workloads of Bucket applications are deleted by hand. Upgrading again after such a rollback converges without deleting another bucket's `Workload` (see above). **Known limitation.** Taking over an ownerless `Workload` keeps the sizes recorded on it as the last known ones, as the controller always has. If such an object was left behind by another bucket (for example `kubectl delete bucketclaim --cascade=orphan`), its sizes count for the new claim until the next successful size poll. **Effect on billing, worth knowing before the upgrade:** `Workload` names are the identity billing integrates over. A bucket's consumer changes name once, at the upgrade, so a usage report whose window spans the upgrade lists the bucket twice: under the old name up to the switch and under the new one after it. The totals are right; the split is cosmetic. Reports for windows entirely before or after the upgrade are unaffected. Tests (`internal/controller/workloadmonitor_controller_test.go`): - `TestBucketWorkloadName`: the naming rule for a Bucket application claim, a plain claim and a Harbor claim. - `TestReconcileBucketApp_NoDoublePrefix`: a Bucket application gets `bucket-cozy-backups` and no `bucket-bucket-cozy-backups`. - `TestReconcileBucketApp_ReplacesLegacyWorkload`: an existing `bucket-bucket-cozy-backups` owned by the claim is deleted, and its recorded logical and physical sizes (no metrics endpoint in the test) are carried over to the new `Workload`. - `TestReconcileBucketApp_KeepsLegacyNameOwnedByAnotherClaim`: a `Workload` under the legacy name owned by a different claim is left alone and its size is not taken. - `TestReconcileBucketApps_LegacyNameIsAnotherAppsNewName`: Bucket applications `x` and `bucket-x` with their legacy `Workload`s, reconciled in both orders, end with `bucket-x` and `bucket-bucket-x`, each with one owner and its own size, and the legacy `bucket-bucket-bucket-x` gone. - `TestReconcileBucketClaim_DoesNotAdoptAnotherClaimsWorkload`: a claim whose name is held by another claim's `Workload` leaves it untouched, emits `BucketWorkloadNameTaken` and is metered under its legacy name. - `TestReconcileBucketClaim_KeepsMeteringUnderLegacyNameWhileNewNameIsHeld`: the legacy `Workload` keeps being updated, with its size and owner, across repeated reconciles while the new name is held. - `TestReconcileBucketClaim_UnprefixedClaimWithHeldNameIsSkipped` and `TestReconcileBucketClaim_BothNamesHeldIsSkipped`: when no free name is left, nothing is taken over and the event says the bucket is not metered. - `TestReconcileBucketClaim_UnprefixedClaimKeepsItsWorkload`: for a claim without the prefix, where the legacy and the new name coincide, its own `Workload` and size survive repeated reconciles. - `TestReconcileBucketApp_RecordedSizeWinsOverLegacy`: a size already recorded on the new `Workload` takes precedence over the legacy one. - `TestReconcileBucketClaim_UpdatesItsOwnWorkload`: a claim's own `Workload`, with its owner reference carrying the `BucketClaim` kind as a live cluster's cache records it, keeps being updated and raises no warning (the fake client leaves that kind empty on objects the controller creates, so the other tests do not exercise this). - `TestReconcileBucketClaim_AdoptsOwnerlessWorkload`: a `Workload` that lost all owner references is taken over by the claim whose name it carries, rather than blocking that name forever. - `TestReconcileBucketApp_ReleasesSharedLegacyWorkload`: a legacy `Workload` that another claim also owns (the rollback case) keeps its size and its other owner; only this claim's reference is dropped, and its size is not carried over. - `TestReconcileBucketApp_KeepsWorkloadThatReplacedLegacy`: a `Workload` that replaces the legacy one under the same name between the lookup and the delete survives. - `TestReconcileBucketApp_RemovesOrphanedLegacyWorkload`, `TestReconcileBucketApp_KeepsOrphanOfAnotherMonitor`: an ownerless legacy `Workload` with this monitor's label is removed without carrying its sizes; one labelled for another monitor is left alone. - `TestReconcileBucketApp_LegacySizeWinsOverOwnerlessNewWorkload`: a stale size on an ownerless `Workload` at the new name loses to the migrating legacy `Workload`'s size. - `TestReconcileBucketApp_LegacyAlreadyGone`: a legacy `Workload` that disappears between the lookup and the delete (NotFound) does not fail the per-claim reconcile, and the size is still carried over. Reverting the naming rule, the legacy delete, the size carry-over, the owner check, the refusal to adopt, the same-name guard, the size precedence ignoring NotFound on the legacy delete, the own-claim match in the adoption check, the ownerless case, the physical size carry-over, the sole-owner rule for legacy Workloads, the delete preconditions, the legacy-name fallback or its own held check, the ownerless-legacy rule or the size precedence on takeover each fails one of these. `go test ./internal/... -count=1` and `go vet ./internal/controller/` pass. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, which touches only `internal/controller/workloadmonitor_controller.go` and its test: no package, values, schema, CRD, variant, `hack/` tooling, telemetry metric or cozy-proxy contract changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(workloadmonitor): Workloads of Bucket applications are no longer named with a doubled "bucket-" prefix (bucket-bucket-<name> becomes bucket-<name>). Existing Workloads are replaced on upgrade, keeping their recorded size; billing usage reports whose window spans the upgrade list such a bucket once under each name. Rolling the controller back afterwards requires deleting the bucket-<name> Workloads of Bucket applications, or the buckets are metered twice. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Bucket workloads now use a consistent name without duplicating the `bucket-` prefix. * Existing workload size data is retained when workloads are renamed or updated. * Workloads owned by another claim are preserved. When a naming conflict occurs, metering continues under the legacy name when possible; if both names are unavailable, metering is skipped and a warning is emitted. * Legacy workloads are cleaned up or updated according to their ownership, helping prevent duplicate or orphaned workloads. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 0b3f5dd + a6c14f6 commit 07301c5

2 files changed

Lines changed: 680 additions & 4 deletions

File tree

‎internal/controller/workloadmonitor_controller.go‎

Lines changed: 138 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -587,6 +587,59 @@ func (r *WorkloadMonitorReconciler) queryAllBucketMetrics(ctx context.Context, p
587587
return result, nil
588588
}
589589

590+
// bucketWorkloadName names the Workload of a BucketClaim. A Bucket application
591+
// names its claim after its release, which already starts with "bucket-", so
592+
// the prefix is added only when the claim name does not carry it.
593+
func bucketWorkloadName(claimName string) string {
594+
if strings.HasPrefix(claimName, "bucket-") {
595+
return claimName
596+
}
597+
return "bucket-" + claimName
598+
}
599+
600+
func ownedBy(w *cozyv1alpha1.Workload, bc *cosiv1alpha1.BucketClaim) bool {
601+
for _, ref := range w.OwnerReferences {
602+
if ref.UID == bc.UID {
603+
return true
604+
}
605+
}
606+
return false
607+
}
608+
609+
// ownedBySomeoneElse reports whether w has owners and bc is not one of them.
610+
func ownedBySomeoneElse(w *cozyv1alpha1.Workload, bc *cosiv1alpha1.BucketClaim) bool {
611+
return len(w.OwnerReferences) > 0 && !ownedBy(w, bc)
612+
}
613+
614+
// heldBySomeoneElse reports whether the Workload called name exists and has
615+
// owners other than bc.
616+
func (r *WorkloadMonitorReconciler) heldBySomeoneElse(ctx context.Context, bc *cosiv1alpha1.BucketClaim, name string) (bool, error) {
617+
w := &cozyv1alpha1.Workload{}
618+
if err := r.Get(ctx, client.ObjectKey{Namespace: bc.Namespace, Name: name}, w); err != nil {
619+
return false, client.IgnoreNotFound(err)
620+
}
621+
return ownedBySomeoneElse(w, bc), nil
622+
}
623+
624+
// legacyBucketWorkload returns the Workload a previous version created for bc
625+
// under the unconditionally prefixed name, if it is still there and either bc
626+
// is among its owners or it has lost every owner (a claim deleted with
627+
// --cascade=orphan and recreated) while still carrying monitor's label.
628+
func (r *WorkloadMonitorReconciler) legacyBucketWorkload(ctx context.Context, monitor *cozyv1alpha1.WorkloadMonitor, bc *cosiv1alpha1.BucketClaim) (*cozyv1alpha1.Workload, error) {
629+
name := "bucket-" + bc.Name
630+
if name == bucketWorkloadName(bc.Name) {
631+
return nil, nil
632+
}
633+
legacy := &cozyv1alpha1.Workload{}
634+
if err := r.Get(ctx, client.ObjectKey{Namespace: bc.Namespace, Name: name}, legacy); err != nil {
635+
return nil, client.IgnoreNotFound(err)
636+
}
637+
if ownedBy(legacy, bc) || (len(legacy.OwnerReferences) == 0 && legacy.Labels[workloadMonitorLabel] == monitor.Name) {
638+
return legacy, nil
639+
}
640+
return nil, nil
641+
}
642+
590643
// reconcileBucketClaimForMonitor creates or updates a Workload object for the given BucketClaim and WorkloadMonitor.
591644
func (r *WorkloadMonitorReconciler) reconcileBucketClaimForMonitor(
592645
ctx context.Context,
@@ -595,16 +648,68 @@ func (r *WorkloadMonitorReconciler) reconcileBucketClaimForMonitor(
595648
allMetrics map[string]*bucketMetrics,
596649
) error {
597650
logger := log.FromContext(ctx)
651+
652+
// The name may be held by another owner's Workload: Bucket applications
653+
// "x" and "bucket-x" make the legacy name of the first the new name of the
654+
// second, and a claim "x" outside the Bucket application maps to the same
655+
// name as the application "x". Adopting it would meter this bucket with
656+
// the other one's size. Keep metering the bucket under its legacy name
657+
// until the new one is free, and skip the bucket only when that name is
658+
// held as well.
659+
name := bucketWorkloadName(bc.Name)
660+
migrate := true
661+
held, err := r.heldBySomeoneElse(ctx, &bc, name)
662+
if err != nil {
663+
return err
664+
}
665+
if held {
666+
legacyName := "bucket-" + bc.Name
667+
legacyHeld := legacyName == name
668+
if !legacyHeld {
669+
if legacyHeld, err = r.heldBySomeoneElse(ctx, &bc, legacyName); err != nil {
670+
return err
671+
}
672+
}
673+
if legacyHeld {
674+
logger.Info("Workload name is held by another owner; not adopting it", "workload", name, "bucketClaim", bc.Name)
675+
if r.Recorder != nil {
676+
r.Recorder.Eventf(monitor, corev1.EventTypeWarning, "BucketWorkloadNameTaken",
677+
"Workload %s for BucketClaim %s is owned by another object; the bucket is not metered", name, bc.Name)
678+
}
679+
return nil
680+
}
681+
logger.Info("Workload name is held by another owner; keeping the legacy name", "workload", name, "legacy", legacyName, "bucketClaim", bc.Name)
682+
if r.Recorder != nil {
683+
r.Recorder.Eventf(monitor, corev1.EventTypeWarning, "BucketWorkloadNameTaken",
684+
"Workload %s for BucketClaim %s is owned by another object; metering the bucket as %s until it is free", name, bc.Name, legacyName)
685+
}
686+
name, migrate = legacyName, false
687+
}
598688
workload := &cozyv1alpha1.Workload{
599689
ObjectMeta: metav1.ObjectMeta{
600-
Name: fmt.Sprintf("bucket-%s", bc.Name),
690+
Name: name,
601691
Namespace: bc.Namespace,
602692
Labels: make(map[string]string, len(bc.Labels)),
603693
},
604694
}
605695

696+
var legacy *cozyv1alpha1.Workload
697+
if migrate {
698+
if legacy, err = r.legacyBucketWorkload(ctx, monitor, &bc); err != nil {
699+
return err
700+
}
701+
}
702+
// A legacy Workload with other owners is another bucket's renamed one that
703+
// a rolled-back controller also attached bc to: neither its sizes nor the
704+
// object are bc's to take. An ownerless one is removed, but its sizes may
705+
// be older than the claim it is recognised for.
706+
soleLegacy := legacy != nil && len(legacy.OwnerReferences) == 1
707+
orphanLegacy := legacy != nil && len(legacy.OwnerReferences) == 0
708+
606709
monitorLabels := r.getMonitorLabels(monitor)
607-
_, err := ctrl.CreateOrUpdate(ctx, r.Client, workload, func() error {
710+
_, err = ctrl.CreateOrUpdate(ctx, r.Client, workload, func() error {
711+
// sizes on a Workload bc did not own yet were not written for it
712+
ownedBefore := ownedBy(workload, &bc)
608713
updateOwnerReferences(workload.GetObjectMeta(), &bc)
609714

610715
if workload.Labels == nil {
@@ -628,11 +733,17 @@ func (r *WorkloadMonitorReconciler) reconcileBucketClaimForMonitor(
628733
// Start from the sizes already recorded on the Workload: when the
629734
// metrics endpoint is unreachable or reports nothing for this bucket,
630735
// the last known good values must survive rather than collapse to
631-
// "no size", which billing would read as an empty bucket.
736+
// "no size", which billing would read as an empty bucket. A Workload
737+
// renamed from its legacy name starts from the legacy sizes, so the
738+
// rename leaves no gap in the bucket's metered size.
632739
resources := make(map[string]resource.Quantity)
633740
for _, key := range []string{resourceS3StorageBytes, resourceS3PhysicalStorageBytes} {
634-
if q, ok := workload.Status.Resources[key]; ok {
741+
if q, ok := workload.Status.Resources[key]; ok && (ownedBefore || !soleLegacy) {
635742
resources[key] = q
743+
} else if soleLegacy {
744+
if q, ok := legacy.Status.Resources[key]; ok {
745+
resources[key] = q
746+
}
636747
}
637748
}
638749

@@ -660,6 +771,29 @@ func (r *WorkloadMonitorReconciler) reconcileBucketClaimForMonitor(
660771
return err
661772
}
662773

774+
// Billing reads every Workload as a consumer, so the legacy one must go
775+
// once its replacement exists, or the bucket is metered twice. The
776+
// preconditions delete only the object that was checked, not one that
777+
// took over the name since the lookup; a conflict is retried next pass.
778+
switch {
779+
case soleLegacy || orphanLegacy:
780+
err = r.Delete(ctx, legacy, client.Preconditions{UID: &legacy.UID, ResourceVersion: &legacy.ResourceVersion})
781+
case legacy != nil:
782+
orig := legacy.DeepCopy()
783+
owners := legacy.OwnerReferences[:0]
784+
for _, ref := range legacy.OwnerReferences {
785+
if ref.UID != bc.UID {
786+
owners = append(owners, ref)
787+
}
788+
}
789+
legacy.OwnerReferences = owners
790+
err = r.Patch(ctx, legacy, client.MergeFromWithOptions(orig, client.MergeFromWithOptimisticLock{}))
791+
}
792+
if client.IgnoreNotFound(err) != nil && !apierrors.IsConflict(err) {
793+
logger.Error(err, "Failed to release legacy bucket Workload", "workload", legacy.Name)
794+
return err
795+
}
796+
663797
return nil
664798
}
665799

0 commit comments

Comments
 (0)