Skip to content

Commit 0f1cc54

Browse files
fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover (#3034)
## What this PR does Fixes two failure modes in the COSI control plane built from upstream container-object-storage-interface v0.2.2, both carried as image patches. **BucketClaim never converges to ready (#3032).** The controller does not watch Bucket objects, and the generic controller drops no-op resync deltas, so a BucketClaim whose backend Bucket flips to ready *after* the first reconcile stayed `bucketReady=false` indefinitely — BucketAccess was never granted and the consuming app (bucket, harbor) got no credentials. The existing re-read of the live Bucket only ran during a reconcile that nothing re-triggered. The controller now returns an error while the Bucket is not ready, forcing a rate-limited requeue that re-reads the Bucket until its readiness propagates, so the claim converges deterministically instead of racing the async sidecar. **Slow provisioner failover on pod recreate (part of #2429).** The controller and the per-driver sidecar run single-replica, so leader election adds no availability — only failover latency. The upstream 150s lease made a recreated provisioner pod stall reconciliation for up to ~150s. The lease is lowered to the standard Kubernetes values (15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver v0.3.1 self-heal; the silent "Bucket ready with no backend bucket" finding on that issue is a separate driver bug and is left tracked there. Coverage: a unit test in the patched controller pins the requeue/propagation contract (fails without the requeue), and the bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound instead of tolerating the lag, with scoped diagnostics on the failure path. Closes #3032 Refs #2429 ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved object storage bucket readiness propagation so bucket claims now track the backend bucket’s actual `bucketReady` status and requeue when readiness isn’t yet observed. * Reduced and made readiness waits more reliable in end-to-end checks, with explicit match conditions and clearer debug output (including claim YAML, backend bucket list, and controller pod status). * Shortened leader-election timing to speed up recovery after pod restarts. * **Chores** * Updated automated PR labeling to map the object storage controller scope to the storage area label. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 0018a50 + 5991808 commit 0f1cc54

5 files changed

Lines changed: 218 additions & 11 deletions

File tree

‎.github/workflows/pr-labeler.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ jobs:
105105
// area/storage
106106
'seaweedfs': 'area/storage',
107107
'seaweedfs-cosi-driver': 'area/storage',
108+
'objectstorage-controller': 'area/storage',
108109
'bucket': 'area/storage',
109110
'linstor': 'area/storage',
110111
'velero': 'area/storage',

‎hack/e2e-apps/bucket.bats‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,24 @@ EOF
2121
# Wait for the bucket to be ready
2222
kubectl -n tenant-test wait hr bucket-${name} --timeout=5m --for=condition=ready
2323
timeout 60 sh -ec "until kubectl -n tenant-test get bucketclaims.objectstorage.k8s.io bucket-${name} >/dev/null 2>&1; do sleep 2; done"
24-
kubectl -n tenant-test wait bucketclaims.objectstorage.k8s.io bucket-${name} --timeout=300s --for=jsonpath='{.status.bucketReady}'
24+
# The COSI controller requeues the BucketClaim until the backend Bucket's
25+
# readiness propagates, so this converges within tens of seconds. Assert
26+
# bucketReady=true explicitly: a bare jsonpath match also matches the literal
27+
# "false", so the unqualified form passed against an unready claim. The tight
28+
# bound makes a propagation-race regression fail fast instead of hiding.
29+
kubectl -n tenant-test wait bucketclaims.objectstorage.k8s.io bucket-${name} --timeout=120s --for=jsonpath='{.status.bucketReady}'=true || {
30+
echo "=== BucketClaim did not converge to bucketReady=true ==="
31+
kubectl -n tenant-test get bucketclaims.objectstorage.k8s.io bucket-${name} -o yaml 2>&1 || true
32+
echo "=== backend Buckets (cluster-scoped) ==="
33+
kubectl get buckets.objectstorage.k8s.io -o wide 2>&1 || true
34+
echo "=== objectstorage-controller ==="
35+
kubectl -n cozy-objectstorage-controller get pods 2>&1 || true
36+
false
37+
}
2538
timeout 60 sh -ec "until kubectl -n tenant-test get bucketaccesses.objectstorage.k8s.io bucket-${name}-admin >/dev/null 2>&1; do sleep 2; done"
26-
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io bucket-${name}-admin --timeout=300s --for=jsonpath='{.status.accessGranted}'
39+
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io bucket-${name}-admin --timeout=300s --for=jsonpath='{.status.accessGranted}'=true
2740
timeout 60 sh -ec "until kubectl -n tenant-test get bucketaccesses.objectstorage.k8s.io bucket-${name}-viewer >/dev/null 2>&1; do sleep 2; done"
28-
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io bucket-${name}-viewer --timeout=300s --for=jsonpath='{.status.accessGranted}'
41+
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io bucket-${name}-viewer --timeout=300s --for=jsonpath='{.status.accessGranted}'=true
2942

3043
# Get admin (readwrite) credentials
3144
kubectl -n tenant-test get secret bucket-${name}-admin -ojsonpath='{.data.BucketInfo}' | base64 -d > bucket-admin-credentials.json

‎hack/e2e-apps/harbor.bats‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,22 @@ EOF
4343
timeout 60 sh -ec "until kubectl -n tenant-test get hr $release >/dev/null 2>&1; do sleep 2; done"
4444
kubectl -n tenant-test wait hr $release --timeout=5m --for=condition=ready
4545

46-
# Wait for COSI to provision bucket. The driver creates the Bucket and grants
47-
# access quickly, but the central COSI controller's propagation of the Bucket's
48-
# readiness back onto the namespaced BucketClaim can lag several minutes on a
49-
# loaded runner, so allow the same 10m budget the dependent HelmRelease gets.
46+
# Wait for COSI to provision the bucket. The driver creates the backend Bucket
47+
# and grants access; the central COSI controller requeues the BucketClaim until
48+
# the Bucket's readiness has propagated, so it converges within tens of seconds.
49+
# A frozen claim (the propagation race this guards against) never converges, so
50+
# the tight bound surfaces that regression instead of masking it.
5051
timeout 60 sh -ec "until kubectl -n tenant-test get bucketclaims.objectstorage.k8s.io $release-registry >/dev/null 2>&1; do sleep 2; done"
5152
kubectl -n tenant-test wait bucketclaims.objectstorage.k8s.io $release-registry \
52-
--timeout=600s --for=jsonpath='{.status.bucketReady}'=true
53+
--timeout=120s --for=jsonpath='{.status.bucketReady}'=true || {
54+
echo "=== BucketClaim did not converge to bucketReady=true ==="
55+
kubectl -n tenant-test get bucketclaims.objectstorage.k8s.io $release-registry -o yaml 2>&1 || true
56+
echo "=== backend Buckets (cluster-scoped) ==="
57+
kubectl get buckets.objectstorage.k8s.io -o wide 2>&1 || true
58+
echo "=== objectstorage-controller ==="
59+
kubectl -n cozy-objectstorage-controller get pods 2>&1 || true
60+
false
61+
}
5362
timeout 60 sh -ec "until kubectl -n tenant-test get bucketaccesses.objectstorage.k8s.io $release-registry >/dev/null 2>&1; do sleep 2; done"
5463
kubectl -n tenant-test wait bucketaccesses.objectstorage.k8s.io $release-registry \
5564
--timeout=60s --for=jsonpath='{.status.accessGranted}'=true

‎packages/system/objectstorage-controller/images/objectstorage/patches/92-bucketclaim-propagate-ready.diff‎

Lines changed: 164 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
diff --git a/controller/pkg/bucketclaim/bucketclaim.go b/controller/pkg/bucketclaim/bucketclaim.go
2-
index 2b55c49..038cf28 100644
2+
index 2b55c49..9195195 100644
33
--- a/controller/pkg/bucketclaim/bucketclaim.go
44
+++ b/controller/pkg/bucketclaim/bucketclaim.go
55
@@ -213,8 +213,23 @@ func (b *BucketClaimListener) provisionBucketClaimOperation(ctx context.Context,
66
return b.recordError(inputBucketClaim, v1.EventTypeWarning, v1alpha1.FailedCreateBucket, err)
77
}
8-
8+
99
+ // The Bucket for this claim may already exist from an earlier reconcile,
1010
+ // in which case the sidecar may have provisioned the backend and flipped
1111
+ // the Bucket to ready in the meantime. The controller does not watch
@@ -25,5 +25,166 @@ index 2b55c49..038cf28 100644
2525
- bucketClaim.Status.BucketReady = false
2626
+ bucketClaim.Status.BucketReady = createdBucket.Status.BucketReady
2727
}
28-
28+
2929
// Update status with retry logic for conflict errors
30+
@@ -282,6 +297,22 @@ func (b *BucketClaimListener) provisionBucketClaimOperation(ctx context.Context,
31+
return b.recordError(inputBucketClaim, v1.EventTypeWarning, v1alpha1.FailedCreateBucket, err)
32+
}
33+
34+
+ // The backend Bucket is provisioned asynchronously by the sidecar, which
35+
+ // flips Bucket.status.bucketReady on an object this controller does not
36+
+ // watch. The periodic informer resync cannot re-drive convergence either:
37+
+ // the generic controller drops no-op resync deltas (reflect.DeepEqual in
38+
+ // internal/runtime/controller.go) and nothing mutates the BucketClaim after
39+
+ // this point. So while the Bucket is not ready, return an error to force a
40+
+ // rate-limited requeue; each retry re-reads the live Bucket above until its
41+
+ // readiness has been propagated onto the claim. Surface a Warning event so
42+
+ // an operator can see why the claim is still pending (the returned error is
43+
+ // logged at klog.V(3), suppressed at default verbosity); event aggregation
44+
+ // folds the per-requeue repetition into a single counted entry.
45+
+ if !statusBucketReady {
46+
+ err := fmt.Errorf("backend Bucket %q for BucketClaim %q not ready yet; requeuing", bucketName, bucketClaim.ObjectMeta.Name)
47+
+ return b.recordError(inputBucketClaim, v1.EventTypeWarning, v1alpha1.WaitingForBucket, err)
48+
+ }
49+
+
50+
klog.V(3).Infof("Finished creating Bucket %v", bucketName)
51+
return nil
52+
}
53+
diff --git a/controller/pkg/bucketclaim/bucketclaim_test.go b/controller/pkg/bucketclaim/bucketclaim_test.go
54+
index fea6f16..837f7e5 100644
55+
--- a/controller/pkg/bucketclaim/bucketclaim_test.go
56+
+++ b/controller/pkg/bucketclaim/bucketclaim_test.go
57+
@@ -3,6 +3,7 @@ package bucketclaim
58+
import (
59+
"context"
60+
"fmt"
61+
+ "strings"
62+
"sync"
63+
"testing"
64+
65+
@@ -379,6 +380,18 @@ func TestRetryOnConflictStatusUpdate(t *testing.T) {
66+
t.Fatalf("Error occurred when creating BucketClaim: %v", err)
67+
}
68+
69+
+ // Pre-create the backend Bucket already flipped to ready so readiness
70+
+ // propagation lands true and Add completes. This test exercises the
71+
+ // status-update conflict-retry path, not the not-ready requeue path
72+
+ // (the latter is covered by TestProvisionRequeuesUntilBucketReady).
73+
+ readyBucket := &v1alpha1.Bucket{
74+
+ ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("bucket-%s", bucketClaim.UID)},
75+
+ Status: v1alpha1.BucketStatus{BucketReady: true},
76+
+ }
77+
+ if _, err := client.ObjectstorageV1alpha1().Buckets().Create(ctx, readyBucket, metav1.CreateOptions{}); err != nil {
78+
+ t.Fatalf("Error occurred when pre-creating ready Bucket: %v", err)
79+
+ }
80+
+
81+
// Cleanup
82+
defer util.DeleteObjects(ctx, client, *bucketClaim, *bucketclass)
83+
84+
@@ -436,8 +449,8 @@ func TestRetryOnConflictStatusUpdate(t *testing.T) {
85+
t.Errorf("Expected BucketName %s, got %s", expectedBucketName, updatedClaim.Status.BucketName)
86+
}
87+
88+
- if updatedClaim.Status.BucketReady != false {
89+
- t.Errorf("Expected BucketReady to be false, got %v", updatedClaim.Status.BucketReady)
90+
+ if updatedClaim.Status.BucketReady != true {
91+
+ t.Errorf("Expected BucketReady to be true, got %v", updatedClaim.Status.BucketReady)
92+
}
93+
94+
// Verify finalizer was added
95+
@@ -445,3 +458,95 @@ func TestRetryOnConflictStatusUpdate(t *testing.T) {
96+
t.Errorf("Expected finalizer to be added, but it was not found")
97+
}
98+
}
99+
+
100+
+// TestProvisionRequeuesUntilBucketReady pins the readiness-propagation contract.
101+
+// The controller does not watch Bucket objects and the generic controller drops
102+
+// no-op resync deltas, so the BucketClaim must be driven to ready by the
103+
+// provision path itself: Add returns an error (forcing a rate-limited requeue)
104+
+// while the backend Bucket is not ready, and returns nil with BucketReady=true
105+
+// once the Bucket's readiness has propagated. Without the requeue the claim
106+
+// would stay BucketReady=false forever and BucketAccess would never be granted.
107+
+func TestProvisionRequeuesUntilBucketReady(t *testing.T) {
108+
+ newListener := func(client *fakebucketclientset.Clientset) *BucketClaimListener {
109+
+ l := NewBucketClaimListener()
110+
+ l.InitializeKubeClient(fakekubeclientset.NewSimpleClientset())
111+
+ l.InitializeBucketClient(client)
112+
+ l.InitializeEventRecorder(record.NewFakeRecorder(3))
113+
+ return l
114+
+ }
115+
+
116+
+ t.Run("requeues while backend Bucket is not ready", func(t *testing.T) {
117+
+ ctx, cancel := context.WithCancel(context.Background())
118+
+ defer cancel()
119+
+
120+
+ client := fakebucketclientset.NewSimpleClientset()
121+
+ if _, err := util.CreateBucketClass(ctx, client, &goldClass); err != nil {
122+
+ t.Fatalf("Error occurred when creating BucketClass: %v", err)
123+
+ }
124+
+ bucketClaim, err := util.CreateBucketClaim(ctx, client, &bucketClaim1)
125+
+ if err != nil {
126+
+ t.Fatalf("Error occurred when creating BucketClaim: %v", err)
127+
+ }
128+
+
129+
+ // The controller creates the Bucket with BucketReady=false and nothing
130+
+ // flips it here, so Add must report an error to force a requeue.
131+
+ err = newListener(client).Add(ctx, bucketClaim)
132+
+ if err == nil {
133+
+ t.Fatal("Add should return an error to requeue while the backend Bucket is not ready")
134+
+ }
135+
+ // Pin the stable substring the production requeue error carries. A
136+
+ // refactor that drops the requeue (returning nil or a different error)
137+
+ // would silently regress convergence; this keeps the contract loud.
138+
+ if !strings.Contains(err.Error(), "not ready yet") {
139+
+ t.Fatalf("Expected requeue error to contain %q, got: %v", "not ready yet", err)
140+
+ }
141+
+
142+
+ updated, err := client.ObjectstorageV1alpha1().BucketClaims(bucketClaim.Namespace).Get(ctx, bucketClaim.Name, metav1.GetOptions{})
143+
+ if err != nil {
144+
+ t.Fatalf("Error occurred when reading BucketClaim: %v", err)
145+
+ }
146+
+ if updated.Status.BucketReady {
147+
+ t.Errorf("Expected BucketReady to stay false while the backend Bucket is not ready")
148+
+ }
149+
+ // Status (name + finalizer) is still persisted before the requeue.
150+
+ if updated.Status.BucketName != fmt.Sprintf("bucket-%s", bucketClaim.UID) {
151+
+ t.Errorf("Expected BucketName to be set even while not ready, got %q", updated.Status.BucketName)
152+
+ }
153+
+ })
154+
+
155+
+ t.Run("converges once the backend Bucket is ready", func(t *testing.T) {
156+
+ ctx, cancel := context.WithCancel(context.Background())
157+
+ defer cancel()
158+
+
159+
+ client := fakebucketclientset.NewSimpleClientset()
160+
+ if _, err := util.CreateBucketClass(ctx, client, &goldClass); err != nil {
161+
+ t.Fatalf("Error occurred when creating BucketClass: %v", err)
162+
+ }
163+
+ bucketClaim, err := util.CreateBucketClaim(ctx, client, &bucketClaim1)
164+
+ if err != nil {
165+
+ t.Fatalf("Error occurred when creating BucketClaim: %v", err)
166+
+ }
167+
+
168+
+ // Simulate the sidecar having already provisioned the backend and
169+
+ // flipped the Bucket to ready before this reconcile.
170+
+ readyBucket := &v1alpha1.Bucket{
171+
+ ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("bucket-%s", bucketClaim.UID)},
172+
+ Status: v1alpha1.BucketStatus{BucketReady: true},
173+
+ }
174+
+ if _, err := client.ObjectstorageV1alpha1().Buckets().Create(ctx, readyBucket, metav1.CreateOptions{}); err != nil {
175+
+ t.Fatalf("Error occurred when pre-creating ready Bucket: %v", err)
176+
+ }
177+
+
178+
+ if err := newListener(client).Add(ctx, bucketClaim); err != nil {
179+
+ t.Fatalf("Add should succeed once the backend Bucket is ready: %v", err)
180+
+ }
181+
+
182+
+ updated, err := client.ObjectstorageV1alpha1().BucketClaims(bucketClaim.Namespace).Get(ctx, bucketClaim.Name, metav1.GetOptions{})
183+
+ if err != nil {
184+
+ t.Fatalf("Error occurred when reading BucketClaim: %v", err)
185+
+ }
186+
+ if !updated.Status.BucketReady {
187+
+ t.Errorf("Expected BucketReady to propagate true once the backend Bucket is ready")
188+
+ }
189+
+ })
190+
+}
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
diff --git a/internal/runtime/controller.go b/internal/runtime/controller.go
2+
index 902116a..4cbe4c3 100644
3+
--- a/internal/runtime/controller.go
4+
+++ b/internal/runtime/controller.go
5+
@@ -171,10 +171,14 @@ func NewObjectStorageControllerWithClientset(identity string, leaderLockName str
6+
threadiness: threads,
7+
8+
ResyncPeriod: 30 * time.Second,
9+
- // leader election
10+
- LeaseDuration: 150 * time.Second,
11+
- RenewDeadline: 120 * time.Second,
12+
- RetryPeriod: 60 * time.Second,
13+
+ // Leader election. The COSI controller and the per-driver sidecar both
14+
+ // run single-replica, so leader election provides no HA — it only gates
15+
+ // how fast a freshly recreated pod takes over. Use the standard
16+
+ // Kubernetes leader-election timings instead of the upstream 150/120/60s
17+
+ // so a recreated pod resumes reconciliation in ~15s rather than ~150s.
18+
+ LeaseDuration: 15 * time.Second,
19+
+ RenewDeadline: 10 * time.Second,
20+
+ RetryPeriod: 2 * time.Second,
21+
22+
opMap: &sync.Map{},
23+
}, nil

0 commit comments

Comments
 (0)