Skip to content

Commit 2f10a08

Browse files
committed
feat(operator): auto-create KubeRay RBAC for Feast service account
When the batch-engine ConfigMap selects the Ray engine in KubeRay mode (type: ray.engine, use_kuberay: true), the Feast service pod uses the CodeFlare SDK to discover RayCluster resources and read mTLS Secrets. Previously the Feast SA had no permissions on either, so the SDK calls returned 403 and users had to apply the Role + RoleBinding by hand before every materialization run. This change makes the operator provision them automatically: - New services/ray_rbac.go reads the batch-engine ConfigMap once per reconcile and, when KubeRay is selected, CreateOrUpdates a namespace-scoped Role + RoleBinding named feast-<crName>-kuberay granting the Feast SA: ray.io/rayclusters: get, list, watch core/secrets: get, list, watch, create, update, delete - Both resources are owner-referenced to the FeatureStore so they GC with the CR. When use_kuberay flips back to false (or the batchEngine block is removed), they are deleted on the next reconcile. - The operator's own kubebuilder RBAC markers are widened to match so it can hand those verbs to the Feast SA (k8s RBAC escalation rules require the granter to hold the granted verbs). config/rbac and dist/install.yaml are regenerated accordingly. - New ginkgo suite covers create-on-enable, delete-on-disable, and no-op when batchEngine is absent. - 06-batch-and-jobs.md documents the new auto-RBAC behavior so users know manual setup is no longer required. Fixes feast-dev#6408 Signed-off-by: Aravind Nidadavolu <[email protected]>
1 parent f77981c commit 2f10a08

8 files changed

Lines changed: 419 additions & 5 deletions

File tree

‎.secrets.baseline‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/how-to-guides/feast-operator/06-batch-and-jobs.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,33 @@ spec:
6363
> For engine-specific YAML options (Spark conf, Ray address, etc.) see the
6464
> [Feast SDK — Compute Engine](../reference/compute-engine/) docs.
6565

66+
### KubeRay clusters — auto-generated RBAC
67+
68+
When the batch-engine ConfigMap selects the Ray engine in KubeRay mode:
69+
70+
```yaml
71+
data:
72+
config: |
73+
type: ray.engine
74+
use_kuberay: true
75+
cluster_name: my-ray-cluster
76+
```
77+
78+
the operator creates a namespace-scoped `Role` and `RoleBinding` named
79+
`feast-<crName>-kuberay`, owner-referenced to the `FeatureStore`. The Role
80+
grants the Feast service account:
81+
82+
| API group | Resource | Verbs |
83+
|-----------|----------|-------|
84+
| `ray.io` | `rayclusters` | `get`, `list`, `watch` |
85+
| (core) | `secrets` | `get`, `list`, `watch`, `create`, `update`, `delete` |
86+
87+
This is what the CodeFlare SDK needs to discover the `RayCluster` and read
88+
the mTLS Secrets used for the Ray client connection. When you flip
89+
`use_kuberay` back to `false` (or remove the `batchEngine` field), the
90+
Role and RoleBinding are deleted on the next reconcile. No manual RBAC
91+
setup is required.
92+
6693
---
6794

6895
## Scheduled Materialization (`spec.cronJob`)

‎infra/feast-operator/config/rbac/role.yaml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ rules:
99
resources:
1010
- configmaps
1111
- persistentvolumeclaims
12+
- secrets
1213
- serviceaccounts
1314
- services
1415
verbs:
@@ -23,7 +24,6 @@ rules:
2324
resources:
2425
- namespaces
2526
- pods
26-
- secrets
2727
verbs:
2828
- get
2929
- list
@@ -124,6 +124,14 @@ rules:
124124
- patch
125125
- update
126126
- watch
127+
- apiGroups:
128+
- ray.io
129+
resources:
130+
- rayclusters
131+
verbs:
132+
- get
133+
- list
134+
- watch
127135
- apiGroups:
128136
- rbac.authorization.k8s.io
129137
resources:

‎infra/feast-operator/dist/install.yaml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20919,6 +20919,7 @@ rules:
2091920919
resources:
2092020920
- configmaps
2092120921
- persistentvolumeclaims
20922+
- secrets
2092220923
- serviceaccounts
2092320924
- services
2092420925
verbs:
@@ -20933,7 +20934,6 @@ rules:
2093320934
resources:
2093420935
- namespaces
2093520936
- pods
20936-
- secrets
2093720937
verbs:
2093820938
- get
2093920939
- list
@@ -21034,6 +21034,14 @@ rules:
2103421034
- patch
2103521035
- update
2103621036
- watch
21037+
- apiGroups:
21038+
- ray.io
21039+
resources:
21040+
- rayclusters
21041+
verbs:
21042+
- get
21043+
- list
21044+
- watch
2103721045
- apiGroups:
2103821046
- rbac.authorization.k8s.io
2103921047
resources:

‎infra/feast-operator/internal/controller/featurestore_controller.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,11 @@ type FeatureStoreReconciler struct {
6666
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;create;update;watch;delete
6767
// +kubebuilder:rbac:groups=core,resources=services;configmaps;persistentvolumeclaims;serviceaccounts,verbs=get;list;create;update;watch;delete
6868
// +kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=roles;rolebindings;clusterroles;clusterrolebindings;subjectaccessreviews,verbs=get;list;create;update;watch;delete
69-
// +kubebuilder:rbac:groups=core,resources=secrets;pods;namespaces,verbs=get;list;watch
69+
// +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;create;update;delete
70+
// +kubebuilder:rbac:groups=core,resources=pods;namespaces,verbs=get;list;watch
7071
// +kubebuilder:rbac:groups=core,resources=pods/exec,verbs=create
7172
// +kubebuilder:rbac:groups=authentication.k8s.io,resources=tokenreviews,verbs=create
73+
// +kubebuilder:rbac:groups=ray.io,resources=rayclusters,verbs=get;list;watch
7274
// +kubebuilder:rbac:groups=route.openshift.io,resources=routes,verbs=get;list;create;update;watch;delete
7375
// +kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
7476
// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete
Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
/*
2+
Copyright 2024 Feast Community.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package controller
18+
19+
import (
20+
"context"
21+
22+
. "github.com/onsi/ginkgo/v2"
23+
. "github.com/onsi/gomega"
24+
corev1 "k8s.io/api/core/v1"
25+
rbacv1 "k8s.io/api/rbac/v1"
26+
"k8s.io/apimachinery/pkg/api/errors"
27+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
28+
"k8s.io/apimachinery/pkg/types"
29+
"sigs.k8s.io/controller-runtime/pkg/reconcile"
30+
31+
feastdevv1 "github.com/feast-dev/feast/infra/feast-operator/api/v1"
32+
"github.com/feast-dev/feast/infra/feast-operator/internal/controller/services"
33+
)
34+
35+
var _ = Describe("FeatureStore Controller-KubeRay RBAC", func() {
36+
const (
37+
batchConfigMapName = "ray-batch-engine"
38+
)
39+
var pullPolicy = corev1.PullAlways
40+
ctx := context.Background()
41+
42+
createBatchEngineConfigMap := func(yamlBody string) {
43+
cm := &corev1.ConfigMap{
44+
ObjectMeta: metav1.ObjectMeta{Name: batchConfigMapName, Namespace: "default"},
45+
Data: map[string]string{"config": yamlBody},
46+
}
47+
Expect(k8sClient.Create(ctx, cm)).To(Succeed())
48+
}
49+
50+
deleteBatchEngineConfigMap := func() {
51+
cm := &corev1.ConfigMap{
52+
ObjectMeta: metav1.ObjectMeta{Name: batchConfigMapName, Namespace: "default"},
53+
}
54+
_ = k8sClient.Delete(ctx, cm)
55+
}
56+
57+
reconcileOnce := func(name string) {
58+
controllerReconciler := &FeatureStoreReconciler{
59+
Client: k8sClient,
60+
Scheme: k8sClient.Scheme(),
61+
}
62+
_, err := controllerReconciler.Reconcile(ctx, reconcile.Request{
63+
NamespacedName: types.NamespacedName{Name: name, Namespace: "default"},
64+
})
65+
Expect(err).NotTo(HaveOccurred())
66+
}
67+
68+
rbacKey := func(fsName string) types.NamespacedName {
69+
return types.NamespacedName{
70+
Name: services.GetFeastName(&feastdevv1.FeatureStore{ObjectMeta: metav1.ObjectMeta{Name: fsName}}) + "-kuberay",
71+
Namespace: "default",
72+
}
73+
}
74+
75+
Context("when batchEngine has type: ray.engine and use_kuberay: true", func() {
76+
const resourceName = "kuberay-rbac-enabled"
77+
78+
BeforeEach(func() {
79+
createBatchEngineConfigMap("type: ray.engine\nuse_kuberay: true\ncluster_name: my-cluster\n")
80+
resource := createFeatureStoreResource(resourceName, image, pullPolicy, &[]corev1.EnvVar{}, nil)
81+
resource.Spec.BatchEngine = &feastdevv1.BatchEngineConfig{
82+
ConfigMapRef: &corev1.LocalObjectReference{Name: batchConfigMapName},
83+
}
84+
Expect(k8sClient.Create(ctx, resource)).To(Succeed())
85+
})
86+
87+
AfterEach(func() {
88+
resource := &feastdevv1.FeatureStore{}
89+
if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceName, Namespace: "default"}, resource); err == nil {
90+
Expect(k8sClient.Delete(ctx, resource)).To(Succeed())
91+
}
92+
deleteBatchEngineConfigMap()
93+
})
94+
95+
It("creates a Role with KubeRay rules and a RoleBinding to the Feast SA", func() {
96+
reconcileOnce(resourceName)
97+
98+
role := &rbacv1.Role{}
99+
Expect(k8sClient.Get(ctx, rbacKey(resourceName), role)).To(Succeed())
100+
Expect(role.Rules).To(ConsistOf(
101+
rbacv1.PolicyRule{
102+
APIGroups: []string{"ray.io"},
103+
Resources: []string{"rayclusters"},
104+
Verbs: []string{"get", "list", "watch"},
105+
},
106+
rbacv1.PolicyRule{
107+
APIGroups: []string{""},
108+
Resources: []string{"secrets"},
109+
Verbs: []string{"get", "list", "watch", "create", "update", "delete"},
110+
},
111+
))
112+
113+
fs := &feastdevv1.FeatureStore{}
114+
Expect(k8sClient.Get(ctx, types.NamespacedName{Name: resourceName, Namespace: "default"}, fs)).To(Succeed())
115+
Expect(role.OwnerReferences).To(HaveLen(1))
116+
Expect(role.OwnerReferences[0].UID).To(Equal(fs.UID))
117+
118+
binding := &rbacv1.RoleBinding{}
119+
Expect(k8sClient.Get(ctx, rbacKey(resourceName), binding)).To(Succeed())
120+
Expect(binding.RoleRef).To(Equal(rbacv1.RoleRef{
121+
APIGroup: rbacv1.GroupName,
122+
Kind: "Role",
123+
Name: rbacKey(resourceName).Name,
124+
}))
125+
Expect(binding.Subjects).To(ConsistOf(rbacv1.Subject{
126+
Kind: rbacv1.ServiceAccountKind,
127+
Name: services.GetFeastName(fs),
128+
Namespace: "default",
129+
}))
130+
Expect(binding.OwnerReferences).To(HaveLen(1))
131+
Expect(binding.OwnerReferences[0].UID).To(Equal(fs.UID))
132+
})
133+
})
134+
135+
Context("when batchEngine ConfigMap changes from use_kuberay: true to false", func() {
136+
const resourceName = "kuberay-rbac-disable"
137+
138+
BeforeEach(func() {
139+
createBatchEngineConfigMap("type: ray.engine\nuse_kuberay: true\n")
140+
resource := createFeatureStoreResource(resourceName, image, pullPolicy, &[]corev1.EnvVar{}, nil)
141+
resource.Spec.BatchEngine = &feastdevv1.BatchEngineConfig{
142+
ConfigMapRef: &corev1.LocalObjectReference{Name: batchConfigMapName},
143+
}
144+
Expect(k8sClient.Create(ctx, resource)).To(Succeed())
145+
})
146+
147+
AfterEach(func() {
148+
resource := &feastdevv1.FeatureStore{}
149+
if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceName, Namespace: "default"}, resource); err == nil {
150+
Expect(k8sClient.Delete(ctx, resource)).To(Succeed())
151+
}
152+
deleteBatchEngineConfigMap()
153+
})
154+
155+
It("deletes the Role and RoleBinding on the next reconcile", func() {
156+
reconcileOnce(resourceName)
157+
Expect(k8sClient.Get(ctx, rbacKey(resourceName), &rbacv1.Role{})).To(Succeed())
158+
159+
cm := &corev1.ConfigMap{}
160+
Expect(k8sClient.Get(ctx, types.NamespacedName{Name: batchConfigMapName, Namespace: "default"}, cm)).To(Succeed())
161+
cm.Data["config"] = "type: ray.engine\nuse_kuberay: false\n"
162+
Expect(k8sClient.Update(ctx, cm)).To(Succeed())
163+
164+
reconcileOnce(resourceName)
165+
166+
err := k8sClient.Get(ctx, rbacKey(resourceName), &rbacv1.Role{})
167+
Expect(errors.IsNotFound(err)).To(BeTrue(), "expected Role to be deleted, got %v", err)
168+
err = k8sClient.Get(ctx, rbacKey(resourceName), &rbacv1.RoleBinding{})
169+
Expect(errors.IsNotFound(err)).To(BeTrue(), "expected RoleBinding to be deleted, got %v", err)
170+
})
171+
})
172+
173+
Context("when no batchEngine is configured", func() {
174+
const resourceName = "kuberay-rbac-absent"
175+
176+
BeforeEach(func() {
177+
resource := createFeatureStoreResource(resourceName, image, pullPolicy, &[]corev1.EnvVar{}, nil)
178+
Expect(k8sClient.Create(ctx, resource)).To(Succeed())
179+
})
180+
181+
AfterEach(func() {
182+
resource := &feastdevv1.FeatureStore{}
183+
if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceName, Namespace: "default"}, resource); err == nil {
184+
Expect(k8sClient.Delete(ctx, resource)).To(Succeed())
185+
}
186+
})
187+
188+
It("does not create the KubeRay Role or RoleBinding", func() {
189+
reconcileOnce(resourceName)
190+
191+
err := k8sClient.Get(ctx, rbacKey(resourceName), &rbacv1.Role{})
192+
Expect(errors.IsNotFound(err)).To(BeTrue(), "expected no Role, got %v", err)
193+
err = k8sClient.Get(ctx, rbacKey(resourceName), &rbacv1.RoleBinding{})
194+
Expect(errors.IsNotFound(err)).To(BeTrue(), "expected no RoleBinding, got %v", err)
195+
})
196+
})
197+
})

0 commit comments

Comments
 (0)