refactor: remove workload field in v1alpha2#281
Conversation
There was a problem hiding this comment.
Code Review
This pull request removes the deprecated Workload field from the v1alpha2 RoleSpec, replacing it with an annotation-based mechanism to maintain compatibility with v1alpha1. It introduces helper methods to retrieve workload specifications and updates the controller and validation logic accordingly. Feedback highlights a conversion bug where missing annotations result in empty workload fields in v1alpha1, and suggests enhancing code safety with nil checks and improving maintainability by using constants instead of hardcoded strings.
There was a problem hiding this comment.
Pull request overview
This PR removes the deprecated workload field from the v1alpha2 RoleSpec API surface and migrates workload selection to an internal/compatibility role annotation (rbg.workloads.x-k8s.io/role-workload-type), primarily to preserve v1alpha1↔v1alpha2 conversion behavior while keeping v1alpha2’s default workload as RoleInstanceSet.
Changes:
- Remove
spec.roles[].workloadfrom v1alpha2 CRDs and generated client/apply configuration; addRoleSpec.GetWorkloadType()/GetWorkloadSpec()helpers. - Update controllers/reconcilers/tests to branch on
GetWorkloadType()/GetWorkloadSpec()and to set/read the role workload type annotation where needed. - Update v1alpha1 conversion logic to store workload type in a v1alpha2 role annotation instead of a v1alpha2 field.
Reviewed changes
Copilot reviewed 36 out of 36 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
api/workloads/constants/annotation.go |
Adds RoleWorkloadTypeAnnotationKey constant used for workload-type annotation. |
api/workloads/v1alpha2/rolebasedgroup_types.go |
Removes RoleSpec.Workload; adds GetWorkloadType()/GetWorkloadSpec() helpers. |
api/workloads/v1alpha2/roletemplate_validation.go |
Switches workload-kind checks to use GetWorkloadSpec(). |
api/workloads/v1alpha2/helper.go |
Switches IsStatefulRole to use GetWorkloadType(). |
api/workloads/v1alpha2/helper_test.go |
Adds unit tests for GetWorkloadType() and GetWorkloadSpec(). |
api/workloads/v1alpha2/zz_generated.deepcopy.go |
Removes deepcopy of deleted Workload field. |
api/workloads/v1alpha1/rolebasedgroup_conversion.go |
Persists/restores workload via role annotation during v1alpha1↔v1alpha2 conversion. |
internal/controller/workloads/rolebasedgroup_controller.go |
Uses role.GetWorkloadSpec() when creating workload reconcilers. |
internal/controller/workloads/pod_controller.go |
Uses role.GetWorkloadSpec() for restart reconciliation. |
internal/controller/workloads/rolebasedgroupscalingadapter_controller.go |
Uses GetWorkloadSpec() to derive GVK for selector extraction. |
pkg/reconciler/svc_reconciler.go |
Switches service workload dispatch to GetWorkloadType(). |
pkg/reconciler/common.go |
Switches orphan cleanup matching to GetWorkloadSpec().Kind. |
pkg/reconciler/deploy_reconciler.go |
Switches orphan cleanup matching to GetWorkloadSpec().Kind. |
pkg/reconciler/lws_reconciler.go |
Switches orphan cleanup matching to GetWorkloadSpec().Kind. |
pkg/dependency/dependency.go |
Uses depRole.GetWorkloadSpec() for reconciler creation. |
config/crd/bases/workloads.x-k8s.io_rolebasedgroups.yaml |
Removes spec.roles[].workload schema from v1alpha2 CRD. |
config/crd/bases/workloads.x-k8s.io_rolebasedgroupsets.yaml |
Removes spec.roles[].workload schema from v1alpha2 RoleBasedGroupSet CRD. |
client-go/applyconfiguration/workloads/v1alpha2/rolespec.go |
Removes Workload from v1alpha2 applyconfiguration RoleSpec. |
client-go/applyconfiguration/workloads/v1alpha2/workloadspec.go |
Deletes now-unused v1alpha2 WorkloadSpec applyconfiguration type. |
client-go/applyconfiguration/utils.go |
Removes WorkloadSpec from kind→applyconfiguration mapping. |
cmd/cli/cmd/llm/run.go |
Stops setting the removed v1alpha2 Workload field when assembling RBGs. |
test/wrappers/v1alpha2/role_wrapper.go |
Updates test wrappers to set workload type via annotation; adds WithWorkload. |
test/e2e/testcase/v1alpha2/debug.go |
Uses GetWorkloadType() for debug switching. |
test/e2e/framework/rbg_v2_expect.go |
Uses GetWorkloadType() when mapping v1alpha2 roles to workload type. |
pkg/reconciler/*_test.go, pkg/discovery/*_test.go, pkg/utils/revision_utils_test.go, internal/controller/workloads/discovery_config_mode_test.go, test/e2e/testcase/v1alpha2/port_allocator.go |
Updates tests/fixtures to stop using removed Workload field and/or to set the new annotation. |
keps/workload-compatibility-mode/README.md |
Adds a KEP describing the refactor and compatibility approach. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
cheyang
left a comment
There was a problem hiding this comment.
The KEP doc examples use strings.Split(workloadType, "/") to parse the annotation value, but the actual implementation uses strings.LastIndex. The two approaches are logically equivalent for well-formed values, but the KEP should reflect the actual code to avoid confusion for future readers. Also worth noting in the KEP: the apiVersion/kind format (e.g. apps/v1/StatefulSet) is inherently ambiguous since / is used as both the group/version separator and the annotation delimiter — LastIndex correctly handles multi-/ apiVersions, but the format choice and its parsing strategy should be explicitly documented.
Consider updating the KEP code examples to use strings.LastIndex consistently, and adding a brief note on why this format was chosen over alternatives (e.g. JSON {apiVersion, kind}) and how edge cases (malformed values, empty values) are handled.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request removes the deprecated Workload field from the v1alpha2 RoleSpec and replaces it with a role-level annotation mechanism (rbg.workloads.x-k8s.io/role-workload-type). This change includes updating the conversion webhook to preserve workload types between v1alpha1 and v1alpha2, adding helper methods (GetWorkloadType and GetWorkloadSpec) to handle annotation parsing and defaults, and updating all controllers, tests, and CLI tools to use these new methods. A KEP has also been added to document this architectural change. Feedback includes a critical fix for the v2 to v1alpha1 conversion logic to ensure default workloads are applied when annotations are missing, and a suggestion to use constants instead of hardcoded strings in the GetWorkloadSpec fallback logic.
PR #281 Code Review SummaryOverall direction is sound: using a role-level annotation to carry v1alpha1 workload compatibility info keeps the v1alpha2 schema clean, and the default 🔴 Blocking Issues1. Annotation may leak to Pod / downstream workloads
Recommendation:
2.
|
aa18e4f to
1bad199
Compare
Signed-off-by: 玖宇 <[email protected]>
Signed-off-by: 玖宇 <[email protected]>
Signed-off-by: 玖宇 <[email protected]>
Signed-off-by: 玖宇 <[email protected]>
05e406b to
7f1b053
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 41 out of 41 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if idx > 0 { | ||
| return WorkloadSpec{ | ||
| APIVersion: wt[:idx], | ||
| Kind: wt[idx+1:], | ||
| } | ||
| } | ||
| // Fallback: derive default from the canonical constant to avoid drift. | ||
| defaultWT := constants.RoleInstanceSetWorkloadType | ||
| idx = strings.LastIndex(defaultWT, "/") | ||
| if idx > 0 { | ||
| return WorkloadSpec{ | ||
| APIVersion: defaultWT[:idx], | ||
| Kind: defaultWT[idx+1:], |
There was a problem hiding this comment.
GetWorkloadSpec() accepts any string from the role annotation, but the current strings.LastIndex(wt, "/") check only validates idx > 0. If the annotation ends with "/" (or otherwise yields an empty Kind), this returns a WorkloadSpec with Kind == "", which then flows into reconciler selection and can cause hard-to-diagnose "unsupported workload" errors. Consider validating both sides (idx > 0 && idx < len(wt)-1) and falling back to the default WorkloadSpec when either APIVersion or Kind is empty.
| if idx > 0 { | |
| return WorkloadSpec{ | |
| APIVersion: wt[:idx], | |
| Kind: wt[idx+1:], | |
| } | |
| } | |
| // Fallback: derive default from the canonical constant to avoid drift. | |
| defaultWT := constants.RoleInstanceSetWorkloadType | |
| idx = strings.LastIndex(defaultWT, "/") | |
| if idx > 0 { | |
| return WorkloadSpec{ | |
| APIVersion: defaultWT[:idx], | |
| Kind: defaultWT[idx+1:], | |
| if idx > 0 && idx < len(wt)-1 { | |
| apiVersion := wt[:idx] | |
| kind := wt[idx+1:] | |
| if apiVersion != "" && kind != "" { | |
| return WorkloadSpec{ | |
| APIVersion: apiVersion, | |
| Kind: kind, | |
| } | |
| } | |
| } | |
| // Fallback: derive default from the canonical constant to avoid drift. | |
| defaultWT := constants.RoleInstanceSetWorkloadType | |
| idx = strings.LastIndex(defaultWT, "/") | |
| if idx > 0 && idx < len(defaultWT)-1 { | |
| apiVersion := defaultWT[:idx] | |
| kind := defaultWT[idx+1:] | |
| if apiVersion != "" && kind != "" { | |
| return WorkloadSpec{ | |
| APIVersion: apiVersion, | |
| Kind: kind, | |
| } |
| // Workload – restore from annotation if present, otherwise default from the | ||
| // v1alpha2 role so v1alpha1 always has workload apiVersion/kind populated. | ||
| workloadSet := false | ||
| if src.Annotations != nil { | ||
| workloadType := src.Annotations[constants.RoleWorkloadTypeAnnotationKey] | ||
| if workloadType != "" { | ||
| // Parse "apiVersion/kind" format. Use LastIndex because apiVersion | ||
| // itself contains "/" (e.g. "leaderworkerset.x-k8s.io/v1"). | ||
| idx := strings.LastIndex(workloadType, "/") | ||
| if idx > 0 { | ||
| dst.Workload.APIVersion = workloadType[:idx] | ||
| dst.Workload.Kind = workloadType[idx+1:] | ||
| workloadSet = true | ||
| } |
There was a problem hiding this comment.
When restoring v1alpha1 WorkloadSpec from the v1alpha2 workload-type annotation, the parsing logic only checks idx > 0. If the annotation is malformed (e.g. trailing slash) this can set an empty Kind and mark workloadSet = true, preventing the intended default fallback. Consider requiring idx > 0 && idx < len(workloadType)-1 (and possibly trimming whitespace) before accepting the parsed values.
| // SystemManagedRoleAnnotations is the set of role-level annotations that are | ||
| // managed by the control plane and must not be propagated to downstream | ||
| // workload/Pod metadata or overridden by user-provided values. | ||
| var SystemManagedRoleAnnotations = map[string]struct{}{ | ||
| RoleSizeAnnotationKey: {}, | ||
| RoleWorkloadTypeAnnotationKey: {}, | ||
| } | ||
|
|
||
| // IsSystemManagedRoleAnnotation reports whether the given annotation key is | ||
| // a system-managed role annotation that should be filtered out when copying | ||
| // role annotations to downstream resources. | ||
| func IsSystemManagedRoleAnnotation(key string) bool { | ||
| _, ok := SystemManagedRoleAnnotations[key] | ||
| return ok |
There was a problem hiding this comment.
SystemManagedRoleAnnotations is an exported, mutable map, so any external package can modify it at runtime and accidentally change which annotations are filtered. To avoid global mutable state, consider making the map unexported (e.g. systemManagedRoleAnnotations) and exposing only IsSystemManagedRoleAnnotation, or returning a copy if you need to export the set.
| // GetCommonAnnotationsFromRole returns common annotations for a role. | ||
| // System-managed role annotations (e.g. RoleWorkloadTypeAnnotationKey, | ||
| // RoleSizeAnnotationKey) are filtered out to prevent them from leaking | ||
| // into downstream workload/Pod metadata or being overridden by user values. | ||
| func (rbg *RoleBasedGroup) GetCommonAnnotationsFromRole(role *RoleSpec) map[string]string { | ||
| return map[string]string{ | ||
| annotations := map[string]string{ | ||
| constants.RoleSizeAnnotationKey: fmt.Sprintf("%d", *role.Replicas), | ||
| } | ||
| // Propagate role annotations, but filter out system-managed keys that | ||
| // must not appear on downstream Pods/STS/Deployments. | ||
| for k, v := range role.Annotations { | ||
| if constants.IsSystemManagedRoleAnnotation(k) { | ||
| continue | ||
| } | ||
| annotations[k] = v | ||
| } | ||
| return annotations |
There was a problem hiding this comment.
GetCommonAnnotationsFromRole now attempts to prevent system-managed role annotations (including RoleWorkloadTypeAnnotationKey) from leaking to downstream workloads. However, several reconcilers still merge role.Annotations directly with GetCommonAnnotationsFromRole(role), which will re-introduce RoleWorkloadTypeAnnotationKey into the final workload annotations. To actually enforce the filtering, consider updating call sites to rely solely on GetCommonAnnotationsFromRole (since it already includes filtered role annotations), or provide a dedicated helper that returns the fully filtered/merged annotation set to use everywhere.
| This KEP proposes removing the deprecated `workload` field from v1alpha2 `RoleSpec` and replacing it with a role-level annotation mechanism. The workload type will now be specified via a role annotation `rbg.workloads.x-k8s.io/role-workload-type`, which is primarily used by the conversion webhook for v1alpha1 to v1alpha2 compatibility. New v1alpha2 RoleBasedGroups should not use this annotation as the default workload type (RoleInstanceSet) is appropriate for most use cases. | ||
|
|
||
| ## Motivation | ||
|
|
||
| In v1alpha2 API, the `workload` field in `RoleSpec` is deprecated. However, keeping it in the schema creates confusion for users and doesn't provide a clear deprecation path. By removing the field entirely and using annotations for backward compatibility, we: | ||
|
|
||
| 1. **Clean API Design**: Remove deprecated field from the schema, making the API cleaner | ||
| 2. **Clear Migration Path**: v1alpha1 RoleBasedGroups converted to v1alpha2 will have workload type preserved via role annotations | ||
| 3. **Prevent Misuse**: New v1alpha2 RBGs cannot accidentally use deprecated workload types through the schema | ||
| 4. **Internal Use Only**: The annotation is primarily for conversion webhook compatibility, not for general user consumption | ||
|
|
||
| ### Goals | ||
|
|
||
| - Remove `Workload` field and `WorkloadSpec` struct from v1alpha2 `RoleSpec` | ||
| - Add role annotation `rbg.workloads.x-k8s.io/role-workload-type` for specifying workload type | ||
| - Modify conversion webhook to set the annotation based on v1alpha1 workload field | ||
| - Modify controller to read workload type from annotation (default to RoleInstanceSet if not set) | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| - Change default workload behavior (RoleInstanceSet remains the default) | ||
| - Support user-specified annotation for new RBGs (this is for conversion compatibility only) | ||
| - Modify v1alpha1 API | ||
|
|
||
| ## Proposal | ||
|
|
||
| ### Design Overview | ||
|
|
||
| The implementation consists of three components: | ||
|
|
||
| 1. **API Change**: Remove `Workload` field from v1alpha2 `RoleSpec` | ||
| 2. **Annotation**: Add role-level annotation `rbg.workloads.x-k8s.io/role-workload-type` | ||
| 3. **Conversion Logic**: Set annotation during v1alpha1 -> v1alpha2 conversion | ||
| 4. **Controller Logic**: Read workload type from annotation instead of field |
There was a problem hiding this comment.
The KEP states the workload-type role annotation is "primarily" for conversion and that new v1alpha2 RoleBasedGroups "should not use" it. But after removing spec.roles[].workload, the annotation appears to be the only mechanism left for selecting non-default workloads (Deployment/StatefulSet/LeaderWorkerSet), and the v1alpha2 e2e tests/wrappers now set it explicitly. Please clarify in this KEP whether the annotation is a supported user-facing API surface for workload selection in v1alpha2, or document an alternative supported way to choose those workloads.
| This KEP proposes removing the deprecated `workload` field from v1alpha2 `RoleSpec` and replacing it with a role-level annotation mechanism. The workload type will now be specified via a role annotation `rbg.workloads.x-k8s.io/role-workload-type`, which is primarily used by the conversion webhook for v1alpha1 to v1alpha2 compatibility. New v1alpha2 RoleBasedGroups should not use this annotation as the default workload type (RoleInstanceSet) is appropriate for most use cases. | |
| ## Motivation | |
| In v1alpha2 API, the `workload` field in `RoleSpec` is deprecated. However, keeping it in the schema creates confusion for users and doesn't provide a clear deprecation path. By removing the field entirely and using annotations for backward compatibility, we: | |
| 1. **Clean API Design**: Remove deprecated field from the schema, making the API cleaner | |
| 2. **Clear Migration Path**: v1alpha1 RoleBasedGroups converted to v1alpha2 will have workload type preserved via role annotations | |
| 3. **Prevent Misuse**: New v1alpha2 RBGs cannot accidentally use deprecated workload types through the schema | |
| 4. **Internal Use Only**: The annotation is primarily for conversion webhook compatibility, not for general user consumption | |
| ### Goals | |
| - Remove `Workload` field and `WorkloadSpec` struct from v1alpha2 `RoleSpec` | |
| - Add role annotation `rbg.workloads.x-k8s.io/role-workload-type` for specifying workload type | |
| - Modify conversion webhook to set the annotation based on v1alpha1 workload field | |
| - Modify controller to read workload type from annotation (default to RoleInstanceSet if not set) | |
| ### Non-Goals | |
| - Change default workload behavior (RoleInstanceSet remains the default) | |
| - Support user-specified annotation for new RBGs (this is for conversion compatibility only) | |
| - Modify v1alpha1 API | |
| ## Proposal | |
| ### Design Overview | |
| The implementation consists of three components: | |
| 1. **API Change**: Remove `Workload` field from v1alpha2 `RoleSpec` | |
| 2. **Annotation**: Add role-level annotation `rbg.workloads.x-k8s.io/role-workload-type` | |
| 3. **Conversion Logic**: Set annotation during v1alpha1 -> v1alpha2 conversion | |
| 4. **Controller Logic**: Read workload type from annotation instead of field | |
| This KEP proposes removing the deprecated `workload` field from v1alpha2 `RoleSpec` and replacing it with a role-level annotation mechanism. The workload type will now be specified via a role annotation `rbg.workloads.x-k8s.io/role-workload-type`, which is used both by the conversion webhook for v1alpha1 to v1alpha2 compatibility and by v1alpha2 clients that need to select a non-default workload type. New v1alpha2 RoleBasedGroups do not need to set this annotation unless they want a workload type other than the default `RoleInstanceSet`. | |
| ## Motivation | |
| In v1alpha2 API, the `workload` field in `RoleSpec` is deprecated. However, keeping it in the schema creates confusion for users and doesn't provide a clear deprecation path. By removing the field entirely and using annotations for backward compatibility and non-default workload selection, we: | |
| 1. **Clean API Design**: Remove deprecated field from the schema, making the API cleaner | |
| 2. **Clear Migration Path**: v1alpha1 RoleBasedGroups converted to v1alpha2 will have workload type preserved via role annotations | |
| 3. **Single Mechanism**: Use one role-level annotation as the supported way to express non-default workload types in v1alpha2 | |
| 4. **Sensible Default**: Most new v1alpha2 RBGs can omit the annotation and use the default `RoleInstanceSet` | |
| ### Goals | |
| - Remove `Workload` field and `WorkloadSpec` struct from v1alpha2 `RoleSpec` | |
| - Add role annotation `rbg.workloads.x-k8s.io/role-workload-type` for specifying workload type | |
| - Modify conversion webhook to set the annotation based on v1alpha1 workload field | |
| - Modify controller to read workload type from annotation (default to `RoleInstanceSet` if not set) | |
| - Document the annotation as the supported v1alpha2 mechanism for selecting non-default workload types such as `Deployment`, `StatefulSet`, and `LeaderWorkerSet` | |
| ### Non-Goals | |
| - Change default workload behavior (`RoleInstanceSet` remains the default) | |
| - Re-introduce a dedicated `spec.roles[].workload` field in the v1alpha2 schema | |
| - Modify v1alpha1 API | |
| ## Proposal | |
| ### Design Overview | |
| The implementation consists of four components: | |
| 1. **API Change**: Remove `Workload` field from v1alpha2 `RoleSpec` | |
| 2. **Annotation**: Add role-level annotation `rbg.workloads.x-k8s.io/role-workload-type` as the supported way to select a non-default workload type in v1alpha2 | |
| 3. **Conversion Logic**: Set annotation during v1alpha1 -> v1alpha2 conversion | |
| 4. **Controller Logic**: Read workload type from annotation instead of field, defaulting to `RoleInstanceSet` when the annotation is absent |
Ⅰ. Motivation
Ⅱ. Modifications
Ⅲ. Does this pull request fix one issue?
fixes #XXXX
Ⅳ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅴ. Describe how to verify it
VI. Special notes for reviews
Checklist
make fmt.