fix: normalize annotation side of validateDeploymentMode before compare - #6069

Fixes #5885

Problem

validateDeploymentMode normalizes oldIsvc.Status.DeploymentMode through
constants.ParseDeploymentMode but compares it against the raw
newIsvc.Annotations[constants.DeploymentMode] string. Legacy alias
annotations (Serverless / RawDeployment) never normalize-equal their raw
form, so once the controller records the normalized status
(Knative / Standard), every subsequent update — including a completely
unchanged kubectl apply — is rejected:

update rejected: deploymentMode cannot be changed from 'Knative' to 'Serverless'

On versions without the DeletionTimestamp bypass in ValidateUpdate
(v0.16.x), this also blocks the finalizer-removal update the controller
issues during deletion, so the InferenceService deadlocks in Terminating
and no kubectl patch can remove the finalizer either, since the same
webhook path validates that update too.

This is the sibling of #5793/#5799: that pair fixed the empty status case
of the same comparison; this is the legacy alias case, left over because
only the status side was ever normalized.

Fix

Normalize the annotation the same way the status side already is, before
comparing:

rawAnnotationDeploymentMode, ok := newIsvc.Annotations[constants.DeploymentMode]
annotationDeploymentMode := string(constants.ParseDeploymentMode(rawAnnotationDeploymentMode))
if ok && annotationDeploymentMode != statusDeploymentMode {
    return fmt.Errorf("update rejected: deploymentMode cannot be changed from '%s' to '%s'", statusDeploymentMode, rawAnnotationDeploymentMode)
}

The error message still reports the raw annotation string (what the user
actually wrote), only the comparison itself is normalized. A real mode
change (e.g. legacy status Standard -> legacy annotation Serverless) is
still rejected — only a semantically-unchanged legacy alias now passes.

Sibling pattern found (not fixed here, flagging for maintainers)

validateCanarySpecs (same file, ~line 188) has the identical class of bug:

deploymentMode := isvc.Annotations[constants.DeploymentMode]
if deploymentMode != string(constants.Standard) {
    return fmt.Errorf("canary is only supported in %s deployment mode", constants.Standard)
}

This compares the raw annotation directly against the literal string
"Standard", so an InferenceService using the legacy RawDeployment
annotation (which is semantically Standard) would have its canary spec
rejected even though the deployment mode is correct. Left out of this PR to
keep the diff focused on the reported issue and its exact repro — happy to
open a follow-up if maintainers agree it's in scope.

Testing

Added regression cases to TestDeploymentModeUpdate in
inference_service_validation_test.go:

• unchanged legacy Serverless annotation against normalized Knative
  status -> now accepted (previously rejected, reproduced the fail-before
  the fix)
• unchanged legacy RawDeployment annotation against normalized Standard
  status -> now accepted
• a legacy alias annotation that actually changes the mode -> still rejected
  (guards against over-normalizing away real change detection)

$ go test ./pkg/apis/serving/v1beta1/... -run TestDeploymentModeUpdate -v
=== RUN   TestDeploymentModeUpdate
--- PASS: TestDeploymentModeUpdate (0.00s)
PASS
ok  	github.com/kserve/kserve/pkg/apis/serving/v1beta1	2.714s

$ go test ./pkg/apis/serving/v1beta1/...
ok  	github.com/kserve/kserve/pkg/apis/serving/v1beta1	1.447s

$ go vet ./pkg/apis/serving/v1beta1/...
(clean)

Before the fix, the added test failed with exactly the error from the issue
report:

update rejected: deploymentMode cannot be changed from 'Knative' to 'Serverless'

I have not reproduced the deletion-deadlock symptom itself end-to-end
(that needs envtest/a real cluster on a v0.16.x build without the
DeletionTimestamp bypass); the unit tests cover the underlying comparison
bug that causes it.