kubectl delete doesn't account for list/watch permissions on resource being deleted since Kubernetes 1.25

What happened:

With the introduction of 77c921a the kubectl delete operation will indefinitely wait on the "resource deleted?" condition by means of list and watch when the user doesn't have permissions on the verb list and watch.

For example, if user is not given privilege to list or watch resources of kind "kind-foo", kubectl delete kind-foo resource-foo operation will never terminate even after resource-foo is removed from the server.

Not certain if the above change is intentional, however, the delete operation should account for the the case where user is forbidden to list and watch resources to resolve above side-effects.

What you expected to happen:

When user is not given list and watch access on kind-foo,kubectl delete kind-foo resource-foo should exit after it finds the reosurce-foo is removed in the apiserver.

How to reproduce it (as minimally and precisely as possible):

Grant user to create resource-foo (say demo-ns) of kind-foo (say ns or namespace) and revoke list and watch verbs in order not to see other user's resources.
kubectl delete kind-foo resource-foo # This step will never terminate even after resource-foo is deleted in server.

Anything else we need to know?:

NA

Environment:

Kubernetes client and server versions (use kubectl version):
Kubernetes server and kubectl version based on v1.25.6

@mpuckett159

@mpuckett159 what do you think?

Hm ok, I'll investigate further. Thank you for opening this.

@mpuckett159

/triage accepted
/priority important-soon

We should get the fix backported.

Marly thinks it's because visitor ignore errors is set incorrectly.

/assign @mpuckett159

I've tried to reproduce the issue but failed.

I've followed the below steps:

Created a namespace called aiotrix
Created a user and associated the role with only the create access to the services
Create a service using the user context
Finally delete the service using the same user context

I'm getting the below error response with immediately

Error from server (Forbidden): services "nginx" is forbidden: User "user1" cannot delete resource "services" in API group "" in the namespace "aiotrix"

Please correct me if I'm wrong.

Role:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: aiotrix
  name: pod-reader
rules:
- apiGroups: [""] # the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list", "delete"]
- apiGroups: [""] # the core API group
  resources: ["services"]
  verbs: ["create"]

Console:

➜  rbac kubectl config use-context user1-context
Switched to context "user1-context".
➜  rbac kubectl delete service nginx
Error from server (Forbidden): services "nginx" is forbidden: User "user1" cannot delete resource "services" in API group "" in the namespace "aiotrix"
➜  rbac kubectl get services
Error from server (Forbidden): services is forbidden: User "user1" cannot list resource "services" in API group "" in the namespace "aiotrix"

Kubectl version:

  buildDate: "2023-06-14T09:53:42Z"
  compiler: gc
  gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598
  gitTreeState: clean
  gitVersion: v1.27.3
  goVersion: go1.20.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2023-03-15T13:33:12Z"
  compiler: gc
  gitCommit: 9e644106593f3f4aa98f8a84b23db5fa378900bd
  gitTreeState: clean
  gitVersion: v1.26.3
  goVersion: go1.19.7
  major: "1"
  minor: "26"
  platform: linux/amd64```

@narendramadanapalli

@narendramadanapalli Could you please help me to reproduce this issue?

I got that problem too, my user can delete a namespace but is not allowed to list / watch others - so kubectl delete namespace XX will result in an endless error flow and never terminates:

$ kubectl delete namespace project-release 
namespace "project-release" deleted
W0801 11:07:28.867409  745140 reflector.go:533] vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: namespaces "project-release" is forbidden: User "u-87672762" cannot list resource "namespaces" in API group "" at the cluster scope
E0801 11:07:28.867560  745140 reflector.go:148] vendor/k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: namespaces "project-release" is forbidden: User "u-87672762" cannot list resource "namespaces" in API group "" at the cluster scope

... and so on ...

That will run endlessly and never terminate.
Version used is: v1.27.4

@soltysh

cc @soltysh for visibility to impact of wait changes on delete

as a workaround, you can indicate you don't want to wait for deletion with --wait=false

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

Confirm that this issue is still relevant with /triage accepted (org members only)
Deprioritize it with /priority important-longterm or /priority backlog
Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@mpuckett159

@mpuckett159 any update on this issue ?
We recently started hitting this on k8s version 1.28+

bash-4.4$ kubectl delete pod -n yb-dev-anab-test-readiness-probe          ybanab-test-r-us-west1-c-xfun-yb-tserver-0
pod "ybanab-test-r-us-west1-c-xfun-yb-tserver-0" deleted
E0801 21:08:59.640755 493522 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *unstructured.Unstructured: unknown
E0801 21:09:00.891846 493522 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *unstructured.Unstructured: unknown
E0801 21:09:03.706254 493522 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *unstructured.Unstructured: unknown
E0801 21:09:08.205278 493522 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *unstructured.Unstructured: unknown

Verified that the SA has role to delete the pod but not watch it.

bash-4.4$ kubectl version
Client Version: v1.28.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.10-gke.1075001

[PLAT-14793] [K8s]Rolling restart on k8s universe hangs at 6%

This is also happening to me when deleting a namespace using github actions:

Run:
kubectl delete namespace $PR_NAMESPACE
namespace "pr-33" deleted
E0912 02:18:00.967377    1[5](https://github.com/***/actions/runs/10806417576/job/30026001219#step:5:5)28 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/tools/watch/informerwatcher.go:14[6](https://github.com/***/actions/runs/10806417576/job/30026001219#step:5:6): Failed to watch *unstructured.Unstructured: the server has asked for the client to provide credentials"
W0912 02:18:02.362545    1528 reflector.go:561] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Unauthorized
E0912 02:18:02.362601    1528 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Unauthorized"
W0912 02:18:05.061413    1528 reflector.go:561] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Unauthorized
E0912 02:18:05.061454    1528 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Unauthorized"
W0912 02:18:08.91926[7](https://github.com/***/actions/runs/10806417576/job/30026001219#step:5:7)    1528 reflector.go:561] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Unauthorized
E0912 02:1[8](https://github.com/***/actions/runs/10806417576/job/30026001219#step:5:8):08.919313    1528 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Unauthorized"

The delete actually works fine, but my cleanup pipeline is always marked failed because of infinite error output.

Navigation Menu

Repository navigation

kubectl delete doesn't account for list/watch permissions on resource being deleted since Kubernetes 1.25 #1411

I've followed the below steps:

Role:

Kubectl version:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

kubectl delete doesn't account for list/watch permissions on resource being deleted since Kubernetes 1.25 #1411

Description

Activity

ardaguclu commented on Apr 14, 2023

mpuckett159 commented on Apr 14, 2023

eddiezane commented on Apr 26, 2023

praem90 commented on Jun 27, 2023

I've followed the below steps:

Role:

Kubectl version:

praem90 commented on Jul 10, 2023

tkrah commented on Aug 1, 2023

liggitt commented on Aug 1, 2023

k8s-triage-robot commented on Jan 19, 2024

k8s-triage-robot commented on Apr 18, 2024

k8s-triage-robot commented on May 18, 2024

amannijhawan commented on Aug 1, 2024

Jyosua commented on Sep 12, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions