Skip to content

fix bug on kubectl deleting uninitialized resources #51186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 6, 2017
Merged

fix bug on kubectl deleting uninitialized resources #51186

merged 2 commits into from
Sep 6, 2017

Conversation

dixudx
Copy link
Member

@dixudx dixudx commented Aug 23, 2017

What this PR does / why we need it:

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #51185

Special notes for your reviewer:
/assign @caesarxuchao @ahmetb

Release note:

fix bug on kubectl deleting uninitialized resources

@k8s-ci-robot
Copy link
Contributor

Hi @dixudx. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 23, 2017
@k8s-github-robot k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Aug 23, 2017
liggitt
liggitt reviewed Aug 23, 2017
View reviewed changes
pkg/kubectl/delete.go Outdated
@@ -226,6 +226,17 @@ func (reaper *ReplicaSetReaper) Stop(namespace, name string, timeout time.Durati
timeout = Timeout + time.Duration(10*rs.Spec.Replicas)*time.Second
}

_, err = reaper.updateReplicaSetWithRetries(namespace, name, func(rs *extensions.ReplicaSet) {
// set replicaset's initializers to nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chasing disabling initializers on everything seems very error-prone. @smarterclayton, what would you expect to happen here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely think that this is a bug in the reaper. Why do we still have the reaper?

A normal user isn't going to be able to perform this operation, so this isn't the correct fix. I think if the object hasn't been initialized yet the reaper should just call delete.

@dims
Copy link
Member

dims commented Aug 25, 2017

/ok-to-test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 25, 2017
@caesarxuchao
Copy link
Member

@dixudx could you explain what's the reason of the deletion failure? Is it because kubectl is waiting for the controller's status (like observedGeneration, replicas etc) to update? If that's the case, you can early return in the Scale() call. That way you don't need an extra GET/UPDATE.

The entire problem will be gone when we move kubectl delete to use the server-side garbage collection (tracked in #50340).

@dixudx
Copy link
Member Author

dixudx commented Aug 26, 2017

Is it because kubectl is waiting for the controller's status (like observedGeneration, replicas etc) to update?

@caesarxuchao Yes, current kubectl will keeps watching the status changes. While for such uninitialized objects with pending initializers, nothing will get changed.

If that's the case, you can early return in the Scale() call.

Yes, this is a nicer way to solve this problem. That means we have to make special pass for Deployment, ReplicaSet, and StatefulSet in the Scale() call, like

if rs.Initializers != nil {
     return nil
}

@liggitt @caesarxuchao WDYT?

@caesarxuchao
Copy link
Member

@dixudx that looks right to me.

@smarterclayton the removal of the reapers is tracked at: #50340

@dixudx dixudx force-pushed the fix_delete_uninitialized_resources branch from 95bddf0 to 493f063 Compare August 27, 2017 00:49
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 27, 2017
@dixudx
Copy link
Member Author

dixudx commented Aug 27, 2017

ping @caesarxuchao @smarterclayton @liggitt

Updated. PTAL. Thanks.

@dixudx
Copy link
Member Author

dixudx commented Aug 27, 2017

/retest

@smarterclayton
Copy link
Contributor

Much simpler

@dixudx
Copy link
Member Author

dixudx commented Aug 28, 2017

/retest

caesarxuchao
caesarxuchao reviewed Aug 30, 2017
View reviewed changes
pkg/kubectl/delete.go Outdated
@@ -400,6 +400,9 @@ func (reaper *DeploymentReaper) Stop(namespace, name string, timeout time.Durati
d.Spec.RevisionHistoryLimit = &rhl
d.Spec.Replicas = 0
d.Spec.Paused = true
if d.Initializers != nil {
d.Initializers = nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't assume kubectl user has the permission to set initializers. Just get the deployment at the beginnig at Stop and delete it immediately if initializers!=nil.

@caesarxuchao
Copy link
Member

Could you add some tests? Maybe one test case for kubectl delete rs and another one for kubectl delete deployment, since the mechanisms of theirs reapers are different.

@dixudx dixudx force-pushed the fix_delete_uninitialized_resources branch from 493f063 to 7448365 Compare August 31, 2017 06:28
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 31, 2017
@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 31, 2017
@dixudx dixudx force-pushed the fix_delete_uninitialized_resources branch from 7448365 to 41c8799 Compare August 31, 2017 06:34
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 31, 2017
@dixudx
Copy link
Member Author

dixudx commented Aug 31, 2017

@caesarxuchao Add the tests. PTAL.

caesarxuchao
caesarxuchao reviewed Aug 31, 2017
View reviewed changes
hack/make-rules/test-cmd-util.sh Outdated
# Pre-condition: no deployment exists
kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''
# Create a deployment
kubectl create -f hack/testdata/deployment-with-initializer.yaml "${kube_flags[@]}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#51436 disabled the initializer field if the feature gate is off, perhaps you need to enable the feature gate in this test.

@dixudx dixudx force-pushed the fix_delete_uninitialized_resources branch from d5471b5 to 5e120cf Compare September 1, 2017 02:46
@caesarxuchao
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017
@caesarxuchao
Copy link
Member

/assign @liggitt @fabianofranz for approval

Thanks.

@k8s-ci-robot
Copy link
Contributor

@caesarxuchao: GitHub didn't allow me to assign the following users: for, approval.

Note that only kubernetes members can be assigned.

In response to this:

/assign @liggitt @fabianofranz for approval

Thanks.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@caesarxuchao
Copy link
Member

This is a bug fix, so it's ok we don't get approval before the code freeze deadline.

@dixudx
Copy link
Member Author

dixudx commented Sep 2, 2017

ping @liggitt @fabianofranz for approval. Thanks.

@liggitt
Copy link
Member

liggitt commented Sep 2, 2017

/approve

@liggitt liggitt added kind/bug Categorizes issue or PR as related to a bug. release-note-none Denotes a PR that doesn't merit a release note. sig/cli Categorizes an issue or PR as relevant to SIG CLI. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Sep 2, 2017
@liggitt liggitt added this to the v1.8 milestone Sep 2, 2017
@liggitt
Copy link
Member

liggitt commented Sep 2, 2017

@fabianofranz @pwittrock for approval and approved-for-milestone tagging if this is required for 1.8

@fabianofranz
Copy link
Contributor

/test pull-kubernetes-e2e-kops-aws

@fabianofranz
Copy link
Contributor

/approve

@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: caesarxuchao, dixudx, fabianofranz, liggitt

Associated issue: 51185

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 5, 2017
@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 51186, 50350, 51751, 51645, 51837)

@k8s-github-robot k8s-github-robot merged commit ee4e4a5 into kubernetes:master Sep 6, 2017
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Sep 6, 2017 •
edited
Loading

@dixudx: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-kops-aws 5e120cf link /test pull-kubernetes-e2e-kops-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@dixudx dixudx deleted the fix_delete_uninitialized_resources branch September 6, 2017 01:14
@dixudx dixudx mentioned this pull request Sep 18, 2017
Merged
@shyamjvs shyamjvs mentioned this pull request Sep 28, 2017
Closed
@shyamjvs shyamjvs mentioned this pull request Oct 10, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liggitt liggitt

@smarterclayton smarterclayton

@caesarxuchao caesarxuchao

Assignees

@fabianofranz fabianofranz

@ahmetb ahmetb

@liggitt liggitt

@caesarxuchao caesarxuchao

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. sig/cli Categorizes an issue or PR as relevant to SIG CLI. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.8
Development

Successfully merging this pull request may close these issues.

kubectl failed to delete uninitialized resources
9 participants
@dixudx @k8s-ci-robot @dims @caesarxuchao @smarterclayton @liggitt @fabianofranz @k8s-github-robot @ahmetb