Repository navigation
Description
What happened:
We are trying to deprecate and remove some CRD API versions and as part of that to ensure that users who have originally deployed Custom Resources at the legacy versions can upgrade.
For a Custom Resource X that is a dependent on another Custom Resource Y and they both have been originally created using a legacy API version v1alpha2, we observe that after an upgrade to a controller version where v1alpha2 is no longer served, X is not garbage collected after Y has been deleted.
X after the upgrade:
apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
annotations:
cert-manager.io/certificate-name: test1
...
name: test1-4d8sc
namespace: default
ownerReferences:
- apiVersion: cert-manager.io/v1alpha2
blockOwnerDeletion: true
controller: true
kind: Certificate
name: test1
uid: a20001e7-4793-42bf-955a-a6216cbdc933
...
Y after the upgrade:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
...
name: test1
namespace: default
resourceVersion: "12276"
uid: a20001e7-4793-42bf-955a-a6216cbdc933
In the above config CertificateRequest.metadata.ownerReferences[0].apiVersion is incorrect. I thought that perhaps this is the issue, so I tried editing it v1alpha2 -> v1 just before running the delete command. The edited CertificateRequest was still not deleted.
Does the metadata.ownerReferences[0].apiVersion matter- I seem to remember seeing somewhere on #sig-api-machinery Slack that it is not actually used?
What you expected to happen:
I expected that X would be deleted after Y is deleted as it happens normally with these resources.
How to reproduce it (as minimally and precisely as possible):
- Deploy
cert-managerversionv0.16where the storage version for all CRDs isv1alpha2 - Create an
Certificate(Y) Custom Resource using thev1alpha2API. ACertificateRequestX will be created for Y with Y set as X's owner reference. - Upgrade to a newer version of
cert-managerwhere the only storage version isv1(v1alpha2is still being served) - Upgrade stored versions of Custom Resources and
status.storedVersionsof CRDs as described in Kubernetes docs here - Upgrade to a version of
cert-managerwherev1is the only served version - Delete Y
kubectl delete certificate test1 - Observe that X is not deleted even after ~30 minutes, and
deletionTimestampis not set.
Anything else we need to know?:
Please let me know how I can go about debugging this or any additional information I could provide?
Environment:
- Kubernetes version (use
kubectl version):
on GKE:
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.6-gke.1000", GitCommit:"3ae0998c5052f420a17cb96bacf860ec5d6822a3", GitTreeState:"clean", BuildDate:"2021-04-29T09:17:16Z", GoVersion:"go1.15.10b5", Compiler:"gc", Platform:"linux/amd64"}
on Kind:
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-18T01:10:20Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: GKE (1.20.6-gke.1000), Kind
- OS (e.g:
cat /etc/os-release): - Kernel (e.g.
uname -a):Linux irbe-XPS-13-9310 5.8.0-53-generic #60-Ubuntu SMP Thu May 6 07:46:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux - Install tools: kubectl, helm
- Network plugin and version (if this is a network-related bug):
- Others:
/cc @munnerz
Activity
irbekrm commentedon Jun 6, 2021
/sig api-machinery
[-]child Custom Resources not garbage collected after CRD API version removal[/-][+]owned Custom Resources not garbage collected after CRD API version removal[/+]fedebongio commentedon Jun 8, 2021
/assign @caesarxuchao
/triage accepted
caesarxuchao commentedon Jun 9, 2021
The apiversion is used in gc here.
Since v1apha1 did not exist anymore, that should trigger this error message. If you ran the apiserver with
-v=5, I would expect this error message in the controller manager log. If that's the case, then it explained why X was not deleted.But you said that even if you manually fixed the ownerReference before deleting Y, X still wouldn't get gc'ed. I don't know why that's the case. Did you wait for some time between fixing the ownerRef and deleting Y? Did X always not get gc'ed or was the behavior flaky?
cc @liggitt who might know similar issues.
irbekrm commentedon Jun 9, 2021
Hi @caesarxuchao ,
Thank you for the reply.
I tried to increase the log level as you suggested. From the errors I saw, it appears that somehow the garbage collector has the wrong API version for the owned resource. I've copied the steps from the issue description with where the errors appear:
Steps followed (here
CertificateRequestX is the owned resource andCertificateY is the owner resource):v0.16where the storage version for all CRDs isv1alpha2,Certificate(Y) Custom Resource using thev1alpha2API. A CertificateRequest X will be created for Y with Y set as X's owner reference.v1(v1alpha2is still being served, so isv1beta1)status.storedVersionsof CRDs as described in Kubernetes docs. Also manually edit `metadata.ownerReferences[x].apiVersion for X. Wait for about ~10 minutesv1is the only served versionkubectl delete certificate test2At this point the
v1beta1,v1alpha2APIs are no longer served. In step 4 I had rankubectl get certificaterequest test2-tx5zn -oyaml | kubectl apply -f -to enforce update of the stored version in etcd and checked that thekubectl.kubernetes.io/last-applied-configurationannotation now shows thev1API version. I assumed that since I can still view and retrieve theCertificateRequestafter the alpha and beta APIs are no longer served the stored version got updated correctly? I also tried deleting thekube-controller-manager-kind-control-planepod (onkind) in case the old version was somehow cashed by the garbage controller- but still saw the same errors.This is the owned
CertificateRequestat a point where the owningCertificatehas been deleted and the garbage collector has thrown the above error.Would you have any suggestion how to go further about debugging this?
(This is on
kind, k8sv1.21)caesarxuchao commentedon Jun 10, 2021
@irbekrm thanks for the detailed logs.
I skimmed through graph_builder.go and it looks like that the gc controller never updates node.APIVersion.
I reformatted the node in the error message in your previous comment so it's easier to read:
As you mentioned, the dependent's APIVersion is not updated (
APIVersion:"cert-manager.io/v1beta1", Kind:"CertificateRequest"). Somehow the Owner's APIVersion is updated (APIVersion:"cert-manager.io/v1", Kind:"Certificate"). I actually don't know why the APIVersion of the owner is up-to-date.Also I wonder why we haven't seen similar GC issues when Kubernetes stopped serving apps/v1beta1 and apiextensions/v1beta1 in the past releases. @liggitt do you know?
Perhaps we can fix the issue here, here GC observes an object is updated, we can let GC checks if the APIVersion in the update event is the same as the APIVersion recorded in the in-memory node, if not, updates the in-memory APIVersion. @irbekrm do you have time to write&test a patch?
irbekrm commentedon Jun 11, 2021
@caesarxuchao thank you for looking at this!
I should have time to for this in the next couple of days- thanks for pointing me in the right direction!
30 remaining items
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall
fix(kafka): delete the release's StrimziPodSets on uninstall