Skip to content

Namespace removal is blocked by cozystack-api server #1674

New issue
New issue
Closed
Closed
Namespace removal is blocked by cozystack-api server#1674
Labels
kind/bugCategorizes issue or PR as related to a bug

Description

@WanzenBug
Moritz Tanner (WanzenBug)
opened on Dec 1, 2025

Describe the bug
I created a three namespaces, creating a volume and STS in each. Then, I deleted all the resources, including the namespace. The namespaces are now stuck in Terminating state.

Environment

  • Cozystack version v0.38.1
  • Provider: on-prem (virtual)

To Reproduce
Steps to reproduce the behavior:

  1. kubectl create ns stuck-ns
  2. kubectl delete ns stuck-ns
  3. kubectl get ns stuck-ns -oyaml will show the error

Expected behaviour
The namespace should be removed.

Actual behaviour
The namespace remains stuck in Terminating state. It seems that the cozystack-api is cause for the issue (see logs). On the status we can see:

  - lastTransitionTime: "2025-12-01T12:51:16Z"
    message: 'Failed to delete all resource types, 25 remaining: content is not a
      list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure

Those 25 resource types would correspond to the 25 apps.cozystack.io/v1alpha1 api-resources.

Logs
The cozystack-api continously generates these logs messages, which seems relevant.

2025-12-01T12:53:15Z	ERROR	klog	Unhandled Error	{"error": "apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"content is not a list: []unstructured.Unstructured\"}: content is not a list: []unstructured.Unstructured"}
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorToAPIStatus
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/handlers/responsewriters/status.go:71
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/handlers/responsewriters/writers.go:316
k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/handlers/rest.go:113
k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/handlers/response.go:339
k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/handlers/get.go:307
k8s.io/apiserver/pkg/endpoints.(*APIInstaller).registerResourceHandlers.restfulListResource.func6
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/installer.go:1270
k8s.io/apiserver/pkg/endpoints.(*APIInstaller).registerResourceHandlers.InstrumentRouteFunc.func7
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/metrics/metrics.go:600
github.com/emicklei/go-restful/v3.(*Container).dispatch
	/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/container.go:299
github.com/emicklei/go-restful/v3.(*Container).Dispatch
	/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP
	/go/pkg/mod/k8s.io/[email protected]/pkg/server/handler.go:146
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func22
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filters.withAuthorization.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/authorization.go:83
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func23
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle.func9
	/go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/priority-and-fairness.go:292
k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle.func2
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/flowcontrol/apf_filter.go:192
k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset.(*request).Finish.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go:391
k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset.(*request).Finish
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go:392
k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/flowcontrol/apf_filter.go:179
k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle.func10
	/go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/priority-and-fairness.go:298
k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle
	/go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/priority-and-fairness.go:299
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func24
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithImpersonation.func4
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/impersonation.go:50
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func25
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func27
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/authentication.go:120
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filterlatency/filterlatency.go:94
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithWarningRecorder.func11
	/go/pkg/mod/k8s.io/[email protected]/pkg/endpoints/filters/warning.go:35
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/server/filters/timeout.go:115

Additional context
.

Checklist

  • I have checked the documentation
  • I have searched for similar issues
  • I have included all required information
  • I have provided clear steps to reproduce
  • I have included relevant logs

Activity

  1. Moritz Tanner (WanzenBug)
    added
    kind/bugCategorizes issue or PR as related to a bug
    on Dec 1, 2025

  2. dosubot commented on Dec 1, 2025

    @dosubot
    dosubotBot
    on Dec 1, 2025 – with DosuBot

    Hi Moritz Tanner (@WanzenBug)! I'm Dosu and I’m helping the cozystack team.

    This is a known bug in Cozystack v0.38.1 (and v0.38.0) where the API server returns typed objects instead of unstructured objects for its custom resources, causing a type mismatch during resource cleanup. This leads to the error content is not a list: []unstructured.Unstructured and blocks namespace deletion for any namespace that ever contained one of the 25 apps.cozystack.io/v1alpha1 resources. The root cause is how the API server registers and returns list responses for these resources, which breaks Kubernetes' expectations during namespace finalization and cleanup. Fixing this requires a code-level change in Cozystack so that list responses always use the correct unstructured format—there is no known workaround at this time. For more details, see GitHub issue #1673.

    If you need to urgently remove stuck namespaces, you may attempt Kubernetes-generic force deletion methods (like manually editing the namespace object to remove finalizers), but these are not guaranteed to work if the API server cannot list and clean up the resources. The permanent fix will require an upgrade once a patch is released.

    To reply, just mention @dosu.

    How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

    Chat with Dosu Open in Cursor Join Discord Share on X

  3. kvaps commented on Dec 8, 2025

    @kvaps
    Andrei Kvapil (kvaps)
    on Dec 8, 2025
    Member

    Hey, thank you this was fixed by #1630 and released in v0.38.2

  4. Andrei Kvapil (kvaps)
    closed this as completedon Dec 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions