Skip to content

pusher deadlocks on error #6242

New issue
New issue
Closed
#6243
Closed
pusher deadlocks on error#6242
#6243
Labels
kind/bug
@tonistiigi

Description

@tonistiigi
tonistiigi
opened on Nov 13, 2021

Description

Since containerd 1.5 (#5379 looks to be the regression point) push code checks if Pusher implements Ingester

containerd/remotes/handlers.go

Lines 172 to 173 in 5b09dc5

if cs, ok := pusher.(content.Ingester); ok {
cw, err = content.OpenWriter(ctx, cs, content.WithRef(MakeRefKey(ctx, desc)), content.WithDescriptor(desc))
and if it does (default) then opens a writer. Should that push receive an error then next time when pusher (or another pusher from same resolver) tries to push the same blob the call to OpenWriter will block indefinitely with pusher returning unavailable

containerd/remotes/docker/pusher.go

Lines 81 to 86 in 5b09dc5

if unavailableOnFail {
// Another push of this ref is happening elsewhere. The rest of function
// will continue only when `errdefs.IsNotFound(err) == true` (i.e. there
// is no actively-tracked ref already).
return nil, errors.Wrap(errdefs.ErrUnavailable, "push is on-going")
}
records because the status tracker has already seen that ref.

@ktock

Steps to reproduce the issue

  1. Push one time
  2. Get error
  3. Push again

As I think ctr always fails the whole command and destroys resolver after push error it probably doesn't reproduce there.

Describe the results you received and expected

No deadlock would be nice

What version of containerd are you using?

1.5

Any other relevant information

This seems to be the cause for docker/buildx#834

Show configuration if it is related to CRI plugin.

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions