Skip to content

Push fails when cross-repository blob mount returns 403 (Forbidden) instead of 401 (Unauthorized) #14086

Open

Description

@bainsy88

Description

When pushing an image where containerd attempts a cross-repository blob mount, some registries return 403 Forbidden rather than 401 Unauthorized when the authenticated user does not have read access to the mount source repository. containerd currently treats a 403 as a failure and aborts the push, rather than falling back to a normal (non-mounted) layer upload.

This causes Docker daemon (when using the containerd snapshotter) to fail pushes that succeed when the classic moby storage backend is used, creating a behavioural regression.

The legacy moby push code handles this correctly (arguably by accident) in push_v2.go:

// when error is unauthorizedError and user don't hasAuthInfo that's the case user don't has right to push layer to register
// and he hasn't login either, in this case candidate cache should be removed
if mountCandidate.SourceRepository != "" &&
    (!isUnauthorizedError || pd.pushState.hasAuthInfo) &&
    (metadata.CheckV2MetadataHMAC(&mountCandidate, pd.hmacKey) ||
        mountCandidate.HMAC == "") {

The logic here is: if the mount failed and the request was already authenticated (hasAuthInfo), remove the mount association and fall back to a direct upload. A 403 from an authenticated request falls into this path and is handled gracefully — the layer is pushed without the mount optimisation.

containerd does not implement an equivalent fallback, so a 403 on the mount attempt causes the push to be aborted entirely.

Steps to reproduce the issue

  1. docker pull alpine
  2. docker tag alpine registry.example.com/account1/alpine
  3. docker login registry.example.com (as account 1)
  4. docker push registry.example.com/account1/alpine
  5. docker logout registry.example.com
  6. docker login registry.example.com (as account 2)
  7. docker tag registry.example.com/account1/alpine registry.example.com/account2/alpine
  8. docker push registry.example.com/account2/alpine

Describe the results you received and expected

Expected: The mount attempt for layers from account1/alpine fails with 403; containerd falls back to a direct upload and the push succeeds.

Actual: The push fails. The 403 from the mount attempt is not treated as a recoverable condition and no fallback upload is attempted.

What version of containerd are you using?

containerd: Version: v2.2.0 GitCommit: 1c4457e

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

Activity

  1. added a commit that references this issue on Sep 5, 2026
    8206483
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions