Skip to content

oras cp attributes destination "not found" errors to the source #2156

Closed

Description

@TerryHowe

What happened in your environment?

When oras cp fails with a "not found" error originating at the destination, the message is attributed to the source and printed without any host, operation, or side information. Users reasonably conclude the source registry is broken.

Split out of #2148 at the maintainers' convenience — that issue is about a copy bug that happens to trigger this, but the attribution problem is independent of it and will outlive that fix.

What did you expect to happen?

A destination-side failure should say so, the way copy errors already do:

Error from destination registry for "localhost:5000/gitlab-workhorse-ee:v19.3.0": sha256:11e4068…: not found

BinaryTarget.ModifyError already knows how to produce exactly that. It just never gets the chance on this path.

How can we reproduce it?

Publicly reproducible today (this is the #2148 scenario, which is the instance that exposed the problem):

$ docker run -d -p 5000:5000 --name registry registry:3
$ oras cp -r --to-insecure \
    registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v19.3.0 \
    localhost:5000/gitlab-workhorse-ee:v19.3.0
Error response from registry: sha256:11e4068791cb99f2871ddbbeb5b3c213814a7cba4299d33acc5d31b1171064cb: not found

Run it with -d and the underlying 404 is plainly GET http://localhost:5000/v2/gitlab-workhorse-ee/manifests/sha256:11e4068… — the local registry. Nothing in the message says so, and the generic prefix reads as though the remote source answered.

For comparison, the same failure shape in #1728 on ORAS 1.3.0-beta.3 did carry the attribution:

Error response from registry: failed to perform "Tag" on destination: sha256:1c955…: application/vnd.oci.image.index.v1+json: not found

Root cause

Two independent contributing spots.

1. The failing call is outside any copy operation, so it never becomes a CopyError.

recursiveCopy tags the root with a bare oras-go call (cmd/oras/root/cp.go:279):

if dstRef != "" && dstRef != root.Digest.String() {
	return dst.Tag(ctx, root, dstRef)
}

BinaryTarget.ModifyError (cmd/oras/internal/option/binary_target.go:72-95) branches on *oras.CopyError to recover the side:

var copyErr *oras.CopyError
if !errors.As(err, &copyErr) {
	return target.modifyError(cmd, err)
}
...
cmd.SetErrPrefix(fmt.Sprintf("Error from %s %s for %q:", copyErr.Origin, errTarget.Type, errTarget.RawReference))

A raw dst.Tag error is not a CopyError, so it falls through to the generic handler and the good prefix is never applied.

2. The generic handler then hands the error to the wrong target.

Target.ModifyError (cmd/oras/internal/option/target.go:251-255):

if errors.Is(err, errdef.ErrNotFound) {
	// special handling for not found error returned by registry target
	cmd.SetErrPrefix(oerrors.RegistryErrorPrefix)
	return true, err
}

There is no check that the error actually came from this target — unlike the errcode.ErrorResponse branch immediately below (target.go:262-275), which compares errResp.URL.Host against the target's host and returns false, err when it does not match. Since BinaryTarget.modifyError consults target.From first (binary_target.go:98-101), the source unconditionally claims any ErrNotFound on a registry-to-registry copy.

This is reproducible at unit level: Target.ModifyError on a From target returns (true, err) for an ErrNotFound raised by a completely unrelated host.

Suggested fix

Either spot alone improves the message; fixing both is better, since they fail independently.

  1. Wrap non-copy destination calls so they carry origin. oras.CopyError has exported fields, so this is available to the CLI directly:

    if err := dst.Tag(ctx, root, dstRef); err != nil {
        return &oras.CopyError{Op: "Tag", Origin: oras.CopyErrorOriginDestination, Err: err}
    }

    That routes the error into the existing copyErr.Origin path and yields the Error from destination registry for "…": prefix with no other changes.

  2. Give the ErrNotFound branch in Target.ModifyError the same host check the ErrorResponse branch already has, so a target only claims errors that came from it. Where the error carries no host information, returning false, err and letting the other target try is strictly better than mislabeling it.

Worth noting this looks like residual surface from #1469 / #1773. The CopyError-aware prefixing added there is a genuine improvement; it just does not cover errors raised outside a copy operation, and the ErrNotFound shortcut predates it.

What is the version of your ORAS CLI?

Reproduced on main (and reported originally on 1.3.4). The relevant code is unchanged on main.

Are you willing to submit PRs to fix it?

  • Yes, I am willing to fix it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions