Repository navigation
isManifestUnknownError fails against Harbor registries, breaking sigstore signature upload #2203
Description
Skopeo is planned to be used to copy already signed images between two or more private registries.
In my situation, the images are already signed in a private source registry. This signature is also shown as existing within the source registry.
If this signed image is copied directly between the two registries - there is a graphical front end in the registry product for this - the image includes the signature appearing on the target registry.
However, this solution cannot be used due to other disadvantages. Thats the reason trying skopeo for a potential solution.
A $skopeo copy docker://192.168.x.y/images/test:0.1 dir:test_0.1 copies the image including the signature into the test_0.1 directory - into signature-1.
However, a copy directly between two registries does not work:
$skopeo copy docker://192.168.x.y/images/test:0.1 docker://192.168.y.z/images/test:0.1
Getting image source signatures
Checking if image destination supports signatures
Copying blob 96526aa774ef skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
Copying config 33b8df73a9 done
Writing manifest to image destination
Storing signatures
FATA[0000] writing signatures: reading manifest sha256-blablabla.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-blablabla.sig not found
Also, there is no positiv result trying to copy the content from inside the local directory. The same message is the result.
My question is: might it be that Skopeo cannot be used to copy images between two or more registries that are already signed in the source registry? There are no plans to re-sign the image with the private key in between again. This signature should be adopted 1:1 into the target registry - just as the registry product itself can do, although there are other reasons against using this solution.
May I ask for assistance or is this some kind of issue or enhancement?
Activity
mtrmac commentedon Nov 27, 2023
Thanks for your report.
As you see, the code is trying to copy signatures. This is supposed to work.
AFAICS the issue here is interoperability with the specific destination registry implementation; a signed push with a sigstore signature to that registry would fail just the same.
First, please make sure you are using the recently-released Skopeo 1.14.0; the behavior in this area has changed, quite possibly fixing this.
If that’s not the case, we’ll need to fix that in c/image; for starters, please provide a full log from
skopeo --debug copy …, and there might be more debugging information necessary.Plat0nicS0lid commentedon Nov 29, 2023
Hello,
tried the skopeo version within arch linux docker image.
Please see the skopeo -v output:
[root@6bbacdc9b6e4 ~]# skopeo -v
skopeo version 1.14.0 commit: 6abf96bb82666fbb3d4ad9faf1812e5ae2d31a74
[root@6bbacdc9b6e4 ~]#
To give you some more information about the environment. Both source and destination are harbor registries - please see goharbor.io if you need more information about the product.
Harbor is able to handle cosign signed images per default. As stated in my first entry, using the internal image replication leads to the result that signed images within the source registry will be copied together with the signature to the target registry. But for our use case, using the graphical tool is not an option.
The "test:0.1" image is a very simple image only containing an alpine image with a simple echo hello line as cmd. That image has been signed with cosign and a local self created key. The image has been uploaded to registry 1 into the project "images".
That is the starting situation.
Please see the following sysouts. First one copying the image into a local dir into the running container that provides skopeo version 1.14.0:
---snip---
[root@6bbacdc9b6e4 ~]# skopeo copy docker://192.168.a.b/images/test:0.1 dir:test_0-1
Getting image source signatures
Checking if image destination supports signatures
Copying blob 5b088f1e648c done |
Copying blob 96526aa774ef done |
Copying config 33b8df73a9 done |
Writing manifest to image destination
Storing signatures
[root@6bbacdc9b6e4 ~]# cd test_0-1/
[root@6bbacdc9b6e4 test_0-1]# ls -la
total 49052
drwxr-xr-x 2 root root 4096 Nov 29 21:29 .
drwxr-x--- 1 root root 4096 Nov 29 21:29 ..
-rw-r--r-- 1 root root 1127 Nov 29 21:29 33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
-rw-r--r-- 1 root root 46789009 Nov 29 21:29 5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
-rw-r--r-- 1 root root 3401967 Nov 29 21:29 96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
-rw-r--r-- 1 root root 740 Nov 29 21:29 manifest.json
-rw-r--r-- 1 root root 1629 Nov 29 21:29 signature-1
-rw-r--r-- 1 root root 33 Nov 29 21:29 version
[root@6bbacdc9b6e4 test_0-1]#
[root@6bbacdc9b6e4 ~]# skopeo --debug copy dir:test_0-1 docker://192.168.y.z/images/test:0.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.y.z/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.y.z
DEBU[0000] No signature storage configuration found for 192.168.y.z/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.y.z
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.y.z
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] IsRunningImageAllowed for image dir:/root/test_0-1
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
Checking if image destination supports signatures
DEBU[0000] GET https://192.168.y.z/v2/
DEBU[0000] Ping https://192.168.y.z/v2/ err Get "https://192.168.y.z/v2/": dial tcp 192.168.y.z:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.y.z/v2/", Err:(*net.OpError)(0xc0000403c0)})
DEBU[0000] GET http://192.168.y.z/v2/
DEBU[0000] Ping http://192.168.y.z/v2/ status 401
DEBU[0000] We can't modify the manifest, hoping for the best...
DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] ... already exists
DEBU[0000] Skipping blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa (already present):
DEBU[0000] Checking if we can reuse blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
Copying blob 96526aa774ef skipped: already exists
DEBU[0000] ... already exists
Copying blob 96526aa774ef skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0000] Using original blob without modification
DEBU[0000] Checking /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0000] ... already exists
Copying config 33b8df73a9 done |
Writing manifest to image destination
DEBU[0000] PUT http://192.168.y.z/v2/images/test/manifests/0.1
Storing signatures
DEBU[0000] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
FATA[0000] writing signatures: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
[root@6bbacdc9b6e4 ~]#
---snap---
If I understand the last line correctly, instead of "signature-1" a file named sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig will be expected - is this correct?
Second one showing sysout copying the image directly between harbor 1 and harbor 2:
---snip---
[root@6bbacdc9b6e4 ~]# skopeo --debug copy docker://192.168.a.b/images/test:0.1 docker://192.168.y.z/images/test:0.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.y.z/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.y.z
DEBU[0000] No signature storage configuration found for 192.168.y.z/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.y.z
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.y.z
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "192.168.a.b/images/test:0.1"
DEBU[0000] Found credentials for 192.168.a.b/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.a.b
DEBU[0000] No signature storage configuration found for 192.168.a.b/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.a.b
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.a.b
DEBU[0000] GET https://192.168.a.b/v2/
DEBU[0001] Ping https://192.168.a.b/v2/ err Get "https://192.168.a.b/v2/": dial tcp 192.168.a.b:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.a.b/v2/", Err:(*net.OpError)(0xc000578780)})
DEBU[0001] GET http://192.168.a.b/v2/
DEBU[0001] Ping http://192.168.a.b/v2/ status 401
DEBU[0001] GET http://nuc02/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull&service=harbor-registry
DEBU[0001] GET http://192.168.a.b/v2/images/test/manifests/0.1
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0001] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0001] IsRunningImageAllowed for image docker:192.168.a.b/images/test:0.1
DEBU[0001] Using default policy section
DEBU[0001] Requirement 0: allowed
DEBU[0001] Overall: allowed
Getting image source signatures
DEBU[0001] Reading /var/lib/containers/sigstore/images/test@sha256=3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db/signature-1
DEBU[0001] Looking for sigstore attachments in 192.168.a.b/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0001] GET http://192.168.a.b/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0001] Found a sigstore attachment manifest with 1 layers
DEBU[0001] Fetching sigstore attachment 1/1: sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
DEBU[0001] Downloading /v2/images/test/blobs/sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
DEBU[0001] GET http://192.168.a.b/v2/images/test/blobs/sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
Checking if image destination supports signatures
DEBU[0001] GET https://192.168.y.z/v2/
DEBU[0002] Ping https://192.168.y.z/v2/ err Get "https://192.168.y.z/v2/": dial tcp 192.168.y.z:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.y.z/v2/", Err:(*net.OpError)(0xc000578050)})
DEBU[0002] GET http://192.168.y.z/v2/
DEBU[0002] Ping http://192.168.y.z/v2/ status 401
DEBU[0002] We can't modify the manifest, hoping for the best...
DEBU[0002] Checking if we can reuse blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0002] Checking /v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0002] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0002] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0002] Checking /v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0002] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0002] ... already exists
DEBU[0002] ... already exists
DEBU[0002] Skipping blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef (already present):
Copying blob 5b088f1e648c skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
Copying blob 96526aa774ef skipped: already exists
DEBU[0002] Downloading /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] GET http://192.168.a.b/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] No compression detected
DEBU[0002] Compression change for blob sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0002] Using original blob without modification
DEBU[0002] Checking /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] ... already exists
Copying config 33b8df73a9 done |
Writing manifest to image destination
DEBU[0002] PUT http://192.168.y.z/v2/images/test/manifests/0.1
Storing signatures
DEBU[0002] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0002] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0002] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0002] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
FATA[0002] writing signatures: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
[root@6bbacdc9b6e4 ~]#
---snap---
Second attempt showing the same result.
Added the following lines to /etc/containers/registries.d/default.yaml:
docker:
192.168.a.b:
use-sigstore-attachments: true
192.168.y.z:
use-sigstore-attachments: true
Hope that helps. Please let me know if any further information is needed.
mtrmac commentedon Nov 29, 2023
Thanks!
OK, the registry (ref. https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/controller/artifact/controller.go#L298) apparently uses a non-standard
"code"valueNOT_FOUND: https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/lib/errors/const.go#L19 (and that turns intounknown) but that value should turn into a 404, looking at https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/lib/http/error.go#L39.No, that name refers to a tag on the destination registry (and a component of the URLs quoted above). That works as expected.
Either way, we’ll want to capture the full HTTP response of the above-quoted HTTP request (with the relevant authentication headers). Is that easy to do for you in the current environment? If not, we’ll need a source code patch to capture that in Skopeo.
[-]Using Skopeo to copy already signed images between two or more private registries - keeping the source signature[/-][+]isManifestUnknownError fails against Harbor registries, breaking sigstore signature upload[/+]mtrmac commentedon Nov 29, 2023
… looking more closely (and compare goharbor/harbor#19179 , not the same request but a sample of the error message):
"code"value.our
isManifestUknowncan, currently, only check for status 404 if the registry does not use the standard JSON format; if the standard JSON format is used, the HTTP status value is not currently available inisManifestUnknown.Plat0nicS0lid commentedon Dec 9, 2023
Hello again,
many thanks for your help and looking for the cause. Apologies for the delay and not answered earlier. May I ask for some guidance how to deliver the full http response? If this is something a sopeo option handles, could you please tell me how to manage it? Or is this something that can be recorded with a running tcpdump, or similar? If yes, which options have I to use, to record the necessary information for you?
If I understand you correctly, the situation is that a not expected response will be returned to prevent the signature to be delivered to harbor. Is this something that can be shipped around inside skopeo or must this be changed on harbor side? As the image copy works between two harbor registries directly, also if the source image has a signature, the functionality is available in general.
Just to see the copy result between two registries working if copying with skopeo, could you help me out with a registry product which would work? Basically, there will be no option to switch to another product than harbor, but it is my personal interest to see that process working in general.
STARRY-S commentedon Apr 28, 2024
Hi @mtrmac, I use the Harbor server (
v2.10.2) and debug theskopeoto capture the HTTP response body content when the non-standard "code" valueNOT_FOUNDappared:Hope this information helps and feel free to ask me if you need me to capture additional debug information, thanks.
By refer to goharbor/harbor#19179, if I understand correctly, it seems like this will be fixed in harbor
v2.11.0.mtrmac commentedon Apr 29, 2024
@STARRY-S Thanks; that’s the request body, but not full request headers. Could you perhaps apply https://github.com/containers/image/pull/201/files (with
s/Trace/Error/g) to Skopeo, and capture the response from the output of that?(I don’t expect the response to contain credentials or secrets, but please double-check.)
STARRY-S commentedon May 8, 2024
@mtrmac Hi, here's the log with captured response output:
Details
HTTP response header debug:
HTTP response body:
"{\"errors\":[{\"code\":\"NOT_FOUND\",\"message\":\"artifact test/alpine:sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig not found\"}]}\n"Recognize "manifest unknown" errors reported by Harbor
mtrmac commentedon May 13, 2024
@STARRY-S Thanks for the data!
#2413 should help fix this. Would you be willing to test and confirm, please?
Recognize "manifest unknown" errors reported by Harbor
Recognize "manifest unknown" errors reported by Harbor
STARRY-S commentedon May 16, 2024
Using PR #2413, replacing the
github.com/containers/image/v5depency withgithub.com/mtrmac/image/[email protected], test code.Test Result
Command:
skopeo copy \ docker://docker.io/library/alpine:3 \ docker://harbor.example.local/library/alpine:3 \ --src-tls-verify=false --dest-tls-verify=false \ --sign-by-sigstore-private-key=sigstore.private \ --sign-passphrase-file=password.txt \ --multi-arch=all \ --preserve-digestsLog outputs:
Details
Result
The error message
unknown: artifact images/test:sha256-xxx.sig not foundno longer exists, passed.Recognize "manifest unknown" errors reported by Harbor
mtrmac commentedon May 16, 2024
Thanks for the testing! #2413 was merged.
Plat0nicS0lid commentedon May 31, 2024
Hello @mtrmac, Hello @STARRY-S ,
unfortunatelly, I cannot agree that the problem is solved.
I also tested the image replication from one harbor instance to a second one. Basically the same situation as from the beginning.
Let me describe the setup:
VM 0: Docker host - Scopeo Image v 1.15.0 - pulled via quay.io/skopeo/stable:v1.15.0, current latest one (28 days ago)
VM 1: Docker host - Docker Image(s) Harbor version 2.10.2
VM 2: Docker host - Docker Image(s) Harbor version 2.10.2
Test case:
Question: is the skopeo image v.1.15.0 to old and does not contain the changes described above, or is the working outcome the result of the different steps, replicating an image from docker hub to an own harbor instance and just sign that one after replication. This is another approach as mine.
If I need a newer skopeo image, please tell me where to pull it from. As the v1.15.0 is older than 28 days, I assume that this latest one does not contain the changes.
skopeo copy --debug docker://192.168.210.91/images/alpine:3.18.4 docker://192.168.210.93/images/alpine:3.18.4 DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" DEBU[0000] Found credentials for 192.168.210.93/images/alpine in credential helper containers-auth.json in file /tmp/auth.json DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.210.93 DEBU[0000] No signature storage configuration found for 192.168.210.93/images/alpine:3.18.4, using built-in default file:///var/lib/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.210.93 DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.210.93 DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "192.168.210.91/images/alpine:3.18.4" DEBU[0000] Found credentials for 192.168.210.91/images/alpine in credential helper containers-auth.json in file /tmp/auth.json DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.210.91 DEBU[0000] No signature storage configuration found for 192.168.210.91/images/alpine:3.18.4, using built-in default file:///var/lib/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.210.91 DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.210.91 DEBU[0000] GET https://192.168.210.91/v2/ DEBU[0000] Ping https://192.168.210.91/v2/ err Get "https://192.168.210.91/v2/": dial tcp 192.168.210.91:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.210.91/v2/", Err:(*net.OpError)(0xc000516190)}) DEBU[0000] GET http://192.168.210.91/v2/ DEBU[0000] Ping http://192.168.210.91/v2/ status 401 DEBU[0000] GET http://nuc02/service/token?account=sascha&scope=repository%3Aimages%2Falpine%3Apull&service=harbor-registry DEBU[0000] GET http://192.168.210.91/v2/images/alpine/manifests/3.18.4 DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite DEBU[0000] IsRunningImageAllowed for image docker:192.168.210.91/images/alpine:3.18.4 DEBU[0000] Using default policy section DEBU[0000] Requirement 0: allowed DEBU[0000] Overall: allowed Getting image source signatures DEBU[0000] Reading /var/lib/containers/sigstore/images/alpine@sha256=48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86/signature-1 DEBU[0000] Looking for sigstore attachments in 192.168.210.91/images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig DEBU[0000] GET http://192.168.210.91/v2/images/alpine/manifests/sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" DEBU[0000] Found a sigstore attachment manifest with 1 layers DEBU[0000] Fetching sigstore attachment 1/1: sha256:73915660087de524a0f777380d9d6f6b7b7df31c9c63f77d5375bdb5d3c1ad54 DEBU[0000] Downloading /v2/images/alpine/blobs/sha256:73915660087de524a0f777380d9d6f6b7b7df31c9c63f77d5375bdb5d3c1ad54 DEBU[0000] GET http://192.168.210.91/v2/images/alpine/blobs/sha256:73915660087de524a0f777380d9d6f6b7b7df31c9c63f77d5375bdb5d3c1ad54 Checking if image destination supports signatures DEBU[0000] GET https://192.168.210.93/v2/ DEBU[0000] Ping https://192.168.210.93/v2/ err Get "https://192.168.210.93/v2/": dial tcp 192.168.210.93:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.210.93/v2/", Err:(*net.OpError)(0xc0005163c0)}) DEBU[0000] GET http://192.168.210.93/v2/ DEBU[0000] Ping http://192.168.210.93/v2/ status 401 DEBU[0000] We can't modify the manifest, hoping for the best... DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0000] Checking /v2/images/alpine/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa DEBU[0000] GET http://nuc07/service/token?account=sascha&scope=repository%3Aimages%2Falpine%3Apull%2Cpush&service=harbor-registry DEBU[0000] HEAD http://192.168.210.93/v2/images/alpine/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa DEBU[0000] ... already exists DEBU[0000] Skipping blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa (already present): Copying blob 96526aa774ef skipped: already exists DEBU[0000] Downloading /v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 DEBU[0000] GET http://192.168.210.91/v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 DEBU[0000] No compression detected DEBU[0000] Compression change for blob sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 ("application/vnd.docker.container.image.v1+json") not supported DEBU[0000] Using original blob without modification DEBU[0000] Checking /v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 DEBU[0000] HEAD http://192.168.210.93/v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 DEBU[0000] ... already exists Copying config 8ca4688f4f done | Writing manifest to image destination DEBU[0000] PUT http://192.168.210.93/v2/images/alpine/manifests/3.18.4 Storing signatures DEBU[0000] Looking for sigstore attachments in 192.168.210.93/images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig DEBU[0000] GET http://192.168.210.93/v2/images/alpine/manifests/sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8" DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig in 192.168.210.93/images/alpine: unknown: artifact images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig not found FATA[0000] writing signatures: reading manifest sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig in 192.168.210.93/images/alpine: unknown: artifact images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig not found [root@c86d2ed07d47 /]#If I copy within the container from one directory to another, the signature is available:
skopeo copy --debug dir:alpine_3-18-4 dir:alpine_3.18.4 DEBU[0000] overwriting existing container image directory "/root/alpine_3.18.4" DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite DEBU[0000] IsRunningImageAllowed for image dir:/root/alpine_3-18-4 DEBU[0000] Using default policy section DEBU[0000] Requirement 0: allowed DEBU[0000] Overall: allowed Getting image source signatures Checking if image destination supports signatures DEBU[0000] We can't modify the manifest, hoping for the best... DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0000] Detected compression format gzip DEBU[0000] Using original blob without modification Copying blob 96526aa774ef done | DEBU[0000] No compression detected DEBU[0000] Compression change for blob sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 ("application/vnd.docker.container.image.v1+json") not supported DEBU[0000] Using original blob without modification Copying config 8ca4688f4f done | Writing manifest to image destination Storing signaturesMay I ask for another test or check?
Best regards
mtrmac commentedon May 31, 2024
Yes; this was included in c/image v5.31.0, and that isn’t in a Skopeo release yet.
Per https://github.com/containers/image_build/blob/main/README.md , those unreleased versions should have (unstable! testing-only!) builds at https://quay.io/repository/skopeo/upstream, but https://quay.io/repository/skopeo/upstream?tab=history shows the tag being deleted. @cevich is that just me forgetting about an announced change, or is that unexpected?
cevich commentedon May 31, 2024
Oh! That's wrong, there should always be a "latest" tag there. Checking podman and buildah, it's the same deal. Something is wrong. I'll look into it.
Plat0nicS0lid commentedon May 31, 2024
Thanks @mtrmac and @cevich,
found and pulled quay.io/skopeo/upstream:latest which shows version:
Test result:
.Used command:
The image will be replicated with the already existing signature to vm02 harbor instance. Inside the project, the image now shows an existing signature.
Many thanks for your help.