Skip to content

docker: Mirantis Secure Registry doesn't support POST#5288

Closed
kzys wants to merge 1 commit intocontainerd:masterfrom
kzys:msr-500
Closed

docker: Mirantis Secure Registry doesn't support POST#5288
kzys wants to merge 1 commit intocontainerd:masterfrom
kzys:msr-500

Conversation

@kzys
Copy link
Copy Markdown
Member

@kzys kzys commented Mar 30, 2021

Mirantis Secure Registry (MSR) doesn't support POST and returns 500.

While retrying on 500 could be problematic, it is hard to detect MSR
since the Server header on the HTTP response is just "nginx/1.16.1".

Since the retry here is only once with GET, the negative impact is
negligible.

https://www.mirantis.com/software/docker/image-registry/

@kzys kzys force-pushed the msr-500 branch 2 times, most recently from 8a7b672 to 1cd2de5 Compare March 30, 2021 18:49
@kzys kzys changed the title Mirantis Secure Registry doesn't support POST docker: Mirantis Secure Registry doesn't support POST Mar 30, 2021
Mirantis Secure Registry (MSR) doesn't support POST and returns 500.

While retrying on 500 could be problematic, it is hard to detect MSR
since the Server header on the HTTP response is just "nginx/1.16.1".

Since the retry here is only once with GET, the negative impact is
negligible.

https://www.mirantis.com/software/docker/image-registry/

Signed-off-by: Kazuyoshi Kato <[email protected]>
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Mar 30, 2021

Build succeeded.

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Mar 30, 2021

@AkihiroSuda I will submit the cherry-pick PR once this PR has been merged.

Copy link
Copy Markdown
Member

@dmcgowan dmcgowan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

500 is not a status code we want to fallback on. This is definitely one that needs to be fixed by the registry operator.

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Mar 30, 2021

I do agree we should not retry on 500. However MSR is a product that people can install and update on their own cadence. Even Mirantis fixes the issue, it may take weeks/months to actually have the fix in all MSR installations.

One thing we could do is checking MSR based on a heuristic logic regarding the JSON response like below and only have a fallback when it is like MSR. It wouldn't address the risk of misbehaving though.

DEBU[0000] token request failed                          body="{\"details\":\"internal error\",\"message\":\"please contact an administrator to resolve this issue\",\"request_id\":\"6b05d3a1-73a5-4397-8e97-47b5caada2f6\"}\n" host=example.com status="500 Internal Server Error" url="https://example.com/v2/admin/busybox/manifests/latest"

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Mar 30, 2021

Another path we could take is having a configuration option to use GET instead.

@estesp
Copy link
Copy Markdown
Member

estesp commented Mar 30, 2021

@dmcgowan This is the DTR codebase, from what I understand, but maybe it is being fronted by something else as I don't think we had any issues reported with original DTR/Docker EE and containerd?

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Mar 31, 2021

Sorry, seems "MSR doesn't support POST" is oversimplification of the situation. Here is what Docker does. POST to /auth/token actually works here.

72.21.198.64 - - [31/Mar/2021:16:17:41 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))"
72.21.198.64 - - [31/Mar/2021:16:17:41 +0000] "POST /auth/token HTTP/1.1" 200 713 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))"
72.21.198.64 - - [31/Mar/2021:16:17:42 +0000] "HEAD /v2/admin/busybox/manifests/latest HTTP/1.1" 401 0 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))"
72.21.198.64 - - [31/Mar/2021:16:17:42 +0000] "GET /v2/admin/busybox/manifests/latest HTTP/1.1" 401 156 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))"

And here is what containerd does.

54.148.61.113 - - [31/Mar/2021:16:19:43 +0000] "HEAD /v2/admin/busybox/manifests/latest HTTP/1.1" 401 0 "-" "containerd/v1.5.0-beta.4-92-gceb67edfd.m"
54.148.61.113 - admin [31/Mar/2021:16:19:43 +0000] "POST /auth/token HTTP/1.1" 500 147 "-" "Go-http-client/1.1"

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Mar 31, 2021

Docker's POST above was actually getting an refresh token, whereas containerd was getting an access token. So the correct summary of the issue is "Mirantis Secure Registry doesn't support POST for getting an access token".

@kzys
Copy link
Copy Markdown
Member Author

kzys commented Mar 31, 2021

@dmcgowan How about adding another AuthorizerOpt that let authHandler uses FetchToken() before FetchTokenWithOAuth() instead of the current order of the operations (FetchTokenWithOAuth() -> FetchToken())?

@thaJeztah
Copy link
Copy Markdown
Member

I tried to reach out to the Mirantis team through our joined slack channel to have a look at this

@thaJeztah
Copy link
Copy Markdown
Member

Oh, actually; I have some GitHub handles; @corhere @squizzi could one of you have a look? (I think you are working on Mirantis Secure Registry ?)

@dmcgowan
Copy link
Copy Markdown
Member

dmcgowan commented Apr 2, 2021

How about adding another AuthorizerOpt that let authHandler uses FetchToken() before FetchTokenWithOAuth() instead of the current order of the operations (FetchTokenWithOAuth() -> FetchToken())?

The Authorizer is already interfaced to allow for using a custom authorizer. If there needs to be more options to override the authorizer, we should make that change. This implementation of the authorizer shouldn't have any changes which alter the protocol though.

I'm going to close this one, but this can be tracked further in an issue until we find an acceptable solution or the bug is fixed by the registry operators.

@dmcgowan dmcgowan closed this Apr 2, 2021
@corhere
Copy link
Copy Markdown
Member

corhere commented Apr 26, 2021

@thaJeztah the bug has been fixed in MSR 2.9.0.

@thaJeztah
Copy link
Copy Markdown
Member

Thanks @corhere 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants