Skip to content

adds some debug log for each failed host#5351

Merged
estesp merged 1 commit intocontainerd:masterfrom
mikebrow:resolver-debug-improvement
Apr 14, 2021
Merged

adds some debug log for each failed host#5351
estesp merged 1 commit intocontainerd:masterfrom
mikebrow:resolver-debug-improvement

Conversation

@mikebrow
Copy link
Copy Markdown
Member

@mikebrow mikebrow commented Apr 14, 2021 •
edited
Loading

When debugging multiple host mirrors additional debug output is desired.

Example debug output:

INFO[2021-04-14T13:44:53.182712069-05:00] PullImage "gcr.io/hello-world"               
DEBU[2021-04-14T13:44:53.182820627-05:00] PullImage using normalized image ref: "gcr.io/hello-world:latest" 
DEBU[2021-04-14T13:44:53.183979170-05:00] resolving                                     host="172.18.0.7:5000"
DEBU[2021-04-14T13:44:53.184026001-05:00] do request                                    host="172.18.0.7:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.0-rc.0-5-g3bbd78b.m request.method=HEAD url="http://172.18.0.7:5000/v2/hello-world/manifests/latest?ns=gcr.io"
INFO[2021-04-14T13:44:56.185820263-05:00] trying next host                              error="failed to do request: Head \"http://172.18.0.7:5000/v2/hello-world/manifests/latest?ns=gcr.io\": dial tcp 172.18.0.7:5000: connect: no route to host" host="172.18.0.7:5000"
DEBU[2021-04-14T13:44:56.186012857-05:00] resolving                                     host="172.18.0.30:5010"
DEBU[2021-04-14T13:44:56.186091912-05:00] do request                                    host="172.18.0.30:5010" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.0-rc.0-5-g3bbd78b.m request.method=HEAD url="https://172.18.0.30:5010/v2/hello-world/manifests/latest?ns=gcr.io"
INFO[2021-04-14T13:44:59.188681251-05:00] trying next host                              error="failed to do request: Head \"https://172.18.0.30:5010/v2/hello-world/manifests/latest?ns=gcr.io\": dial tcp 172.18.0.30:5010: connect: no route to host" host="172.18.0.30:5010"
DEBU[2021-04-14T13:44:59.188816554-05:00] resolving                                     host="172.18.0.20:5007"
DEBU[2021-04-14T13:44:59.188878065-05:00] do request                                    host="172.18.0.20:5007" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.0-rc.0-5-g3bbd78b.m request.method=HEAD url="https://172.18.0.20:5007/v2/hello-world/manifests/latest?ns=gcr.io"
INFO[2021-04-14T13:45:02.193126271-05:00] trying next host                              error="failed to do request: Head \"https://172.18.0.20:5007/v2/hello-world/manifests/latest?ns=gcr.io\": dial tcp 172.18.0.20:5007: connect: no route to host" host="172.18.0.20:5007"
DEBU[2021-04-14T13:45:02.193203677-05:00] resolving                                     host=gcr.io
DEBU[2021-04-14T13:45:02.193225578-05:00] do request                                    host=gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.0-rc.0-5-g3bbd78b.m request.method=HEAD url="https://gcr.io/v2/hello-world/manifests/latest"
DEBU[2021-04-14T13:45:02.503561868-05:00] fetch response received                       host=gcr.io response.header.accept-ranges=none response.header.alt-svc="h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.cache-control=private response.header.content-type=application/json response.header.date="Wed, 14 Apr 2021 18:45:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.vary=Accept-Encoding response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="400 Bad Request" url="https://gcr.io/v2/hello-world/manifests/latest"
ERRO[2021-04-14T13:45:02.504448856-05:00] PullImage "gcr.io/hello-world" failed         error="failed to pull and unpack image \"gcr.io/hello-world:latest\": failed to resolve reference \"gcr.io/hello-world:latest\": unexpected status code [manifests latest]: 400 Bad Request"

example default output:

INFO[2021-04-14T13:46:49.311397415-05:00] PullImage "gcr.io/hello-world"               
INFO[2021-04-14T13:46:52.316663978-05:00] trying next host                              error="failed to do request: Head \"http://172.18.0.7:5000/v2/hello-world/manifests/latest?ns=gcr.io\": dial tcp 172.18.0.7:5000: connect: no route to host" host="172.18.0.7:5000"
INFO[2021-04-14T13:46:55.318201690-05:00] trying next host                              error="failed to do request: Head \"https://172.18.0.30:5010/v2/hello-world/manifests/latest?ns=gcr.io\": dial tcp 172.18.0.30:5010: connect: no route to host" host="172.18.0.30:5010"
INFO[2021-04-14T13:46:58.315027921-05:00] trying next host                              error="failed to do request: Head \"https://172.18.0.20:5007/v2/hello-world/manifests/latest?ns=gcr.io\": dial tcp 172.18.0.20:5007: connect: no route to host" host="172.18.0.20:5007"
ERRO[2021-04-14T13:46:58.497219002-05:00] PullImage "gcr.io/hello-world" failed         error="failed to pull and unpack image \"gcr.io/hello-world:latest\": failed to resolve reference \"gcr.io/hello-world:latest\": unexpected status code [manifests latest]: 400 Bad Request"

note the additional log entry for why each resolve attempt failed...

Typical issue where I'd like to ask them to run with debug logging enabled... #5330

Signed-off-by: Mike Brown [email protected]

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Apr 14, 2021

Build succeeded.

dmcgowan
dmcgowan reviewed Apr 14, 2021
View reviewed changes
Comment thread remotes/docker/resolver.go Outdated
@mikebrow mikebrow force-pushed the resolver-debug-improvement branch from 32aabf8 to 3bbd78b Compare April 14, 2021 13:15
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Apr 14, 2021

Build succeeded.

@mikebrow mikebrow force-pushed the resolver-debug-improvement branch from 3bbd78b to d21fe46 Compare April 14, 2021 18:24
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci Bot commented Apr 14, 2021

Build succeeded.

estesp
estesp approved these changes Apr 14, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@estesp estesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@estesp estesp merged commit ce5d42f into containerd:master Apr 14, 2021
This was referenced Apr 14, 2021
Closed
Merged
@thaJeztah thaJeztah added cherry-picked/1.4.x PR commits are cherry picked into the release/1.4 branch and removed cherry-pick/1.4.x labels Apr 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dmcgowan dmcgowan dmcgowan approved these changes

@estesp estesp estesp approved these changes

@ktock ktock ktock approved these changes

Assignees

No one assigned

Labels

cherry-picked/1.4.x PR commits are cherry picked into the release/1.4 branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mikebrow @dmcgowan @estesp @ktock @thaJeztah