Private registry push fail: 500 error. "s3aws: AuthorizationHeaderMalformed: The authorization header is malformed;" #2033
Description
I am configuring a docker registry to run behind a NGINX proxy. I was able to authenticate. But a problem occurs when I try to push a image to the registry.
docker push registry.mydomain.com.br/busyboxI get this error message:
time="2016-11-01T10:51:50.960847082Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AuthorizationHeaderMalformed: The authorization header is malformed; the Credential is mal-formed; expecting \"<YOUR-AKID>/YYYYMMDD/REGION/SERVICE/aws4_request\".\n\tstatus code: 400, request id: A1896A22345B3CD8" err.message="unknown error" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=e90033d4-d377-4dc1-87f4-5f13e9f0ad9e http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=32.585103ms http.response.status=500 http.response.written=117 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
But apparently I have everything configured right. It is like is not passing some auth info to the s3. But I have confirmed my key, secret and region for my s3 are valid.
/data/registry/config.yaml
version: 0.1
log:
level: debug
fields:
service: registry
http:
addr: :5000
host: https://registry.mydomain.com.br
storage:
cache:
layerinfo: inmemory
s3:
accesskey: myawskey
secretkey: myawssecret
region: us-east-1
bucket: my-s3-bucket
encrypt: true
secure: true
v4auth: true
chunksize: 5242880
rootdirectory: /Running
docker run --rm --name registry \
-v /data/registry/config.yaml:/etc/docker/registry/config.yml \
-p 5000 registry:2Registry log
time="2016-11-01T10:51:19.90279961Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.6.3 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry version=v2.5.1
time="2016-11-01T10:51:19.90286376Z" level=info msg="redis not configured" go.version=go1.6.3 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry version=v2.5.1
time="2016-11-01T10:51:19.903420589Z" level=info msg="Starting upload purge in 1m0s" go.version=go1.6.3 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry version=v2.5.1
time="2016-11-01T10:51:19.935437384Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.6.3 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry version=v2.5.1
time="2016-11-01T10:51:19.936112542Z" level=info msg="listening on [::]:5000" go.version=go1.6.3 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry version=v2.5.1
time="2016-11-01T10:51:45.889450622Z" level=debug msg="authorizing request" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=c7d15dea-2bd1-4802-8a69-60fcaa266e62 http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
time="2016-11-01T10:51:45.889560532Z" level=debug msg=GetBlob go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=c7d15dea-2bd1-4802-8a69-60fcaa266e62 http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
time="2016-11-01T10:51:45.917756023Z" level=debug msg="s3aws.GetContent(\"/docker/registry/v2/repositories/busybox/_layers/sha256/56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190/link\")" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=c7d15dea-2bd1-4802-8a69-60fcaa266e62 http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry trace.duration=28.117459ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=003b53e0-7b7d-417a-940c-972404241b42 trace.line=82 vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
time="2016-11-01T10:51:45.91792986Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AuthorizationHeaderMalformed: The authorization header is malformed; the Credential is mal-formed; expecting \"<YOUR-AKID>/YYYYMMDD/REGION/SERVICE/aws4_request\".\n\tstatus code: 400, request id: DFE2A7C70E32C322" err.message="unknown error" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=c7d15dea-2bd1-4802-8a69-60fcaa266e62 http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=29.855287ms http.response.status=500 http.response.written=117 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
172.17.0.3 - - [01/Nov/2016:10:51:45 +0000] "HEAD /v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190 HTTP/1.1" 500 117 "" "docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64"
time="2016-11-01T10:51:50.929522596Z" level=debug msg="authorizing request" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=e90033d4-d377-4dc1-87f4-5f13e9f0ad9e http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
time="2016-11-01T10:51:50.929609491Z" level=debug msg=GetBlob go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=e90033d4-d377-4dc1-87f4-5f13e9f0ad9e http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
time="2016-11-01T10:51:50.96071846Z" level=debug msg="s3aws.GetContent(\"/docker/registry/v2/repositories/busybox/_layers/sha256/56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190/link\")" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=e90033d4-d377-4dc1-87f4-5f13e9f0ad9e http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry trace.duration=31.035061ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=594b98b9-4e1c-4ebb-8bbd-c2526253b1cc trace.line=82 vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
time="2016-11-01T10:51:50.960847082Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: AuthorizationHeaderMalformed: The authorization header is malformed; the Credential is mal-formed; expecting \"<YOUR-AKID>/YYYYMMDD/REGION/SERVICE/aws4_request\".\n\tstatus code: 400, request id: A1896A22345B3CD8" err.message="unknown error" go.version=go1.6.3 http.request.host=registry.mydomain.com.br http.request.id=e90033d4-d377-4dc1-87f4-5f13e9f0ad9e http.request.method=HEAD http.request.remoteaddr=52.55.236.208 http.request.uri="/v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" http.request.useragent="docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=32.585103ms http.response.status=500 http.response.written=117 instance.id=2dbcb6bd-1e3a-43b7-a1a5-42c97d4290f8 service=registry vars.digest="sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" vars.name=busybox version=v2.5.1
172.17.0.3 - - [01/Nov/2016:10:51:50 +0000] "HEAD /v2/busybox/blobs/sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190 HTTP/1.1" 500 117 "" "docker/1.10.3 go/go1.5.4 git-commit/1f8f545 kernel/4.7.0-coreos-r1 os/linux arch/amd64"
Deamon log
Nov 01 11:10:08 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:08.831384762Z" level=debug msg="Calling GET /_ping"
Nov 01 11:10:08 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:08.831422196Z" level=debug msg="GET /_ping"
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.725027043Z" level=debug msg="Calling POST /v1.22/images/registry.mydomain.com.br/busybox/push"
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.725064365Z" level=debug msg="POST /v1.22/images/registry.mydomain.com.br/busybox/push?tag="
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.726115465Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.mydomain.com.br"
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.726565277Z" level=debug msg="hostDir: /etc/docker/certs.d/registry.mydomain.com.br"
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.726625385Z" level=debug msg="Trying to push registry.mydomain.com.br/busybox to https://registry.mydomain.com.br v2"
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.737121910Z" level=debug msg="Pushing repository: registry.mydomain.com.br/busybox:latest"
Nov 01 11:10:10 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:10.762947677Z" level=error msg="Upload failed, retrying: Received unexpected HTTP status: 500 Internal Server Error"
Nov 01 11:10:15 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:15.887727168Z" level=error msg="Upload failed, retrying: Received unexpected HTTP status: 500 Internal Server Error"
Nov 01 11:10:17 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:17.888059572Z" level=info msg="Pull session cancelled"
Nov 01 11:10:17 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:17.888118184Z" level=debug msg="Not continuing with error: context canceled"
Nov 01 11:10:18 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:18.832602646Z" level=debug msg="Calling GET /_ping"
Nov 01 11:10:18 ip-10-1-14-30.ec2.internal dockerd[1647]: time="2016-11-01T11:10:18.832640958Z" level=debug msg="GET /_ping"
Registry Version
docker run --rm registry:2 --version
registry github.com/docker/distribution v2.5.1Docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.4
Git commit: 1f8f545
Built:
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.4
Git commit: 1f8f545
Built:
OS/Arch: linux/amd64Docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.10.3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.7.0-coreos-r1
Operating System: CoreOS 1122.3.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.862 GiB
Name: ip-10-1-14-30.ec2.internal
ID: NNS5:WQJG:3WGS:XMME:XQBE:UEN3:CSIM:X52U:K7UJ:HPNU:X6EC:CECC
Debug mode (server): true
File Descriptors: 36
Goroutines: 54
System Time: 2016-11-01T11:06:43.936196975Z
EventsListeners: 1
Init SHA1: 2877c6642ddf0db27b616fc3271258333ef9f1ee
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker