rustfs as docker registry backend encounters errors. #815
Description
Using docker registry backend with rustfs encounters errors, previously using minio didn't have this error.
client log:
{"level":"error","ts":1762510154.5430245,"logger":"http.log.error","msg":"readfrom tcp 10.0.0.101:47854->10.0.0.117:5000: unexpected EOF","request":{"remote_ip":"127.0.0.1","remote_port":"60026","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"PATCH","host":"127.0.0.1:5555","uri":"/v2/demo1/demo1/blobs/uploads/cf1b856e-f9a7-4ae2-9cc8-6ceffa1e744d?_state=F9AUspr4tOH9k9MuCV0GCIJrb1VWI_ahdyNapimnqM97Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJjZjFiODU2ZS1mOWE3LTRhZTItOWNjOC02Y2VmZmExZTc0NGQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTEtMDdUMTA6MDk6MTQuMjM3MDA4NDM2WiJ9","headers":{"User-Agent":["go-containerregistry/v0.20.6"],"Content-Length":["278153"],"Authorization":["REDACTED"],"Content-Type":["application/octet-stream"],"Accept-Encoding":["gzip"]}},"duration":0.079357271,"status":502,"err_id":"jc9cu2ufd","err_trace":"reverseproxy.statusError (reverseproxy.go:1390)"}
[exporter] *** Images (sha256:31b683d006c9f1f62e8c6afb3d8849e68739a8bf7857ff2dae2a228f799f8397):
[exporter] 127.0.0.1:5555/demo1/demo1:git-34f83068 - PUT http://127.0.0.1:5555/v2/demo1/demo1/blobs/uploads/a1dad4d3-fa58-47e5-ac8d-90948093b969?_state=REDACTED&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b: RANGE_INVALID: invalid content range
[exporter] 127.0.0.1:5555/demo1/demo1:latest - PUT http://127.0.0.1:5555/v2/demo1/demo1/blobs/uploads/aa93edf3-8872-48e0-904d-8521c882ebbf?_state=REDACTED&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b: RANGE_INVALID: invalid content range
[exporter] ERROR: failed to export: failed to write image to the following tags: [127.0.0.1:5555/demo1/demo1:git-34f83068: PUT http://127.0.0.1:5555/v2/demo1/demo1/blobs/uploads/a1dad4d3-fa58-47e5-ac8d-90948093b969?_state=REDACTED&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b: RANGE_INVALID: invalid content range],[127.0.0.1:5555/demo1/demo1:latest: PUT http://127.0.0.1:5555/v2/demo1/demo1/blobs/uploads/aa93edf3-8872-48e0-904d-8521c882ebbf?_state=REDACTED&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b: RANGE_INVALID: invalid content range]
ERROR: failed to build: executing lifecycle: failed with status code: 62
docker registry log:
time="2025-11-07T10:09:14.540693662Z" level=error msg="upload resumed at wrong offset: 0 != 434" auth.user.name=bkTTVM0oLKlg0pDz3yX04ZrAncQtcr60 go.version=go1.25.1 http.request.contenttype=application/octet-stream http.request.host="127.0.0.1:5555" http.request.id=4cf16b26-54d8-45d2-8d2c-a2d9f535555a http.request.method=PUT http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/demo1/demo1/blobs/uploads/a1dad4d3-fa58-47e5-ac8d-90948093b969?_state=dRmbFajiOHC6MgSFakc5eZjWwP2cZT7QIAFky6HeTBh7Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJhMWRhZDRkMy1mYTU4LTQ3ZTUtYWM4ZC05MDk0ODA5M2I5NjkiLCJPZmZzZXQiOjQzNCwiU3RhcnRlZEF0IjoiMjAyNS0xMS0wN1QxMDowOToxNFoifQ%3D%3D&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b" http.request.useragent=go-containerregistry/v0.20.6 instance.id=4549be94-0a6f-4566-8202-20f40fc4b75a service=registry vars.name=demo1/demo1 vars.uuid=a1dad4d3-fa58-47e5-ac8d-90948093b969 version=v3.0.0+unknown
time="2025-11-07T10:09:14.540873873Z" level=error msg="response completed with error" auth.user.name=bkTTVM0oLKlg0pDz3yX04ZrAncQtcr60 err.code="range invalid" err.message="invalid content range" go.version=go1.25.1 http.request.contenttype=application/octet-stream http.request.host="127.0.0.1:5555" http.request.id=4cf16b26-54d8-45d2-8d2c-a2d9f535555a http.request.method=PUT http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/demo1/demo1/blobs/uploads/a1dad4d3-fa58-47e5-ac8d-90948093b969?_state=dRmbFajiOHC6MgSFakc5eZjWwP2cZT7QIAFky6HeTBh7Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJhMWRhZDRkMy1mYTU4LTQ3ZTUtYWM4ZC05MDk0ODA5M2I5NjkiLCJPZmZzZXQiOjQzNCwiU3RhcnRlZEF0IjoiMjAyNS0xMS0wN1QxMDowOToxNFoifQ%3D%3D&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b" http.request.useragent=go-containerregistry/v0.20.6 http.response.contenttype=application/json http.response.duration=198.239917ms http.response.status=416 http.response.written=72 instance.id=4549be94-0a6f-4566-8202-20f40fc4b75a service=registry vars.name=demo1/demo1 vars.uuid=a1dad4d3-fa58-47e5-ac8d-90948093b969 version=v3.0.0+unknown
10.0.0.101 - - [07/Nov/2025:10:09:14 +0000] "PUT /v2/demo1/demo1/blobs/uploads/a1dad4d3-fa58-47e5-ac8d-90948093b969?_state=dRmbFajiOHC6MgSFakc5eZjWwP2cZT7QIAFky6HeTBh7Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJhMWRhZDRkMy1mYTU4LTQ3ZTUtYWM4ZC05MDk0ODA5M2I5NjkiLCJPZmZzZXQiOjQzNCwiU3RhcnRlZEF0IjoiMjAyNS0xMS0wN1QxMDowOToxNFoifQ%3D%3D&digest=sha256%3A04a30ed22d465eafc2c7e07093100f519f311a4746330c28ad65f90a94dc344b HTTP/1.1" 416 72 "" "go-containerregistry/v0.20.6"
time="2025-11-07T10:09:14.542865031Z" level=error msg="client disconnected during blob PATCH" auth.user.name=bkTTVM0oLKlg0pDz3yX04ZrAncQtcr60 contentLength=278153 copied=36326 error="unexpected EOF" go.version=go1.25.1 http.request.contenttype=application/octet-stream http.request.host="127.0.0.1:5555" http.request.id=d286160a-20fb-4992-88f5-4c4bfea5a292 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/demo1/demo1/blobs/uploads/cf1b856e-f9a7-4ae2-9cc8-6ceffa1e744d?_state=F9AUspr4tOH9k9MuCV0GCIJrb1VWI_ahdyNapimnqM97Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJjZjFiODU2ZS1mOWE3LTRhZTItOWNjOC02Y2VmZmExZTc0NGQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTEtMDdUMTA6MDk6MTQuMjM3MDA4NDM2WiJ9" http.request.useragent=go-containerregistry/v0.20.6 instance.id=4549be94-0a6f-4566-8202-20f40fc4b75a service=registry vars.name=demo1/demo1 vars.uuid=cf1b856e-f9a7-4ae2-9cc8-6ceffa1e744d version=v3.0.0+unknown
2025/11/07 10:09:14 http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/internal/request.(*RespWriterWrapper).writeHeader (resp_writer_wrapper.go:80)
10.0.0.101 - - [07/Nov/2025:10:09:14 +0000] "PATCH /v2/demo1/demo1/blobs/uploads/cf1b856e-f9a7-4ae2-9cc8-6ceffa1e744d?_state=F9AUspr4tOH9k9MuCV0GCIJrb1VWI_ahdyNapimnqM97Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJjZjFiODU2ZS1mOWE3LTRhZTItOWNjOC02Y2VmZmExZTc0NGQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTEtMDdUMTA6MDk6MTQuMjM3MDA4NDM2WiJ9 HTTP/1.1" 500 89 "" "go-containerregistry/v0.20.6"
time="2025-11-07T10:09:14.543477878Z" level=error msg="response completed with error" auth.user.name=bkTTVM0oLKlg0pDz3yX04ZrAncQtcr60 err.code=unknown err.detail="client disconnected" err.message="unknown error" go.version=go1.25.1 http.request.contenttype=application/octet-stream http.request.host="127.0.0.1:5555" http.request.id=d286160a-20fb-4992-88f5-4c4bfea5a292 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/demo1/demo1/blobs/uploads/cf1b856e-f9a7-4ae2-9cc8-6ceffa1e744d?_state=F9AUspr4tOH9k9MuCV0GCIJrb1VWI_ahdyNapimnqM97Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiJjZjFiODU2ZS1mOWE3LTRhZTItOWNjOC02Y2VmZmExZTc0NGQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTEtMDdUMTA6MDk6MTQuMjM3MDA4NDM2WiJ9" http.request.useragent=go-containerregistry/v0.20.6 http.response.contenttype=application/json http.response.duration=79.245851ms http.response.status=500 http.response.written=89 instance.id=4549be94-0a6f-4566-8202-20f40fc4b75a service=registry vars.name=demo1/demo1 vars.uuid=cf1b856e-f9a7-4ae2-9cc8-6ceffa1e744d version=v3.0.0+unknown
time="2025-11-07T10:09:14.543487102Z" level=info msg="response completed" auth.user.name=bkTTVM0oLKlg0pDz3yX04ZrAncQtcr60 go.version=go1.25.1 http.request.contenttype=application/octet-stream http.request.host="127.0.0.1:5555" http.request.id=c4e46059-73c0-4ca8-992b-98b8f5c30738 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/demo1/demo1/blobs/uploads/1bd62626-6a4d-4dbe-95dc-8a576461f4bd?_state=bLqR9nWxBFJoan6LkxNcbGpUwtvdzFrjuyqBcE3_hWp7Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiIxYmQ2MjYyNi02YTRkLTRkYmUtOTVkYy04YTU3NjQ2MWY0YmQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTEtMDdUMTA6MDk6MTQuMjMzNzUwMDJaIn0%3D" http.request.useragent=go-containerregistry/v0.20.6 http.response.duration=196.279749ms http.response.status=202 http.response.written=0 instance.id=4549be94-0a6f-4566-8202-20f40fc4b75a service=registry vars.name=demo1/demo1 vars.uuid=1bd62626-6a4d-4dbe-95dc-8a576461f4bd version=v3.0.0+unknown
10.0.0.101 - - [07/Nov/2025:10:09:14 +0000] "PATCH /v2/demo1/demo1/blobs/uploads/1bd62626-6a4d-4dbe-95dc-8a576461f4bd?_state=bLqR9nWxBFJoan6LkxNcbGpUwtvdzFrjuyqBcE3_hWp7Ik5hbWUiOiJkZW1vMS9kZW1vMSIsIlVVSUQiOiIxYmQ2MjYyNi02YTRkLTRkYmUtOTVkYy04YTU3NjQ2MWY0YmQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTEtMDdUMTA6MDk6MTQuMjMzNzUwMDJaIn0%3D HTTP/1.1" 202 0 "" "go-containerregistry/v0.20.6"
10.0.0.101 - - [07/Nov/2025:10:09:14 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "Go-http-client/1.1"
time="2025-11-07T10:09:14.586811989Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"basic-realm\": invalid authorization credential" go.version=go1.25.1 http.request.host="127.0.0.1:5555" http.request.id=f7cc0058-b2c4-42a1-9515-47f5dccc3891 http.request.method=GET http.request.remoteaddr=127.0.0.1 http.request.uri=/v2/ http.request.useragent=Go-http-client/1.1 instance.id=4549be94-0a6f-4566-8202-20f40fc4b75a service=registry version=v3.0.0+unknown
The underlying error should be rustfs failing to parse range: invalid content range