Fatal error: concurrent map read and map write #1045
Description
1. What version of Caddy are you running (caddy -version)?
v0.9.1
2. What are you trying to do?
Refresh heavily a web site served by caddy reverse proxy, i.e. hold F5 key for a while in Chrome / Firefox.
3. What is your entire Caddyfile?
https://:443 {
tls self_signed
proxy / 127.0.0.1:580 {
transparent
max_fails 0
}
bind 127.0.0.1
}
4. How did you run Caddy (give the full command and describe the execution environment)?
caddy.exe -conf Caddyfile
5. What did you expect to see?
I expected the web site to appear once the heavy refreshing is done.
6. What did you see instead (give full error messages and/or log)?
The web site kept loading indefinitely (spinning wheel). Refresh or restart of browser did not help. Then Caddy failed with fatal error (see attachment for complete output):
Activating privacy features... done.
https:// (only accessible on this machine)
fatal error: concurrent map read and map write
goroutine 19217 [running]:
runtime.throw(0xa3a054, 0x21)
/usr/local/go/src/runtime/panic.go:566 +0x9c fp=0xc0458ffc38 sp=0xc0458ffc18
runtime.mapaccess1_faststr(0x9ca040, 0xc04217d9b0, 0xa2aff8, 0xa, 0x94ce9aa0244a93f8)
/usr/local/go/src/runtime/hashmap_fast.go:201 +0x4fa fp=0xc0458ffc98 sp=0xc0458ffc38
net/textproto.MIMEHeader.Get(0xc04217d9b0, 0xa2aff8, 0xa, 0xc04232eb00, 0x0)
/usr/local/go/src/net/textproto/header.go:33 +0x7c fp=0xc0458ffcd0 sp=0xc0458ffc98
net/http.Header.Get(0xc04217d9b0, 0xa2aff8, 0xa, 0xa, 0xc04232eb88)
/usr/local/go/src/net/http/header.go:39 +0x46 fp=0xc0458ffd08 sp=0xc0458ffcd0
net/http.(*Request).write(0xc042324d20, 0xcbbd20, 0xc04472c6c0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:507 +0xabf fp=0xc0458ffe48 sp=0xc0458ffd08
net/http.(*persistConn).writeLoop(0xc042c71700)
/usr/local/go/src/net/http/transport.go:1644 +0x1b3 fp=0xc0458fff98 sp=0xc0458ffe48
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc0458fffa0 sp=0xc0458fff98
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1058 +0x515
...
7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?
Unfortunately, I could not reproduce the issue by proxying to another Caddy instance or a public Internet site. I can only reproduce the issue with our application as upstream host running on localhost. It seems to be dependent on timing and number of requests from browser.
The issue does not occur when running caddy with HTTP2 disabled, when running without proxy or when running with proxy but without max_fails directive. I also could not reproduce the issue with boom benchmark tool (tried w/ and w/o -h2 switch).