Repository navigation
Description
What version of Go are you using (go version)?
$ go version
go version go1.10.1 linux/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/test/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GORACE=""
GOROOT="/home/test/go"
GOTMPDIR=""
GOTOOLDIR="/home/test/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build614182611=/tmp/go-build -gno-record-gcc-switches"
and nginx version: nginx/1.13.3
What did you do?
Here is my client:
package main
import (
"fmt"
"net/http"
"sync"
"time"
"errors"
"bytes"
"io/ioutil"
)
type Client struct {
urlBase string
httpClient *http.Client
}
func NewClient(url string) *Client {
cc := &Client{
urlBase: url,
httpClient: &http.Client{Transport: &http.Transport{MaxIdleConnsPerHost: 100, MaxIdleConns: 100, IdleConnTimeout: 90 * time.Second, ExpectContinueTimeout: 5 * time.Second}},
}
cc.httpClient.Timeout = 5 * time.Second
return cc
}
func (self *Client) Send(content []byte) error {
if content == nil {
return errors.New("nil content")
}
body := &bytes.Buffer{}
body.Write(content)
begin := time.Now()
req, err := http.NewRequest("POST", self.urlBase, body)
if err != nil {
return fmt.Errorf("Send %+v (from %v to %v)", err, begin, time.Now())
}
req.Header = make(http.Header)
req.Header.Set("Content-Type", "application/json")
begin = time.Now()
resp, err := self.httpClient.Do(req)
if err != nil {
return fmt.Errorf("Send %+v (from %v to %v)", err, begin, time.Now())
}
defer resp.Body.Close()
if resp.StatusCode >= http.StatusBadRequest {
bs, _ := ioutil.ReadAll(resp.Body)
return fmt.Errorf("Send : HTTP : %d , BODY : %s , REQ : %s", resp.StatusCode, string(bs), string(content))
}
bs, err := ioutil.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("Send : ERR : %v", err)
}
fmt.Println("Send : Resp : ", string(bs))
return nil
}
func main() {
client := NewClient(
"https://test.cn", // I use /etc/hosts to bind the domain
)
var content = []byte(`{"test":1}`)
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
wg.Add(1)
go func() {
defer wg.Done()
client.Send(content)
}()
}
wg.Wait()
}
And here is my nginx configure of http2:
http2_max_concurrent_streams 40960;
http2_max_requests 3;
http2_max_field_size 8k;
http2_max_header_size 32k;
http2_recv_buffer_size 1024k;
http2_recv_timeout 120s;
What did you expect to see?
I expect all requests are successfully.
What did you see instead?
But I got 400 occasionally and found client prematurely closed stream: only 0 out of 10 bytes of request body received line in nginx log.
I used GODEBUG to try to figure out, and found data not sent after END_HEADERS:
2018/04/23 18:32:25 http2: Transport encoding header ":method" = "POST"
2018/04/23 18:32:25 http2: Transport encoding header ":path" = "/"
2018/04/23 18:32:25 http2: Transport encoding header ":scheme" = "https"
2018/04/23 18:32:25 http2: Transport encoding header "content-type" = "application/json"
2018/04/23 18:32:25 http2: Transport encoding header "content-length" = "10"
2018/04/23 18:32:25 http2: Transport encoding header "accept-encoding" = "gzip"
2018/04/23 18:32:25 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2018/04/23 18:32:25 http2: Framer 0xc4205480e0: wrote HEADERS flags=END_HEADERS stream=5 len=8
2018/04/23 18:32:25 http2: Framer 0xc4205480e0: wrote DATA flags=END_STREAM stream=5 len=0 data=""
That's the reason why nginx responsed 400 directly.
Activity
[-]net/http2 doesn't send body while content-length is not zero which cause 400 in nginx[/-][+]net/http: http2 client doesn't send body while content-length is not zero which causes 400 in nginx[/+]agnivade commentedon Apr 23, 2018
Does this happen only with http2 ?
Is it possible to for you to write a quick http2 server in Go and make this entire issue to be self-contained and fully reproducible ?
ictar commentedon Apr 24, 2018
@agnivade Yes, it's fine with http1.1. Here is a quite simple server:
And nginx is needed.
agnivade commentedon Apr 24, 2018
I don't understand. The whole point of having a Go server was to have the repro self-contained. If this is a client issue, why do we need nginx ? Is the Go client-server not enough to cause the repro ?
ictar commentedon Apr 24, 2018
@agnivade Nginx is needed because it will check the body length against the header
content-length. That's why nginx reponse400 Bad Requestdirectly. I don't think this is nginx issue because according to the tcpdump and GODEBUG, no body is sent indeedAnd please note that in order to make this issue reproducible,
http2_max_requestsis set to 3 for nginx.agnivade commentedon Apr 24, 2018
The point of having a Go server is to allow us to reproduce the issue without nginx. You can simply add the equivalent check in the Go server so that it's behavior is same as nginx.
odeke-em commentedon May 12, 2018
Thank you @ictar for the report, we'll investigate it.
@agnivade thank you for handling this. I just wanted to add that I can relate to @ictar's need to have nginx for a repro(many companies that I know heavily use it) and that the question of absolving the issue only because Go's servers can't reproduce it might be tricky. This is because many companies will usually put nginx or some other web server as their frontend to their web apps, and from there proxy the content to polyglot services on their backend e.g Go, Node.js, Python(etc) web server. Interoperability between protocols and servers matters a lot and discrepancies between Go clients talking to various servers creep in :)
agnivade commentedon May 12, 2018
Yes I understand. I was trying to isolate the issue.
I did not mean to say to try to reproduce the issue with a plain Go HTTP server. Rather I meant to write equivalent code in Go to mimic nginx behavior so that we hit the bug. If that is hard to do, then we obviously need nginx. But if a simple check of verifying the content-length header and the body shows the issue, then we theoretically don't need nginx. Kind of like the test case that one will write if this indeed is a bug.
ernado commentedon May 31, 2018
Seems like just verifying content-length to body length does not show the issue and we need nginx for that.
Also does not reproduce if
http.Clientis used just by single goroutine (in a non-concurrent way).[-]net/http: http2 client doesn't send body while content-length is not zero which causes 400 in nginx[/-][+]x/net/http2: client doesn't send body while content-length is not zero which causes 400 in nginx[/+]agnivade commentedon May 31, 2018
/cc @bradfitz , @tombergan
odeke-em commentedon Jun 15, 2018
@ictar I just got sometime this evening to try to reproduce this:
ssg --host localhostwith my nginx file looking like this
and I ran it 50 times in a row
for ((i=0; i<50; i++)) do go run client.go >> output;doneand even with curl as a control
but no failures and the client and server look like this
Client
server.go
@ictar with the setup that I've provided above, could you please try it locally too and see if this can reproduce the bug? Thank you.
ernado commentedon Jun 15, 2018
@odeke-em
Can you try a http server instead of https?
Also, you can take a look at #25652 where there are client and server code.
It is likely that 50 requests are not enough.
52 remaining items
fraenkel commentedon Aug 27, 2018
@ernado Your fix in http/transport.go looks correct. We still would need a test case to go along with the change.
ernado commentedon Aug 28, 2018
I've implemented a test case for transport and currently preparing the patch.
gopherbot commentedon Aug 28, 2018
Change https://golang.org/cl/131755 mentions this issue:
net/http: rewind request body unconditionallyernado commentedon Sep 6, 2018
Gentle ping for @bradfitz @ianlancetaylor
Please take a look, seems like we are close to resolving that issue.
ernado commentedon May 27, 2019
@kaspergyselinck what version of go are you using?
This problem should be fixed in go1.12.
kwestein commentedon Sep 25, 2019
Hi @ernado,
I am still seeing this issue. I have tested on both Go 1.12.9 and 1.13
It seems that others are as well in #32441
ernado commentedon Sep 25, 2019
I can confirm that my go-issue-25009 repro still fails on go1.12 and go1.13 for some reason.
Seems like issue is not fixed (or there is a regression), I will investigate it further.
ernado commentedon Sep 26, 2019
Thank you @kwestein, I've found the cause and commented in referenced issue with possible solution.
gopherbot commentedon Dec 8, 2019
Change https://golang.org/cl/210123 mentions this issue:
net/http: rewind request body on ErrSkipAltProtocolFelix3Felix commentedon Jan 16, 2020
@ernado hello, i have meet the same solution, have you fix it ?
ernado commentedon Jan 20, 2020
Hey @Centhero, I'm trying to fix in https://golang.org/cl/210123, but it seems like fix is not as trivial as I expected, sorry.