Post es http err:http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

Please use the following questions as a guideline to help me answer
your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

github.com/olivere/elastic/v7 v7.0.22

Please describe the expected behavior

Post "https://aaaes.amazonaws.com/statistics_%2A/_search": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
server/ChessCronApi/es.(*FundData).FundMoneySearch

Please describe the actual behavior

Online requests for es addresses often report errors on stage

Any steps to reproduce the behavior?

no

I need to further investigate. There's an ancient issue at golang/go#25009 which describes this bug but is closed at least after Go 1.12. I've never seen this message after years of running in production.

I'm also seeing this error, but we're using a GET request

Get "https://xyz.us-east-1.es.amazonaws.com/xyz/_doc/1157": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

github.com/olivere/elastic/v6 v6.2.16
go version go1.15 linux/amd64

Same error in bulk write request. It looks like we are all using aws?

Post https://xxxxxxxxx.us-west-2.es.amazonaws.com/_bulk: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error.

@olivere

@olivere seems like usage of go 1.15 version should help to avoid this problem according to this thread: golang/go#32441

Any updates on this ? Im facing the similar issue and also upgrade of go to 1.15 did not help.

I'm also seeing this regularly on go 1.16.

I am seeing this in go 1.14 and 1.15 both. Any updates on this?

It looks like this is the source of the error. I'm looking into this in the next release.

It should be a simple fix.

P.S.: Go 1.14 is no longer supported, and 1.15 is on the edge (once 1.17 comes out in August 2021). I think it's a good idea to update anyway—even if it doesn't fix your issue.

Maybe it's http2. You can set http.Transport.ForceAttemptHTTP2 to false.

Anybody can confirm this still happens in the supported Go versions, i.e. Go 1.16 and Go 1.17 (at the time I'm writing this)?

@olivere

@olivere 1.17.1 is the version

Do you have any idea how we can solve the issue? Is happening really often.

Happening multiple times a day for me on go 1.17.1 + AWS OpenSearch (Elasticsearch 7.9, connections from ECS, through VPN)

We experienced this issue on go 1.16 also and worked-around it by building with:
go build -tags nethttpomithttp2

I am seeing this in go 1.14.

Yes its observed in aws opensearch and aws elasticsearch.

Issue can be reproduced when you try to write huge data or simultaneous data writes to elastic search

@hexa00

We were not running into this issue when using only http, but once we started using https, this issue cropped up for us. This appeared to occur very intermittently, but it was reproducible and occurred when using basic auth or AWS IAM Auth4.

@hexa00 's solution above did indeed work for us, but due to our codebase, we were hesitant to to use a compiler directive to affect all httpClients.

After some research, I believe I have a solution. I believe the piece of info we are looking for is outlined here: https://pkg.go.dev/net/http#pkg-overview. "Programs that must disable HTTP/2 can do so by setting Transport.TLSNextProto (for clients) or Server.TLSNextProto (for servers) to a non-nil, empty map. Alternatively, the following GODEBUG environment variables are currently supported."

Our full solution below appears (thus far) to work for us:

httpClient := &http.Client{
	Transport: &http.Transport{
		TLSNextProto: map[string]func(string, *tls.Conn) http.RoundTripper{},
	},
}

signingClient := elasticaws.NewV4SigningClientWithHTTPClient(credentials.NewStaticCredentials(env.ESKeyID, env.ESSecretKey, ""), env.AWSRegion, httpClient)
options := []elastic.ClientOptionFunc{elastic.SetURL(env.ESURL), elastic.SetHttpClient(signingClient), elastic.SetHealthcheck(false), elastic.SetSniff(false)}

client := elastic.NewClient(options...)

Hello guys,
So basically we need to disable http2 to fix this problem ?
On my side, it's not on a post request but a get request.
Would be great to have a fix on this, thanks guys

Navigation Menu

Repository navigation

Post es http err:http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error #1443

Which version of Elastic are you using?

Please describe the expected behavior

Please describe the actual behavior

Any steps to reproduce the behavior?

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Post es http err:http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error #1443

Description

Which version of Elastic are you using?

Please describe the expected behavior

Please describe the actual behavior

Any steps to reproduce the behavior?

Activity

olivere commented on Dec 4, 2020

GrahamWalters commented on Dec 14, 2020

fengyuxx commented on Dec 15, 2020

rbUUbr commented on Dec 17, 2020

Atply234 commented on May 6, 2021

juhovuori commented on Aug 2, 2021

kahluw2 commented on Aug 11, 2021

olivere commented on Aug 12, 2021

Hhhha commented on Aug 13, 2021

olivere commented on Sep 16, 2021

3 remaining items

vladiacob commented on Sep 19, 2021

vladiacob commented on Sep 19, 2021

jaakkotuosa commented on Sep 23, 2021

hexa00 commented on Nov 17, 2021

ceverus commented on Nov 29, 2021

pgvishnuram commented on Dec 9, 2021

pgvishnuram commented on Dec 9, 2021

nblizard commented on Dec 9, 2021

JoydS commented on Mar 27, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions