golang.org/x/net/http2/h2c: GCP Cloud Run Sporadic 502 Bad Gateway

Go version

go1.23.6 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/user/.cache/go-build'
GOENV='/home/user/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/user/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/user/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.6'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/user/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build460801396=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I have many Cloud Run applications in GCP that are all golang. I have Cloud Run HTTP2 enabled which means my go servers need h2c.

Around January of this year, all of my go apps started sporadically producing 502 Bad Gateway responses. This indicates something changed on Google's side. However, through trial and error I've narrowed down at least part of the issue to golang's h2c package. I opened a GCP community discussion here but the jist of it is:

The 502 Bad Gateways mainly seem to happen when uploading files (POST + request body)
Size of the request body seems to exacerbate issue
Issue doesn't happen frequently on my or any other user's PCs, maybe 1 out of 20 uploads or less
The issue happens extremely frequently in GCP's "Cloud Build" environment (sometimes 50% of the time or more)
This is how I discovered the issue - starting January of this year uploads in our Cloud Build pipelines started failing with 502s at an annoyingly high rate (50% or more)
Cloud Build has very fast upload speeds (1Gbps), so I wonder if that could be a factor
If I use HTTP1 the problem goes away and I can't get any 502s
If I use Node.js instead of golang the problem also goes away and I can't get any 502s no matter how big of files I upload

Here's the go code I deployed to Cloud Run:

package main

import (
	"io"
	"log"
	"net/http"

	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
)

func main() {
	http.HandleFunc("/upload", uploadDataHandler)
	server := &http.Server{
		Addr:    ":8080",
		Handler: h2c.NewHandler(http.DefaultServeMux, &http2.Server{}),
	}
	log.Fatal(server.ListenAndServe())
}

func uploadDataHandler(w http.ResponseWriter, r *http.Request) {
	if r.Method != http.MethodPost {
		http.Error(w, "Invalid request method", http.StatusMethodNotAllowed)
		return
	}

	maxUploadSize := int64(100 * 1024 * 1024) // 100MB
	err := r.ParseMultipartForm(maxUploadSize)
	if err != nil {
		http.Error(w, "Failed to parse multipart form", http.StatusBadRequest)
		log.Println("Error parsing form:", err)
		return
	}

	file, _, err := r.FormFile("file")
	if err != nil {
		http.Error(w, "Could not read 'file' form value in payload", http.StatusBadRequest)
		log.Println("Error reading form file:", err)
		return
	}
	defer file.Close()

	data, err := io.ReadAll(file)
	if err != nil {
		http.Error(w, "Failed to read file", http.StatusInternalServerError)
		log.Println("Error reading file:", err)
		return
	}

	log.Printf("Received file of size %d bytes\n", len(data))

	w.WriteHeader(http.StatusOK)
	w.Write([]byte(`{"message": "file uploaded successfully"}`))
}

Dockerfile:

from scratch

ADD ./h2cissue /h2cissue
EXPOSE 8080
ENTRYPOINT ["/h2cissue"]

Command to run from Cloud Build to trigger failure:

dd if=/dev/urandom of=random_file bs=1M count=5
curl -X POST -F "file=@random_file" https://h2cissue-664297616426.us-central1.run.app/upload

About 20% of the time you'll get:

upstream connect error or disconnect/reset before headers. reset reason: protocol error

This Node.js implementation, however, never fails:

const http2 = require('http2');
const PORT = process.env.PORT || 8080;

const server = http2.createServer();

server.on('stream', (stream, headers) => {
  const method = headers[':method']
  const path = headers[':path']

  if (method === 'POST' && path === '/upload') {
    // Read in form data file and print the length
    const chunks = [];
    stream.on('data', chunk => {
      chunks.push(chunk);
    });
    stream.on('end', () => {
      const data = Buffer.concat(chunks);
      console.log(`Received ${data.length} bytes of data`);
      stream.respond({ ':status': 200 });
      stream.end('File received');
    });
  }
});

server.listen(PORT, () => {
  console.log(`Server is listening on port ${PORT}`);
})

What did you see happen?

I'm seeing sporadic protocol errors floating out of golang's h2c implementation

What did you expect to see?

I expected golang to be more robust than Node.js! In all seriousness though, I am eager to help investigate/debug further, any tips would be appreciated.

Related Issues

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

I think this needs to be a bug report with GCP, the error indicates its a problem with their load balancers, there isn't much the go h2c package can do about "being more robust".

I opened the issue because other http2 implementations such as Node.js don't have the issue. So to me that means golang's h2c implementation is fragile in a way that Node.js is not. If I file a bug report with GCP and they find out it's working for Node.js and not golang what's to stop GCP maintainers from just pointing the finger back at you?

GCP issue for reference: https://issuetracker.google.com/issues/410174597

We hit this exact issue on Cloud Run with Go + Gin: sporadic "client disconnected" on multipart uploads over h2c. Confirmed it goes away with HTTP/1.1, but Cloud Run has a 32MB payload limit on HTTP/1.1 so that's not an option for us.
What worked: deploying an Nginx sidecar on Cloud Run that accepts h2c from the Google Frontend and proxies to the Go app over HTTP/1.1 on localhost (no payload limit on loopback). This is a documented Google Cloud pattern: https://cloud.google.com/run/docs/internet-proxy-nginx-sidecar
The Go app no longer needs to handle h2c at all, and the upload failures stopped.

Nice workaround, I'll have to try it. Slightly tangential, but I noticed that recently you can do payloads >32MB if you go through a GCP Load Balancer --> Cloud Run instead of Cloud Run directly. It appears the load balancer's internal Envoy proxy is configured to do something similar to the nginx sidecar pattern you posted (downgrade to HTTP 1.1), so it could be the case that you could turn off h2c here as well to workaround the issue.

Fable 5.1 figured it out for me—

What's going on: Go's HTTP/2 server advertises a 1 MiB SETTINGS_MAX_FRAME_SIZE by default, and Envoy's newer oghttp2 codec (which Google's frontend switched to) honors it, so on a fast link Cloud Run sends DATA frames up to 1 MiB each. Go's server then replies with a connection+stream WINDOW_UPDATE pair for roughly every 4 KiB the handler consumes, so a single 1 MiB DATA frame triggers ~500 WINDOW_UPDATEs against Envoy's default upstream flood budget of 5 + 2*(streams + 10*DATA_frames_sent), i.e. about 20. Envoy treats that as a flood, kills the upstream connection with a codec error, and surfaces it as the 502 upstream connect error or disconnect/reset before headers. reset reason: protocol error, which is why fast uploaders fail more often while HTTP/1.1 and Node (16 KiB default frame size) are unaffected.

Workaround: advertise the HTTP/2 default frame size (16 KiB) instead of Go's 1 MiB, which keeps the ratio at ~8 WINDOW_UPDATEs per DATA frame, well under Envoy's limit:

srv := &http.Server{
  Addr:    ":8080",
  Handler: handler,
  // Go 1.24+: cap incoming DATA frames at the HTTP/2 default (16 KiB)
  // instead of Go's 1 MiB default.
  HTTP2: &http.HTTP2Config{MaxReadFrameSize: 16 << 10},
}

On older Go, or if you configure golang.org/x/net/http2 directly, the equivalent is http2.ConfigureServer(srv, &http2.Server{MaxReadFrameSize: 16 << 10}).

fix(api): advertise a 16 KiB HTTP/2 max frame size (#1700)

Navigation Menu

Repository navigation

golang.org/x/net/http2/h2c: GCP Cloud Run Sporadic 502 Bad Gateway #73343

Go version

Output of `go env` in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

golang.org/x/net/http2/h2c: GCP Cloud Run Sporadic 502 Bad Gateway #73343

Description

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Activity

gabyhelp commented on Apr 12, 2025

seankhliao commented on Apr 12, 2025

RPGillespie6 commented on Apr 12, 2025

RPGillespie6 commented on Apr 12, 2025

Serhii-Kozik commented on Feb 19, 2026

RPGillespie6 commented on Feb 19, 2026

taylorhughes commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `go env` in your module/workspace: