fix(api): advertise a 16 KiB HTTP/2 max frame size by geoherna · Pull Request #1700 · gotenberg/gotenberg

geoherna · 2026-10-23T21:40:32Z

Summary

Set http.Server.HTTP2 = &http.HTTP2Config{MaxReadFrameSize: 16 << 10} in the API module's BeforeServeFunc, so Gotenberg's HTTP/2 server advertises the protocol default SETTINGS_MAX_FRAME_SIZE (16 KiB) instead of Go's 1 MiB default. This stops a class of sporadic connection resets on multipart uploads when Gotenberg runs behind an HTTP/2 proxy such as Google Cloud Run's, and addresses the symptom reported in #1112.

One line of configuration plus a comment. No flag, no form field, no endpoint change. The setting applies to both the h2c path and the TLS path, since both share the same http.Server.

Symptom

Deployments of Gotenberg on Cloud Run with the container port named h2c (required for request bodies over 32 MiB) see intermittent failures on POST routes that upload files, for example /forms/chromium/convert/html and /forms/pdfengines/merge:

The client receives 502 with the body upstream connect error or disconnect/reset before headers. reset reason: protocol error.
Gotenberg logs create request context: get multipart form: client disconnected for the same request, or nothing at all when the reset lands before the handler runs.
Retrying the identical request usually succeeds. Larger bodies and faster uploaders fail more often. Switching the port back to HTTP/1 makes the failures disappear but reinstates the 32 MiB limit.

This is the thread in #1112, which was closed without a root cause, and the same symptom as golang/go#73343 for any Go h2c server on Cloud Run.

Mechanism

Go's HTTP/2 server advertises SETTINGS_MAX_FRAME_SIZE = 1 MiB by default (defaultMaxReadFrameSize in net/http's bundled http2). A proxy that honors the advertised value is free to send DATA frames far larger than the 16 KiB protocol default, and Cloud Run's proxy does: with a buffered request body it sends DATA frames of tens to hundreds of KiB on a fast link, up to roughly 250 KiB in the trace below.

Go's server refreshes flow control in 4 KiB steps (inflowMinRefresh): for roughly every 4 KiB the handler consumes it sends one stream-level and one connection-level WINDOW_UPDATE. A single 250 KiB DATA frame therefore produces on the order of 120 WINDOW_UPDATE frames in reply. The proxy budgets the number of WINDOW_UPDATE frames it will accept per DATA frame it has sent; that burst exceeds the budget, the proxy treats it as a flood and drops the TCP connection without a GOAWAY. Gotenberg sees the client disconnect mid-body; the client sees the 502.

Advertising 16 KiB frames keeps the ratio at about 8 WINDOW_UPDATE frames per DATA frame, well inside the proxy's allowance. 16 KiB is the minimum value net/http accepts for MaxReadFrameSize and the value every HTTP/2 peer must support, so no client needs to change. Frame size does not limit request body size or throughput; it only changes how the body is chunked on the wire, and the A/B below showed no latency difference.

The mechanism and this exact workaround were identified in a 2026-10-01 comment on golang/go#73343 (HTTP2: &http.HTTP2Config{MaxReadFrameSize: 16 << 10} on Go 1.24+). This PR applies it to Gotenberg's server, where most users cannot change the Go code themselves.

Evidence

A/B on a Cloud Run service running Gotenberg 8.37.0 with the port named h2c and concurrency 1. Client: a Go harness sending a 6,515,078-byte multipart body over one shared HTTP/2 connection, in rounds of 6 or 12 parallel uploads. Order of runs: control, patched, control, so instance churn and time of day do not favour either arm.

Arm	Build	Parallelism x rounds	Uploads	502 resets
control	official `gotenberg/gotenberg:8.37.0`	6 x 9	54	3
patched	8.37.0 + this change	6 x 9	54	0
patched	8.37.0 + this change	12 x 6	72	0
control	official `gotenberg/gotenberg:8.37.0`	6 x 9	54	1
control	official `gotenberg/gotenberg:8.37.0`	12 x 6	72	3

Totals: 7 resets in 180 uploads on the official image, 0 in 126 with the patched build. Earlier the same day the official image produced 5/18 and 3/18 in two smaller runs. Per-upload latency was the same in both arms (about 9 s per upload at 6 in parallel, about 17 s at 12).

A frame trace of the official image with GODEBUG=http2debug=2 shows the pattern above directly: DATA frames of 40 KiB to about 248 KiB arriving from the proxy, a run of 4 KiB-step WINDOW_UPDATE frames after each (over 400 per connection), then the connection closing with no GOAWAY.

Changes

pkg/modules/api/api.go: set s.HTTP2 = &http.HTTP2Config{MaxReadFrameSize: 16 << 10} in BeforeServeFunc, before the h2c protocol block so it covers both the h2c and TLS servers, with a comment explaining why.

Backward compatibility

No CLI flag, environment variable, form field or endpoint changes. The only observable change is the SETTINGS_MAX_FRAME_SIZE value the server advertises; HTTP/2 peers must honour it, and 16 KiB is the value every peer already supports. Clients that previously sent larger frames will send more, smaller frames. No flag is added on purpose: nothing in Gotenberg benefits from 1 MiB frames, and a fixed default is what removes the failure for every deployment.

Testing

go build ./...
go test ./pkg/modules/api/... and go test -race ./pkg/modules/api/...
The staging A/B above.

No unit test is added; the change is a server setting that only shows on the wire as a SETTINGS frame. A test that opens an h2c connection and asserts the advertised value is possible with golang.org/x/net/http2 (already a dependency) if you would like one.

References

golang.org/x/net/http2/h2c: GCP Cloud Run Sporadic 502 Bad Gateway golang/go#73343, "x/net/http2/h2c: GCP Cloud Run Sporadic 502 Bad Gateway", in particular the comment of 2026-10-01 describing the mechanism and this workaround.
Cloud Run: 502 error #1112, "Cloud Run: 502 error". This change addresses the symptom reported there.

Fixes #1112.

Go's HTTP/2 server advertises SETTINGS_MAX_FRAME_SIZE of 1 MiB by default. Proxies that honour it, such as Cloud Run's, send DATA frames of hundreds of KiB on a fast link, and Go's server answers with a WINDOW_UPDATE pair for roughly every 4 KiB it consumes. That exceeds the proxy's per-frame WINDOW_UPDATE allowance, so it drops the connection mid-upload: the client sees a 502 "protocol error" and Gotenberg logs "get multipart form: client disconnected". Advertise the protocol default (16 KiB) instead. On a Cloud Run service with the port named h2c, the official 8.37.0 image produced 7 resets in 180 multipart uploads; the same image with this change produced 0 in 126, at the same latency. See golang/go#73343 (comment of 2026-10-01) and gotenberg#1112. Fixes gotenberg#1112

gulien · 2026-10-25T16:23:35Z

Thanks @geoherna, great investigation and write-up! 🙏

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [gotenberg/gotenberg](https://github.com/gotenberg/gotenberg) | minor | `8.38.1` → `8.39.0` | --- ### Release Notes <details> <summary>gotenberg/gotenberg</summary> ### [`v8.39.0`](https://github.com/gotenberg/gotenberg/releases/tag/v8.39.0): 8.39.0 [Compare Source](gotenberg/gotenberg@v8.38.1...v8.39.0) ##### Security Fixes ⚠️ - **Reject ExifTool's `j` option alias and stop loading ExifTool config files.** The metadata check blocked `json` but not its alias `j`. A metadata key could import a JSON file carrying a `HardLink` pseudo-tag, plant `~/.ExifTool_config`, and have the next ExifTool run execute it as Perl. The keys `j`, `addtagsfromfile`, and `alltagsfromfile` now return `400`, and Gotenberg always runs ExifTool with `-config ""`. - **Accept only `http://` and `https://` URLs on the URL routes.** The default deny-list only covered `file://`, so `devtools://devtools/bundled/devtools_app.html` loaded Chromium's privileged DevTools page, and `waitForExpression` ran there with access to local files. `/forms/chromium/convert/url` and `/forms/chromium/screenshot/url` now return `400` for any other scheme. - **Apply path and `wss://` deny rules to WebSocket handshakes.** The pinning proxy matched CONNECT tunnels against `https://host:port`, so a deny rule anchored on a path, such as `^https?://host/`, or on `wss://` refused the plain request but let the handshake through. The proxy now matches tunnels against each scheme's root URL, and a handshake whose full URL is denied fails the conversion with a generic `403`. - **Drop the compromised eMudhra roots** ([#1699](gotenberg/gotenberg#1699)). The images trusted `emSign ECC Root CA - G3` and `AAA Certificate Services`, whose intermediates have recoverable private keys. The OS bundle now comes from `ca-certificates` `20250419+deb13u1`, the Java trust store used by PDFtk drops both roots, and Chromium 156 rejects them. Thanks [@hrueger](https://github.com/hrueger). ##### New Features - **`failOnConsoleMessageTypes` for Chromium** ([#1667](gotenberg/gotenberg#1667)). Fails the conversion with `409` when the page logs a console message of one of the given types (`log`, `debug`, `info`, `warning`, `error`, `assert`). The response lists the messages. Thanks [@jschneider](https://github.com/jschneider). - **PDF/A-4** ([#1703](gotenberg/gotenberg#1703)). The `pdfa` form field now accepts `PDF/A-4` on every route that supports it, through the LibreOffice PDF engine. Thanks [@henningda](https://github.com/henningda). - **`Gotenberg-Version` response header** ([#1693](gotenberg/gotenberg#1693)). Set on every authenticated route, error responses included, and on webhook deliveries. It is not set on `401` responses or `/health` ([#1694](gotenberg/gotenberg#1694)). Thanks [@fragkakis](https://github.com/fragkakis). ##### Behavior Changes - **The URL routes reject non-HTTP schemes.** `chrome://`, `devtools://`, `data:`, and any other scheme besides `http` and `https` now return `400`. - **ExifTool ignores `~/.ExifTool_config`.** The metadata keys `j`, `addtagsfromfile`, and `alltagsfromfile` now return `400`. Prefix them with a group, for example `XMP:j`, to write a tag of that name. - **The `XRECHNUNG` Factur-X profile embeds `xrechnung.xml`.** See the bug fix below. ##### Bug Fixes - **Sporadic `502` on uploads behind HTTP/2 proxies** ([#1112](gotenberg/gotenberg#1112)). Behind Cloud Run with an `h2c` port, the proxy sent large DATA frames, Go answered with a flood of WINDOW_UPDATE frames, and the proxy dropped the connection mid-upload. Gotenberg now advertises the 16 KiB HTTP/2 default frame size ([#1700](gotenberg/gotenberg#1700)). Thanks [@TessilimiTheo](https://github.com/TessilimiTheo) and [@geoherna](https://github.com/geoherna). - **No listener with absolute TLS paths** ([#1690](gotenberg/gotenberg#1690)). Since 8.38.0, `--api-tls-cert-file` and `--api-tls-key-file` with absolute paths left Gotenberg running without accepting connections. Both paths work again, and a missing or unreadable file now fails the startup ([#1691](gotenberg/gotenberg#1691)). Thanks [@n-rodriguez](https://github.com/n-rodriguez). - **Empty `stampSource` or `watermarkSource` returned `400`** ([#1692](gotenberg/gotenberg#1692)). Since 8.36.0, a blank source failed the merge, split, Chromium, and LibreOffice routes. An empty source is skipped again. Thanks [@APX403](https://github.com/APX403). - **OIDC rejected `bearer` in lower case** ([#1696](gotenberg/gotenberg#1696)). The Authorization scheme is now matched case-insensitively, as RFC 9110 requires. Thanks [@MaxFreedomPollard](https://github.com/MaxFreedomPollard). - **The `XRECHNUNG` Factur-X profile embedded `factur-x.xml`** ([#1701](gotenberg/gotenberg#1701)). The spec requires `xrechnung.xml` for this profile, both as the embedded file and in `fx:DocumentFileName`. Other levels keep `factur-x.xml`. Thanks [@nhh](https://github.com/nhh). ##### Chore - Updated Chromium to `156.0.8078.47`. - Updated pdfcpu to `v0.16.3` and bumped Go to `1.27.3`, which fix several HIGH CVEs in the Go standard library and `golang.org/x` modules ([#1695](gotenberg/gotenberg#1695)). Thanks [@sergeykad](https://github.com/sergeykad). - Updated the AWS Lambda Web Adapter to `1.1.0`, which supports Lambda SnapStart ([#1704](gotenberg/gotenberg#1704)). Thanks [@michael-follmann](https://github.com/michael-follmann). - Bumped golangci-lint to `v2.15.0`. - Updated Go dependencies. </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.hayden.moe/hayden/phoebe/pulls/1021

… 8.39.0) (#2479) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.io/gotenberg/gotenberg](https://github.com/gotenberg/gotenberg) | minor | `8.38.1` → `8.39.0` | --- ### Release Notes <details> <summary>gotenberg/gotenberg (docker.io/gotenberg/gotenberg)</summary> ### [`v8.39.0`](https://github.com/gotenberg/gotenberg/releases/tag/v8.39.0): 8.39.0 [Compare Source](gotenberg/gotenberg@v8.38.1...v8.39.0) ##### Security Fixes ⚠️ - **Reject ExifTool's `j` option alias and stop loading ExifTool config files.** The metadata check blocked `json` but not its alias `j`. A metadata key could import a JSON file carrying a `HardLink` pseudo-tag, plant `~/.ExifTool_config`, and have the next ExifTool run execute it as Perl. The keys `j`, `addtagsfromfile`, and `alltagsfromfile` now return `400`, and Gotenberg always runs ExifTool with `-config ""`. - **Accept only `http://` and `https://` URLs on the URL routes.** The default deny-list only covered `file://`, so `devtools://devtools/bundled/devtools_app.html` loaded Chromium's privileged DevTools page, and `waitForExpression` ran there with access to local files. `/forms/chromium/convert/url` and `/forms/chromium/screenshot/url` now return `400` for any other scheme. - **Apply path and `wss://` deny rules to WebSocket handshakes.** The pinning proxy matched CONNECT tunnels against `https://host:port`, so a deny rule anchored on a path, such as `^https?://host/`, or on `wss://` refused the plain request but let the handshake through. The proxy now matches tunnels against each scheme's root URL, and a handshake whose full URL is denied fails the conversion with a generic `403`. - **Drop the compromised eMudhra roots** ([#1699](gotenberg/gotenberg#1699)). The images trusted `emSign ECC Root CA - G3` and `AAA Certificate Services`, whose intermediates have recoverable private keys. The OS bundle now comes from `ca-certificates` `20250419+deb13u1`, the Java trust store used by PDFtk drops both roots, and Chromium 156 rejects them. Thanks [@hrueger](https://github.com/hrueger). ##### New Features - **`failOnConsoleMessageTypes` for Chromium** ([#1667](gotenberg/gotenberg#1667)). Fails the conversion with `409` when the page logs a console message of one of the given types (`log`, `debug`, `info`, `warning`, `error`, `assert`). The response lists the messages. Thanks [@jschneider](https://github.com/jschneider). - **PDF/A-4** ([#1703](gotenberg/gotenberg#1703)). The `pdfa` form field now accepts `PDF/A-4` on every route that supports it, through the LibreOffice PDF engine. Thanks [@henningda](https://github.com/henningda). - **`Gotenberg-Version` response header** ([#1693](gotenberg/gotenberg#1693)). Set on every authenticated route, error responses included, and on webhook deliveries. It is not set on `401` responses or `/health` ([#1694](gotenberg/gotenberg#1694)). Thanks [@fragkakis](https://github.com/fragkakis). ##### Behavior Changes - **The URL routes reject non-HTTP schemes.** `chrome://`, `devtools://`, `data:`, and any other scheme besides `http` and `https` now return `400`. - **ExifTool ignores `~/.ExifTool_config`.** The metadata keys `j`, `addtagsfromfile`, and `alltagsfromfile` now return `400`. Prefix them with a group, for example `XMP:j`, to write a tag of that name. - **The `XRECHNUNG` Factur-X profile embeds `xrechnung.xml`.** See the bug fix below. ##### Bug Fixes - **Sporadic `502` on uploads behind HTTP/2 proxies** ([#1112](gotenberg/gotenberg#1112)). Behind Cloud Run with an `h2c` port, the proxy sent large DATA frames, Go answered with a flood of WINDOW_UPDATE frames, and the proxy dropped the connection mid-upload. Gotenberg now advertises the 16 KiB HTTP/2 default frame size ([#1700](gotenberg/gotenberg#1700)). Thanks [@TessilimiTheo](https://github.com/TessilimiTheo) and [@geoherna](https://github.com/geoherna). - **No listener with absolute TLS paths** ([#1690](gotenberg/gotenberg#1690)). Since 8.38.0, `--api-tls-cert-file` and `--api-tls-key-file` with absolute paths left Gotenberg running without accepting connections. Both paths work again, and a missing or unreadable file now fails the startup ([#1691](gotenberg/gotenberg#1691)). Thanks [@n-rodriguez](https://github.com/n-rodriguez). - **Empty `stampSource` or `watermarkSource` returned `400`** ([#1692](gotenberg/gotenberg#1692)). Since 8.36.0, a blank source failed the merge, split, Chromium, and LibreOffice routes. An empty source is skipped again. Thanks [@APX403](https://github.com/APX403). - **OIDC rejected `bearer` in lower case** ([#1696](gotenberg/gotenberg#1696)). The Authorization scheme is now matched case-insensitively, as RFC 9110 requires. Thanks [@MaxFreedomPollard](https://github.com/MaxFreedomPollard). - **The `XRECHNUNG` Factur-X profile embedded `factur-x.xml`** ([#1701](gotenberg/gotenberg#1701)). The spec requires `xrechnung.xml` for this profile, both as the embedded file and in `fx:DocumentFileName`. Other levels keep `factur-x.xml`. Thanks [@nhh](https://github.com/nhh). ##### Chore - Updated Chromium to `156.0.8078.47`. - Updated pdfcpu to `v0.16.3` and bumped Go to `1.27.3`, which fix several HIGH CVEs in the Go standard library and `golang.org/x` modules ([#1695](gotenberg/gotenberg#1695)). Thanks [@sergeykad](https://github.com/sergeykad). - Updated the AWS Lambda Web Adapter to `1.1.0`, which supports Lambda SnapStart ([#1704](gotenberg/gotenberg#1704)). Thanks [@michael-follmann](https://github.com/michael-follmann). - Bumped golangci-lint to `v2.15.0`. - Updated Go dependencies. </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2479

gulien merged commit 42affb1 into gotenberg:main Oct 25, 2026

Navigation Menu

Repository navigation

fix(api): advertise a 16 KiB HTTP/2 max frame size - #1700

fix(api): advertise a 16 KiB HTTP/2 max frame size#1700
gulien merged 1 commit into
gotenberg:mainfrom
geoherna:api-h2c-max-read-frame-size

geoherna commented Oct 23, 2026

Uh oh!

gulien commented Oct 25, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

geoherna commented Oct 23, 2026

Summary

Symptom

Mechanism

Evidence

Changes

Backward compatibility

Testing

References

Uh oh!

gulien commented Oct 25, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants