Skip to content

Cloud Run: 502 error #1112

Closed
Closed

Description

@TessilimiTheo

I'm experiencing an issue when running Gotenberg (8-cloudrun) on Google Cloud Run behind Cloudflare. Intermittently, I receive a 502: create request context: get multipart form: client disconnected when making POST requests to /forms/pdfengines/merge and /forms/chromium/convert/html.

This started happening around January 13th. Retrying the call multiple time can resolve the issue, but the root cause is unclear.

The command I'm using to run Gotenberg:

gotenberg --chromium-auto-start --chromium-disable-web-security --api-timeout=120s  --pdfengines-engines=qpdf,pdftk

Environment details:

  • Platform: Google Cloud Run
  • Proxy: Cloudflare

It seems like the issue could be related to timeout or protocol mismatches between Cloudflare, Cloud Run, and Gotenberg. Any guidance or suggestions on how to resolve this would be appreciated!

I try different versions of gotenberg.

If you have any insights into why these issues are occurring that might help me solve them.

Activity

  1. added
    bugSomething isn't working
    help wantedExtra attention is needed
    on Jan 27, 2025
  2. changed the title [-]502 error on /forms/pdfengines/merge[/-] [+]Cloud Run: 502 error on /forms/pdfengines/merge[/+] on Jan 27, 2025
  3. gulien commented on Jan 27, 2025

    @gulien
    Member

    Hello @TessilimiTheo,

    I'm not familiar with your configuration, so I cannot help much. I guess it's working fine when calling the Gotenberg instance via Cloud Run directly?

  4. TessilimiTheo commented on Jan 27, 2025

    @TessilimiTheo
    Author

    Hello @TessilimiTheo,

    I'm not familiar with your configuration, so I cannot help much. I guess it's working fine when calling the Gotenberg instance via Cloud Run directly?

    Hi, thank you for your response. I removed Cloudflare, and my application now calls Gotenberg directly on Cloud Run. However, I still encounter the same error on the path /forms/chromium/convert/html:

    502 create request context: get multipart form: client disconnected

  5. gulien commented on Jan 27, 2025

    @gulien
    Member

    Does it work when you use HTTP/2?

  6. TessilimiTheo commented on Jan 27, 2025

    @TessilimiTheo
    Author

    No, I'm already on HTTP/2

  7. gulien commented on Jan 27, 2025

    @gulien
    Member

    I'm kinda out of ideas now 🤔 maybe @fank has some experience on this issue?

  8. fank commented on Jan 28, 2025

    @fank

    Not sure if helpful but I am using https://github.com/starwalkn/gotenberg-go-client to connect to gotenberg.
    The command of how I deployed it:

    gcloud run deploy gotenberg-prod
              --image=gotenberg/gotenberg:8.15.3-cloudrun
              --region=europe-west3
              --no-allow-unauthenticated
              --args=gotenberg
              --args="--api-port-from-env=PORT"
              --args="--api-timeout=300s"
              --args="--webhook-disable=true"
              --args="--chromium-restart-after=50"
              --memory=4Gi
    

    --no-allow-unauthenticated is just for authentication, does not make a different of how stable it is.

    So far the instance is running without any kind of connection issues.

    The URL I am using to connect to the instance is the .run.app domain.

  9. novelnet commented on Feb 2, 2025

    @novelnet

    Disabling HTTP/2 solved it for me mostly, also disabled NGINX in proxy redirect and changed the startup probe with delay (all together works for me):

    Image

  10. sagikazarmark commented on Feb 4, 2025

    @sagikazarmark

    I have the same problem, except I don't use CloudFlare. It also started recently.

    I just enabled debug logging. Info logs only reported 200 on the Gotenberg side and 502 on the Cloud Run side.

    I will try disabling HTTP/2 and report back. If that's indeed the problem, I wonder if the recent Cloud Run v2 migration is related.

  11. changed the title [-]Cloud Run: 502 error on /forms/pdfengines/merge[/-] [+]Cloud Run: 502 error[/+] on Feb 4, 2025
  12. 5 remaining items

  13. reopened this on Feb 6, 2025
  14. stijntratsaertit commented on Feb 6, 2025

    @stijntratsaertit

    According to these docs it is not advised - even prohibited - to use such connection headers when using HTTP/2.

  15. stranavad commented on Feb 6, 2025

    @stranavad

    I have also run into this issue today, gotenberg on cloud run, http/2 (we need more than 32MB), 502 Bad Gateway error

  16. gulien commented on Feb 6, 2025

    @gulien
    Member

    What kind of HTTP/2 are you all using? H2C? Classic?

  17. stijntratsaertit commented on Feb 6, 2025

    @stijntratsaertit

    H2C.

    This is the Cloud Run spec we're running.

    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: gotenberg
      annotations:
        run.googleapis.com/ingress: internal
      labels:
        cloud.googleapis.com/location: europe-west1
    spec:
      template:
        metadata:
          annotations:
            autoscaling.knative.dev/minScale: "0"
            autoscaling.knative.dev/maxScale: "100"
            run.googleapis.com/execution-environment: gen2
            run.googleapis.com/vpc-access-connector: projects/.../locations/europe-west1/connectors/...
            run.googleapis.com/vpc-access-egress: private-ranges-only
            run.googleapis.com/startup-cpu-boost: "true"
        spec:
          containerConcurrency: 1
          containers:
            - image: europe-docker.pkg.dev/.../gotenberg:8.15.3
              command:
                - gotenberg
              args:
                - "--libreoffice-auto-start"
                - "--api-timeout"
                - "450s"
                - "--pdfengines-disable-routes"
                - "true"
                - "--webhook-disable"
                - "true"
                - "--prometheus-disable-route-logging"
                - "true"
                - "--prometheus-disable-collect"
                - "true"
              ports:
                - containerPort: 3000
                  name: h2c
              resources:
                limits:
                  cpu: 1000m
                  memory: 2Gi
          serviceAccountName: [email protected]
          timeoutSeconds: 600
      traffic:
        - latestRevision: true
          percent: 100

  18. gulien commented on Feb 6, 2025

    @gulien
    Member

    @fank are you using HTTP2 too?

  19. sagikazarmark commented on Feb 7, 2025

    @sagikazarmark

    Debug logs didn't reveal anything. Looks like on Gotenberg's side everything works, the Cloud Run proxy still replies with 502 for some reason.

    Disabling HTTP/2 solves the problem (or is at least a workaround for now).

  20. TessilimiTheo commented on Feb 10, 2025

    @TessilimiTheo
    Author

    Cloudrun's 30mo limit prevents me from disabling HTTP/2. Any suggestions before I move Gotenberg outside Cloudrun are welcome.

  21. TessilimiTheo commented on Feb 14, 2025

    @TessilimiTheo
    Author

    any update or solution?

  22. gulien commented on Aug 29, 2025

    @gulien
    Member

    As I don't have any clue of whats happening, I'm waiting for a concrete solution or explanation for re-opening this ticket.

  23. v36u commented on Mar 2, 2026

    @v36u

    My team also encountered this problem:

    We worked around this by leveraging Gotenberg's capability to download files from provided URLs: https://gotenberg.dev/docs/webhook-download#download-from. So we first upload them to some bucket dedicated for temp files (low retention), and then send the pre-signed URLs to Gotenberg, instead of sending the actual files over the network.

    It is important to keep h2c enabled though, because otherwise GCP's response size limit (https://docs.cloud.google.com/run/quotas#request_limits) will kick in and prevent Gotenberg from returning the response -- this appears in Gotenberg's logs as a 500 with no message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationDocumentation update requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions