Description:
We just tried upgrading our gateway using helm from 1.3.3 to 1.4.0 and notice that our grpcroutes are no longer working as expected. When we call them using native grpc, we always get a "13 - internal" error. When we call them using grpc-web it works fine still. On 1.3.3 both native grpc and grpc-web work correctly and we didn't touch any other configuration.

Repro steps:
Run a very basic grpc backend on the same k8s cluster as envoy (we used grpcbin for testing), try connecting using both grpc-web and grpc. This is our grpcroute config:

apiVersion: gateway.networking.k8s.io/v1 kind: GRPCRoute metadata: name: datahub-api-grpcbin spec: parentRefs: - name: dm-gateway namespace: dm-gateway hostnames: - "api...." rules: - matches: - method: service: grpcbin.GRPCBin filters: - type: RequestHeaderModifier requestHeaderModifier: add: - name: x-test-header value: foo backendRefs: - kind: Service name: grpcbin namespace: default port: 9000

Environment:
Gateway 1.4.0

Logs:
The access logs show a response code 200 to these calls, but a connectrpc go client returns the error "protocol error: no Grpc-Status trailer: unexpected EOF" , except when we force grpcweb then it works fine. A postman client using grpc protocol is also returning a 13 - INTERNAL error.

Reverting back to 1.3.3 solves the issue.