Skip to content

bug: proxy-mirror sends mirrored gRPC requests with :path /proxy_mirror_grpc instead of the original method path #13498

Description

@nic-6443

Current Behavior

When proxy-mirror is configured with a grpc:// host, the mirrored request arrives at the mirror target with :path = /proxy_mirror_grpc (the name of the internal nginx location) instead of the real gRPC method path. A gRPC server receiving such a call just rejects it with UNIMPLEMENTED, so gRPC mirroring is effectively unusable — the mirror backend never sees a valid call.

The root cause is in the generated nginx config: the HTTP mirror location forwards with proxy_pass $upstream_mirror_uri which carries the full URI, but the gRPC mirror location only has grpc_pass $upstream_mirror_host. grpc_pass cannot carry a URI, so the grpc module builds :path from the subrequest's own URI, which is the mirror location name /proxy_mirror_grpc.

The existing test case in t/plugin/proxy-mirror3.t (added in #9388) only asserts Connection refused against a closed port, so it never verified what the mirrored gRPC request actually contains and this went unnoticed.

There is also a phase-timing aspect: plugins like grpc-web rewrite the request URI in their access phase, while proxy-mirror captures the URI in its rewrite phase, so the mirror path must be taken after all access phase plugins have run.

Expected Behavior

The mirrored gRPC request keeps the same method path as the upstream request, including rewrites done by access phase plugins (e.g. grpc-web).

Error Logs

No error on the APISIX side. The mirror target receives POST /proxy_mirror_grpc HTTP/2.0 (verified with an h2c server logging the incoming :path).

Steps to Reproduce

  1. Create a route with a grpc scheme upstream and proxy-mirror pointing to a grpc:// mirror host:
routes:
  - uris:
      - /helloworld.Greeter/SayHello
    plugins:
      proxy-mirror:
        host: grpc://127.0.0.1:19797
    upstream:
      scheme: grpc
      nodes:
        "127.0.0.1:10051": 1
      type: roundrobin
  1. Run any h2c server on 19797 that logs the request path (the mirror target).
  2. Send a request: grpcurl -plaintext -d '{"name":"apisix"}' 127.0.0.1:1984 helloworld.Greeter.SayHello
  3. The main upstream receives POST /helloworld.Greeter/SayHello HTTP/2.0, while the mirror target receives POST /proxy_mirror_grpc HTTP/2.0.

Environment

  • APISIX version: reproduced on master and 3.14.1
  • Operating system: Ubuntu 22.04
  • OpenResty / Nginx version: apisix-runtime (openresty 1.29)

I have a fix ready: capture the mirror path into a new $upstream_mirror_grpc_path variable in the before_proxy phase (which runs after all access phase plugins have rewritten the URI), and rewrite the mirror subrequest URI with it in the proxy_mirror_grpc location before grpc_pass. Will submit a PR shortly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    ✅ Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions