Current Behavior
When proxy-mirror is configured with a grpc:// host, the mirrored request arrives at the mirror target with :path = /proxy_mirror_grpc (the name of the internal nginx location) instead of the real gRPC method path. A gRPC server receiving such a call just rejects it with UNIMPLEMENTED, so gRPC mirroring is effectively unusable — the mirror backend never sees a valid call.
The root cause is in the generated nginx config: the HTTP mirror location forwards with proxy_pass $upstream_mirror_uri which carries the full URI, but the gRPC mirror location only has grpc_pass $upstream_mirror_host. grpc_pass cannot carry a URI, so the grpc module builds :path from the subrequest's own URI, which is the mirror location name /proxy_mirror_grpc.
The existing test case in t/plugin/proxy-mirror3.t (added in #9388) only asserts Connection refused against a closed port, so it never verified what the mirrored gRPC request actually contains and this went unnoticed.
There is also a phase-timing aspect: plugins like grpc-web rewrite the request URI in their access phase, while proxy-mirror captures the URI in its rewrite phase, so the mirror path must be taken after all access phase plugins have run.
Expected Behavior
The mirrored gRPC request keeps the same method path as the upstream request, including rewrites done by access phase plugins (e.g. grpc-web).
Error Logs
No error on the APISIX side. The mirror target receives POST /proxy_mirror_grpc HTTP/2.0 (verified with an h2c server logging the incoming :path).
Steps to Reproduce
- Create a route with a
grpc scheme upstream and proxy-mirror pointing to a grpc:// mirror host:
routes:
- uris:
- /helloworld.Greeter/SayHello
plugins:
proxy-mirror:
host: grpc://127.0.0.1:19797
upstream:
scheme: grpc
nodes:
"127.0.0.1:10051": 1
type: roundrobin
- Run any h2c server on 19797 that logs the request path (the mirror target).
- Send a request:
grpcurl -plaintext -d '{"name":"apisix"}' 127.0.0.1:1984 helloworld.Greeter.SayHello
- The main upstream receives
POST /helloworld.Greeter/SayHello HTTP/2.0, while the mirror target receives POST /proxy_mirror_grpc HTTP/2.0.
Environment
- APISIX version: reproduced on master and 3.14.1
- Operating system: Ubuntu 22.04
- OpenResty / Nginx version: apisix-runtime (openresty 1.29)
I have a fix ready: capture the mirror path into a new $upstream_mirror_grpc_path variable in the before_proxy phase (which runs after all access phase plugins have rewritten the URI), and rewrite the mirror subrequest URI with it in the proxy_mirror_grpc location before grpc_pass. Will submit a PR shortly.
Current Behavior
When
proxy-mirroris configured with agrpc://host, the mirrored request arrives at the mirror target with:path = /proxy_mirror_grpc(the name of the internal nginx location) instead of the real gRPC method path. A gRPC server receiving such a call just rejects it withUNIMPLEMENTED, so gRPC mirroring is effectively unusable — the mirror backend never sees a valid call.The root cause is in the generated nginx config: the HTTP mirror location forwards with
proxy_pass $upstream_mirror_uriwhich carries the full URI, but the gRPC mirror location only hasgrpc_pass $upstream_mirror_host.grpc_passcannot carry a URI, so the grpc module builds:pathfrom the subrequest's own URI, which is the mirror location name/proxy_mirror_grpc.The existing test case in
t/plugin/proxy-mirror3.t(added in #9388) only assertsConnection refusedagainst a closed port, so it never verified what the mirrored gRPC request actually contains and this went unnoticed.There is also a phase-timing aspect: plugins like
grpc-webrewrite the request URI in their access phase, whileproxy-mirrorcaptures the URI in its rewrite phase, so the mirror path must be taken after all access phase plugins have run.Expected Behavior
The mirrored gRPC request keeps the same method path as the upstream request, including rewrites done by access phase plugins (e.g.
grpc-web).Error Logs
No error on the APISIX side. The mirror target receives
POST /proxy_mirror_grpc HTTP/2.0(verified with an h2c server logging the incoming:path).Steps to Reproduce
grpcscheme upstream andproxy-mirrorpointing to agrpc://mirror host:grpcurl -plaintext -d '{"name":"apisix"}' 127.0.0.1:1984 helloworld.Greeter.SayHelloPOST /helloworld.Greeter/SayHello HTTP/2.0, while the mirror target receivesPOST /proxy_mirror_grpc HTTP/2.0.Environment
I have a fix ready: capture the mirror path into a new
$upstream_mirror_grpc_pathvariable in thebefore_proxyphase (which runs after all access phase plugins have rewritten the URI), and rewrite the mirror subrequest URI with it in theproxy_mirror_grpclocation beforegrpc_pass. Will submit a PR shortly.