perf(ai): reuse raw request body when unchanged#13406
Conversation
dosubot
Bot
added
size:L
There was a problem hiding this comment.
Pull request overview
This PR optimizes the AI proxy request pipeline by reusing the original raw client request body when the outgoing body is unchanged, avoiding unnecessary JSON re-encoding and preventing retry attempts from accidentally reverting to the original raw body after a prior rewrite.
Changes:
- Track whether the request body was modified during provider request building, and reuse the original raw body when unchanged.
- Update AI protocol adapters’
prepare_outgoing_requestto return a boolean indicating whether they mutated the body. - Add tests to ensure ai-proxy / ai-proxy-multi forward the original body when not rewritten, and that retries don’t reuse raw after a rewrite.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| t/plugin/ai-proxy-request-body-override.t | Adds coverage for the unchanged-body forward path and retry behavior. |
| apisix/plugins/ai-request-rewrite.lua | Marks sidecar-generated requests as “body changed” to prevent raw-body reuse. |
| apisix/plugins/ai-providers/base.lua | Implements change tracking and raw-body reuse/caching logic in build_request. |
| apisix/plugins/ai-protocols/passthrough.lua | Makes prepare_outgoing_request explicitly report “no mutation”. |
| apisix/plugins/ai-protocols/openai-chat.lua | Returns whether stream-related injection mutated the body. |
| apisix/plugins/ai-protocols/bedrock-converse.lua | Returns whether stripping the stream field mutated the body. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
|
|
||
|
|
||
| === TEST 4b: ai-proxy-multi forwards the original body when it is not rewritten |
There was a problem hiding this comment.
I think lint will fail here.
There was a problem hiding this comment.
strange that lint didn't fail here, I remember doing something similar and lint failed, lol.
There was a problem hiding this comment.
===TEST 4a
...
===TEST 4b
At this point, Lint will fail.
This avoids re-encoding the ai-proxy request body when the outgoing body is unchanged. If protocol conversion, provider preparation, model options, request body overrides, or provider cleanup changes the body, the existing encoded table path is still used.
It also covers ai-proxy-multi through the shared provider build path and keeps retry behavior from falling back to the original raw body after an earlier attempt has rewritten the parsed body. The raw body is read lazily in the unchanged-body branch and cached on the request context, so retries do not read it again.
The sidecar LLM request path used by ai-request-rewrite is explicitly marked as generated, so it continues to send the JSON-encoded LLM request instead of reusing the original client body.
Tests run: