feat(ai-proxy): add per-protocol request_body override and rename max_tokens mapping to llm_options#13269
Merged
nic-6443 merged 2 commits intoApr 22, 2026
Conversation
…_tokens mapping to llm_options - Rename override.request_body (max_tokens only) to override.llm_options which always force-overwrites the client value - Restore per-protocol override.request_body deep-merge from PR apache#13251's predecessor, keyed by target protocol name (openai-chat, openai-responses, anthropic-messages, openai-embeddings) - Add override.request_body_force_override for request_body deep-merge - Precedence: model_options -> llm_options (always force) -> request_body (per-protocol deep merge) - Recreate merge.lua with force-aware deep_merge that always recurses into nested objects regardless of force flag - Restore protocols.names() for dynamic schema enum - Update en/zh docs for ai-proxy and ai-proxy-multi
dosubot
Bot
added
size:XL
There was a problem hiding this comment.
Pull request overview
Reintroduces per-target-protocol override.request_body deep-merge for ai-proxy/ai-proxy-multi while renaming the provider-aware max_tokens override mechanism to override.llm_options and documenting the new precedence/semantics.
Changes:
- Add
override.llm_options(provider-aware mapping; always force-overwrites) and restoreoverride.request_bodyas a per-target-protocol deep-merge map withrequest_body_force_override. - Introduce a shared
deep_mergehelper and apply overrides in the order:model_options→llm_options→request_body. - Update schemas, docs (EN/ZH), and add/adjust tests to cover the new configuration and behaviors.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| t/plugin/ai-proxy-request-body-override.t | Expands test coverage for llm_options mapping + per-protocol request_body deep-merge semantics and precedence. |
| docs/zh/latest/plugins/ai-proxy.md | Updates ZH docs for llm_options + per-protocol request_body override semantics and precedence. |
| docs/zh/latest/plugins/ai-proxy-multi.md | Updates ZH multi-instance docs to reflect llm_options + per-protocol request_body. |
| docs/en/latest/plugins/ai-proxy.md | Updates EN docs for renamed llm_options and restored per-protocol request_body override behavior. |
| docs/en/latest/plugins/ai-proxy-multi.md | Updates EN multi-instance docs to reflect llm_options + per-protocol request_body. |
| apisix/plugins/ai-proxy/schema.lua | Extends schema with override.llm_options and protocol-keyed override.request_body validation. |
| apisix/plugins/ai-proxy/merge.lua | Adds deep_merge helper used to apply per-protocol request body patches. |
| apisix/plugins/ai-proxy/base.lua | Passes new override fields (llm_options, request_body map, force flag) into provider request building. |
| apisix/plugins/ai-providers/base.lua | Applies llm_options via provider hook (forced) and applies per-protocol request_body via deep merge. |
| apisix/plugins/ai-protocols/init.lua | Restores protocols.names() to enumerate registered protocols for schema building. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
nic-6443
requested review from
AlinsRan,
Baoyuantop,
membphis and
shreemaan-abhishek
April 21, 2026 09:47
membphis
approved these changes
Apr 21, 2026
AlinsRan
approved these changes
Apr 22, 2026
shreemaan-abhishek
approved these changes
Apr 22, 2026
wistefan
pushed a commit
to wistefan/apisix
that referenced
this pull request
Jun 16, 2026
…_tokens mapping to llm_options (apache#13269)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR brings back the per-protocol
override.request_bodydeep-merge capability (from the predecessor of #13251) while keeping the provider-awaremax_tokensmapping — renamed tooverride.llm_options.Why
Users want both override mechanisms:
llm_options: Simple, provider-awaremax_tokensmapping (set one value, APISIX maps it to the right field per provider). Always force-overwrites.request_body: Fine-grained, per-protocol deep-merge for arbitrary fields liketemperature,stop_sequences,stream_options, etc.Changes
override.request_body→override.llm_options(always force-overwrites, no force flag)override.request_bodyas per-protocol deep-merge keyed by target protocol (openai-chat,openai-responses, etc.)override.request_body_force_override(boolean, controls request_body deep-merge only)model_options→llm_options→request_bodymerge.luawith force-awaredeep_mergeprotocols.names()for dynamic schema enumConfig example
{ "override": { "llm_options": { "max_tokens": 500 }, "request_body": { "openai-chat": { "temperature": 0.2, "stop": ["Human:"] }, "anthropic-messages": { "max_tokens": 500 } }, "request_body_force_override": false } }