Skip to content

Commit 1d599d9

Browse files
yangjj-isoredyyang
andauthored
[ISSUE #7383] Harden provider profiles and capability validation. (#7580)
* refactor(ai-proxy): harden provider profiles and capability validation. * fix(ai-proxy): enforce declared provider capabilities before adaptation --------- Co-authored-by: redyyang <[email protected]>
1 parent ad89a88 commit 1d599d9

16 files changed

Lines changed: 1256 additions & 89 deletions

File tree

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/README.md‎

Lines changed: 82 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,85 @@ generic; neither URLs nor model names are used to guess a dialect. DeepSeek's
9797
profile maps token-limit and reasoning/thinking fields and rejects incompatible
9898
thinking/tool configurations rather than silently changing their meaning.
9999

100+
## Provider profiles and capabilities
101+
102+
The explicit Provider registry remains the extension point. Built-in identifiers are
103+
`OpenAI`, `OpenAICompatible`, `DeepSeek`, `ALiYun`, `OpenAPI`, and `Moonshot`; separators
104+
and case are normalized by the registry. Both existing client protocols are supported.
105+
The protocol decoder also declares the request operation (`CHAT_COMPLETION` or
106+
`RESPONSE`), so a profile can restrict operations separately from protocol support.
107+
Responses clients continue to use conversion to a Chat upstream; these declarations
108+
do not enable native Responses endpoints or other HTTP operations.
109+
110+
`AiProviderCapabilities` declares streaming, function/custom tools, JSON mode, JSON
111+
schema output, stream usage and audio output. The default `supports()` follows the
112+
declared protocols. The engine validates decoded protocol, operation and effective
113+
stream settings before invoking any provider, including custom implementations.
114+
Providers also validate the final converted payload, including provider defaults. Unsupported
115+
features or operations fail as request errors before HTTP execution. Generic capabilities
116+
describe the service, not a guarantee for every model it hosts; configure a narrower
117+
profile for a model or deployment with additional restrictions. Unknown fields that
118+
need no transformation remain in the payload.
119+
120+
Built-in profiles enumerate their supported operations and features explicitly. Adding
121+
an enum value does not enable it for existing profiles; review and opt in each profile.
122+
123+
OpenAI now emits one Chat token-limit field, `max_completion_tokens`. When both token
124+
fields are supplied, the modern field takes precedence; gateway `maxTokens` overrides
125+
it. This follows the [OpenAI Chat API](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create).
126+
Generic compatible profiles preserve the client's field choice unless configured
127+
otherwise. `withMaxTokensField("max_tokens")` selects a legacy upstream dialect through
128+
Connect's provider adaptation phase, after Responses-to-Chat conversion. DeepSeek uses
129+
its existing Connect profile, including token and thinking mappings. Its Chat capability
130+
profile accepts function tools and JSON mode and rejects JSON-schema output, custom
131+
tools and audio output, following the [DeepSeek Chat API](https://api-docs.deepseek.com/api/create-chat-completion/).
132+
133+
Provider paths may be relative to a base URL, or the base URL may already include the
134+
complete configured operation path. `/v1` is not appended twice. DeepSeek uses
135+
`/chat/completions`, including when the configured base URL ends in `/v1` or `/beta`.
136+
Base URLs retain the existing HTTP(S), no-userinfo/query/fragment restriction; profile
137+
query parameters are encoded separately. Header credentials support arbitrary header
138+
names and prefixes (including Bearer); query credentials and unauthenticated local
139+
services are also available. Generated content-type, accept and credential headers
140+
replace conflicting profile headers case-insensitively.
141+
142+
Register another Chat-compatible service as a Spring bean without changing Engine,
143+
Protocol or Transport. For example:
144+
145+
```java
146+
@Bean
147+
public AiProxyProvider tenantProvider() {
148+
ObjectNode defaults = JsonNodeFactory.instance.objectNode();
149+
defaults.put("temperature", 0.7);
150+
OpenAiProviderOptions options = new OpenAiProviderOptions(
151+
"/generate", OpenAiProviderOptions.Authentication.HEADER, "api-key", "",
152+
Map.of("X-Tenant", "tenant-a"), Map.of("api-version", "2026-01"),
153+
Map.of("client-alias", "upstream-model"), defaults,
154+
AiProviderCapabilities.chat(), null, "max_tokens");
155+
return new OpenAiCompatible("tenant-provider", options);
156+
}
157+
```
158+
159+
Set the selector's `provider` to `tenant-provider`, its `baseUrl` to the service base
160+
URL and its `apiKey` to the credential. The profile also works with a collection-based
161+
`AiProxyProviderFactory` without Spring. Existing custom implementations keep their
162+
interface contract and the `OpenAi(ProviderProfile)` constructor remains available.
163+
164+
Model precedence is configured routed model, then client model, then profile default.
165+
Alias mapping applies to the selected model. Defaults fill only absent fields; explicit
166+
client values survive unless a gateway override is configured. Defaults cannot replace
167+
`messages`, `input`, `stream` or internal fallback configuration. Streaming selection
168+
therefore remains consistent with Engine retry/response policy. A Connect dialect
169+
that changes the selected streaming mode is rejected before HTTP execution. Profile maps and defaults
170+
are copied and per-request changes do not mutate the registered profile. Profile string
171+
representations omit headers, query values and body defaults.
172+
173+
Provider response differences remain in the request-bound Connect response adapter or
174+
the existing `AiProxyProvider.adaptResponse` hook. DeepSeek Chat response fields, including
175+
`reasoning_content` and vendor usage fields, remain passthrough for Chat clients; Connect
176+
converts reasoning and usage for Responses clients. Providers do not open connections,
177+
configure pools/timeouts, write client responses, or implement retry/fallback policy.
178+
100179
## Compatibility and limits
101180

102181
- Existing Chat routes retain their protocol identifier, upstream endpoint,
@@ -119,6 +198,7 @@ thinking/tool configurations rather than silently changing their meaning.
119198
be inferred from synthetic or missing usage values.
120199

121200
No database/configuration migration is required. Existing Chat routes need no
122-
change. Responses routes opt in through the selector handle; DeepSeek routes
123-
should review the stricter dialect validation. Roll back by reverting the
201+
change except for OpenAI token-field normalization described above. Responses routes
202+
opt in through the selector handle; DeepSeek routes should review the capability and
203+
dialect validation. Roll back by reverting the
124204
integration commit and restoring any opt-in selectors to `openai-chat`.

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/model/ShenyuAiRequest.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ public final class ShenyuAiRequest {
3333

3434
private String protocol;
3535

36+
private Operation operation = Operation.CHAT_COMPLETION;
37+
3638
private String model;
3739

3840
private Integer maxTokens;
@@ -68,6 +70,24 @@ public String getProtocol() {
6870
return protocol;
6971
}
7072

73+
/**
74+
* Get the operation established by the protocol decoder.
75+
*
76+
* @return requested operation
77+
*/
78+
public Operation getOperation() {
79+
return operation;
80+
}
81+
82+
/**
83+
* Set the operation established by the protocol decoder.
84+
*
85+
* @param operation requested operation
86+
*/
87+
public void setOperation(final Operation operation) {
88+
this.operation = operation;
89+
}
90+
7191
/**
7292
* Get requested model.
7393
*
@@ -175,4 +195,11 @@ public List<ShenyuAiMessage> getMessages() {
175195
public void setMessages(final List<ShenyuAiMessage> messages) {
176196
this.messages = messages;
177197
}
198+
199+
/**
200+
* Generation operations supported by the current protocol decoders.
201+
*/
202+
public enum Operation {
203+
CHAT_COMPLETION, RESPONSE
204+
}
178205
}

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/protocol/OpenAiChat.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ public ShenyuAiRequest decodeRequest(final String rawBody, final ObjectNode payl
7373
new ShenyuAiRequest();
7474

7575
request.setProtocol(NAME);
76+
request.setOperation(ShenyuAiRequest.Operation.CHAT_COMPLETION);
7677
request.setRawBody(rawBody);
7778
request.setRawPayload(payload.deepCopy());
7879
request.setModel(readOptionalText(

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/protocol/OpenAiResponses.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ public ShenyuAiRequest decodeRequest(final String rawBody, final ObjectNode payl
6565
}
6666
ShenyuAiRequest request = new ShenyuAiRequest();
6767
request.setProtocol(NAME);
68+
request.setOperation(ShenyuAiRequest.Operation.RESPONSE);
6869
request.setRawBody(rawBody);
6970
request.setRawPayload(payload.deepCopy());
7071
request.setModel(Objects.isNull(model) || model.isNull() ? null : model.textValue());
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.shenyu.plugin.ai.proxy.enhanced.provider;
19+
20+
import com.fasterxml.jackson.databind.JsonNode;
21+
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.AiProxyException;
22+
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.ShenyuAiRequest;
23+
import org.apache.shenyu.plugin.ai.proxy.enhanced.protocol.OpenAiChat;
24+
import org.apache.shenyu.plugin.ai.proxy.enhanced.protocol.OpenAiResponses;
25+
26+
import java.util.EnumSet;
27+
import java.util.Objects;
28+
import java.util.Set;
29+
30+
/**
31+
* Declared provider capabilities, independent of HTTP execution and model discovery.
32+
*
33+
* @param protocols supported client protocols
34+
* @param operations supported AI operations
35+
* @param features supported request features
36+
*/
37+
public record AiProviderCapabilities(Set<String> protocols, Set<ShenyuAiRequest.Operation> operations, Set<Feature> features) {
38+
39+
/**
40+
* Copy declarations so request handling never mutates provider configuration.
41+
*/
42+
public AiProviderCapabilities {
43+
protocols = Set.copyOf(protocols);
44+
operations = Set.copyOf(operations);
45+
features = Set.copyOf(features);
46+
}
47+
48+
/**
49+
* General Chat upstream capabilities, including conversion from Responses clients.
50+
*
51+
* @return general capabilities
52+
*/
53+
public static AiProviderCapabilities chat() {
54+
return new AiProviderCapabilities(Set.of(OpenAiChat.NAME, OpenAiResponses.NAME),
55+
EnumSet.of(ShenyuAiRequest.Operation.CHAT_COMPLETION, ShenyuAiRequest.Operation.RESPONSE),
56+
EnumSet.of(Feature.STREAMING, Feature.TOOL_CALLING, Feature.CUSTOM_TOOLS, Feature.JSON_MODE,
57+
Feature.JSON_SCHEMA, Feature.STREAM_USAGE, Feature.AUDIO_OUTPUT));
58+
}
59+
60+
/**
61+
* Validate the decoded request and effective gateway streaming setting.
62+
*
63+
* @param request decoded request
64+
* @param stream effective streaming setting
65+
*/
66+
public void validate(final ShenyuAiRequest request, final boolean stream) {
67+
if (!protocols.contains(request.getProtocol())) {
68+
reject("protocol " + request.getProtocol());
69+
}
70+
if (Objects.isNull(request.getOperation()) || !operations.contains(request.getOperation())) {
71+
reject("operation " + request.getOperation());
72+
}
73+
if (stream) {
74+
require(Feature.STREAMING);
75+
}
76+
}
77+
78+
/**
79+
* Validate features in the final Chat payload, after conversion and defaults.
80+
*
81+
* @param payload final provider payload
82+
*/
83+
public void validatePayload(final JsonNode payload) {
84+
if (nonempty(payload.get("tools")) || nonempty(payload.get("functions"))) {
85+
require(Feature.TOOL_CALLING);
86+
}
87+
for (String field : new String[] {"tool_choice", "function_call"}) {
88+
JsonNode choice = payload.get(field);
89+
if (Objects.nonNull(choice) && !choice.isNull() && !(choice.isTextual() && "none".equals(choice.asText()))) {
90+
require(Feature.TOOL_CALLING);
91+
}
92+
}
93+
JsonNode tools = payload.get("tools");
94+
if (Objects.nonNull(tools) && tools.isArray()) {
95+
for (JsonNode tool : tools) {
96+
if ("custom".equals(tool.path("type").asText())) {
97+
require(Feature.CUSTOM_TOOLS);
98+
}
99+
}
100+
}
101+
String format = payload.path("response_format").path("type").asText();
102+
if ("json_object".equals(format)) {
103+
require(Feature.JSON_MODE);
104+
} else if ("json_schema".equals(format)) {
105+
require(Feature.JSON_SCHEMA);
106+
}
107+
if (payload.path("stream_options").path("include_usage").asBoolean()) {
108+
require(Feature.STREAM_USAGE);
109+
}
110+
JsonNode modalities = payload.get("modalities");
111+
if (Objects.nonNull(modalities) && modalities.isArray()) {
112+
for (JsonNode modality : modalities) {
113+
if ("audio".equals(modality.asText())) {
114+
require(Feature.AUDIO_OUTPUT);
115+
}
116+
}
117+
}
118+
}
119+
120+
private boolean nonempty(final JsonNode node) {
121+
return Objects.nonNull(node) && !node.isNull() && !node.isEmpty();
122+
}
123+
124+
private void require(final Feature feature) {
125+
if (!features.contains(feature)) {
126+
reject("feature " + feature);
127+
}
128+
}
129+
130+
private void reject(final String capability) {
131+
throw new AiProxyException(AiProxyException.Kind.REQUEST, "AI provider does not support " + capability, null);
132+
}
133+
134+
/**
135+
* Features checked when ShenYu constructs the final upstream request.
136+
*/
137+
public enum Feature {
138+
STREAMING, TOOL_CALLING, CUSTOM_TOOLS, JSON_MODE, JSON_SCHEMA, STREAM_USAGE, AUDIO_OUTPUT
139+
}
140+
}

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/provider/AiProxyProvider.java‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.AiUpstreamResponse;
2323
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.ShenyuAiRequest;
2424

25+
import java.util.Objects;
26+
2527
/**
2628
* Contract for adapting a ShenYu AI request to a provider request.
2729
*/
@@ -43,14 +45,24 @@ public interface AiProxyProvider {
4345
*/
4446
AiUpstreamRequest buildRequest(ShenyuAiRequest request, AiCommonConfig config);
4547

48+
/**
49+
* Get provider feature and operation declarations.
50+
* Existing custom providers retain the general Chat capability default.
51+
*
52+
* @return provider capabilities
53+
*/
54+
default AiProviderCapabilities capabilities() {
55+
return AiProviderCapabilities.chat();
56+
}
57+
4658
/**
4759
* Check whether the provider supports a client protocol.
4860
*
4961
* @param protocol client protocol identifier
5062
* @return whether supported
5163
*/
5264
default boolean supports(final String protocol) {
53-
return true;
65+
return Objects.nonNull(protocol) && capabilities().protocols().contains(protocol);
5466
}
5567

5668
/**

0 commit comments

Comments
 (0)