Skip to content

Commit 83866af

Browse files
committed
fix(ai-proxy): address structured-output review findings (#8257)
- repair after a fallback re-asks the upstream that produced the failing response (active config) instead of always the primary - a repaired tool-call answer is forwarded like a primary one - validated answers drop the stale Content-Length/Content-Encoding framing - legacy functions/function_call requests also disable repair - compiled validators are cached by schema text (no per-request event-loop compilation); schema budget exhaustion is its own non-repairable outcome - repair generations default to a 1024 token bound; nested config copies are deep-copied; shared body materialization; readability and visibility fixes
1 parent c156eab commit 83866af

12 files changed

Lines changed: 333 additions & 48 deletions

File tree

‎shenyu-common/src/main/java/org/apache/shenyu/common/dto/convert/rule/AiStructuredOutputConfig.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,13 @@ public class AiStructuredOutputConfig implements Serializable {
3636
*/
3737
public static final int MAX_REPAIR_ATTEMPTS_LIMIT = 3;
3838

39+
/**
40+
* Conservative default token bound of one repair generation (issue #8257
41+
* requires an explicit conservative default); an operator raises it by
42+
* configuring {@code repairMaxTokens} explicitly.
43+
*/
44+
public static final int REPAIR_DEFAULT_MAX_TOKENS = 1024;
45+
3946
private static final long serialVersionUID = -4216685160670534827L;
4047

4148
/**
@@ -49,7 +56,7 @@ public class AiStructuredOutputConfig implements Serializable {
4956

5057
private Integer repairMaxAttempts = 1;
5158

52-
private Integer repairMaxTokens;
59+
private Integer repairMaxTokens = REPAIR_DEFAULT_MAX_TOKENS;
5360

5461
public String getSchema() {
5562
return schema;

‎shenyu-plugin/shenyu-plugin-ai/AI_STRUCTURED_OUTPUT.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ existing configuration pipeline:
1919
| `repairEnabled` | whether repairable failures regenerate | `false` (reject only) |
2020
| `repairPrompt` | the operator's repair instruction; required when repair is enabled | — |
2121
| `repairMaxAttempts` | bounded regeneration budget | `1`, hard-capped at `3` |
22-
| `repairMaxTokens` | optional upper bound applied to `max_tokens` of a repair generation | unset |
22+
| `repairMaxTokens` | upper bound applied to `max_tokens` of a repair generation (the provider dialect may rename the field, e.g. `max_completion_tokens`) | `1024` |
2323

2424
Misconfiguration fails fast (unsupported schema, repair without prompt, out-of-range
2525
budget) — nothing silently degrades. Without a schema the response bytes are
@@ -38,6 +38,7 @@ upstream attempt.
3838
| TOOL_CALLS_PRESENT | `finish_reason: tool_calls` or message `tool_calls` | forwarded untouched — the client's workflow decision | never |
3939
| INVALID_JSON | content field present but not parseable | REJECT (HTTP 502, reason `CONTENT`) | yes |
4040
| SCHEMA_MISMATCH | bounded validator rejects the content | REJECT (HTTP 502) | yes |
41+
| SCHEMA_BUDGET_EXCEEDED | content exceeds the bounded validator's size/complexity budget — it may be perfectly valid, just too large to validate | REJECT (HTTP 502) | no — a regeneration of a merely oversized document wastes billable work |
4142
| MALFORMED_ENVELOPE | outer response is not a parseable chat completion | REJECT (HTTP 502) | no |
4243
| EMPTY | content absent or blank | REJECT (HTTP 502) | no |
4344
| TRUNCATED | `finish_reason: length` | REJECT (HTTP 502) | no |
@@ -52,8 +53,17 @@ A repairable failure regenerates by re-preparing the original request with one
5253
appended user message: the operator's `repairPrompt` (the only instruction source),
5354
the authoritative schema, the bounded classification evidence, and the previous
5455
output as clearly labeled **data** — model text cannot inject repair instructions.
55-
`stream` is forced off, model/provider references are preserved, and an optional
56-
`repairMaxTokens` clamps `max_tokens`.
56+
`stream` is forced off, model/provider references are preserved, and
57+
`repairMaxTokens` clamps the generation's token bound. The repair re-asks the
58+
upstream that produced the failing response (primary after a primary failure,
59+
the fallback after a fallback failure). Tool-bearing traffic — the `tools` array
60+
as well as the legacy `functions`/`function_call` shapes — never enters repair.
61+
62+
Refusal detection is deliberately protocol-level only (`finish_reason:
63+
content_filter`): a refusal delivered as plain text with `finish_reason: stop`
64+
is a semantic judgement owned by the semantic-safety boundary (#7623), not
65+
something the gateway guesses — such traffic is classified like any other
66+
content, never specially repaired.
5767

5868
Every actual regeneration is a correlated billable attempt: it has its own
5969
`Phase.REPAIR` attempt identity (`AiAttemptEvent.Kind.REPAIR`), its own usage tap,

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-common/src/main/java/org/apache/shenyu/plugin/ai/common/config/AiCommonConfig.java‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,24 @@ public AiCommonConfig(final AiCommonConfig other) {
109109
this.protocol = other.getProtocol();
110110
this.requestBodyOverrides = copyRequestBodyOverrides(other.getRequestBodyOverrides());
111111
this.requestBodyForceOverride = other.getRequestBodyForceOverride();
112-
this.structuredOutput = other.getStructuredOutput();
112+
this.structuredOutput = copyStructuredOutput(other.getStructuredOutput());
113+
}
114+
115+
/**
116+
* Defensive copy of the nested structured-output policy: the DTO is Gson-parsed
117+
* and mutable, so a config copy must never share the nested instance.
118+
*/
119+
private static AiStructuredOutputConfig copyStructuredOutput(final AiStructuredOutputConfig config) {
120+
if (Objects.isNull(config)) {
121+
return null;
122+
}
123+
AiStructuredOutputConfig copy = new AiStructuredOutputConfig();
124+
copy.setSchema(config.getSchema());
125+
copy.setRepairEnabled(config.getRepairEnabled());
126+
copy.setRepairPrompt(config.getRepairPrompt());
127+
copy.setRepairMaxAttempts(config.getRepairMaxAttempts());
128+
copy.setRepairMaxTokens(config.getRepairMaxTokens());
129+
return copy;
113130
}
114131

115132
/**

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/conversion/ConnectResponsesToChatConverter.java‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,14 +33,14 @@
3333
import io.github.agenticai.connect.StreamOptions;
3434
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.AiProxyException;
3535
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.AiUpstreamResponse;
36+
import org.apache.shenyu.plugin.ai.proxy.enhanced.service.AiProxyBodies;
3637
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.ShenyuAiRequest;
3738
import org.apache.shenyu.plugin.ai.proxy.enhanced.protocol.OpenAiChat;
3839
import org.apache.shenyu.plugin.ai.proxy.enhanced.protocol.OpenAiResponses;
3940
import org.slf4j.Logger;
4041
import org.slf4j.LoggerFactory;
4142
import reactor.core.publisher.Flux;
4243

43-
import java.io.ByteArrayOutputStream;
4444
import java.nio.charset.StandardCharsets;
4545
import java.util.LinkedHashMap;
4646
import java.util.List;
@@ -122,14 +122,9 @@ private AiUpstreamResponse convertResponse(final AiUpstreamResponse response, fi
122122
return new AiUpstreamResponse(response.getStatusCode(), responseHeaders(response, "text/event-stream"),
123123
convertStream(framed.getBody(), context));
124124
}
125-
Flux<byte[]> body = response.getBody().reduceWith(ByteArrayOutputStream::new, (buffer, bytes) -> {
126-
if (bytes.length > MAX_CONVERSION_BYTES - buffer.size()) {
127-
throw new AiProxyException(AiProxyException.Kind.PROTOCOL,
128-
"AI response exceeds the conversion size limit", null);
129-
}
130-
buffer.writeBytes(bytes);
131-
return buffer;
132-
}).map(buffer -> convertCompleteResponse(buffer.toString(StandardCharsets.UTF_8), context)).flux();
125+
Flux<byte[]> body = AiProxyBodies.materialize(response.getBody(), () -> new AiProxyException(
126+
AiProxyException.Kind.PROTOCOL, "AI response exceeds the conversion size limit", null))
127+
.map(buffer -> convertCompleteResponse(buffer, context)).flux();
133128
return new AiUpstreamResponse(response.getStatusCode(), responseHeaders(response, "application/json"), body);
134129
}
135130

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/model/AiContentCompliance.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,13 @@ public enum AiContentCompliance {
4545
*/
4646
SCHEMA_MISMATCH,
4747

48+
/**
49+
* The generated content exceeded the bounded validator's size or complexity
50+
* budget; it may be perfectly valid, just too large to validate, so it is
51+
* rejected without a repair regeneration.
52+
*/
53+
SCHEMA_BUDGET_EXCEEDED,
54+
4855
/**
4956
* The generated content field was absent or blank.
5057
*/

‎shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-proxy/src/main/java/org/apache/shenyu/plugin/ai/proxy/enhanced/service/AiGeneratedContentInspector.java‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
import com.fasterxml.jackson.databind.JsonNode;
2323
import com.fasterxml.jackson.databind.ObjectMapper;
2424
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.AiContentCompliance;
25+
import org.apache.shenyu.plugin.base.schema.BoundedJsonSchemaValidator;
2526

2627
import java.util.Objects;
2728

@@ -39,13 +40,13 @@ public final class AiGeneratedContentInspector {
3940
* Maximum characters of prior model output carried into a repair request as
4041
* data; longer outputs are represented by their prefix.
4142
*/
42-
public static final int MAX_PREVIOUS_OUTPUT_CHARS = 2048;
43+
private static final int MAX_PREVIOUS_OUTPUT_CHARS = 2048;
4344

4445
/**
4546
* Maximum characters of the classification summary carried into a repair
4647
* request and failure messages.
4748
*/
48-
public static final int MAX_SUMMARY_CHARS = 512;
49+
private static final int MAX_SUMMARY_CHARS = 512;
4950

5051
private static final ObjectMapper MAPPER = new ObjectMapper().enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS);
5152

@@ -87,7 +88,7 @@ public static AiContentInspection classify(final AiStructuredOutputPolicy policy
8788
return new AiContentInspection(AiContentCompliance.TRUNCATED, "upstream declared finish_reason length", null);
8889
}
8990
JsonNode content = message.path("content");
90-
if (content.isMissingNode() || content.isNull() || content.isTextual() && content.asText().isBlank()) {
91+
if (content.isMissingNode() || content.isNull() || (content.isTextual() && content.asText().isBlank())) {
9192
return new AiContentInspection(AiContentCompliance.EMPTY, "generated content is absent or blank", null);
9293
}
9394
if (!content.isTextual()) {
@@ -104,6 +105,12 @@ public static AiContentInspection classify(final AiStructuredOutputPolicy policy
104105
}
105106
try {
106107
policy.validator().validateValue(generatedJson);
108+
} catch (BoundedJsonSchemaValidator.SchemaBudgetExceededException error) {
109+
// an oversized document may be perfectly valid: rejecting it is fail-closed,
110+
// but regenerating it can only waste billable work
111+
return new AiContentInspection(AiContentCompliance.SCHEMA_BUDGET_EXCEEDED,
112+
bounded("generated content exceeds the bounded validation budget: " + error.getMessage()),
113+
boundedOutput(generated));
107114
} catch (IllegalArgumentException error) {
108115
return new AiContentInspection(AiContentCompliance.SCHEMA_MISMATCH,
109116
bounded("generated content violates the operator schema: " + error.getMessage()),
@@ -118,7 +125,7 @@ public static AiContentInspection classify(final AiStructuredOutputPolicy policy
118125
* @param generated the prior generated content
119126
* @return the bounded prefix
120127
*/
121-
public static String boundedOutput(final String generated) {
128+
private static String boundedOutput(final String generated) {
122129
if (Objects.isNull(generated) || generated.length() <= MAX_PREVIOUS_OUTPUT_CHARS) {
123130
return generated;
124131
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.shenyu.plugin.ai.proxy.enhanced.service;
19+
20+
import org.apache.shenyu.plugin.ai.proxy.enhanced.model.AiProxyException;
21+
import reactor.core.publisher.Flux;
22+
import reactor.core.publisher.Mono;
23+
24+
import java.io.ByteArrayOutputStream;
25+
import java.nio.charset.StandardCharsets;
26+
import java.util.function.Supplier;
27+
28+
/**
29+
* Shared bounded materialization of a full upstream body (issue #8257 review:
30+
* one limit, one accumulation, one overflow contract instead of duplicated
31+
* per-caller copies).
32+
*/
33+
public final class AiProxyBodies {
34+
35+
/**
36+
* Maximum number of bytes an AI response body may occupy when a stage needs
37+
* the full body in memory; the validation and the protocol conversion share it.
38+
*/
39+
public static final int MAX_BODY_BYTES = 5 * 1024 * 1024;
40+
41+
private AiProxyBodies() {
42+
}
43+
44+
/**
45+
* Materialize a full body into a UTF-8 string, bounded by {@link #MAX_BODY_BYTES}.
46+
*
47+
* @param body the response body flux, consumed exactly once
48+
* @param overflowError the classified failure to raise when the limit is exceeded
49+
* @return the materialized body
50+
*/
51+
public static Mono<String> materialize(final Flux<byte[]> body, final Supplier<AiProxyException> overflowError) {
52+
return body.reduceWith(ByteArrayOutputStream::new, (buffer, bytes) -> {
53+
if (bytes.length > MAX_BODY_BYTES - buffer.size()) {
54+
throw overflowError.get();
55+
}
56+
buffer.writeBytes(bytes);
57+
return buffer;
58+
}).map(buffer -> buffer.toString(StandardCharsets.UTF_8));
59+
}
60+
}

0 commit comments

Comments
 (0)