Repository navigation
[Feature][AI] Support Anthropic Messages and Gemini payloads in AI governance plugins #8458
Description
Feature Request
Make AI governance plugins understand Anthropic Messages and Gemini payloads: usage metering, token limiting, PII and the normalized request view. Then they keep working when those protocols are routed, natively or through conversion.
Is your feature request related to a problem? Please describe
On master @ 9dd9a12, the request and response inspection in the AI plugins only covers OpenAI Chat and OpenAI Responses:
| Component | Location | Current coverage |
|---|---|---|
| Usage metering | shenyu-plugin-ai-proxy/.../service/AiUpstreamUsageTap.java L196-L207 |
Reads usage.prompt_tokens / completion_tokens / total_tokens (OpenAI Chat). Responses usage is handled in OpenAiResponses. |
| Token limiter | shenyu-plugin-ai-token-limiter/.../AiTokenLimiterPlugin.java usageOf / outputTokensOf (L306-L328) |
Looks for a root usage, or response.usage for Responses terminal events, and reads completion_tokens or output_tokens. Anthropic stream input tokens are in message_start.message.usage, so they are missed. Gemini usageMetadata (promptTokenCount, candidatesTokenCount, totalTokenCount) is not read. |
| PII | shenyu-plugin-ai-pii/.../profile/PiiProtocolProfiles.java |
Only OpenAiChatProfile and OpenAiResponsesProfile. |
| Normalized request view (prompt, request transformer and other view consumers) | shenyu-plugin-ai-common/.../view/AiRequestViewParser.java L76-L90 |
messages maps to chat. input maps to Responses, structural only ("deep coverage is a follow-up"). An Anthropic body is treated as Chat. Gemini contents is not recognized. |
The sensitive-word plugin is not affected. JsonTextValues walks every string member of the JSON body, so it does not depend on the protocol shape.
So when Anthropic or Gemini traffic flows through the gateway, token limits and usage accounting under-count or skip it, and PII masking does not apply. That traffic can come from the native providers in #8388 / #8389, the protocol conversion proposed in #8457, or the passthrough mode in #8454. Nothing reports the gap.
Describe the solution you'd like
-
Usage extraction. Add Anthropic and Gemini usage parsing to the shared usage path ([Task] Integrate shared per-attempt LLM usage metering across supported protocols #7955 / [Task][AI Proxy] Integrate AI Proxy attempt lifecycle with shared usage and invocation telemetry #8125) and to the token limiter, covering JSON and SSE:
- Anthropic:
usage.input_tokens/output_tokens. In streams, input usage comes frommessage_start.message.usageand cumulative output usage frommessage_delta.usage. - Gemini:
usageMetadata.promptTokenCount/candidatesTokenCount/totalTokenCount. In streams it appears in the final chunk and may be cumulative.
Take care not to sum cumulative values or invent zeros.
- Anthropic:
-
PII profiles. Add profiles for Anthropic Messages (
system,messages[].contenttext blocks,tool_useinput) and Gemini (systemInstruction,contents[].parts[].text,functionCall.args), with response-side handling where PII masks responses. -
Request view. Have
AiRequestViewParserproduce bounded views for both protocols, selecting the protocol from the shared detector of [Task] Unify AI protocol detection and support ai-prompt for more protocols #8371 / feat(ai): unify AI protocol detection and support ai-prompt for openai-response #8409 instead of the body shape. -
Explicit coverage. When a plugin cannot inspect the detected protocol, report it as unsupported through observations or logs. Do not silently pass the request through as covered. Publish a plugin × protocol coverage matrix in the docs.
Describe alternatives you've considered
- Converting everything to OpenAI Chat before governance. This works for the conversion path, but not for native or passthrough traffic, and accounting should reflect the actual upstream usage.
Additional context
Related: #7955, #8125 (shared usage metering), #8371 / #8409 (protocol detection), #8388, #8389 (native providers), #8454 (passthrough), #8457 (Anthropic/Gemini converters).
Activity
konlue commentedon Nov 14, 2026
/assign
konlue commentedon Nov 14, 2026
Taking this — the governance read paths are ours to extend, and the plugin set is small enough to land as one coherent change. Proposed approach:
Naming:
anthropic-messagesandgemini-generate-content, aligned with the existingopenai-chat/openai-responsesconvention (AiProxyProtocol.name(),AiCommonConfig.protocol,PiiProtocolProfile.id(), viewprotocolhint all reuse the same strings).detect(JsonNode)classifyingmessages+top-levelsystem→ anthropic,messages→ openai-chat,input→ openai-responses,contents→ gemini; the view parser and PII selection both delegate to it instead of their own ad-hoc shape checks (per the issue's shared-detector direction).messagesbranch (protocolanthropic-messages); the top-levelsystem(string or text blocks) surfaces as a leading systemBoundedMessage— zero contract change toAiNormalizedRequestView. Gemini branch oncontents(+systemInstruction.parts) →gemini-generate-content.AiUpstreamUsageTap): thenativeChatctor flag becomes the target protocol string;capture()resolves usage fields by shape chain (openai → anthropic → gemini — the field names are mutually exclusive): Anthropicinput_tokens→prompt /output_tokens→completion with total left null (never invented); GeminiusageMetadata.{promptTokenCount,candidatesTokenCount,totalTokenCount}; snapshot-replacement semantics unchanged (never summing cumulative stream counters). Stream-completion evidence extends to Anthropic viamessage_stop(no[DONE]there); Gemini streams carry no terminal marker and keep today's evidence behavior.usageOflearns Anthropic stream events (message_start.message.usage,message_delta.usage— the latter's root usage already matches the existing branch) and GeminiusageMetadata;outputTokensOfgainscandidatesTokenCount. Last-event-wins matches the cumulative counters. Note Anthropic non-streaming already works by field-name coincidence — covered by a regression test.AnthropicMessagesProfile(top-levelsystemstring/blocks,messages[].contentstring/text-blocks,tool_use.inputstring leaves via a bounded walker; responsecontent[]blocks) andGeminiGenerateContentProfile(systemInstruction/contents[].parts[].text; responsecandidates[].content.parts[].text). Anthropic selection is checked before the generic messages→CHAT shape sosystemis not silently dropped.Scope notes: no converter registration (#8457), no native provider transports (#8388/#8389), no passthrough mode (#8454); sensitive-word is unaffected (
JsonTextValueswalks every string regardless of shape). A plugin × protocol coverage matrix lands in the plugin docs, including the explicit-gap reporting that already exists (PII null-profile, viewunsupported, usageUNKNOWN).[ISSUE apache#8458] Drop the unused input-tokens constant