feat(ai-proxy): support aws bedrock#13249
Conversation
Signed-off-by: Abhishek Choudhary <[email protected]>
Rename double_encode_path to encode_path_for_canonical_uri since the function only does a single ngx.escape_uri pass. The double-encoding required by AWS SigV4 is achieved by combining this pass with the upstream encoding done by bedrock.lua on the model ID. Update the function comment to accurately describe this two-stage flow.
construct_forward_headers() in http.lua lowercases all header keys, so writing signed headers (Authorization, X-Amz-Date, X-Amz-Security-Token, Host) in mixed case could result in duplicate headers depending on the HTTP client's behavior. Write them back as lowercase to match the convention used elsewhere.
Add an explicit nil/empty check on params.path at the start of sign_request() to return a clear error instead of crashing in encode_path_for_canonical_uri() with "attempt to index a nil value". This can happen when the Bedrock provider's path capability returns nil (e.g., when options.model is missing).
Bedrock Converse API only accepts 'user' and 'assistant' roles in
messages[*].role. System prompts must go in the top-level body.system
field as an array of {text: "..."} blocks.
Update prepend_messages and append_messages to split incoming messages
by role: system entries are routed to body.system as text blocks, while
user/assistant entries go to body.messages as before. Without this,
prompt-decorator plugins inserting canonical system messages would
produce invalid Bedrock requests.
Bedrock builds the upstream URL path from the model ID. Without options.model and without override.endpoint, the path resolves to nil and the request fails later with a low-signal transport error. Add a schema-level rule to require options.model when provider=bedrock and override.endpoint is not set, so misconfigurations fail fast at config validation time.
TEST 2-4 depend on the route created in TEST 1, and TEST 7 depends on the route created in TEST 6. Add no_shuffle() so tests run in declared order and these dependencies are honored consistently.
Switch oneOf to anyOf in the conditional schemas for vertex-ai and bedrock. With oneOf, users couldn't set both provider_conf and override.endpoint together. This blocked valid use cases like AWS PrivateLink (custom endpoint) combined with explicit signing region, since the code does not parse the region from the override URL. anyOf still requires at least one of the two to be set.
… endpoint Add test scenarios covering edge cases introduced by the Bedrock provider: - TEST 8/9: model as inference profile ARN (validates URL encoding of ":" and "/" in the path and SigV4 canonical URI handling) - TEST 10/11: auth.aws.session_token propagation as x-amz-security-token header to the upstream - TEST 12: route with provider_conf.region but no override.endpoint (validates schema accepts default endpoint construction) Mock server appends session_token to the response body so TEST 11 can assert propagation via response_body regex.
Move all SigV4 path encoding into auth-aws.lua so callers don't need to know about the AWS double-encoding rule. The previous design split encoding between bedrock.lua (pre-encoded the model) and auth-aws.lua (added the second encoding pass), which produced an incorrect canonical URI when params.path came from a raw source like override.endpoint. Now bedrock.lua passes the raw model in the path, and auth-aws.lua normalizes the input (decode-then-encode is idempotent for both raw and once-encoded inputs) and produces: - params.path: once-encoded for HTTP wire transport - canonicalURI: twice-encoded for SigV4 canonical request This handles both default-endpoint and override.endpoint paths correctly regardless of whether the input is raw or pre-encoded.
There was a problem hiding this comment.
Pull request overview
Adds Amazon Bedrock support to the ai-proxy / ai-proxy-multi plugin by introducing a Bedrock provider implementation, Bedrock Converse protocol handling, and AWS SigV4 request signing.
Changes:
- Introduces a new
bedrockprovider +bedrock-converseprotocol adapter (detection, request shaping, response parsing). - Adds AWS SigV4 signing support (incl. session token propagation) and updates HTTP transport to accept pre-serialized bodies.
- Extends ai-proxy schemas and log sanitization, plus adds an end-to-end Bedrock-focused test suite.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
t/plugin/ai-proxy-bedrock.t |
New test coverage for Bedrock Converse routing, schema validation, ARN model handling, and session token propagation. |
apisix/utils/log-sanitize.lua |
Redacts x-amz-security-token in logged headers. |
apisix/plugins/ai-transport/http.lua |
Allows params.body to be a pre-serialized string (needed after SigV4 signing). |
apisix/plugins/ai-transport/auth-aws.lua |
New AWS SigV4 signing helper for outbound AI provider requests. |
apisix/plugins/ai-proxy/schema.lua |
Adds AWS auth schema + encrypt_fields updates; adds Bedrock provider_conf schema and conditional validation. |
apisix/plugins/ai-providers/schema.lua |
Registers bedrock as a known provider. |
apisix/plugins/ai-providers/bedrock.lua |
New Bedrock provider implementation (host/path generation; remove model from body). |
apisix/plugins/ai-providers/base.lua |
Hooks AWS SigV4 signing into the shared request build pipeline. |
apisix/plugins/ai-protocols/init.lua |
Registers bedrock-converse and adjusts protocol detection order. |
apisix/plugins/ai-protocols/bedrock-converse.lua |
New Bedrock Converse protocol adapter (non-streaming). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Previously the override schema didn't require any fields, so a config
like override = {} would pass validation despite being meaningless.
This was particularly problematic because the schema's anyOf accepted
"override" as an alternative to "provider_conf" — letting an empty
override silently bypass the provider_conf requirement and break
downstream request construction.
Make endpoint required in the override schema since it's the only
field and the only purpose of override is to set the endpoint.
Add minLength = 1 to auth.aws.access_key_id, secret_access_key, session_token, and provider_conf.region for the Bedrock provider. The 'required' constraint only checks key presence; without minLength, empty strings pass schema validation and fail later during signing or upstream URL construction. Catch these as admin-side validation errors instead.
Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
…nsert prepend_messages and append_messages only checked for nil before calling core.table.insert(body.system/messages, ...), which would crash if those fields existed but weren't tables (e.g., a string). Use type(...) ~= "table" to normalize, preventing request-time exceptions on malformed input.
Bedrock provider falls back to AWS_REGION env var (or us-east-1 default) when provider_conf.region is not set, so the schema should not require provider_conf or override. The inner rule still requires options.model when override.endpoint is not set, ensuring a valid upstream URL can always be constructed.
The previous code passed n=1 to normalize_and_encode_path for the canonical URI, but n=1 means decode-then-encode-once, which cancels out and produces the same once-encoded form as params.path. AWS SigV4 requires reserved characters to be encoded TWICE in the canonical URI (e.g., raw ":" → "%3A" on the wire → "%253A" in canonicalURI). Pass n=2 so the function decodes each segment then encodes it twice, producing the required double-encoded form. Tests pass because the local mock doesn't validate signatures, but real AWS requests would have failed with signature mismatch.
Application inference profile ARNs contain "/" (e.g.
"...:application-inference-profile/abc") which would be split as path
segments if interpolated raw into "/model/{id}/converse". Encode the
model with ngx.escape_uri so "/" becomes "%2F" and ":" becomes "%3A",
keeping the model as a single path segment.
This is safe to combine with auth-aws.lua's normalize_and_encode_path
since that function decodes-then-encodes (idempotent for both raw and
once-encoded inputs).
Update the override.endpoint description to: - Recommend host-only overrides for the typical use case (PrivateLink, reverse proxies) so the plugin computes the path with proper encoding - Document that any reserved characters in an included path (e.g., Bedrock inference profile ARNs with ':' or '/') must be URL-encoded
- Fix test URIs to end with /converse so bedrock-converse protocol detection matches (previous URIs like /ai/converse-arn don't end with /converse and would fall through to openai-chat) - Log the request URI in the mock so tests can assert specific paths - Update TEST 8 override.endpoint to use a properly URL-encoded ARN (matching what users should provide per the schema documentation) - Add an error_log assertion in TEST 9 verifying the wire path keeps the ARN as a single percent-encoded segment (%3A and %2F preserved)
The mock server extracted text from the first user message into first_content but never used it. Remove the dead code to make the mock handler clearer.
Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
Remove the AWS_REGION env var fallback and "us-east-1" default for
the Bedrock provider's region. Silently defaulting to us-east-1 was a
footgun — requests would route to the wrong region and fail SigV4
signature validation with confusing errors.
Now provider_conf.region must be set explicitly for Bedrock at the
schema level. The runtime in base.lua also returns a clear error
("missing region for AWS SigV4 signing (provider_conf.region required
for bedrock)") as a defense-in-depth check.
Also remove the hanging schema comment that explained the (now
removed) fallback behavior. Update docs (en + zh) to remove env var /
default mentions and clarify provider_conf.region is required.
Tests updated to set provider_conf.region in all bedrock instance
configs; AWS_REGION env var dropped from the test main_config.
Validate the structure of the SigV4 Authorization and X-Amz-Date headers instead of just checking presence. A broken signer would now fail tests instead of silently passing. Checks: - Authorization starts with "AWS4-HMAC-SHA256 " - Credential matches AKIAIOSFODNN7EXAMPLE/<DATE>/us-east-1/bedrock/aws4_request - SignedHeaders=... is present - Signature= is followed by exactly 64 hex chars - X-Amz-Date matches YYYYMMDDTHHMMSSZ format
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Abhishek Choudhary <[email protected]>
Co-authored-by: Copilot <[email protected]>
…edrock Signed-off-by: Abhishek Choudhary <[email protected]>
…apisix into feat-support-bedrock Signed-off-by: Abhishek Choudhary <[email protected]>
Implement the rewrite_request_body capability hook for the bedrock provider so override.request_body.max_tokens is mapped to Bedrock's native field (inferenceConfig.maxTokens). This brings Bedrock to parity with other providers (openai, deepseek, gemini, vertex-ai, etc.) that already implement this hook. Honors request_body_force_override: when false (default), respects client-supplied inferenceConfig.maxTokens; when true, overwrites it.
The previous comment "Remove fields Bedrock doesn't accept" was misleading — Bedrock DOES accept stream, it just uses a separate /converse-stream endpoint with the AWS EventStream binary protocol. Replace with a TODO that explains we strip the field because we don't yet implement that endpoint.
…to Lua
The schema's allOf/if/then/not blocks for vertex-ai and bedrock
provider requirements were hard to read and produced unhelpful error
messages ("allOf 2 failed: then clause did not match"). Move the
conditional logic to a Lua function in schema.lua and call it from
each plugin's check_schema.
- ai_instance_schema and ai_proxy_schema now have provider_conf as a
permissive top-level property (no provider-specific required fields)
- New schema.validate_provider_requirements(conf) enforces:
* vertex-ai: provider_conf (project_id + region) OR override.endpoint
* bedrock: provider_conf.region + auth.aws (always required)
* bedrock without override.endpoint: also requires options.model
- ai-proxy.check_schema and ai-proxy-multi.check_schema call the
validator after the JSON schema check
- multi mode prefixes per-instance errors with instance name
…edrock Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
The bedrock provider previously required `options.model` on the route,
forcing one route per model. Drop that schema requirement so clients
can supply `model` in the request body — `before_proxy` already prefers
`options.model` and falls back to body.model, populating
`ctx.var.llm_model` used by the bedrock path callback.
- Treat empty `ctx.var.llm_model` ("") as "no model" in the bedrock
path callback (it defaults to "" via APISIX.pm `set $llm_model '';`,
which would otherwise produce `/model//converse`).
- Return a clear 400 in the provider base when neither target_path nor
override.endpoint can resolve a path, instead of failing opaquely.
- Rename the rewrite hook to `rewrite_converse_request_body` so the
name reflects its protocol (bedrock may host other protocols later).
Tests cover schema acceptance without options.model, body-supplied
model success path, and the runtime 400 error case.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| endpoint = { | ||
| type = "string", | ||
| description = "To be specified to override the endpoint of the AI Instance", | ||
| description = "Override the endpoint of the AI Instance. " | ||
| .. "Typically used for custom hosts (e.g., AWS " | ||
| .. "PrivateLink, reverse proxies). You may provide " | ||
| .. "only the scheme + host, in which case the plugin " | ||
| .. "computes the provider-specific path, or provide " | ||
| .. "a full endpoint including path and query, in " | ||
| .. "which case the plugin uses the supplied path/query. " | ||
| .. "If your custom path or query contains reserved " | ||
| .. "characters (e.g., Bedrock inference profile ARNs " | ||
| .. "containing ':' or '/'), they must be URL-encoded.", | ||
| }, |
There was a problem hiding this comment.
In validate_provider_requirements(), has_override treats any non-nil override.endpoint as satisfying provider requirements. In Lua, an empty string is truthy, so override.endpoint: "" would pass validation even though it can’t be parsed into a usable URL at runtime. Consider adding minLength = 1 to override.endpoint in the schema (and/or checking endpoint ~= "" when computing has_override).
| -- in the request body for per-request model selection. The bedrock | ||
| -- provider falls back to options.model only when body.model is absent. | ||
| -- If neither is set at request time, build_request returns 400. |
There was a problem hiding this comment.
The Bedrock-specific comment here says the provider “falls back to options.model only when body.model is absent”, but the current model selection logic in apisix/plugins/ai-proxy/base.lua sets llm_model to options.model whenever it’s configured and only uses body.model when options.model is missing. Please either update this comment to match the actual precedence (route config wins), or change the precedence if per-request model override is intended.
| -- in the request body for per-request model selection. The bedrock | |
| -- provider falls back to options.model only when body.model is absent. | |
| -- If neither is set at request time, build_request returns 400. | |
| -- in the request body when no route-level model is configured. If | |
| -- options.model is set, it takes precedence; body.model is only used | |
| -- when options.model is absent. If neither is set at request time, | |
| -- build_request returns 400. |
…ence - Add `minLength = 1` to `override.endpoint`. An empty string is truthy in Lua, so without this the schema validator and `validate_provider_requirements`'s `has_override` check would treat `override.endpoint: ""` as set even though it can't parse into a usable URL. - Fix the bedrock options.model comment in `validate_provider_requirements` to match the real precedence in `before_proxy`: route-level `options.model` wins; `body.model` is only used when options.model is absent.
Add Amazon Bedrock support to ai-proxy and ai-proxy-multi
Adds a new
bedrockprovider that proxies requests to Amazon Bedrock's Converse API, with full AWS SigV4 request signing.What's new
Provider & protocol:
bedrockprovider that signs requests with SigV4 and routes them tobedrock-runtime.{region}.amazonaws.com/model/{modelId}/conversebedrock-converseprotocol module that parses Bedrock's native message/content-block formatmessagesarray sent to a URI ending with/converseAuthentication:
auth.awsblock withaccess_key_id,secret_access_key(required), andsession_token(optional, for temporary credentials from STS / assumed roles)secret_access_key,session_token) are encrypted at the schema levelx-amz-security-token) are redacted from logsRegion & endpoint:
provider_conf.regionis required forbedrock(no env-var fallback or hardcoded default — fails fast on misconfiguration)override.endpointsupported for AWS PrivateLink, custom proxies, etc.Model identifiers (
options.model):anthropic.claude-3-5-sonnet-20240620-v1:0)us.anthropic.claude-3-5-sonnet-20240620-v1:0)arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123)ARN model IDs are URL-encoded automatically so reserved characters (
:,/) stay as a single path segment.Example
Implementation highlights
resty.aws.request.sign(already a dependency):→%3Aon the wire →%253Ain canonical URI)path:gsubpreserves slash structure including trailing slashes and empty segmentsconstruct_forward_headers()and avoid duplicatesLimitations / future work
ConverseStream(which uses AWS EventStream binary protocol, not SSE) is not yet supported.Checklist