Skip to content

feat(ai-proxy): support aws bedrock#13249

Merged
shreemaan-abhishek merged 47 commits into
apache:masterfrom
shreemaan-abhishek:feat-support-bedrock
Apr 27, 2026
Merged

feat(ai-proxy): support aws bedrock#13249
shreemaan-abhishek merged 47 commits into
apache:masterfrom
shreemaan-abhishek:feat-support-bedrock

Conversation

@shreemaan-abhishek

@shreemaan-abhishek shreemaan-abhishek commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Add Amazon Bedrock support to ai-proxy and ai-proxy-multi

Adds a new bedrock provider that proxies requests to Amazon Bedrock's Converse API, with full AWS SigV4 request signing.

What's new

Provider & protocol:

  • New bedrock provider that signs requests with SigV4 and routes them to bedrock-runtime.{region}.amazonaws.com/model/{modelId}/converse
  • New bedrock-converse protocol module that parses Bedrock's native message/content-block format
  • Detection: requests with body containing messages array sent to a URI ending with /converse

Authentication:

  • New auth.aws block with access_key_id, secret_access_key (required), and session_token (optional, for temporary credentials from STS / assumed roles)
  • Sensitive fields (secret_access_key, session_token) are encrypted at the schema level
  • All credential headers (including x-amz-security-token) are redacted from logs

Region & endpoint:

  • provider_conf.region is required for bedrock (no env-var fallback or hardcoded default — fails fast on misconfiguration)
  • override.endpoint supported for AWS PrivateLink, custom proxies, etc.

Model identifiers (options.model):

  • Foundation model IDs (e.g., anthropic.claude-3-5-sonnet-20240620-v1:0)
  • Cross-region inference profile IDs (e.g., us.anthropic.claude-3-5-sonnet-20240620-v1:0)
  • Application inference profile ARNs (e.g., arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123)

ARN model IDs are URL-encoded automatically so reserved characters (:, /) stay as a single path segment.

Example

curl http://127.0.0.1:9180/apisix/admin/routes/1 -X PUT \
  -H 'X-API-KEY: ...' -d '{
    "uri": "/ai/converse",
    "plugins": {
      "ai-proxy": {
        "provider": "bedrock",
        "auth": {
          "aws": {
            "access_key_id": "AKIA...",
            "secret_access_key": "..."
          }
        },
        "provider_conf": { "region": "us-east-1" },
        "options": {
          "model": "anthropic.claude-3-5-sonnet-20240620-v1:0"
        }
      }
    }
  }'
curl http://127.0.0.1:9080/ai/converse -X POST \
  -H 'Content-Type: application/json' -d '{
    "messages": [{"role": "user", "content": [{"text": "What is 1+1?"}]}],
    "inferenceConfig": {"maxTokens": 256}
  }'

Implementation highlights

  • AWS SigV4 signing uses resty.aws.request.sign (already a dependency)
  • Path encoding handles AWS's double-encoding requirement for canonical URIs (e.g., :%3A on the wire → %253A in canonical URI)
  • path:gsub preserves slash structure including trailing slashes and empty segments
  • Header values from the signer are written back using lowercase keys to match construct_forward_headers() and avoid duplicates

Limitations / future work

  • Non-streaming only. Bedrock's ConverseStream (which uses AWS EventStream binary protocol, not SSE) is not yet supported.
  • Bedrock is the only AWS service currently — other AWS services (e.g., SageMaker) would need separate provider modules.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

Signed-off-by: Abhishek Choudhary <[email protected]>
Rename double_encode_path to encode_path_for_canonical_uri since the
function only does a single ngx.escape_uri pass. The double-encoding
required by AWS SigV4 is achieved by combining this pass with the
upstream encoding done by bedrock.lua on the model ID.

Update the function comment to accurately describe this two-stage flow.
construct_forward_headers() in http.lua lowercases all header keys, so
writing signed headers (Authorization, X-Amz-Date, X-Amz-Security-Token,
Host) in mixed case could result in duplicate headers depending on the
HTTP client's behavior. Write them back as lowercase to match the
convention used elsewhere.
Add an explicit nil/empty check on params.path at the start of
sign_request() to return a clear error instead of crashing in
encode_path_for_canonical_uri() with "attempt to index a nil value".
This can happen when the Bedrock provider's path capability returns
nil (e.g., when options.model is missing).
Bedrock Converse API only accepts 'user' and 'assistant' roles in
messages[*].role. System prompts must go in the top-level body.system
field as an array of {text: "..."} blocks.

Update prepend_messages and append_messages to split incoming messages
by role: system entries are routed to body.system as text blocks, while
user/assistant entries go to body.messages as before. Without this,
prompt-decorator plugins inserting canonical system messages would
produce invalid Bedrock requests.
Bedrock builds the upstream URL path from the model ID. Without
options.model and without override.endpoint, the path resolves to nil
and the request fails later with a low-signal transport error. Add a
schema-level rule to require options.model when provider=bedrock and
override.endpoint is not set, so misconfigurations fail fast at
config validation time.
TEST 2-4 depend on the route created in TEST 1, and TEST 7 depends on
the route created in TEST 6. Add no_shuffle() so tests run in declared
order and these dependencies are honored consistently.
Switch oneOf to anyOf in the conditional schemas for vertex-ai and
bedrock. With oneOf, users couldn't set both provider_conf and
override.endpoint together. This blocked valid use cases like AWS
PrivateLink (custom endpoint) combined with explicit signing region,
since the code does not parse the region from the override URL.
anyOf still requires at least one of the two to be set.
… endpoint

Add test scenarios covering edge cases introduced by the Bedrock
provider:

- TEST 8/9: model as inference profile ARN (validates URL encoding of
  ":" and "/" in the path and SigV4 canonical URI handling)
- TEST 10/11: auth.aws.session_token propagation as
  x-amz-security-token header to the upstream
- TEST 12: route with provider_conf.region but no override.endpoint
  (validates schema accepts default endpoint construction)

Mock server appends session_token to the response body so TEST 11 can
assert propagation via response_body regex.
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request labels Apr 17, 2026
Move all SigV4 path encoding into auth-aws.lua so callers don't need
to know about the AWS double-encoding rule. The previous design split
encoding between bedrock.lua (pre-encoded the model) and auth-aws.lua
(added the second encoding pass), which produced an incorrect canonical
URI when params.path came from a raw source like override.endpoint.

Now bedrock.lua passes the raw model in the path, and auth-aws.lua
normalizes the input (decode-then-encode is idempotent for both raw
and once-encoded inputs) and produces:
- params.path: once-encoded for HTTP wire transport
- canonicalURI: twice-encoded for SigV4 canonical request

This handles both default-endpoint and override.endpoint paths
correctly regardless of whether the input is raw or pre-encoded.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Amazon Bedrock support to the ai-proxy / ai-proxy-multi plugin by introducing a Bedrock provider implementation, Bedrock Converse protocol handling, and AWS SigV4 request signing.

Changes:

  • Introduces a new bedrock provider + bedrock-converse protocol adapter (detection, request shaping, response parsing).
  • Adds AWS SigV4 signing support (incl. session token propagation) and updates HTTP transport to accept pre-serialized bodies.
  • Extends ai-proxy schemas and log sanitization, plus adds an end-to-end Bedrock-focused test suite.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
t/plugin/ai-proxy-bedrock.t New test coverage for Bedrock Converse routing, schema validation, ARN model handling, and session token propagation.
apisix/utils/log-sanitize.lua Redacts x-amz-security-token in logged headers.
apisix/plugins/ai-transport/http.lua Allows params.body to be a pre-serialized string (needed after SigV4 signing).
apisix/plugins/ai-transport/auth-aws.lua New AWS SigV4 signing helper for outbound AI provider requests.
apisix/plugins/ai-proxy/schema.lua Adds AWS auth schema + encrypt_fields updates; adds Bedrock provider_conf schema and conditional validation.
apisix/plugins/ai-providers/schema.lua Registers bedrock as a known provider.
apisix/plugins/ai-providers/bedrock.lua New Bedrock provider implementation (host/path generation; remove model from body).
apisix/plugins/ai-providers/base.lua Hooks AWS SigV4 signing into the shared request build pipeline.
apisix/plugins/ai-protocols/init.lua Registers bedrock-converse and adjusts protocol detection order.
apisix/plugins/ai-protocols/bedrock-converse.lua New Bedrock Converse protocol adapter (non-streaming).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/plugins/ai-transport/auth-aws.lua Outdated
Comment thread apisix/plugins/ai-proxy/schema.lua Outdated
Comment thread apisix/plugins/ai-protocols/bedrock-converse.lua
Comment thread apisix/plugins/ai-protocols/bedrock-converse.lua
Comment thread apisix/plugins/ai-protocols/bedrock-converse.lua
Comment thread t/plugin/ai-proxy-bedrock.t Outdated
Previously the override schema didn't require any fields, so a config
like override = {} would pass validation despite being meaningless.
This was particularly problematic because the schema's anyOf accepted
"override" as an alternative to "provider_conf" — letting an empty
override silently bypass the provider_conf requirement and break
downstream request construction.

Make endpoint required in the override schema since it's the only
field and the only purpose of override is to set the endpoint.
Add minLength = 1 to auth.aws.access_key_id, secret_access_key,
session_token, and provider_conf.region for the Bedrock provider.
The 'required' constraint only checks key presence; without minLength,
empty strings pass schema validation and fail later during signing or
upstream URL construction. Catch these as admin-side validation errors
instead.
Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
…nsert

prepend_messages and append_messages only checked for nil before
calling core.table.insert(body.system/messages, ...), which would
crash if those fields existed but weren't tables (e.g., a string).
Use type(...) ~= "table" to normalize, preventing request-time
exceptions on malformed input.
Bedrock provider falls back to AWS_REGION env var (or us-east-1
default) when provider_conf.region is not set, so the schema should
not require provider_conf or override. The inner rule still requires
options.model when override.endpoint is not set, ensuring a valid
upstream URL can always be constructed.
The previous code passed n=1 to normalize_and_encode_path for the
canonical URI, but n=1 means decode-then-encode-once, which cancels
out and produces the same once-encoded form as params.path. AWS SigV4
requires reserved characters to be encoded TWICE in the canonical
URI (e.g., raw ":" → "%3A" on the wire → "%253A" in canonicalURI).

Pass n=2 so the function decodes each segment then encodes it twice,
producing the required double-encoded form. Tests pass because the
local mock doesn't validate signatures, but real AWS requests would
have failed with signature mismatch.
Application inference profile ARNs contain "/" (e.g.
"...:application-inference-profile/abc") which would be split as path
segments if interpolated raw into "/model/{id}/converse". Encode the
model with ngx.escape_uri so "/" becomes "%2F" and ":" becomes "%3A",
keeping the model as a single path segment.

This is safe to combine with auth-aws.lua's normalize_and_encode_path
since that function decodes-then-encodes (idempotent for both raw and
once-encoded inputs).
Update the override.endpoint description to:
- Recommend host-only overrides for the typical use case (PrivateLink,
  reverse proxies) so the plugin computes the path with proper encoding
- Document that any reserved characters in an included path (e.g.,
  Bedrock inference profile ARNs with ':' or '/') must be URL-encoded
- Fix test URIs to end with /converse so bedrock-converse protocol
  detection matches (previous URIs like /ai/converse-arn don't end
  with /converse and would fall through to openai-chat)
- Log the request URI in the mock so tests can assert specific paths
- Update TEST 8 override.endpoint to use a properly URL-encoded ARN
  (matching what users should provide per the schema documentation)
- Add an error_log assertion in TEST 9 verifying the wire path keeps
  the ARN as a single percent-encoded segment (%3A and %2F preserved)
The mock server extracted text from the first user message into
first_content but never used it. Remove the dead code to make the
mock handler clearer.
Signed-off-by: Abhishek Choudhary <[email protected]>
Signed-off-by: Abhishek Choudhary <[email protected]>
Remove the AWS_REGION env var fallback and "us-east-1" default for
the Bedrock provider's region. Silently defaulting to us-east-1 was a
footgun — requests would route to the wrong region and fail SigV4
signature validation with confusing errors.

Now provider_conf.region must be set explicitly for Bedrock at the
schema level. The runtime in base.lua also returns a clear error
("missing region for AWS SigV4 signing (provider_conf.region required
for bedrock)") as a defense-in-depth check.

Also remove the hanging schema comment that explained the (now
removed) fallback behavior. Update docs (en + zh) to remove env var /
default mentions and clarify provider_conf.region is required.

Tests updated to set provider_conf.region in all bedrock instance
configs; AWS_REGION env var dropped from the test main_config.
Validate the structure of the SigV4 Authorization and X-Amz-Date
headers instead of just checking presence. A broken signer would now
fail tests instead of silently passing.

Checks:
- Authorization starts with "AWS4-HMAC-SHA256 "
- Credential matches AKIAIOSFODNN7EXAMPLE/<DATE>/us-east-1/bedrock/aws4_request
- SignedHeaders=... is present
- Signature= is followed by exactly 64 hex chars
- X-Amz-Date matches YYYYMMDDTHHMMSSZ format

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

shreemaan-abhishek and others added 4 commits April 20, 2026 13:59
Signed-off-by: Abhishek Choudhary <[email protected]>
…apisix into feat-support-bedrock

Signed-off-by: Abhishek Choudhary <[email protected]>
Comment thread apisix/plugins/ai-proxy/schema.lua Outdated
Comment thread apisix/plugins/ai-protocols/bedrock-converse.lua
Implement the rewrite_request_body capability hook for the bedrock
provider so override.request_body.max_tokens is mapped to Bedrock's
native field (inferenceConfig.maxTokens). This brings Bedrock to
parity with other providers (openai, deepseek, gemini, vertex-ai,
etc.) that already implement this hook.

Honors request_body_force_override: when false (default), respects
client-supplied inferenceConfig.maxTokens; when true, overwrites it.
The previous comment "Remove fields Bedrock doesn't accept" was
misleading — Bedrock DOES accept stream, it just uses a separate
/converse-stream endpoint with the AWS EventStream binary protocol.
Replace with a TODO that explains we strip the field because we
don't yet implement that endpoint.
…to Lua

The schema's allOf/if/then/not blocks for vertex-ai and bedrock
provider requirements were hard to read and produced unhelpful error
messages ("allOf 2 failed: then clause did not match"). Move the
conditional logic to a Lua function in schema.lua and call it from
each plugin's check_schema.

- ai_instance_schema and ai_proxy_schema now have provider_conf as a
  permissive top-level property (no provider-specific required fields)
- New schema.validate_provider_requirements(conf) enforces:
  * vertex-ai: provider_conf (project_id + region) OR override.endpoint
  * bedrock: provider_conf.region + auth.aws (always required)
  * bedrock without override.endpoint: also requires options.model
- ai-proxy.check_schema and ai-proxy-multi.check_schema call the
  validator after the JSON schema check
- multi mode prefixes per-instance errors with instance name
Signed-off-by: Abhishek Choudhary <[email protected]>
Comment thread apisix/plugins/ai-providers/bedrock.lua Outdated
Comment thread apisix/plugins/ai-proxy/schema.lua Outdated
Comment thread t/fixtures/bedrock/converse-basic.json
Signed-off-by: Abhishek Choudhary <[email protected]>
f
Signed-off-by: Abhishek Choudhary <[email protected]>
The bedrock provider previously required `options.model` on the route,
forcing one route per model. Drop that schema requirement so clients
can supply `model` in the request body — `before_proxy` already prefers
`options.model` and falls back to body.model, populating
`ctx.var.llm_model` used by the bedrock path callback.

- Treat empty `ctx.var.llm_model` ("") as "no model" in the bedrock
  path callback (it defaults to "" via APISIX.pm `set $llm_model '';`,
  which would otherwise produce `/model//converse`).
- Return a clear 400 in the provider base when neither target_path nor
  override.endpoint can resolve a path, instead of failing opaquely.
- Rename the rewrite hook to `rewrite_converse_request_body` so the
  name reflects its protocol (bedrock may host other protocols later).

Tests cover schema acceptance without options.model, body-supplied
model success path, and the runtime 400 error case.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 130 to 142
endpoint = {
type = "string",
description = "To be specified to override the endpoint of the AI Instance",
description = "Override the endpoint of the AI Instance. "
.. "Typically used for custom hosts (e.g., AWS "
.. "PrivateLink, reverse proxies). You may provide "
.. "only the scheme + host, in which case the plugin "
.. "computes the provider-specific path, or provide "
.. "a full endpoint including path and query, in "
.. "which case the plugin uses the supplied path/query. "
.. "If your custom path or query contains reserved "
.. "characters (e.g., Bedrock inference profile ARNs "
.. "containing ':' or '/'), they must be URL-encoded.",
},

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In validate_provider_requirements(), has_override treats any non-nil override.endpoint as satisfying provider requirements. In Lua, an empty string is truthy, so override.endpoint: "" would pass validation even though it can’t be parsed into a usable URL at runtime. Consider adding minLength = 1 to override.endpoint in the schema (and/or checking endpoint ~= "" when computing has_override).

Copilot uses AI. Check for mistakes.
Comment thread apisix/plugins/ai-proxy/schema.lua Outdated
Comment on lines +388 to +390
-- in the request body for per-request model selection. The bedrock
-- provider falls back to options.model only when body.model is absent.
-- If neither is set at request time, build_request returns 400.

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Bedrock-specific comment here says the provider “falls back to options.model only when body.model is absent”, but the current model selection logic in apisix/plugins/ai-proxy/base.lua sets llm_model to options.model whenever it’s configured and only uses body.model when options.model is missing. Please either update this comment to match the actual precedence (route config wins), or change the precedence if per-request model override is intended.

Suggested change
-- in the request body for per-request model selection. The bedrock
-- provider falls back to options.model only when body.model is absent.
-- If neither is set at request time, build_request returns 400.
-- in the request body when no route-level model is configured. If
-- options.model is set, it takes precedence; body.model is only used
-- when options.model is absent. If neither is set at request time,
-- build_request returns 400.

Copilot uses AI. Check for mistakes.
…ence

- Add `minLength = 1` to `override.endpoint`. An empty string is
  truthy in Lua, so without this the schema validator and
  `validate_provider_requirements`'s `has_override` check would treat
  `override.endpoint: ""` as set even though it can't parse into a
  usable URL.
- Fix the bedrock options.model comment in `validate_provider_requirements`
  to match the real precedence in `before_proxy`: route-level
  `options.model` wins; `body.model` is only used when options.model is
  absent.
@shreemaan-abhishek
shreemaan-abhishek merged commit 9986502 into apache:master Apr 27, 2026
25 of 26 checks passed
wistefan pushed a commit to wistefan/apisix that referenced this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants