A Rust dynamic module that resolves virtual models with NVIDIA NeMo Switchyard routing algorithms.
Clients send a virtual model name, such as coding-auto. For every request, the filter runs a
Switchyard algorithm in process and rewrites the model field of the request body to the model
that should serve it. For example, the auto algorithm sends agent steps that only read tool output
to an efficient model, and escalates to a capable model when tool calls fail.
The filter only changes the request. Routing to the selected model is left to the filters and routes that follow it:
- With Agent Router, nothing else is needed: Agent Router reads the model from the body after this filter has rewritten it. See Using with Agent Router.
- On plain Envoy, set
model_headerandclear_route_cache, and match routes on that header. - To let a later filter make the final choice, set
rewrite_bodyto false. The body keeps the virtual model, and the decision is only carried in headers, dynamic metadata and filter state. See Decision-only mode.
- On
POSTrequests with a JSON body to a configured endpoint, the filter holds the headers and buffers the body. Other requests are not buffered. - If the body's
modelis a configured virtual model (and itsmatch_headersmatch), the filter decodes the request with Switchyard's translation layer and runs the algorithm. - Unless
rewrite_bodyis false, the filter replaces the body'smodelwith the selected model and updatescontent-length. When the algorithm changes the request beyond the model, the filter sends Switchyard's encoding of the request. Otherwise it keeps every other byte of the client's body. - It records the decision in dynamic metadata, filter state and, when enabled, the decision headers, and lets the request continue.
Requests for any other model pass through unchanged.
The algorithms run on a small Tokio runtime owned by the module. Envoy worker threads block on it for the duration of a decision, which never waits on I/O: only algorithms that decide from the request alone are supported.
{
"virtual_models": [
{
"name": "coding-auto",
"match_headers": [{"name": "x-tier", "value": "free"}],
"targets": {"efficient": "gpt-4.1-mini"},
"algorithm": {"type": "passthrough", "target": "efficient"}
},
{
"name": "coding-auto",
"targets": {"efficient": "gpt-4.1-mini", "capable": "gpt-5"},
"algorithm": {"type": "auto", "capable_target": "capable", "efficient_target": "efficient"},
"default_model": "gpt-4.1-mini",
"failure_mode": "use_default"
}
],
"session": {
"id_header": "x-switchyard-session-id",
"fallback_id_headers": ["agent-session-id"],
"scope_headers": ["x-tenant-id"],
"agent_id_header": "x-switchyard-agent-id",
"subagent_header": "x-switchyard-is-subagent"
},
"endpoints": {
"/v1/chat/completions": "openai_chat",
"/v1/responses": "openai_responses",
"/v1/messages": "anthropic_messages"
},
"model_header": null,
"clear_route_cache": false,
"decision_headers": true,
"rewrite_body": true
}Only virtual_models is required. The other values above are the defaults. Judges, model calls
and eligible models use the optional fields described in Judges and model calls
and Eligible models.
| Field | Description |
|---|---|
name |
The model name clients send. Several entries can share a name; the first whose match_headers all match wins. |
targets |
Switchyard target names mapped to the model names they route to. The algorithm refers to the target names. |
algorithm |
The Switchyard algorithm, in Switchyard's route format. |
default_model |
Model used when routing fails. Must be one of the target models. |
failure_mode |
use_default (route to default_model), passthrough (leave the request unchanged), or reject (reply with a 503). Defaults to use_default when default_model is set, and to passthrough otherwise. |
match_headers |
Exact request header matches, for example to give a tenant or a tier its own policy. |
| Type | Behaviour |
|---|---|
auto |
Switchyard's recommended preset: a stage_router that starts on the efficient tier and escalates on errors and stalls. Needs capable_target and efficient_target. |
stage_router |
Picks a tier per turn by scoring recent tool results, optionally with a classifier judge. |
plan_execute |
Plans on the capable target and hands off to the efficient one after the first edit, or when a judge approves. |
random |
Weighted split across targets. |
passthrough |
Always the same target. |
llm_classifier |
A judge picks the target. |
composite |
A judge picks the tier at each user turn, and a stage router runs the turns within it. |
vgr |
Answers with the efficient model, unless a verifier escalates the request to the capable model. |
advisor |
An executor model answers, and an advisor model reviews some answers first. |
noop |
Answers OK without calling a model, for smoke tests. |
prefill_router, which runs a PyTorch model in process, is not supported.
Algorithms that call a model reach it in one of two ways:
decision_targetsandjudge_targetshold judges with their own endpoint and API key, such as Jev or a provider's chat API. The filter never sends them client headers.model_callssends calls to the gateway itself, so that its credentials, rate limits, costs and telemetry apply. Judges named by a key oftargets, and the modelsvgrandadvisorcall, go this way.
An algorithm names a judge as in Switchyard, for example with classifier_target. The filter looks
the name up in decision_targets, then judge_targets, then the virtual model's targets. A
decision judge, set with an algorithm decision block, must be in decision_targets. Config load
fails when a name is in both judge maps or in none, or when an algorithm needs model_calls and it
isn't set.
decision_targets and judge_targets map a judge name to:
| Field | Description |
|---|---|
format |
system_one or openai_decisions for decision targets. openai_chat, openai_responses or anthropic_messages for judge targets. |
model |
The provider model sent in the request body, for example jev-latest. |
cluster |
The Envoy cluster the calls go to. |
authority |
The Host header of the calls. It must match the host name the judge's TLS certificate expects. |
path |
Defaults to /v1/systemone, /v1/decisions, /v1/chat/completions, /v1/responses or /v1/messages, by format. |
api_key |
Optional. {"inline": "..."} or {"file": "/path"}, read once at config load. Sent as authorization: Bearer <key>, or as x-api-key for anthropic_messages. |
headers |
Extra request headers. |
timeout_ms |
Defaults to 5000 for decision targets and 30000 for judge targets. |
trace_context |
w3c, b3 or none. Defaults to w3c. |
model_calls:
| Field | Description |
|---|---|
cluster |
Required. The Envoy cluster the calls go to, usually one that loops back to the gateway. |
authority |
Defaults to the client request's. |
path |
Defaults to the client request's path, or to the standard path of format when it differs from the client's. |
format |
openai_chat, openai_responses or anthropic_messages. Defaults to the client request's. |
timeout_ms |
Defaults to 10000. |
forward_headers |
Client request headers copied to the calls, such as credentials and tenant headers, so that the gateway authenticates and charges each call like the client request. |
headers |
Extra request headers. |
trace_context |
w3c, b3 or none. Defaults to w3c. |
This example routes coding-auto with Jev. The typesafe cluster must exist in Envoy and connect
to api.typesafe.ai:443 with TLS:
{
"decision_targets": {
"jev": {
"format": "system_one",
"model": "jev-latest",
"cluster": "typesafe",
"authority": "api.typesafe.ai",
"api_key": {"file": "/etc/switchyard/jev-api-key"}
}
},
"virtual_models": [{
"name": "coding-auto",
"targets": {"efficient": "gpt-4.1-mini", "capable": "gpt-5"},
"algorithm": {
"type": "llm_classifier", "mode": "capability", "classifier_target": "jev",
"strong_target": "capable", "weak_target": "efficient", "classify_trigger": "new_session",
"decision": {"cutoff": 0.05, "candidates": {"candidate_a": "capable", "candidate_b": "efficient"}, "evidence": {}}
},
"default_model": "gpt-4.1-mini"
}]
}For model_calls, loopback.yaml adds a cluster that loops
back to the Gateway listener. Agent Router routes those calls by their model, so a gateway judge
must be a model of an AIGatewayRoute, not a virtual model.
Calls are non-blocking Envoy HTTP callouts, but the request waits for the algorithm, so each call
adds latency. Bound it with the algorithm's own deadline, such as judge_deadline_ms, and with a
classify_trigger such as new_session instead of every_request. A failed model call moves to
the next candidate model on a transport error, a 5xx, 403, 408 or 429 status, or an exceeded context
window. Otherwise the algorithm applies its own fallback, such as fail_open, and when the
algorithm fails, the virtual model's failure_mode applies. When the client goes away, the filter
stops the run.
Every call carries x-switchyard-call (judge, model or decision), x-switchyard-parent-id
(the client request's x-request-id) and, when Envoy traces the request, trace context in the
format of its trace_context. Calls are sampled unless the client's trace context says it is not.
The algorithms of a virtual model that calls models through model_calls cannot route to another
virtual model: the call would loop back through the filter and start another routing run.
When vgr, advisor or noop writes the answer, the filter sends it to the client as a 200
response in the client's API format, as server-sent events when the client asked for a stream.
Set eligible_models_header to limit the models of each request, for example to the models of a
tenant's plan. The header holds comma-separated model names, and the algorithm only sees the
eligible targets. Judges are never limited. Routing fails, and failure_mode applies, when no
target is eligible. A trusted filter, such as an ext_authz service, must set the header. Never let
the client set it.
Stateful algorithms, such as auto and stage_router, keep per-session state, for example to hold
the capable tier for a few turns after an escalation. The session ID is the first non-empty header
among id_header and fallback_id_headers. agent-session-id is the header Agent Router already
records as the session of a request. The values of scope_headers prefix the session ID, so that
sessions of different tenants never share state.
Session state lives in the memory of each Envoy process. Send the requests of a session to the same Envoy instance, for example with session affinity on the session header. Without a session ID, requests are routed from their own content only.
Requests whose subagent_header is true or 1 are sub-agent requests. When the algorithm has a
subagents block, they are routed by it instead of by the parent algorithm. A sub-agent request
must also carry the agent ID in agent_id_header to keep its own session state; without it, the
request is routed statelessly.
By default, the filter rewrites the body, so the filters after it only see the selected model. Set
rewrite_body to false to keep the virtual model in the body and only record the decision. A later
filter then sees both the virtual model and the decision, and can apply its own policy, such as
quotas, budgets or fallbacks, before it picks the backend:
{
"virtual_models": [{
"name": "coding-auto",
"targets": {"efficient": "gpt-4.1-mini", "capable": "gpt-5"},
"algorithm": {"type": "auto", "capable_target": "capable", "efficient_target": "efficient"}
}],
"rewrite_body": false,
"decision_headers": false
}The decision is always available to later filters as:
- Dynamic metadata in the
io.builtonenvoy.switchyardnamespace. - Filter state, as plain strings under the
io.builtonenvoy.switchyard.virtual_model,io.builtonenvoy.switchyard.selected_modelandio.builtonenvoy.switchyard.reasonkeys. Filter state is never sent upstream, and is available to other filters, to access logs with%FILTER_STATE(io.builtonenvoy.switchyard.selected_model:PLAIN)%, to header formatters, and to CEL expressions asfilter_state['io.builtonenvoy.switchyard.selected_model'].
The decision headers and model_header carry it too, when enabled. Headers are easy to consume
from any filter, including ext_proc and ext_authz services, but they are also sent to the upstream
unless a later filter removes them.
Clients can never set the decision: the filter removes client-sent decision headers, and dynamic metadata and filter state are only set by filters. Envoy does not, however, protect filter state or dynamic metadata from other filters: since Envoy removed read-only filter state, a filter can overwrite any key. Treat the decision as trusted only if the filters after this one are.
When the algorithm changes the request beyond the model, those changes are dropped in this mode, as only the decision is passed on.
With Agent Router, the body's model is what Agent Router routes on, so in this mode Agent Router routes the virtual model unless a filter between the two acts on the decision.
- Request headers, when
decision_headersis true:x-switchyard-virtual-model,x-switchyard-selected-modelandx-switchyard-reason. They are also sent to the upstream; setdecision_headersto false to avoid that. These headers are always removed from the client request, so the upstream only sees the values set by the filter. - Dynamic metadata in the
io.builtonenvoy.switchyardnamespace, with the keysvirtual_model,selected_modelandreason. Use them in access logs with%DYNAMIC_METADATA(io.builtonenvoy.switchyard:selected_model)%. - Filter state under the
io.builtonenvoy.switchyard.virtual_model,io.builtonenvoy.switchyard.selected_modelandio.builtonenvoy.switchyard.reasonkeys. Use them in access logs with%FILTER_STATE(io.builtonenvoy.switchyard.selected_model:PLAIN)%. - The
switchyard_decisionscounter, with the labelsvirtual_model,selected_modelandoutcome(routed,answered,default,passthroughorrejected). Answers sent by the filter have the response detailsswitchyard_answered. - For judge algorithms, the judge's evidence as compact JSON, in the
evidencedynamic metadata key and theio.builtonenvoy.switchyard.evidencefilter state key. - The
switchyard_judge_callscounter, with the labelsjudgeandoutcome(ok,errorortimeout), one per call. - For algorithms that call models, the dynamic metadata keys
routing_calls,routing_input_tokens,routing_output_tokens,routing_total_tokens,routing_models, androuting_latency_ms(from the start of routing to its outcome). Whenrouting_headersis true, the calls and tokens also go in thex-switchyard-routing-calls,-input-tokens,-output-tokensand-total-tokensresponse headers.
boe run --local extensions/switchyard --config '{
"virtual_models": [{
"name": "chat-split",
"targets": {"a": "model-a", "b": "model-b"},
"algorithm": {"type": "random", "targets": ["a", "b"], "weights": [0.5, 0.5]}
}],
"model_header": "x-model",
"clear_route_cache": true
}'
curl localhost:10000/anything/v1/chat/completions -H 'content-type: application/json' \
-d '{"model": "chat-split", "messages": [{"role": "user", "content": "hi"}]}'The default upstream echoes the request, so the response shows the rewritten model and the
decision headers.
Agent Router (formerly Envoy AI Gateway) runs on Envoy Gateway. The extension works with an unmodified Agent Router: it is added to the Gateway with Envoy Gateway's dynamic module support, and Agent Router routes the rewritten request.
client ─▶ … ─▶ switchyard ─▶ Agent Router ext_proc ─▶ … ─▶ router ─▶ selected model's backend
(this module) (envoy.filters.http.ext_proc/aigateway)
Agent Router's ext_proc filter reads the model from the request body, sets the x-ai-eg-model
header that Agent Router routes match, and makes Envoy pick the route again. Because this filter runs
first and rewrites the body, Agent Router sees the selected model, as if the client had asked for it.
Its credentials, request translation, rate limits, costs and metrics then apply to the selected
model's backend.
Two settings make this work:
- The filter must come before Agent Router's filter. Agent Router inserts its ext_proc filter
before the first
lua,ext_procorwasmfilter of the listener. Envoy Gateway places dynamic modules after those, so theEnvoyProxysetsfilterOrderto move the module before them. Without any such filter on the listener, the module is already in the right place. - The policy must target the whole Gateway. Envoy picks the filters for a request from the
first route it matches, before the model is known. For Agent Router routes, that is usually the
catch-all
route-not-foundrule, which a policy targeting specific routes or rules would miss.
The examples/agent-router directory has a complete setup:
gateway.yaml: theGateway, and anEnvoyProxythat registers the module, sets the filter order, and mounts the extension image as a volume.policies/: oneEnvoyExtensionPolicyper algorithm and feature, with the filter configuration.policies/auto.yamlis the default.judge.yamlandloopback.yaml: a fake judge and decision API, and the clusters that reach them and the gateway itself.routes.yaml: theAIGatewayRoutewith a rule per target model, like any other Agent Router setup. A rule for the virtual model is optional: it lists the virtual model in/v1/models, and serves requests the filter leaves unchanged.testupstream.yaml: fake model servers for local tests.kind.sh: creates a kind cluster, installs Envoy Gateway and Agent Router, applies a policy, and checks the routing../kind.sh allchecks every policy.
examples/judge-routing runs Jev and an LLM classifier side by side
against real models, and writes an HTML report.
The key parts of the EnvoyProxy:
spec:
dynamicModules:
- name: switchyard
source:
type: Local
local:
path: /var/boe/extensions/switchyard/libswitchyard.so
doNotClose: true
filterOrder:
- name: envoy.filters.http.dynamic_modules
before: envoy.filters.http.lua
- name: envoy.filters.http.dynamic_modules
before: envoy.filters.http.ext_proc
- name: envoy.filters.http.dynamic_modules
before: envoy.filters.http.wasm
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
pod:
volumes:
- name: switchyard
image:
reference: ghcr.io/tetratelabs/built-on-envoy/extension-switchyard:0.1.1
container:
volumeMounts:
- name: switchyard
mountPath: /var/boe/extensions/switchyard
readOnly: trueThe example mounts the extension's OCI image as a Kubernetes
image volume, which is enabled by
default since Kubernetes 1.35. The image also contains libgcc_s.so.1, which the module needs and
which distroless Envoy images, such as Envoy Gateway's default, do not have.
Without image volumes, build an Envoy image that contains the module and set it as the
envoyDeployment.container.image of the EnvoyProxy:
FROM ghcr.io/tetratelabs/built-on-envoy/extension-switchyard:0.1.1 AS switchyard
FROM docker.io/envoyproxy/envoy:distroless-v1.39.1
COPY --from=switchyard /libswitchyard.so /libgcc_s.so.1 /var/boe/extensions/switchyard/The module can also be served over HTTP and referenced with a Remote source and its SHA-256, but
then libgcc_s.so.1 must already be in the Envoy image.
Agent Router can add the decision headers to its metrics, spans and access logs. For example, with the Agent Router helm chart:
--set controller.requestHeaderAttributes='x-switchyard-virtual-model:switchyard.virtual_model\,x-switchyard-reason:switchyard.reason'Agent Router records the selected model as both the original and the request model, because it never
sees the virtual model. The switchyard.virtual_model attribute keeps it.
- Built with the dynamic modules SDK of Envoy 1.39.1. The dynamic modules ABI is unchanged across Envoy 1.39 patch releases, including 1.39.6, the default of the Agent Router CLI.
- The Switchyard crates are pinned to a specific commit, as Switchyard has not reached 1.0.