Skip to content

docs: recipe for OCI Generative AI as a Switchyard backend #1292

Open

Description

@fede-kamel

Summary

Add a recipe documenting Oracle Cloud Infrastructure (OCI) Generative AI as a Switchyard backend. OCI exposes an OpenAI-compatible endpoint (https://inference.generativeai.<region>.oci.oraclecloud.com/openai/v1, Chat Completions and Responses) authenticated with a compartment-scoped Generative AI API key sent as a bearer token, so the released v0.3.0 server and Python bindings work with no code change. The repository currently has no OCI mention; this closes that documentation gap the same way the single-provider coding-agent recipe does for OpenAI and Anthropic logins.

What the recipe covers

  • Endpoint, model ID form (vendor.model, e.g. meta.llama-4-scout-17b-16e-instruct, xai.grok-4.7, openai.gpt-5.5), and how to list the regional catalog (the endpoint does not serve GET /models).
  • Credential setup in the order that works: IAM policy first, then the API key (a key created before its policy can stay unauthorized; OCI returns the same 401 for unknown and unauthorized keys).
  • A llm_classifier capability route and a stage_router route over OCI targets, with two clients: openai_chat for Meta/xAI/Google models and openai_responses for OpenAI gpt-5.x/gpt-6.x, which reject max_tokens on Chat Completions.
  • The Python host path with the released bindings.
  • Known limits: 404 for catalog models not served on demand, Cohere embeddings unsupported, judge timeout_ms, cost table needs an override for OCI IDs, dedicated clusters and signed OCI principals need a signing gateway in front.

Verification

  • switchyard-server --config oci-routing.toml --dry-run on v0.3.0 accepts the configuration (oci-task, oci-execution, plus two passthrough baselines).
  • Live runs against us-chicago-1 with the judge and answers served by OCI are being recorded now and will be attached to the PR before it leaves draft.

Proposed placement

docs/recipes/oci_generative_ai.md, linked in mkdocs.yml under Integrate Switchyard → Model Providers (a new group next to Gateways). Happy to move it if you prefer it under Local Demo with the other recipe.

Opening this first because the page is about 140 lines, per CONTRIBUTING's guidance for larger changes. A follow-up, separate from this one, would address #665 (serving judge calls in the LiteLLM plugin), which I have working against OCI.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions