Skip to content

feat(realtime): OpenAI Realtime GA support and beta compatibility#27110

Merged
mateo-berri merged 13 commits into
litellm_internal_stagingfrom
litellm_openai_realtime_ga
May 5, 2026
Merged

feat(realtime): OpenAI Realtime GA support and beta compatibility#27110
mateo-berri merged 13 commits into
litellm_internal_stagingfrom
litellm_openai_realtime_ga

Conversation

@Sameerlite

@Sameerlite Sameerlite commented May 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Supports OpenAI Realtime API (GA) on the proxy while keeping beta-shaped clients working.

Behavior

  • WebSocket upstream (OpenAI): default connection omits OpenAI-Beta: realtime=v1 (GA). If the client includes that header on the proxy WebSocket, it is forwarded upstream for legacy beta deployments.
  • session.update: beta-style flat fields are normalized to GA (e.g. modalitiesoutput_modalities, nested audio, session.type, single modality when both audio+text were sent).
  • Responses: when the client sent the beta header, GA event names can be translated back to beta-style names for old listeners.
  • HTTP: POST /v1/realtime/client_secrets fallback no longer adds the beta header by default.
  • Logging: LiteLLMRealtimeStreamLoggingObject.results uses List[Any] so Pydantic does not reject new GA event payloads.
  • Types: OpenAI realtime TypedDicts extended for GA deltas, conversation.item.added / .done, and output_text / output_audio content types.
  • UI: Realtime playground sends session.type: "realtime" and handles GA delta event names (with fallbacks for beta names).

Tests

  • test_remap_beta_session_to_ga_normalizes_modalities_and_audio, translation + header helper tests in test_realtime_streaming.py
  • OpenAI realtime handler tests updated for GA default headers and optional beta forward

Fixes #24212
Fixes #18969
image


Note

Medium Risk
Touches realtime WebSocket request/response translation and header negotiation, so protocol mismatches could break streaming clients or guardrail gating. Changes are localized and backed by new/updated tests across OpenAI/Azure handlers and streaming logic.

Overview
OpenAI Realtime now defaults to GA behavior by omitting OpenAI-Beta: realtime=v1 on upstream WebSocket connections, and only forwarding that header when the client included it on the proxy WebSocket.

Proxy streaming adds GA↔beta compatibility shims: session.update payloads from beta-shaped clients are remapped to GA’s nested schema when the backend is GA (including required session.type, audio format nesting, and modality normalization), while GA backend events can be translated back to beta event/content type names for legacy clients (dropping GA-only events like conversation.item.done).

Updates Azure realtime handling to correctly treat GA/v1 as not requiring api_version, adjusts client_secrets fallback to stop sending the beta header by default, expands OpenAI realtime TypedDicts for GA event/content names, updates the Realtime Playground to send session.type and accept GA delta event names, and adds/updates tests to cover header detection and translation behavior.

Reviewed by Cursor Bugbot for commit 2cf367f. Bugbot is set up for automated code reviews on this repo. Configure here.

Sameerlite and others added 2 commits May 4, 2026 14:37
- Normalize beta-style session.update to GA for upstream OpenAI; optional GA→beta
  event translation when client sends OpenAI-Beta: realtime=v1
- Default upstream WebSocket without OpenAI-Beta; forward header when client opts in
- Extend OpenAI realtime types for GA event names and conversation item shapes
- Relax LiteLLMRealtimeStreamLoggingObject.results to List[Any] for GA events
- Update proxy client_secrets fallback to omit beta header; dashboard RealtimePlayground
- Add unit tests for remap, translation, and beta header helper

Co-authored-by: Cursor <[email protected]>
@greptile-apps

greptile-apps Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds GA protocol support to the OpenAI Realtime WebSocket proxy: the upstream connection now omits OpenAI-Beta: realtime=v1 by default (forwarding it only when the connecting client included it), beta-shaped session.update payloads are remapped to the GA nested schema, and GA event names are translated back to beta names for legacy clients. The changes are well-tested and localized to the realtime handler stack.

Confidence Score: 5/5

Safe to merge; only P2 findings remain after previous thread resolved the P1 backwards-compatibility concerns.

All P1/P0 concerns from prior review threads were addressed by the maintainer. The remaining finding is a P2 edge case where _remap_beta_session_to_ga injects session.type = "realtime" for GA transcription clients who omit the field. New and updated tests are mock-only and cover the key protocol paths.

litellm/litellm_core_utils/realtime_streaming.py — the _remap_beta_session_to_ga session-type injection.

Important Files Changed

Filename Overview
litellm/litellm_core_utils/realtime_streaming.py Core GA/beta protocol shims added: _detect_beta_header, _remap_beta_session_to_ga, _translate_event_to_beta, and _make_disable_auto_response_message; minor P2 around unconditional session.type = "realtime" injection for absent-type sessions.
litellm/llms/azure/realtime/handler.py Correctly derives backend_uses_beta_protocol from realtime_protocol and passes it to RealTimeStreaming; Azure GA/V1 path now skips the deprecated beta header requirement check.
litellm/llms/custom_httpx/llm_http_handler.py Removes OpenAI-Beta: realtime=v1 from async_realtime_client_secret_handler; already discussed in previous thread and deemed non-breaking by maintainers.
litellm/llms/openai/realtime/handler.py Defaults to GA (no OpenAI-Beta header) and only forwards the header when the connecting client explicitly sent it; consistent with RealTimeStreaming protocol detection.
litellm/llms/xai/realtime/handler.py Signature updated to match the new openai_beta_realtime kwarg on the parent; parameter is correctly accepted but intentionally ignored for xAI.
litellm/types/llms/openai.py TypedDicts extended with GA event types (OpenAIRealtimeConversationItemAdded, OpenAIRealtimeConversationItemDone), GA content type literals (output_text, output_audio), and GA delta/done event name literals; additions are additive and non-breaking.
tests/test_litellm/litellm_core_utils/test_realtime_streaming.py Substantial new test coverage for remapping, translation, header detection, and beta/GA session shape preservation; two modified assertions correctly reflect the new GA-shaped _make_disable_auto_response_message output.
tests/test_litellm/llms/azure/realtime/test_azure_realtime_handler.py Adds assertions that backend_uses_beta_protocol is correctly set (False for GA, True for default beta path); no weakening of existing assertions.
tests/test_litellm/llms/openai/realtime/test_openai_realtime_handler.py Existing test updated to expect no OpenAI-Beta header by default (correct for GA); new test verifies the header is forwarded when the client WebSocket includes it.
ui/litellm-dashboard/src/components/playground/chat_ui/RealtimePlayground.tsx Sends session.type: "realtime" on all session.update calls; handles both GA and beta delta event names; response.done fallback reads `c.text

Reviews (6): Last reviewed commit: "Fix Azure realtime beta session shape" | Re-trigger Greptile

Comment thread litellm/llms/openai/realtime/handler.py
Comment thread ui/litellm-dashboard/src/components/playground/chat_ui/RealtimePlayground.tsx Outdated
@Sameerlite

Copy link
Copy Markdown
Collaborator Author

@greptile-apps re review

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

@greptile-apps re review

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

bugbot run

Comment thread litellm/litellm_core_utils/realtime_streaming.py
Comment thread litellm/litellm_core_utils/realtime_streaming.py Outdated
cursoragent and others added 2 commits May 4, 2026 12:50
…A_TYPES

These frozensets were defined as class-level constants in realtime_streaming.py
but never referenced anywhere in the codebase. Removing dead code.

Co-authored-by: Sameer Kankute <[email protected]>
The guardrail VAD injection code sent a beta-style session.update with a
flat turn_detection field:

  {"session": {"turn_detection": {"create_response": false}}}

When the upstream OpenAI backend operates in GA mode (no OpenAI-Beta
header forwarded), it requires the nested GA shape:

  {"session": {"type": "realtime", "audio": {"input": {"turn_detection": {"create_response": false}}}}}

The _remap_beta_session_to_ga helper was only applied to client-
originated session.update messages in client_ack_messages. Internally-
generated session.updates (sent via _send_to_backend) in two paths:
  - _handle_raw_backend_message (raw/no provider_config path, line 518)
  - backend_to_client_send_messages provider_config path (line 481)
bypassed the remap, so GA upstreams ignored or rejected them, breaking
audio transcription guardrails for all non-beta clients.

Fix: add _make_disable_auto_response_message() helper that always emits
the correct GA-shaped session.update, and replace both injection sites
with it.

Update existing tests to assert the GA nested shape instead of the old
flat beta shape, and add a new unit test for the helper itself.

Co-authored-by: Sameer Kankute <[email protected]>
@Sameerlite

Copy link
Copy Markdown
Collaborator Author

bugbot run

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

@greptile-apps re review

Comment thread litellm/llms/custom_httpx/llm_http_handler.py

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Dead code claims logging but performs no action
    • Replaced the discarded session type lookup with a debug log so the traceability comment matches observable behavior.
Preview (7e3d8ab2e4)
diff --git a/litellm/litellm_core_utils/realtime_streaming.py b/litellm/litellm_core_utils/realtime_streaming.py
--- a/litellm/litellm_core_utils/realtime_streaming.py
+++ b/litellm/litellm_core_utils/realtime_streaming.py
@@ -31,6 +31,8 @@
     "session.created",
     "response.create",
     "response.done",
+    "conversation.item.added",  # GA
+    "conversation.item.done",  # GA
 ]
 
 
@@ -54,6 +56,9 @@
         self.session_tools: List[Dict] = []
         self.tool_calls: List[Dict] = []
 
+        # Detect whether the client is explicitly opting into the beta protocol.
+        self._client_wants_beta = self._detect_beta_header(websocket)
+
         _logged_real_time_event_types = litellm.logged_real_time_event_types
 
         if _logged_real_time_event_types is None:
@@ -76,6 +81,27 @@
         # response.create can be rewritten to include the failure context.
         self._pending_guardrail_message: Optional[str] = None
 
+    _SESSION_EVENT_TYPES = frozenset(["session.created", "session.updated"])
+    _AUDIO_FORMAT_MAP: Dict[str, Dict[str, Any]] = {
+        "pcm16": {"type": "audio/pcm", "rate": 24000},
+        "g711_ulaw": {"type": "audio/G711-ulaw", "rate": 8000},
+        "g711_alaw": {"type": "audio/G711-alaw", "rate": 8000},
+    }
+    # GA name → beta name (when client WebSocket includes OpenAI-Beta: realtime=v1)
+    _GA_TO_BETA_EVENT_TYPES: Dict[str, str] = {
+        "conversation.item.added": "conversation.item.created",
+        "response.output_text.delta": "response.text.delta",
+        "response.output_audio.delta": "response.audio.delta",
+        "response.output_audio_transcript.delta": "response.audio_transcript.delta",
+        "response.output_text.done": "response.text.done",
+        "response.output_audio.done": "response.audio.done",
+        "response.output_audio_transcript.done": "response.audio_transcript.done",
+    }
+    _GA_TO_BETA_CONTENT_TYPES: Dict[str, str] = {
+        "output_text": "text",
+        "output_audio": "audio",
+    }
+
     def _should_store_message(
         self,
         message_obj: Union[dict, OpenAIRealtimeEvents],
@@ -92,24 +118,27 @@
         if isinstance(message, bytes):
             message = message.decode("utf-8")
         if isinstance(message, dict):
-            message_obj = message
+            # TypedDict union members do not narrow to plain dict for mypy.
+            message_obj: Dict[str, Any] = cast(Dict[str, Any], message)
         else:
-            message_obj = json.loads(message)
+            message_obj = cast(Dict[str, Any], json.loads(cast(str, message)))
         self._collect_tool_calls_from_response_done(cast(dict, message_obj))
         try:
-            if (
-                not isinstance(message, dict)
-                or message_obj.get("type") == "session.created"
-                or message_obj.get("type") == "session.updated"
-            ):
-                message_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
-            elif not isinstance(message, dict):
-                message_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
+            event_type = message_obj.get("type", "")
+            if event_type in self._SESSION_EVENT_TYPES:
+                typed_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
+            else:
+                # Use the base object as a safe catch-all for all other event types
+                # (both beta and GA), so unknown/new event names never raise here.
+                typed_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
         except Exception as e:
             verbose_logger.debug(f"Error parsing message for logging: {e}")
-            raise e
-        if self._should_store_message(message_obj):
-            self.messages.append(message_obj)
+            # Don't re-raise — a parse failure must not drop or delay the message
+            if self._should_store_message(message_obj):
+                self.messages.append(message_obj)  # type: ignore[arg-type]
+            return
+        if self._should_store_message(typed_obj):
+            self.messages.append(typed_obj)
 
     def _collect_user_input_from_client_event(self, message: Union[str, dict]) -> None:
         """Extract user text content from client WebSocket events for spend logging."""
@@ -147,6 +176,8 @@
                 tools = session.get("tools")
                 if tools and isinstance(tools, list):
                     self.session_tools = tools
+                # GA: session.type is required; log it for traceability but no action needed
+                verbose_logger.debug(f"Realtime session.type: {session.get('type')}")
         except (json.JSONDecodeError, AttributeError, TypeError):
             pass
 
@@ -228,6 +259,26 @@
         else:
             await self.backend_ws.send(message)  # type: ignore[union-attr, attr-defined]
 
+    def _make_disable_auto_response_message(self) -> str:
+        """Return a GA-shaped session.update that disables VAD auto-response.
+
+        The guardrail injection sites historically sent a beta-style payload
+        with a flat ``turn_detection`` key.  When the upstream operates in GA
+        mode (no ``OpenAI-Beta: realtime=v1`` header), the backend requires
+        the nested shape ``session.audio.input.turn_detection`` and a required
+        ``session.type`` field.  This helper always produces the correct shape
+        so both injection sites are GA-safe regardless of client protocol.
+        """
+        session: Dict[str, Any] = {
+            "type": "realtime",
+            "audio": {
+                "input": {
+                    "turn_detection": {"create_response": False},
+                }
+            },
+        }
+        return json.dumps({"type": "session.update", "session": session})
+
     def _has_realtime_guardrails(self) -> bool:
         """Return True if any callback is registered for realtime guardrail event types."""
         from litellm.integrations.custom_guardrail import CustomGuardrail
@@ -435,14 +486,7 @@
             ):
                 self.store_message(event_str)
                 await self.websocket.send_text(event_str)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 continue
             ## GUARDRAIL: run on transcription events in provider_config path too
             if (
@@ -484,14 +528,7 @@
             ):
                 self.store_message(raw_response)
                 await self.websocket.send_text(raw_response)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 return True
 
             if (
@@ -542,8 +579,21 @@
                         continue
                     ## LOGGING
                     self.store_message(raw_response)
-                    await self.websocket.send_text(raw_response)
 
+                    # If the client opted into beta protocol, translate GA event
+                    # names/shapes back to the beta equivalents before forwarding.
+                    if self._client_wants_beta:
+                        try:
+                            event_dict = json.loads(raw_response)
+                            translated = self._translate_event_to_beta(event_dict)
+                            if translated is None:
+                                continue  # drop GA-only events (e.g. conversation.item.done)
+                            await self.websocket.send_text(json.dumps(translated))
+                        except Exception:
+                            await self.websocket.send_text(raw_response)
+                    else:
+                        await self.websocket.send_text(raw_response)
+
         except websockets.exceptions.ConnectionClosed as e:  # type: ignore
             verbose_logger.exception(
                 f"Connection closed in backend to client send messages - {e}"
@@ -553,6 +603,175 @@
         finally:
             await self.log_messages()
 
+    @staticmethod
+    def _detect_beta_header(websocket: Any) -> bool:
+        """Return True if the client sent 'OpenAI-Beta: realtime=v1'.
+
+        Checks the raw ASGI scope headers so it works for both FastAPI WebSocket
+        objects and any test doubles that expose a .scope dict.
+        """
+        try:
+            headers = websocket.scope.get("headers", [])
+            for name, value in headers:
+                if isinstance(name, bytes):
+                    name = name.decode("latin-1")
+                if isinstance(value, bytes):
+                    value = value.decode("latin-1")
+                if name.lower() == "openai-beta" and "realtime=v1" in value.lower():
+                    return True
+        except Exception:
+            pass
+        return False
+
+    @staticmethod
+    def _remap_beta_session_to_ga(session: dict) -> dict:
+        """
+        Convert a beta-style session.update payload to the GA nested schema.
+
+        Beta → GA field mappings
+        ─────────────────────────────────────────────────────────────────────
+        session.type                    (inject "realtime" if absent)
+        session.modalities              → session.output_modalities
+        session.voice                   → session.audio.output.voice
+        session.input_audio_format      → session.audio.input.format  (with type/rate)
+        session.output_audio_format     → session.audio.output.format (with type/rate)
+        session.turn_detection          → session.audio.input.turn_detection
+        session.input_audio_transcription → session.audio.input.transcription
+        ─────────────────────────────────────────────────────────────────────
+        Fields not in the mapping (instructions, tools, etc.) are passed through.
+        GA clients that already use the nested shape are unaffected.
+        """
+        # Work on a shallow copy so we don't mutate the caller's dict
+        session = dict(session)
+
+        # 1. Ensure session.type is present
+        if "type" not in session:
+            session["type"] = "realtime"
+
+        # 2. Rename modalities → output_modalities and normalise combinations.
+        # Beta allowed ["audio", "text"] together; GA only supports ["audio"] or
+        # ["text"] as single-element lists. When both are present we prefer
+        # ["audio"] because audio mode already delivers transcripts via events.
+        if "modalities" in session:
+            mods = session.pop("modalities")
+            if "output_modalities" not in session:
+                mods_set = {m.lower() for m in (mods or [])}
+                if "audio" in mods_set:
+                    session["output_modalities"] = ["audio"]
+                elif "text" in mods_set:
+                    session["output_modalities"] = ["text"]
+
+        # 3-7. Lift flat audio fields into the nested audio object
+        audio: Dict[str, Any] = {}
+        inp: Dict[str, Any] = {}
+        out: Dict[str, Any] = {}
+
+        # voice → audio.output.voice
+        if "voice" in session:
+            out["voice"] = session.pop("voice")
+
+        # input_audio_format → audio.input.format
+        if "input_audio_format" in session:
+            raw = session.pop("input_audio_format")
+            inp["format"] = RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+
+        # output_audio_format → audio.output.format
+        if "output_audio_format" in session:
+            raw = session.pop("output_audio_format")
+            out["format"] = RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+
+        # turn_detection → audio.input.turn_detection
+        if "turn_detection" in session:
+            inp["turn_detection"] = session.pop("turn_detection")
+
+        # input_audio_transcription → audio.input.transcription
+        if "input_audio_transcription" in session:
+            inp["transcription"] = session.pop("input_audio_transcription")
+
+        if inp:
+            audio["input"] = inp
+        if out:
+            audio["output"] = out
+
+        if audio:
+            # Merge with any existing GA-style `audio` block the client already set,
+            # letting the remapped values take precedence within each sub-key.
+            existing = session.get("audio") or {}
+            for sub_key, sub_val in audio.items():
+                if (
+                    sub_key in existing
+                    and isinstance(existing[sub_key], dict)
+                    and isinstance(sub_val, dict)
+                ):
+                    existing[sub_key] = {**existing[sub_key], **sub_val}
+                else:
+                    existing[sub_key] = sub_val
+            session["audio"] = existing
+
+        return session
+
+    @staticmethod
+    def _translate_event_to_beta(event: dict) -> Optional[dict]:
+        """Translate a single GA event dict to its beta equivalent.
+
+        Returns None if the event should be dropped entirely (e.g. the GA-only
+        conversation.item.done has no beta counterpart).
+        Returns the (possibly mutated copy of the) event otherwise.
+        """
+        event_type = event.get("type", "")
+
+        # conversation.item.done has no beta equivalent — the client already
+        # received conversation.item.created (translated from .added).
+        if event_type == "conversation.item.done":
+            return None
+
+        # Shallow-copy so we don't mutate the stored message
+        translated = dict(event)
+
+        # Rename the type field
+        if event_type in RealTimeStreaming._GA_TO_BETA_EVENT_TYPES:
+            translated["type"] = RealTimeStreaming._GA_TO_BETA_EVENT_TYPES[event_type]
+
+        # Fix content block types inside items (response.done output list,
+        # conversation.item.created item content, etc.)
+        if "item" in translated and isinstance(translated["item"], dict):
+            translated["item"] = RealTimeStreaming._translate_item_content_types(
+                dict(translated["item"])
+            )
+        if "response" in translated and isinstance(translated["response"], dict):
+            resp = dict(translated["response"])
+            if "output" in resp and isinstance(resp["output"], list):
+                resp["output"] = [
+                    (
+                        RealTimeStreaming._translate_item_content_types(dict(o))
+                        if isinstance(o, dict)
+                        else o
+                    )
+                    for o in resp["output"]
+                ]
+            translated["response"] = resp
+
+        return translated
+
+    @staticmethod
+    def _translate_item_content_types(item: dict) -> dict:
+        """Replace GA content type names with beta names inside a single item."""
+        if "content" not in item or not isinstance(item["content"], list):
+            return item
+        new_content = []
+        for block in item["content"]:
+            if (
+                isinstance(block, dict)
+                and block.get("type") in RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES
+            ):
+                block = dict(block)
+                block["type"] = RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES[
+                    block["type"]
+                ]
+            new_content.append(block)
+        item["content"] = new_content
+        return item
+
     async def client_ack_messages(self):
         try:
             while True:
@@ -594,6 +813,17 @@
                         self._pending_guardrail_message = None
                         continue
 
+                    # GA compatibility: remap beta-style session fields to GA shape.
+                    # Always applied so that beta-format clients work against a GA
+                    # backend.  When _client_wants_beta is True the proxy also
+                    # translates GA events back to beta names on the return path.
+                    if msg_type == "session.update":
+                        session = msg_obj.get("session", {})
+                        if isinstance(session, dict):
+                            session = self._remap_beta_session_to_ga(session)
+                            msg_obj["session"] = session
+                            message = json.dumps(msg_obj)
+
                 except (json.JSONDecodeError, AttributeError):
                     pass
 
@@ -627,3 +857,8 @@
                     await forward_task
                 except asyncio.CancelledError:
                     pass
+
+
+def client_sent_openai_beta_realtime_header(websocket: Any) -> bool:
+    """True when the client WebSocket includes ``OpenAI-Beta: realtime=v1``."""
+    return RealTimeStreaming._detect_beta_header(websocket)

diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -5255,7 +5255,6 @@
             headers = {
                 "Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json",
-                "OpenAI-Beta": "realtime=v1",
             }
 
         if extra_headers:

diff --git a/litellm/llms/openai/realtime/handler.py b/litellm/llms/openai/realtime/handler.py
--- a/litellm/llms/openai/realtime/handler.py
+++ b/litellm/llms/openai/realtime/handler.py
@@ -6,12 +6,15 @@
 
 from typing import Any, Optional, cast
 
-from litellm._logging import _redact_string
+from litellm._logging import _redact_string, verbose_logger
 from litellm.constants import REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES
 from litellm.types.realtime import RealtimeQueryParams
 
 from ....litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-from ....litellm_core_utils.realtime_streaming import RealTimeStreaming
+from ....litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from ....llms.custom_httpx.http_handler import get_shared_realtime_ssl_context
 from ..openai import OpenAIChatCompletion
 
@@ -33,21 +36,24 @@
         """
         return "https://api.openai.com/"
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
-        Get additional headers beyond Authorization.
-        Override this in subclasses to customize headers (e.g., remove OpenAI-Beta).
+        Headers for the upstream OpenAI Realtime WebSocket.
 
-        Args:
-            api_key: API key for authentication
-
-        Returns:
-            Dictionary of additional headers
+        When the client sent ``OpenAI-Beta: realtime=v1`` on the proxy WebSocket,
+        ``openai_beta_realtime`` is True and the same header is forwarded upstream
+        so the legacy beta API is used. GA clients omit that header on the client
+        connection and must send GA-shaped ``session.update`` payloads.
         """
-        return {
-            "Authorization": f"Bearer {api_key}",
-            "OpenAI-Beta": "realtime=v1",
-        }
+        headers: dict = {"Authorization": f"Bearer {api_key}"}
+        if openai_beta_realtime:
+            headers["OpenAI-Beta"] = "realtime=v1"
+        return headers
 
     def _get_ssl_config(self, url: str) -> Any:
         """
@@ -120,8 +126,16 @@
             # Get provider-specific SSL configuration
             ssl_config = self._get_ssl_config(url)
 
-            # Get provider-specific headers
-            headers = self._get_additional_headers(api_key)
+            openai_beta_realtime = client_sent_openai_beta_realtime_header(websocket)
+            if not openai_beta_realtime:
+                verbose_logger.debug(
+                    "OpenAI Realtime: connecting with GA protocol (no OpenAI-Beta header). "
+                    "If your client expects beta event names, add 'OpenAI-Beta: realtime=v1' "
+                    "to the WebSocket headers sent to the LiteLLM proxy."
+                )
+            headers = self._get_additional_headers(
+                api_key, openai_beta_realtime=openai_beta_realtime
+            )
 
             # Log a masked request preview consistent with other endpoints.
             logging_obj.pre_call(

diff --git a/litellm/llms/xai/realtime/handler.py b/litellm/llms/xai/realtime/handler.py
--- a/litellm/llms/xai/realtime/handler.py
+++ b/litellm/llms/xai/realtime/handler.py
@@ -28,7 +28,12 @@
         """xAI uses a different API base URL."""
         return XAI_API_BASE
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
         xAI does NOT require the OpenAI-Beta header.
         Only send Authorization header.

diff --git a/litellm/types/llms/openai.py b/litellm/types/llms/openai.py
--- a/litellm/types/llms/openai.py
+++ b/litellm/types/llms/openai.py
@@ -1878,14 +1878,24 @@
 
 class OpenAIRealtimeStreamResponseOutputItemContent(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, used for 'input_audio' content types"""
+    """Base64-encoded audio bytes, used for 'input_audio' / 'audio' / 'output_audio' content types"""
     id: str
     """The ID of the previous conversation item for reference"""
     text: str
-    """The text content, used for 'input_text' and 'text' content types"""
+    """The text content, used for 'input_text' / 'text' / 'output_text' content types"""
     transcript: str
-    """The transcript content, used for 'input_audio' content types"""
-    type: Literal["input_audio", "input_text", "text", "item_reference", "audio"]
+    """The transcript content, used for 'input_audio' / 'audio' content types"""
+    type: Literal[
+        "input_audio",
+        "input_text",
+        # Beta assistant content types
+        "text",
+        "audio",
+        "item_reference",
+        # GA assistant content types (aligns with Responses API)
+        "output_text",
+        "output_audio",
+    ]
     """The type of content"""
 
 
@@ -1945,23 +1955,46 @@
 
 
 class OpenAIRealtimeConversationItemCreated(TypedDict, total=False):
+    """Beta: single event emitted when a conversation item is created."""
+
     type: Required[Literal["conversation.item.created"]]
     item: OpenAIRealtimeStreamResponseOutputItem
     event_id: str
-    previous_item_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
 
 
+class OpenAIRealtimeConversationItemAdded(TypedDict, total=False):
+    """GA: emitted immediately when a conversation item is added (replaces .created)."""
+
+    type: Required[Literal["conversation.item.added"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
+class OpenAIRealtimeConversationItemDone(TypedDict, total=False):
+    """GA: emitted when a conversation item is fully complete (e.g. transcription done)."""
+
+    type: Required[Literal["conversation.item.done"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
 class OpenAIRealtimeResponseContentPart(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, if type is 'audio'"""
+    """Base64-encoded audio bytes, if type is 'audio' or 'output_audio'"""
 
     text: str
-    """The text content, if type is 'text'"""
+    """The text content, if type is 'text' or 'output_text'"""
 
     transcript: str
-    """The transcript content, if type is 'audio'"""
+    """The transcript content, if type is 'audio' or 'output_audio'"""
 
-    type: Literal["audio", "text"]
+    type: Union[
+        Literal["audio", "text"],  # beta
+        Literal["output_audio", "output_text"],  # GA
+    ]
     """The type of content"""
 
 
@@ -1982,7 +2015,14 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Union[Literal["response.text.delta"], Literal["response.audio.delta"]]
+    type: Union[
+        Literal["response.text.delta"],
+        Literal["response.audio.delta"],
+        # GA renamed events
+        Literal["response.output_text.delta"],
+        Literal["response.output_audio.delta"],
+        Literal["response.output_audio_transcript.delta"],
+    ]
 
 
 class OpenAIRealtimeResponseTextDone(TypedDict):
@@ -1992,7 +2032,10 @@
     output_index: int
     response_id: str
     text: str
-    type: Literal["response.text.done"]
+    type: Union[
+        Literal["response.text.done"],
+        Literal["response.output_text.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeResponseAudioDone(TypedDict):
@@ -2001,7 +2044,11 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Literal["response.audio.done"]
+    type: Union[
+        Literal["response.audio.done"],
+        Literal["response.output_audio.done"],  # GA rename
+        Literal["response.output_audio_transcript.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeContentPartDone(TypedDict):
@@ -2046,10 +2093,18 @@
 
 class OpenAIRealtimeEventTypes(Enum):
     SESSION_CREATED = "session.created"
+    # Beta delta event names
     RESPONSE_TEXT_DELTA = "response.text.delta"
     RESPONSE_AUDIO_DELTA = "response.audio.delta"
     RESPONSE_TEXT_DONE = "response.text.done"
     RESPONSE_AUDIO_DONE = "response.audio.done"
+    # GA renamed delta event names
+    RESPONSE_OUTPUT_TEXT_DELTA = "response.output_text.delta"
+    RESPONSE_OUTPUT_AUDIO_DELTA = "response.output_audio.delta"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DELTA = "response.output_audio_transcript.delta"
+    RESPONSE_OUTPUT_TEXT_DONE = "response.output_text.done"
+    RESPONSE_OUTPUT_AUDIO_DONE = "response.output_audio.done"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DONE = "response.output_audio_transcript.done"
     RESPONSE_DONE = "response.done"
     RESPONSE_OUTPUT_ITEM_ADDED = "response.output_item.added"
     RESPONSE_CONTENT_PART_ADDED = "response.content_part.added"
@@ -2060,7 +2115,11 @@
     OpenAIRealtimeStreamSessionEvents,
     OpenAIRealtimeStreamResponseOutputItemAdded,
     OpenAIRealtimeResponseContentPartAdded,
+    # Beta conversation item event
     OpenAIRealtimeConversationItemCreated,
+    # GA conversation item events
+    OpenAIRealtimeConversationItemAdded,
+    OpenAIRealtimeConversationItemDone,
     OpenAIRealtimeConversationCreated,
     OpenAIRealtimeResponseDelta,
     OpenAIRealtimeResponseTextDone,

diff --git a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
--- a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
+++ b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
@@ -13,7 +13,10 @@
 )  # Adds the parent directory to the system path
 
 from litellm.integrations.custom_guardrail import CustomGuardrail
-from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
+from litellm.litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.llms.openai import (
     OpenAIRealtimeStreamResponseBaseObject,
@@ -80,6 +83,68 @@
     assert len(streaming.messages) == 2  # Should not store the new message
 
 
+def test_remap_beta_session_to_ga_normalizes_modalities_and_audio():
+    out = RealTimeStreaming._remap_beta_session_to_ga(
+        {"modalities": ["audio", "text"], "voice": "alloy"}
+    )
+    assert out["type"] == "realtime"
+    assert out["output_modalities"] == ["audio"]
+    assert out["audio"]["output"]["voice"] == "alloy"
+
+
+def test_make_disable_auto_response_message_produces_ga_shape():
+    """_make_disable_auto_response_message must produce a GA-shaped session.update.
+
+    The GA Realtime API requires:
+      - session.type = "realtime"
+      - turn_detection nested at session.audio.input.turn_detection
+    The old beta-style flat ``session.turn_detection`` is rejected by GA upstreams.
+    """
+    websocket = MagicMock()
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert (
+        session.get("type") == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    # turn_detection must NOT be at the flat beta location
+    assert (
+        "turn_detection" not in session
+    ), "turn_detection must not be at the top-level session (beta shape); use audio.input"
+    # turn_detection must be nested under audio.input
+    assert session["audio"]["input"]["turn_detection"]["create_response"] is False
+
+
+def test_translate_event_to_beta_renames_delta_types():
+    ev = RealTimeStreaming._translate_event_to_beta(
+        {"type": "response.output_audio.delta", "delta": "abc", "event_id": "e1"}
+    )
+    assert ev is not None
+    assert ev["type"] == "response.audio.delta"
+
+
+def test_translate_event_to_beta_drops_conversation_item_done():
+    assert (
+        RealTimeStreaming._translate_event_to_beta({"type": "conversation.item.done"})
+        is None
+    )
+
+
+def test_client_sent_openai_beta_realtime_header_detects_header():
+    ws = MagicMock()
+    ws.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    assert client_sent_openai_beta_realtime_header(ws) is True
+    empty = MagicMock()
+    empty.scope = {"headers": []}
+    assert client_sent_openai_beta_realtime_header(empty) is False
+
+
 def test_collect_user_input_from_text_conversation_item():
     """
     Test that conversation.item.create with input_text content is collected as user input.
@@ -761,7 +826,14 @@
     assert (
         len(session_updates) == 1
     ), f"Expected one session.update injected to backend, got: {sent_to_backend}"
-    assert session_updates[0]["session"]["turn_detection"]["create_response"] is False
+    # GA shape: turn_detection must be nested under audio.input, not at top-level session
+    injected_session = session_updates[0]["session"]
+    assert (
+        injected_session["type"] == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    assert (
+        injected_session["audio"]["input"]["turn_detection"]["create_response"] is False
+    ), "GA session.update must nest turn_detection under audio.input"
 
     litellm.callbacks = []  # cleanup
 
@@ -818,7 +890,14 @@
     assert (
         len(session_updates) == 1
     ), f"pre_call guardrail should inject session.update to gate audio responses, got: {sent_to_backend}"
-    assert session_updates[0]["session"]["turn_detection"]["create_response"] is False
+    # GA shape: turn_detection must be nested under audio.input, not at top-level session
+    injected_session = session_updates[0]["session"]
+    assert (
+        injected_session["type"] == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    assert (
+        injected_session["audio"]["input"]["turn_detection"]["create_response"] is False
+    ), "GA session.update must nest turn_detection under audio.input"
 
     litellm.callbacks = []  # cleanup
 

diff --git a/tests/test_litellm/llms/openai/realtime/test_openai_realtime_handler.py b/tests/test_litellm/llms/openai/realtime/test_openai_realtime_handler.py
--- a/tests/test_litellm/llms/openai/realtime/test_openai_realtime_handler.py
+++ b/tests/test_litellm/llms/openai/realtime/test_openai_realtime_handler.py
@@ -213,12 +213,12 @@
         assert called_url.startswith("wss://api.openai.com/v1/realtime?")
         assert f"model={model}" in called_url
 
-        # Verify proper headers were set
+        # Verify proper headers were set (GA default: no OpenAI-Beta unless client sent it)
         called_kwargs = mock_ws_connect.call_args[1]
         assert "additional_headers" in called_kwargs
         additional_headers = called_kwargs["additional_headers"]
         assert additional_headers["Authorization"] == f"Bearer {api_key}"
-        assert additional_headers["OpenAI-Beta"] == "realtime=v1"
+        assert "OpenAI-Beta" not in additional_headers
         # Verify SSL is configured (should be an SSLContext or True, not None or False)
         assert called_kwargs["ssl"] is not None
         assert called_kwargs["ssl"] is not False
@@ -228,6 +228,65 @@
 
 
 @pytest.mark.asyncio
+async def test_async_realtime_forwards_openai_beta_header_when_client_sends_it():
+    """Upstream WS gets OpenAI-Beta: realtime=v1 only when the client WebSocket included it."""
+    from litellm.llms.openai.realtime.handler import OpenAIRealtime
+    from litellm.types.realtime import RealtimeQueryParams
+
+    handler = OpenAIRealtime()
+    api_base = "https://api.openai.com/"
+    api_key = "test-key"
+    model = "gpt-4o-mini-realtime-preview"
+    query_params: RealtimeQueryParams = {"model": model}
... diff truncated: showing 800 of 936 lines

You can send follow-ups to the cloud agent here.

Comment thread litellm/litellm_core_utils/realtime_streaming.py Outdated
@CLAassistant

CLAassistant commented May 4, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ Sameerlite
✅ mateo-berri
❌ cursoragent
You have signed the CLA already but the status is still pending? Let us recheck it.

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

bugbot run

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

@greptile-apps re review

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ Sameerlite
❌ cursoragent
You have signed the CLA already but the status is still pending? Let us recheck it.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: GA session format sent to beta-mode upstream
    • Beta realtime clients now keep flat beta session.update payloads while GA clients still get GA remapping, including guardrail auto-response disabling.
Preview (e33fd0ddcf)
diff --git a/litellm/litellm_core_utils/realtime_streaming.py b/litellm/litellm_core_utils/realtime_streaming.py
--- a/litellm/litellm_core_utils/realtime_streaming.py
+++ b/litellm/litellm_core_utils/realtime_streaming.py
@@ -31,6 +31,8 @@
     "session.created",
     "response.create",
     "response.done",
+    "conversation.item.added",  # GA
+    "conversation.item.done",  # GA
 ]
 
 
@@ -54,6 +56,9 @@
         self.session_tools: List[Dict] = []
         self.tool_calls: List[Dict] = []
 
+        # Detect whether the client is explicitly opting into the beta protocol.
+        self._client_wants_beta = self._detect_beta_header(websocket)
+
         _logged_real_time_event_types = litellm.logged_real_time_event_types
 
         if _logged_real_time_event_types is None:
@@ -76,6 +81,27 @@
         # response.create can be rewritten to include the failure context.
         self._pending_guardrail_message: Optional[str] = None
 
+    _SESSION_EVENT_TYPES = frozenset(["session.created", "session.updated"])
+    _AUDIO_FORMAT_MAP: Dict[str, Dict[str, Any]] = {
+        "pcm16": {"type": "audio/pcm", "rate": 24000},
+        "g711_ulaw": {"type": "audio/G711-ulaw", "rate": 8000},
+        "g711_alaw": {"type": "audio/G711-alaw", "rate": 8000},
+    }
+    # GA name → beta name (when client WebSocket includes OpenAI-Beta: realtime=v1)
+    _GA_TO_BETA_EVENT_TYPES: Dict[str, str] = {
+        "conversation.item.added": "conversation.item.created",
+        "response.output_text.delta": "response.text.delta",
+        "response.output_audio.delta": "response.audio.delta",
+        "response.output_audio_transcript.delta": "response.audio_transcript.delta",
+        "response.output_text.done": "response.text.done",
+        "response.output_audio.done": "response.audio.done",
+        "response.output_audio_transcript.done": "response.audio_transcript.done",
+    }
+    _GA_TO_BETA_CONTENT_TYPES: Dict[str, str] = {
+        "output_text": "text",
+        "output_audio": "audio",
+    }
+
     def _should_store_message(
         self,
         message_obj: Union[dict, OpenAIRealtimeEvents],
@@ -92,24 +118,27 @@
         if isinstance(message, bytes):
             message = message.decode("utf-8")
         if isinstance(message, dict):
-            message_obj = message
+            # TypedDict union members do not narrow to plain dict for mypy.
+            message_obj: Dict[str, Any] = cast(Dict[str, Any], message)
         else:
-            message_obj = json.loads(message)
+            message_obj = cast(Dict[str, Any], json.loads(cast(str, message)))
         self._collect_tool_calls_from_response_done(cast(dict, message_obj))
         try:
-            if (
-                not isinstance(message, dict)
-                or message_obj.get("type") == "session.created"
-                or message_obj.get("type") == "session.updated"
-            ):
-                message_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
-            elif not isinstance(message, dict):
-                message_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
+            event_type = message_obj.get("type", "")
+            if event_type in self._SESSION_EVENT_TYPES:
+                typed_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
+            else:
+                # Use the base object as a safe catch-all for all other event types
+                # (both beta and GA), so unknown/new event names never raise here.
+                typed_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
         except Exception as e:
             verbose_logger.debug(f"Error parsing message for logging: {e}")
-            raise e
-        if self._should_store_message(message_obj):
-            self.messages.append(message_obj)
+            # Don't re-raise — a parse failure must not drop or delay the message
+            if self._should_store_message(message_obj):
+                self.messages.append(message_obj)  # type: ignore[arg-type]
+            return
+        if self._should_store_message(typed_obj):
+            self.messages.append(typed_obj)
 
     def _collect_user_input_from_client_event(self, message: Union[str, dict]) -> None:
         """Extract user text content from client WebSocket events for spend logging."""
@@ -147,6 +176,8 @@
                 tools = session.get("tools")
                 if tools and isinstance(tools, list):
                     self.session_tools = tools
+                # GA: session.type is required; log it for traceability but no action needed
+                verbose_logger.debug(f"Realtime session.type: {session.get('type')}")
         except (json.JSONDecodeError, AttributeError, TypeError):
             pass
 
@@ -228,6 +259,23 @@
         else:
             await self.backend_ws.send(message)  # type: ignore[union-attr, attr-defined]
 
+    def _make_disable_auto_response_message(self) -> str:
+        """Return a session.update that disables VAD auto-response."""
+        if self._client_wants_beta:
+            session: Dict[str, Any] = {
+                "turn_detection": {"create_response": False},
+            }
+        else:
+            session = {
+                "type": "realtime",
+                "audio": {
+                    "input": {
+                        "turn_detection": {"create_response": False},
+                    }
+                },
+            }
+        return json.dumps({"type": "session.update", "session": session})
+
     def _has_realtime_guardrails(self) -> bool:
         """Return True if any callback is registered for realtime guardrail event types."""
         from litellm.integrations.custom_guardrail import CustomGuardrail
@@ -435,14 +483,7 @@
             ):
                 self.store_message(event_str)
                 await self.websocket.send_text(event_str)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 continue
             ## GUARDRAIL: run on transcription events in provider_config path too
             if (
@@ -484,14 +525,7 @@
             ):
                 self.store_message(raw_response)
                 await self.websocket.send_text(raw_response)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 return True
 
             if (
@@ -542,8 +576,21 @@
                         continue
                     ## LOGGING
                     self.store_message(raw_response)
-                    await self.websocket.send_text(raw_response)
 
+                    # If the client opted into beta protocol, translate GA event
+                    # names/shapes back to the beta equivalents before forwarding.
+                    if self._client_wants_beta:
+                        try:
+                            event_dict = json.loads(raw_response)
+                            translated = self._translate_event_to_beta(event_dict)
+                            if translated is None:
+                                continue  # drop GA-only events (e.g. conversation.item.done)
+                            await self.websocket.send_text(json.dumps(translated))
+                        except Exception:
+                            await self.websocket.send_text(raw_response)
+                    else:
+                        await self.websocket.send_text(raw_response)
+
         except websockets.exceptions.ConnectionClosed as e:  # type: ignore
             verbose_logger.exception(
                 f"Connection closed in backend to client send messages - {e}"
@@ -553,6 +600,175 @@
         finally:
             await self.log_messages()
 
+    @staticmethod
+    def _detect_beta_header(websocket: Any) -> bool:
+        """Return True if the client sent 'OpenAI-Beta: realtime=v1'.
+
+        Checks the raw ASGI scope headers so it works for both FastAPI WebSocket
+        objects and any test doubles that expose a .scope dict.
+        """
+        try:
+            headers = websocket.scope.get("headers", [])
+            for name, value in headers:
+                if isinstance(name, bytes):
+                    name = name.decode("latin-1")
+                if isinstance(value, bytes):
+                    value = value.decode("latin-1")
+                if name.lower() == "openai-beta" and "realtime=v1" in value.lower():
+                    return True
+        except Exception:
+            pass
+        return False
+
+    @staticmethod
+    def _remap_beta_session_to_ga(session: dict) -> dict:
+        """
+        Convert a beta-style session.update payload to the GA nested schema.
+
+        Beta → GA field mappings
+        ─────────────────────────────────────────────────────────────────────
+        session.type                    (inject "realtime" if absent)
+        session.modalities              → session.output_modalities
+        session.voice                   → session.audio.output.voice
+        session.input_audio_format      → session.audio.input.format  (with type/rate)
+        session.output_audio_format     → session.audio.output.format (with type/rate)
+        session.turn_detection          → session.audio.input.turn_detection
+        session.input_audio_transcription → session.audio.input.transcription
+        ─────────────────────────────────────────────────────────────────────
+        Fields not in the mapping (instructions, tools, etc.) are passed through.
+        GA clients that already use the nested shape are unaffected.
+        """
+        # Work on a shallow copy so we don't mutate the caller's dict
+        session = dict(session)
+
+        # 1. Ensure session.type is present
+        if "type" not in session:
+            session["type"] = "realtime"
+
+        # 2. Rename modalities → output_modalities and normalise combinations.
+        # Beta allowed ["audio", "text"] together; GA only supports ["audio"] or
+        # ["text"] as single-element lists. When both are present we prefer
+        # ["audio"] because audio mode already delivers transcripts via events.
+        if "modalities" in session:
+            mods = session.pop("modalities")
+            if "output_modalities" not in session:
+                mods_set = {m.lower() for m in (mods or [])}
+                if "audio" in mods_set:
+                    session["output_modalities"] = ["audio"]
+                elif "text" in mods_set:
+                    session["output_modalities"] = ["text"]
+
+        # 3-7. Lift flat audio fields into the nested audio object
+        audio: Dict[str, Any] = {}
+        inp: Dict[str, Any] = {}
+        out: Dict[str, Any] = {}
+
+        # voice → audio.output.voice
+        if "voice" in session:
+            out["voice"] = session.pop("voice")
+
+        # input_audio_format → audio.input.format
+        if "input_audio_format" in session:
+            raw = session.pop("input_audio_format")
+            inp["format"] = RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+
+        # output_audio_format → audio.output.format
+        if "output_audio_format" in session:
+            raw = session.pop("output_audio_format")
+            out["format"] = RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+
+        # turn_detection → audio.input.turn_detection
+        if "turn_detection" in session:
+            inp["turn_detection"] = session.pop("turn_detection")
+
+        # input_audio_transcription → audio.input.transcription
+        if "input_audio_transcription" in session:
+            inp["transcription"] = session.pop("input_audio_transcription")
+
+        if inp:
+            audio["input"] = inp
+        if out:
+            audio["output"] = out
+
+        if audio:
+            # Merge with any existing GA-style `audio` block the client already set,
+            # letting the remapped values take precedence within each sub-key.
+            existing = session.get("audio") or {}
+            for sub_key, sub_val in audio.items():
+                if (
+                    sub_key in existing
+                    and isinstance(existing[sub_key], dict)
+                    and isinstance(sub_val, dict)
+                ):
+                    existing[sub_key] = {**existing[sub_key], **sub_val}
+                else:
+                    existing[sub_key] = sub_val
+            session["audio"] = existing
+
+        return session
+
+    @staticmethod
+    def _translate_event_to_beta(event: dict) -> Optional[dict]:
+        """Translate a single GA event dict to its beta equivalent.
+
+        Returns None if the event should be dropped entirely (e.g. the GA-only
+        conversation.item.done has no beta counterpart).
+        Returns the (possibly mutated copy of the) event otherwise.
+        """
+        event_type = event.get("type", "")
+
+        # conversation.item.done has no beta equivalent — the client already
+        # received conversation.item.created (translated from .added).
+        if event_type == "conversation.item.done":
+            return None
+
+        # Shallow-copy so we don't mutate the stored message
+        translated = dict(event)
+
+        # Rename the type field
+        if event_type in RealTimeStreaming._GA_TO_BETA_EVENT_TYPES:
+            translated["type"] = RealTimeStreaming._GA_TO_BETA_EVENT_TYPES[event_type]
+
+        # Fix content block types inside items (response.done output list,
+        # conversation.item.created item content, etc.)
+        if "item" in translated and isinstance(translated["item"], dict):
+            translated["item"] = RealTimeStreaming._translate_item_content_types(
+                dict(translated["item"])
+            )
+        if "response" in translated and isinstance(translated["response"], dict):
+            resp = dict(translated["response"])
+            if "output" in resp and isinstance(resp["output"], list):
+                resp["output"] = [
+                    (
+                        RealTimeStreaming._translate_item_content_types(dict(o))
+                        if isinstance(o, dict)
+                        else o
+                    )
+                    for o in resp["output"]
+                ]
+            translated["response"] = resp
+
+        return translated
+
+    @staticmethod
+    def _translate_item_content_types(item: dict) -> dict:
+        """Replace GA content type names with beta names inside a single item."""
+        if "content" not in item or not isinstance(item["content"], list):
+            return item
+        new_content = []
+        for block in item["content"]:
+            if (
+                isinstance(block, dict)
+                and block.get("type") in RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES
+            ):
+                block = dict(block)
+                block["type"] = RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES[
+                    block["type"]
+                ]
+            new_content.append(block)
+        item["content"] = new_content
+        return item
+
     async def client_ack_messages(self):
         try:
             while True:
@@ -594,6 +810,16 @@
                         self._pending_guardrail_message = None
                         continue
 
+                    # GA compatibility: remap beta-style session fields only when
+                    # the upstream is in GA mode. Beta upstreams expect the flat
+                    # session shape unchanged.
+                    if msg_type == "session.update" and not self._client_wants_beta:
+                        session = msg_obj.get("session", {})
+                        if isinstance(session, dict):
+                            session = self._remap_beta_session_to_ga(session)
+                            msg_obj["session"] = session
+                            message = json.dumps(msg_obj)
+
                 except (json.JSONDecodeError, AttributeError):
                     pass
 
@@ -627,3 +853,8 @@
                     await forward_task
                 except asyncio.CancelledError:
                     pass
+
+
+def client_sent_openai_beta_realtime_header(websocket: Any) -> bool:
+    """True when the client WebSocket includes ``OpenAI-Beta: realtime=v1``."""
+    return RealTimeStreaming._detect_beta_header(websocket)

diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -5255,7 +5255,6 @@
             headers = {
                 "Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json",
-                "OpenAI-Beta": "realtime=v1",
             }
 
         if extra_headers:

diff --git a/litellm/llms/openai/realtime/handler.py b/litellm/llms/openai/realtime/handler.py
--- a/litellm/llms/openai/realtime/handler.py
+++ b/litellm/llms/openai/realtime/handler.py
@@ -6,12 +6,15 @@
 
 from typing import Any, Optional, cast
 
-from litellm._logging import _redact_string
+from litellm._logging import _redact_string, verbose_logger
 from litellm.constants import REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES
 from litellm.types.realtime import RealtimeQueryParams
 
 from ....litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-from ....litellm_core_utils.realtime_streaming import RealTimeStreaming
+from ....litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from ....llms.custom_httpx.http_handler import get_shared_realtime_ssl_context
 from ..openai import OpenAIChatCompletion
 
@@ -33,21 +36,24 @@
         """
         return "https://api.openai.com/"
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
-        Get additional headers beyond Authorization.
-        Override this in subclasses to customize headers (e.g., remove OpenAI-Beta).
+        Headers for the upstream OpenAI Realtime WebSocket.
 
-        Args:
-            api_key: API key for authentication
-
-        Returns:
-            Dictionary of additional headers
+        When the client sent ``OpenAI-Beta: realtime=v1`` on the proxy WebSocket,
+        ``openai_beta_realtime`` is True and the same header is forwarded upstream
+        so the legacy beta API is used. GA clients omit that header on the client
+        connection and must send GA-shaped ``session.update`` payloads.
         """
-        return {
-            "Authorization": f"Bearer {api_key}",
-            "OpenAI-Beta": "realtime=v1",
-        }
+        headers: dict = {"Authorization": f"Bearer {api_key}"}
+        if openai_beta_realtime:
+            headers["OpenAI-Beta"] = "realtime=v1"
+        return headers
 
     def _get_ssl_config(self, url: str) -> Any:
         """
@@ -120,8 +126,16 @@
             # Get provider-specific SSL configuration
             ssl_config = self._get_ssl_config(url)
 
-            # Get provider-specific headers
-            headers = self._get_additional_headers(api_key)
+            openai_beta_realtime = client_sent_openai_beta_realtime_header(websocket)
+            if not openai_beta_realtime:
+                verbose_logger.debug(
+                    "OpenAI Realtime: connecting with GA protocol (no OpenAI-Beta header). "
+                    "If your client expects beta event names, add 'OpenAI-Beta: realtime=v1' "
+                    "to the WebSocket headers sent to the LiteLLM proxy."
+                )
+            headers = self._get_additional_headers(
+                api_key, openai_beta_realtime=openai_beta_realtime
+            )
 
             # Log a masked request preview consistent with other endpoints.
             logging_obj.pre_call(

diff --git a/litellm/llms/xai/realtime/handler.py b/litellm/llms/xai/realtime/handler.py
--- a/litellm/llms/xai/realtime/handler.py
+++ b/litellm/llms/xai/realtime/handler.py
@@ -28,7 +28,12 @@
         """xAI uses a different API base URL."""
         return XAI_API_BASE
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
         xAI does NOT require the OpenAI-Beta header.
         Only send Authorization header.

diff --git a/litellm/types/llms/openai.py b/litellm/types/llms/openai.py
--- a/litellm/types/llms/openai.py
+++ b/litellm/types/llms/openai.py
@@ -1878,14 +1878,24 @@
 
 class OpenAIRealtimeStreamResponseOutputItemContent(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, used for 'input_audio' content types"""
+    """Base64-encoded audio bytes, used for 'input_audio' / 'audio' / 'output_audio' content types"""
     id: str
     """The ID of the previous conversation item for reference"""
     text: str
-    """The text content, used for 'input_text' and 'text' content types"""
+    """The text content, used for 'input_text' / 'text' / 'output_text' content types"""
     transcript: str
-    """The transcript content, used for 'input_audio' content types"""
-    type: Literal["input_audio", "input_text", "text", "item_reference", "audio"]
+    """The transcript content, used for 'input_audio' / 'audio' content types"""
+    type: Literal[
+        "input_audio",
+        "input_text",
+        # Beta assistant content types
+        "text",
+        "audio",
+        "item_reference",
+        # GA assistant content types (aligns with Responses API)
+        "output_text",
+        "output_audio",
+    ]
     """The type of content"""
 
 
@@ -1945,23 +1955,46 @@
 
 
 class OpenAIRealtimeConversationItemCreated(TypedDict, total=False):
+    """Beta: single event emitted when a conversation item is created."""
+
     type: Required[Literal["conversation.item.created"]]
     item: OpenAIRealtimeStreamResponseOutputItem
     event_id: str
-    previous_item_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
 
 
+class OpenAIRealtimeConversationItemAdded(TypedDict, total=False):
+    """GA: emitted immediately when a conversation item is added (replaces .created)."""
+
+    type: Required[Literal["conversation.item.added"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
+class OpenAIRealtimeConversationItemDone(TypedDict, total=False):
+    """GA: emitted when a conversation item is fully complete (e.g. transcription done)."""
+
+    type: Required[Literal["conversation.item.done"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
 class OpenAIRealtimeResponseContentPart(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, if type is 'audio'"""
+    """Base64-encoded audio bytes, if type is 'audio' or 'output_audio'"""
 
     text: str
-    """The text content, if type is 'text'"""
+    """The text content, if type is 'text' or 'output_text'"""
 
     transcript: str
-    """The transcript content, if type is 'audio'"""
+    """The transcript content, if type is 'audio' or 'output_audio'"""
 
-    type: Literal["audio", "text"]
+    type: Union[
+        Literal["audio", "text"],  # beta
+        Literal["output_audio", "output_text"],  # GA
+    ]
     """The type of content"""
 
 
@@ -1982,7 +2015,14 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Union[Literal["response.text.delta"], Literal["response.audio.delta"]]
+    type: Union[
+        Literal["response.text.delta"],
+        Literal["response.audio.delta"],
+        # GA renamed events
+        Literal["response.output_text.delta"],
+        Literal["response.output_audio.delta"],
+        Literal["response.output_audio_transcript.delta"],
+    ]
 
 
 class OpenAIRealtimeResponseTextDone(TypedDict):
@@ -1992,7 +2032,10 @@
     output_index: int
     response_id: str
     text: str
-    type: Literal["response.text.done"]
+    type: Union[
+        Literal["response.text.done"],
+        Literal["response.output_text.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeResponseAudioDone(TypedDict):
@@ -2001,7 +2044,11 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Literal["response.audio.done"]
+    type: Union[
+        Literal["response.audio.done"],
+        Literal["response.output_audio.done"],  # GA rename
+        Literal["response.output_audio_transcript.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeContentPartDone(TypedDict):
@@ -2046,10 +2093,18 @@
 
 class OpenAIRealtimeEventTypes(Enum):
     SESSION_CREATED = "session.created"
+    # Beta delta event names
     RESPONSE_TEXT_DELTA = "response.text.delta"
     RESPONSE_AUDIO_DELTA = "response.audio.delta"
     RESPONSE_TEXT_DONE = "response.text.done"
     RESPONSE_AUDIO_DONE = "response.audio.done"
+    # GA renamed delta event names
+    RESPONSE_OUTPUT_TEXT_DELTA = "response.output_text.delta"
+    RESPONSE_OUTPUT_AUDIO_DELTA = "response.output_audio.delta"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DELTA = "response.output_audio_transcript.delta"
+    RESPONSE_OUTPUT_TEXT_DONE = "response.output_text.done"
+    RESPONSE_OUTPUT_AUDIO_DONE = "response.output_audio.done"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DONE = "response.output_audio_transcript.done"
     RESPONSE_DONE = "response.done"
     RESPONSE_OUTPUT_ITEM_ADDED = "response.output_item.added"
     RESPONSE_CONTENT_PART_ADDED = "response.content_part.added"
@@ -2060,7 +2115,11 @@
     OpenAIRealtimeStreamSessionEvents,
     OpenAIRealtimeStreamResponseOutputItemAdded,
     OpenAIRealtimeResponseContentPartAdded,
+    # Beta conversation item event
     OpenAIRealtimeConversationItemCreated,
+    # GA conversation item events
+    OpenAIRealtimeConversationItemAdded,
+    OpenAIRealtimeConversationItemDone,
     OpenAIRealtimeConversationCreated,
     OpenAIRealtimeResponseDelta,
     OpenAIRealtimeResponseTextDone,

diff --git a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
--- a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
+++ b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
@@ -13,7 +13,10 @@
 )  # Adds the parent directory to the system path
 
 from litellm.integrations.custom_guardrail import CustomGuardrail
-from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
+from litellm.litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.llms.openai import (
     OpenAIRealtimeStreamResponseBaseObject,
@@ -80,6 +83,121 @@
     assert len(streaming.messages) == 2  # Should not store the new message
 
 
+def test_remap_beta_session_to_ga_normalizes_modalities_and_audio():
+    out = RealTimeStreaming._remap_beta_session_to_ga(
+        {"modalities": ["audio", "text"], "voice": "alloy"}
+    )
+    assert out["type"] == "realtime"
+    assert out["output_modalities"] == ["audio"]
+    assert out["audio"]["output"]["voice"] == "alloy"
+
+
+def test_make_disable_auto_response_message_produces_ga_shape():
+    """_make_disable_auto_response_message must produce a GA-shaped session.update.
+
+    The GA Realtime API requires:
+      - session.type = "realtime"
+      - turn_detection nested at session.audio.input.turn_detection
+    The old beta-style flat ``session.turn_detection`` is rejected by GA upstreams.
+    """
+    websocket = MagicMock()
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert (
+        session.get("type") == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    # turn_detection must NOT be at the flat beta location
+    assert (
+        "turn_detection" not in session
+    ), "turn_detection must not be at the top-level session (beta shape); use audio.input"
+    # turn_detection must be nested under audio.input
+    assert session["audio"]["input"]["turn_detection"]["create_response"] is False
+
+
+def test_make_disable_auto_response_message_produces_beta_shape_for_beta_clients():
+    websocket = MagicMock()
+    websocket.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert session == {"turn_detection": {"create_response": False}}
+
+
+@pytest.mark.asyncio
+async def test_client_ack_messages_keeps_beta_session_shape_for_beta_clients():
+    client_ws = MagicMock()
+    client_ws.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    session_update = json.dumps(
+        {
+            "type": "session.update",
+            "session": {
+                "modalities": ["audio", "text"],
+                "voice": "alloy",
+                "turn_detection": {"create_response": False},
+            },
+        }
+    )
+    client_ws.receive_text = AsyncMock(
+        side_effect=[
+            session_update,
+            Exception("connection closed"),
+        ]
+    )
+    backend_ws = MagicMock()
+    backend_ws.send = AsyncMock()
+    logging_obj = MagicMock()
+    logging_obj.pre_call = MagicMock()
+    streaming = RealTimeStreaming(client_ws, backend_ws, logging_obj)
+
+    await streaming.client_ack_messages()
+
+    sent_to_backend = json.loads(backend_ws.send.call_args_list[0].args[0])
+    session = sent_to_backend["session"]
+    assert session["modalities"] == ["audio", "text"]
+    assert session["voice"] == "alloy"
+    assert session["turn_detection"] == {"create_response": False}
+    assert "type" not in session
+    assert "output_modalities" not in session
+    assert "audio" not in session
+
+
+def test_translate_event_to_beta_renames_delta_types():
+    ev = RealTimeStreaming._translate_event_to_beta(
+        {"type": "response.output_audio.delta", "delta": "abc", "event_id": "e1"}
+    )
+    assert ev is not None
+    assert ev["type"] == "response.audio.delta"
+
+
+def test_translate_event_to_beta_drops_conversation_item_done():
+    assert (
+        RealTimeStreaming._translate_event_to_beta({"type": "conversation.item.done"})
+        is None
+    )
+
+
+def test_client_sent_openai_beta_realtime_header_detects_header():
+    ws = MagicMock()
+    ws.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    assert client_sent_openai_beta_realtime_header(ws) is True
+    empty = MagicMock()
+    empty.scope = {"headers": []}
+    assert client_sent_openai_beta_realtime_header(empty) is False
+
+
 def test_collect_user_input_from_text_conversation_item():
     """
     Test that conversation.item.create with input_text content is collected as user input.
@@ -761,7 +879,14 @@
     assert (
         len(session_updates) == 1
     ), f"Expected one session.update injected to backend, got: {sent_to_backend}"
-    assert session_updates[0]["session"]["turn_detection"]["create_response"] is False
+    # GA shape: turn_detection must be nested under audio.input, not at top-level session
+    injected_session = session_updates[0]["session"]
+    assert (
+        injected_session["type"] == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    assert (
+        injected_session["audio"]["input"]["turn_detection"]["create_response"] is False
+    ), "GA session.update must nest turn_detection under audio.input"
 
     litellm.callbacks = []  # cleanup
 
... diff truncated: showing 800 of 985 lines

You can send follow-ups to the cloud agent here.

Comment thread litellm/litellm_core_utils/realtime_streaming.py
@mateo-berri

Copy link
Copy Markdown
Collaborator

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Uncaught TypeError can crash client message loop
    • Guarded audio format remapping to only use the lookup map for string values, preserving GA-style dict formats without raising TypeError.
Preview (2f743c1d40)
diff --git a/litellm/litellm_core_utils/realtime_streaming.py b/litellm/litellm_core_utils/realtime_streaming.py
--- a/litellm/litellm_core_utils/realtime_streaming.py
+++ b/litellm/litellm_core_utils/realtime_streaming.py
@@ -31,6 +31,8 @@
     "session.created",
     "response.create",
     "response.done",
+    "conversation.item.added",  # GA
+    "conversation.item.done",  # GA
 ]
 
 
@@ -54,6 +56,9 @@
         self.session_tools: List[Dict] = []
         self.tool_calls: List[Dict] = []
 
+        # Detect whether the client is explicitly opting into the beta protocol.
+        self._client_wants_beta = self._detect_beta_header(websocket)
+
         _logged_real_time_event_types = litellm.logged_real_time_event_types
 
         if _logged_real_time_event_types is None:
@@ -76,6 +81,27 @@
         # response.create can be rewritten to include the failure context.
         self._pending_guardrail_message: Optional[str] = None
 
+    _SESSION_EVENT_TYPES = frozenset(["session.created", "session.updated"])
+    _AUDIO_FORMAT_MAP: Dict[str, Dict[str, Any]] = {
+        "pcm16": {"type": "audio/pcm", "rate": 24000},
+        "g711_ulaw": {"type": "audio/G711-ulaw", "rate": 8000},
+        "g711_alaw": {"type": "audio/G711-alaw", "rate": 8000},
+    }
+    # GA name → beta name (when client WebSocket includes OpenAI-Beta: realtime=v1)
+    _GA_TO_BETA_EVENT_TYPES: Dict[str, str] = {
+        "conversation.item.added": "conversation.item.created",
+        "response.output_text.delta": "response.text.delta",
+        "response.output_audio.delta": "response.audio.delta",
+        "response.output_audio_transcript.delta": "response.audio_transcript.delta",
+        "response.output_text.done": "response.text.done",
+        "response.output_audio.done": "response.audio.done",
+        "response.output_audio_transcript.done": "response.audio_transcript.done",
+    }
+    _GA_TO_BETA_CONTENT_TYPES: Dict[str, str] = {
+        "output_text": "text",
+        "output_audio": "audio",
+    }
+
     def _should_store_message(
         self,
         message_obj: Union[dict, OpenAIRealtimeEvents],
@@ -92,24 +118,27 @@
         if isinstance(message, bytes):
             message = message.decode("utf-8")
         if isinstance(message, dict):
-            message_obj = message
+            # TypedDict union members do not narrow to plain dict for mypy.
+            message_obj: Dict[str, Any] = cast(Dict[str, Any], message)
         else:
-            message_obj = json.loads(message)
+            message_obj = cast(Dict[str, Any], json.loads(cast(str, message)))
         self._collect_tool_calls_from_response_done(cast(dict, message_obj))
         try:
-            if (
-                not isinstance(message, dict)
-                or message_obj.get("type") == "session.created"
-                or message_obj.get("type") == "session.updated"
-            ):
-                message_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
-            elif not isinstance(message, dict):
-                message_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
+            event_type = message_obj.get("type", "")
+            if event_type in self._SESSION_EVENT_TYPES:
+                typed_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
+            else:
+                # Use the base object as a safe catch-all for all other event types
+                # (both beta and GA), so unknown/new event names never raise here.
+                typed_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
         except Exception as e:
             verbose_logger.debug(f"Error parsing message for logging: {e}")
-            raise e
-        if self._should_store_message(message_obj):
-            self.messages.append(message_obj)
+            # Don't re-raise — a parse failure must not drop or delay the message
+            if self._should_store_message(message_obj):
+                self.messages.append(message_obj)  # type: ignore[arg-type]
+            return
+        if self._should_store_message(typed_obj):
+            self.messages.append(typed_obj)
 
     def _collect_user_input_from_client_event(self, message: Union[str, dict]) -> None:
         """Extract user text content from client WebSocket events for spend logging."""
@@ -147,6 +176,8 @@
                 tools = session.get("tools")
                 if tools and isinstance(tools, list):
                     self.session_tools = tools
+                # GA: session.type is required; log it for traceability but no action needed
+                verbose_logger.debug(f"Realtime session.type: {session.get('type')}")
         except (json.JSONDecodeError, AttributeError, TypeError):
             pass
 
@@ -228,6 +259,23 @@
         else:
             await self.backend_ws.send(message)  # type: ignore[union-attr, attr-defined]
 
+    def _make_disable_auto_response_message(self) -> str:
+        """Return a session.update that disables VAD auto-response."""
+        if self._client_wants_beta:
+            session: Dict[str, Any] = {
+                "turn_detection": {"create_response": False},
+            }
+        else:
+            session = {
+                "type": "realtime",
+                "audio": {
+                    "input": {
+                        "turn_detection": {"create_response": False},
+                    }
+                },
+            }
+        return json.dumps({"type": "session.update", "session": session})
+
     def _has_realtime_guardrails(self) -> bool:
         """Return True if any callback is registered for realtime guardrail event types."""
         from litellm.integrations.custom_guardrail import CustomGuardrail
@@ -435,14 +483,7 @@
             ):
                 self.store_message(event_str)
                 await self.websocket.send_text(event_str)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 continue
             ## GUARDRAIL: run on transcription events in provider_config path too
             if (
@@ -484,14 +525,7 @@
             ):
                 self.store_message(raw_response)
                 await self.websocket.send_text(raw_response)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 return True
 
             if (
@@ -542,8 +576,21 @@
                         continue
                     ## LOGGING
                     self.store_message(raw_response)
-                    await self.websocket.send_text(raw_response)
 
+                    # If the client opted into beta protocol, translate GA event
+                    # names/shapes back to the beta equivalents before forwarding.
+                    if self._client_wants_beta:
+                        try:
+                            event_dict = json.loads(raw_response)
+                            translated = self._translate_event_to_beta(event_dict)
+                            if translated is None:
+                                continue  # drop GA-only events (e.g. conversation.item.done)
+                            await self.websocket.send_text(json.dumps(translated))
+                        except Exception:
+                            await self.websocket.send_text(raw_response)
+                    else:
+                        await self.websocket.send_text(raw_response)
+
         except websockets.exceptions.ConnectionClosed as e:  # type: ignore
             verbose_logger.exception(
                 f"Connection closed in backend to client send messages - {e}"
@@ -553,6 +600,183 @@
         finally:
             await self.log_messages()
 
+    @staticmethod
+    def _detect_beta_header(websocket: Any) -> bool:
+        """Return True if the client sent 'OpenAI-Beta: realtime=v1'.
+
+        Checks the raw ASGI scope headers so it works for both FastAPI WebSocket
+        objects and any test doubles that expose a .scope dict.
+        """
+        try:
+            headers = websocket.scope.get("headers", [])
+            for name, value in headers:
+                if isinstance(name, bytes):
+                    name = name.decode("latin-1")
+                if isinstance(value, bytes):
+                    value = value.decode("latin-1")
+                if name.lower() == "openai-beta" and "realtime=v1" in value.lower():
+                    return True
+        except Exception:
+            pass
+        return False
+
+    @staticmethod
+    def _remap_beta_session_to_ga(session: dict) -> dict:
+        """
+        Convert a beta-style session.update payload to the GA nested schema.
+
+        Beta → GA field mappings
+        ─────────────────────────────────────────────────────────────────────
+        session.type                    (inject "realtime" if absent)
+        session.modalities              → session.output_modalities
+        session.voice                   → session.audio.output.voice
+        session.input_audio_format      → session.audio.input.format  (with type/rate)
+        session.output_audio_format     → session.audio.output.format (with type/rate)
+        session.turn_detection          → session.audio.input.turn_detection
+        session.input_audio_transcription → session.audio.input.transcription
+        ─────────────────────────────────────────────────────────────────────
+        Fields not in the mapping (instructions, tools, etc.) are passed through.
+        GA clients that already use the nested shape are unaffected.
+        """
+        # Work on a shallow copy so we don't mutate the caller's dict
+        session = dict(session)
+
+        # 1. Ensure session.type is present
+        if "type" not in session:
+            session["type"] = "realtime"
+
+        # 2. Rename modalities → output_modalities and normalise combinations.
+        # Beta allowed ["audio", "text"] together; GA only supports ["audio"] or
+        # ["text"] as single-element lists. When both are present we prefer
+        # ["audio"] because audio mode already delivers transcripts via events.
+        if "modalities" in session:
+            mods = session.pop("modalities")
+            if "output_modalities" not in session:
+                mods_set = {m.lower() for m in (mods or [])}
+                if "audio" in mods_set:
+                    session["output_modalities"] = ["audio"]
+                elif "text" in mods_set:
+                    session["output_modalities"] = ["text"]
+
+        # 3-7. Lift flat audio fields into the nested audio object
+        audio: Dict[str, Any] = {}
+        inp: Dict[str, Any] = {}
+        out: Dict[str, Any] = {}
+
+        # voice → audio.output.voice
+        if "voice" in session:
+            out["voice"] = session.pop("voice")
+
+        # input_audio_format → audio.input.format
+        if "input_audio_format" in session:
+            raw = session.pop("input_audio_format")
+            inp["format"] = (
+                RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+                if isinstance(raw, str)
+                else raw
+            )
+
+        # output_audio_format → audio.output.format
+        if "output_audio_format" in session:
+            raw = session.pop("output_audio_format")
+            out["format"] = (
+                RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+                if isinstance(raw, str)
+                else raw
+            )
+
+        # turn_detection → audio.input.turn_detection
+        if "turn_detection" in session:
+            inp["turn_detection"] = session.pop("turn_detection")
+
+        # input_audio_transcription → audio.input.transcription
+        if "input_audio_transcription" in session:
+            inp["transcription"] = session.pop("input_audio_transcription")
+
+        if inp:
+            audio["input"] = inp
+        if out:
+            audio["output"] = out
+
+        if audio:
+            # Merge with any existing GA-style `audio` block the client already set,
+            # letting the remapped values take precedence within each sub-key.
+            existing = session.get("audio") or {}
+            for sub_key, sub_val in audio.items():
+                if (
+                    sub_key in existing
+                    and isinstance(existing[sub_key], dict)
+                    and isinstance(sub_val, dict)
+                ):
+                    existing[sub_key] = {**existing[sub_key], **sub_val}
+                else:
+                    existing[sub_key] = sub_val
+            session["audio"] = existing
+
+        return session
+
+    @staticmethod
+    def _translate_event_to_beta(event: dict) -> Optional[dict]:
+        """Translate a single GA event dict to its beta equivalent.
+
+        Returns None if the event should be dropped entirely (e.g. the GA-only
+        conversation.item.done has no beta counterpart).
+        Returns the (possibly mutated copy of the) event otherwise.
+        """
+        event_type = event.get("type", "")
+
+        # conversation.item.done has no beta equivalent — the client already
+        # received conversation.item.created (translated from .added).
+        if event_type == "conversation.item.done":
+            return None
+
+        # Shallow-copy so we don't mutate the stored message
+        translated = dict(event)
+
+        # Rename the type field
+        if event_type in RealTimeStreaming._GA_TO_BETA_EVENT_TYPES:
+            translated["type"] = RealTimeStreaming._GA_TO_BETA_EVENT_TYPES[event_type]
+
+        # Fix content block types inside items (response.done output list,
+        # conversation.item.created item content, etc.)
+        if "item" in translated and isinstance(translated["item"], dict):
+            translated["item"] = RealTimeStreaming._translate_item_content_types(
+                dict(translated["item"])
+            )
+        if "response" in translated and isinstance(translated["response"], dict):
+            resp = dict(translated["response"])
+            if "output" in resp and isinstance(resp["output"], list):
+                resp["output"] = [
+                    (
+                        RealTimeStreaming._translate_item_content_types(dict(o))
+                        if isinstance(o, dict)
+                        else o
+                    )
+                    for o in resp["output"]
+                ]
+            translated["response"] = resp
+
+        return translated
+
+    @staticmethod
+    def _translate_item_content_types(item: dict) -> dict:
+        """Replace GA content type names with beta names inside a single item."""
+        if "content" not in item or not isinstance(item["content"], list):
+            return item
+        new_content = []
+        for block in item["content"]:
+            if (
+                isinstance(block, dict)
+                and block.get("type") in RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES
+            ):
+                block = dict(block)
+                block["type"] = RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES[
+                    block["type"]
+                ]
+            new_content.append(block)
+        item["content"] = new_content
+        return item
+
     async def client_ack_messages(self):
         try:
             while True:
@@ -594,6 +818,16 @@
                         self._pending_guardrail_message = None
                         continue
 
+                    # GA compatibility: remap beta-style session fields only when
+                    # the upstream is in GA mode. Beta upstreams expect the flat
+                    # session shape unchanged.
+                    if msg_type == "session.update" and not self._client_wants_beta:
+                        session = msg_obj.get("session", {})
+                        if isinstance(session, dict):
+                            session = self._remap_beta_session_to_ga(session)
+                            msg_obj["session"] = session
+                            message = json.dumps(msg_obj)
+
                 except (json.JSONDecodeError, AttributeError):
                     pass
 
@@ -627,3 +861,8 @@
                     await forward_task
                 except asyncio.CancelledError:
                     pass
+
+
+def client_sent_openai_beta_realtime_header(websocket: Any) -> bool:
+    """True when the client WebSocket includes ``OpenAI-Beta: realtime=v1``."""
+    return RealTimeStreaming._detect_beta_header(websocket)

diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -5255,7 +5255,6 @@
             headers = {
                 "Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json",
-                "OpenAI-Beta": "realtime=v1",
             }
 
         if extra_headers:

diff --git a/litellm/llms/openai/realtime/handler.py b/litellm/llms/openai/realtime/handler.py
--- a/litellm/llms/openai/realtime/handler.py
+++ b/litellm/llms/openai/realtime/handler.py
@@ -6,12 +6,15 @@
 
 from typing import Any, Optional, cast
 
-from litellm._logging import _redact_string
+from litellm._logging import _redact_string, verbose_logger
 from litellm.constants import REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES
 from litellm.types.realtime import RealtimeQueryParams
 
 from ....litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-from ....litellm_core_utils.realtime_streaming import RealTimeStreaming
+from ....litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from ....llms.custom_httpx.http_handler import get_shared_realtime_ssl_context
 from ..openai import OpenAIChatCompletion
 
@@ -33,21 +36,24 @@
         """
         return "https://api.openai.com/"
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
-        Get additional headers beyond Authorization.
-        Override this in subclasses to customize headers (e.g., remove OpenAI-Beta).
+        Headers for the upstream OpenAI Realtime WebSocket.
 
-        Args:
-            api_key: API key for authentication
-
-        Returns:
-            Dictionary of additional headers
+        When the client sent ``OpenAI-Beta: realtime=v1`` on the proxy WebSocket,
+        ``openai_beta_realtime`` is True and the same header is forwarded upstream
+        so the legacy beta API is used. GA clients omit that header on the client
+        connection and must send GA-shaped ``session.update`` payloads.
         """
-        return {
-            "Authorization": f"Bearer {api_key}",
-            "OpenAI-Beta": "realtime=v1",
-        }
+        headers: dict = {"Authorization": f"Bearer {api_key}"}
+        if openai_beta_realtime:
+            headers["OpenAI-Beta"] = "realtime=v1"
+        return headers
 
     def _get_ssl_config(self, url: str) -> Any:
         """
@@ -120,8 +126,16 @@
             # Get provider-specific SSL configuration
             ssl_config = self._get_ssl_config(url)
 
-            # Get provider-specific headers
-            headers = self._get_additional_headers(api_key)
+            openai_beta_realtime = client_sent_openai_beta_realtime_header(websocket)
+            if not openai_beta_realtime:
+                verbose_logger.debug(
+                    "OpenAI Realtime: connecting with GA protocol (no OpenAI-Beta header). "
+                    "If your client expects beta event names, add 'OpenAI-Beta: realtime=v1' "
+                    "to the WebSocket headers sent to the LiteLLM proxy."
+                )
+            headers = self._get_additional_headers(
+                api_key, openai_beta_realtime=openai_beta_realtime
+            )
 
             # Log a masked request preview consistent with other endpoints.
             logging_obj.pre_call(

diff --git a/litellm/llms/xai/realtime/handler.py b/litellm/llms/xai/realtime/handler.py
--- a/litellm/llms/xai/realtime/handler.py
+++ b/litellm/llms/xai/realtime/handler.py
@@ -28,7 +28,12 @@
         """xAI uses a different API base URL."""
         return XAI_API_BASE
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
         xAI does NOT require the OpenAI-Beta header.
         Only send Authorization header.

diff --git a/litellm/types/llms/openai.py b/litellm/types/llms/openai.py
--- a/litellm/types/llms/openai.py
+++ b/litellm/types/llms/openai.py
@@ -1878,14 +1878,24 @@
 
 class OpenAIRealtimeStreamResponseOutputItemContent(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, used for 'input_audio' content types"""
+    """Base64-encoded audio bytes, used for 'input_audio' / 'audio' / 'output_audio' content types"""
     id: str
     """The ID of the previous conversation item for reference"""
     text: str
-    """The text content, used for 'input_text' and 'text' content types"""
+    """The text content, used for 'input_text' / 'text' / 'output_text' content types"""
     transcript: str
-    """The transcript content, used for 'input_audio' content types"""
-    type: Literal["input_audio", "input_text", "text", "item_reference", "audio"]
+    """The transcript content, used for 'input_audio' / 'audio' content types"""
+    type: Literal[
+        "input_audio",
+        "input_text",
+        # Beta assistant content types
+        "text",
+        "audio",
+        "item_reference",
+        # GA assistant content types (aligns with Responses API)
+        "output_text",
+        "output_audio",
+    ]
     """The type of content"""
 
 
@@ -1945,23 +1955,46 @@
 
 
 class OpenAIRealtimeConversationItemCreated(TypedDict, total=False):
+    """Beta: single event emitted when a conversation item is created."""
+
     type: Required[Literal["conversation.item.created"]]
     item: OpenAIRealtimeStreamResponseOutputItem
     event_id: str
-    previous_item_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
 
 
+class OpenAIRealtimeConversationItemAdded(TypedDict, total=False):
+    """GA: emitted immediately when a conversation item is added (replaces .created)."""
+
+    type: Required[Literal["conversation.item.added"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
+class OpenAIRealtimeConversationItemDone(TypedDict, total=False):
+    """GA: emitted when a conversation item is fully complete (e.g. transcription done)."""
+
+    type: Required[Literal["conversation.item.done"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
 class OpenAIRealtimeResponseContentPart(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, if type is 'audio'"""
+    """Base64-encoded audio bytes, if type is 'audio' or 'output_audio'"""
 
     text: str
-    """The text content, if type is 'text'"""
+    """The text content, if type is 'text' or 'output_text'"""
 
     transcript: str
-    """The transcript content, if type is 'audio'"""
+    """The transcript content, if type is 'audio' or 'output_audio'"""
 
-    type: Literal["audio", "text"]
+    type: Union[
+        Literal["audio", "text"],  # beta
+        Literal["output_audio", "output_text"],  # GA
+    ]
     """The type of content"""
 
 
@@ -1982,7 +2015,14 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Union[Literal["response.text.delta"], Literal["response.audio.delta"]]
+    type: Union[
+        Literal["response.text.delta"],
+        Literal["response.audio.delta"],
+        # GA renamed events
+        Literal["response.output_text.delta"],
+        Literal["response.output_audio.delta"],
+        Literal["response.output_audio_transcript.delta"],
+    ]
 
 
 class OpenAIRealtimeResponseTextDone(TypedDict):
@@ -1992,7 +2032,10 @@
     output_index: int
     response_id: str
     text: str
-    type: Literal["response.text.done"]
+    type: Union[
+        Literal["response.text.done"],
+        Literal["response.output_text.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeResponseAudioDone(TypedDict):
@@ -2001,7 +2044,11 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Literal["response.audio.done"]
+    type: Union[
+        Literal["response.audio.done"],
+        Literal["response.output_audio.done"],  # GA rename
+        Literal["response.output_audio_transcript.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeContentPartDone(TypedDict):
@@ -2046,10 +2093,18 @@
 
 class OpenAIRealtimeEventTypes(Enum):
     SESSION_CREATED = "session.created"
+    # Beta delta event names
     RESPONSE_TEXT_DELTA = "response.text.delta"
     RESPONSE_AUDIO_DELTA = "response.audio.delta"
     RESPONSE_TEXT_DONE = "response.text.done"
     RESPONSE_AUDIO_DONE = "response.audio.done"
+    # GA renamed delta event names
+    RESPONSE_OUTPUT_TEXT_DELTA = "response.output_text.delta"
+    RESPONSE_OUTPUT_AUDIO_DELTA = "response.output_audio.delta"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DELTA = "response.output_audio_transcript.delta"
+    RESPONSE_OUTPUT_TEXT_DONE = "response.output_text.done"
+    RESPONSE_OUTPUT_AUDIO_DONE = "response.output_audio.done"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DONE = "response.output_audio_transcript.done"
     RESPONSE_DONE = "response.done"
     RESPONSE_OUTPUT_ITEM_ADDED = "response.output_item.added"
     RESPONSE_CONTENT_PART_ADDED = "response.content_part.added"
@@ -2060,7 +2115,11 @@
     OpenAIRealtimeStreamSessionEvents,
     OpenAIRealtimeStreamResponseOutputItemAdded,
     OpenAIRealtimeResponseContentPartAdded,
+    # Beta conversation item event
     OpenAIRealtimeConversationItemCreated,
+    # GA conversation item events
+    OpenAIRealtimeConversationItemAdded,
+    OpenAIRealtimeConversationItemDone,
     OpenAIRealtimeConversationCreated,
     OpenAIRealtimeResponseDelta,
     OpenAIRealtimeResponseTextDone,

diff --git a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
--- a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
+++ b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
@@ -13,7 +13,10 @@
 )  # Adds the parent directory to the system path
 
 from litellm.integrations.custom_guardrail import CustomGuardrail
-from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
+from litellm.litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.llms.openai import (
     OpenAIRealtimeStreamResponseBaseObject,
@@ -80,6 +83,136 @@
     assert len(streaming.messages) == 2  # Should not store the new message
 
 
+def test_remap_beta_session_to_ga_normalizes_modalities_and_audio():
+    out = RealTimeStreaming._remap_beta_session_to_ga(
+        {"modalities": ["audio", "text"], "voice": "alloy"}
+    )
+    assert out["type"] == "realtime"
+    assert out["output_modalities"] == ["audio"]
+    assert out["audio"]["output"]["voice"] == "alloy"
+
+
+def test_remap_beta_session_to_ga_preserves_ga_audio_format_dicts():
+    input_format = {"type": "audio/pcm", "rate": 24000}
+    output_format = {"type": "audio/G711-ulaw", "rate": 8000}
+
+    out = RealTimeStreaming._remap_beta_session_to_ga(
+        {
+            "input_audio_format": input_format,
+            "output_audio_format": output_format,
+        }
+    )
+
+    assert out["audio"]["input"]["format"] == input_format
+    assert out["audio"]["output"]["format"] == output_format
+
+
+def test_make_disable_auto_response_message_produces_ga_shape():
+    """_make_disable_auto_response_message must produce a GA-shaped session.update.
+
+    The GA Realtime API requires:
+      - session.type = "realtime"
+      - turn_detection nested at session.audio.input.turn_detection
+    The old beta-style flat ``session.turn_detection`` is rejected by GA upstreams.
+    """
+    websocket = MagicMock()
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert (
+        session.get("type") == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    # turn_detection must NOT be at the flat beta location
+    assert (
+        "turn_detection" not in session
+    ), "turn_detection must not be at the top-level session (beta shape); use audio.input"
+    # turn_detection must be nested under audio.input
+    assert session["audio"]["input"]["turn_detection"]["create_response"] is False
+
+
+def test_make_disable_auto_response_message_produces_beta_shape_for_beta_clients():
+    websocket = MagicMock()
+    websocket.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert session == {"turn_detection": {"create_response": False}}
+
+
+@pytest.mark.asyncio
+async def test_client_ack_messages_keeps_beta_session_shape_for_beta_clients():
+    client_ws = MagicMock()
+    client_ws.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    session_update = json.dumps(
+        {
+            "type": "session.update",
+            "session": {
+                "modalities": ["audio", "text"],
+                "voice": "alloy",
+                "turn_detection": {"create_response": False},
+            },
+        }
+    )
+    client_ws.receive_text = AsyncMock(
+        side_effect=[
+            session_update,
+            Exception("connection closed"),
+        ]
+    )
+    backend_ws = MagicMock()
+    backend_ws.send = AsyncMock()
+    logging_obj = MagicMock()
+    logging_obj.pre_call = MagicMock()
+    streaming = RealTimeStreaming(client_ws, backend_ws, logging_obj)
+
+    await streaming.client_ack_messages()
+
+    sent_to_backend = json.loads(backend_ws.send.call_args_list[0].args[0])
+    session = sent_to_backend["session"]
+    assert session["modalities"] == ["audio", "text"]
+    assert session["voice"] == "alloy"
+    assert session["turn_detection"] == {"create_response": False}
+    assert "type" not in session
+    assert "output_modalities" not in session
+    assert "audio" not in session
+
+
+def test_translate_event_to_beta_renames_delta_types():
+    ev = RealTimeStreaming._translate_event_to_beta(
+        {"type": "response.output_audio.delta", "delta": "abc", "event_id": "e1"}
+    )
+    assert ev is not None
+    assert ev["type"] == "response.audio.delta"
+
+
+def test_translate_event_to_beta_drops_conversation_item_done():
+    assert (
+        RealTimeStreaming._translate_event_to_beta({"type": "conversation.item.done"})
+        is None
+    )
+
+
+def test_client_sent_openai_beta_realtime_header_detects_header():
+    ws = MagicMock()
+    ws.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    assert client_sent_openai_beta_realtime_header(ws) is True
+    empty = MagicMock()
... diff truncated: showing 800 of 1008 lines

You can send follow-ups to the cloud agent here.

Comment thread litellm/litellm_core_utils/realtime_streaming.py Outdated
@mateo-berri

Copy link
Copy Markdown
Collaborator

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Session remap breaks Azure beta realtime backend
    • Azure realtime now passes its selected backend protocol into streaming so beta sessions keep beta shape while GA sessions are remapped.
Preview (2cf367f7cd)
diff --git a/litellm/litellm_core_utils/realtime_streaming.py b/litellm/litellm_core_utils/realtime_streaming.py
--- a/litellm/litellm_core_utils/realtime_streaming.py
+++ b/litellm/litellm_core_utils/realtime_streaming.py
@@ -31,6 +31,8 @@
     "session.created",
     "response.create",
     "response.done",
+    "conversation.item.added",  # GA
+    "conversation.item.done",  # GA
 ]
 
 
@@ -44,6 +46,7 @@
         model: str = "",
         user_api_key_dict: Optional[Any] = None,
         request_data: Optional[Dict] = None,
+        backend_uses_beta_protocol: Optional[bool] = None,
     ):
         self.websocket = websocket
         self.backend_ws = backend_ws
@@ -54,6 +57,14 @@
         self.session_tools: List[Dict] = []
         self.tool_calls: List[Dict] = []
 
+        # Detect whether the client is explicitly opting into the beta protocol.
+        self._client_wants_beta = self._detect_beta_header(websocket)
+        self._backend_uses_beta_protocol = (
+            self._client_wants_beta
+            if backend_uses_beta_protocol is None
+            else backend_uses_beta_protocol
+        )
+
         _logged_real_time_event_types = litellm.logged_real_time_event_types
 
         if _logged_real_time_event_types is None:
@@ -76,6 +87,27 @@
         # response.create can be rewritten to include the failure context.
         self._pending_guardrail_message: Optional[str] = None
 
+    _SESSION_EVENT_TYPES = frozenset(["session.created", "session.updated"])
+    _AUDIO_FORMAT_MAP: Dict[str, Dict[str, Any]] = {
+        "pcm16": {"type": "audio/pcm", "rate": 24000},
+        "g711_ulaw": {"type": "audio/G711-ulaw", "rate": 8000},
+        "g711_alaw": {"type": "audio/G711-alaw", "rate": 8000},
+    }
+    # GA name → beta name (when client WebSocket includes OpenAI-Beta: realtime=v1)
+    _GA_TO_BETA_EVENT_TYPES: Dict[str, str] = {
+        "conversation.item.added": "conversation.item.created",
+        "response.output_text.delta": "response.text.delta",
+        "response.output_audio.delta": "response.audio.delta",
+        "response.output_audio_transcript.delta": "response.audio_transcript.delta",
+        "response.output_text.done": "response.text.done",
+        "response.output_audio.done": "response.audio.done",
+        "response.output_audio_transcript.done": "response.audio_transcript.done",
+    }
+    _GA_TO_BETA_CONTENT_TYPES: Dict[str, str] = {
+        "output_text": "text",
+        "output_audio": "audio",
+    }
+
     def _should_store_message(
         self,
         message_obj: Union[dict, OpenAIRealtimeEvents],
@@ -92,24 +124,27 @@
         if isinstance(message, bytes):
             message = message.decode("utf-8")
         if isinstance(message, dict):
-            message_obj = message
+            # TypedDict union members do not narrow to plain dict for mypy.
+            message_obj: Dict[str, Any] = cast(Dict[str, Any], message)
         else:
-            message_obj = json.loads(message)
+            message_obj = cast(Dict[str, Any], json.loads(cast(str, message)))
         self._collect_tool_calls_from_response_done(cast(dict, message_obj))
         try:
-            if (
-                not isinstance(message, dict)
-                or message_obj.get("type") == "session.created"
-                or message_obj.get("type") == "session.updated"
-            ):
-                message_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
-            elif not isinstance(message, dict):
-                message_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
+            event_type = message_obj.get("type", "")
+            if event_type in self._SESSION_EVENT_TYPES:
+                typed_obj = OpenAIRealtimeStreamSessionEvents(**message_obj)  # type: ignore
+            else:
+                # Use the base object as a safe catch-all for all other event types
+                # (both beta and GA), so unknown/new event names never raise here.
+                typed_obj = OpenAIRealtimeStreamResponseBaseObject(**message_obj)  # type: ignore
         except Exception as e:
             verbose_logger.debug(f"Error parsing message for logging: {e}")
-            raise e
-        if self._should_store_message(message_obj):
-            self.messages.append(message_obj)
+            # Don't re-raise — a parse failure must not drop or delay the message
+            if self._should_store_message(message_obj):
+                self.messages.append(message_obj)  # type: ignore[arg-type]
+            return
+        if self._should_store_message(typed_obj):
+            self.messages.append(typed_obj)
 
     def _collect_user_input_from_client_event(self, message: Union[str, dict]) -> None:
         """Extract user text content from client WebSocket events for spend logging."""
@@ -147,6 +182,8 @@
                 tools = session.get("tools")
                 if tools and isinstance(tools, list):
                     self.session_tools = tools
+                # GA: session.type is required; log it for traceability but no action needed
+                verbose_logger.debug(f"Realtime session.type: {session.get('type')}")
         except (json.JSONDecodeError, AttributeError, TypeError):
             pass
 
@@ -228,6 +265,23 @@
         else:
             await self.backend_ws.send(message)  # type: ignore[union-attr, attr-defined]
 
+    def _make_disable_auto_response_message(self) -> str:
+        """Return a session.update that disables VAD auto-response."""
+        if self._backend_uses_beta_protocol:
+            session: Dict[str, Any] = {
+                "turn_detection": {"create_response": False},
+            }
+        else:
+            session = {
+                "type": "realtime",
+                "audio": {
+                    "input": {
+                        "turn_detection": {"create_response": False},
+                    }
+                },
+            }
+        return json.dumps({"type": "session.update", "session": session})
+
     def _has_realtime_guardrails(self) -> bool:
         """Return True if any callback is registered for realtime guardrail event types."""
         from litellm.integrations.custom_guardrail import CustomGuardrail
@@ -435,14 +489,7 @@
             ):
                 self.store_message(event_str)
                 await self.websocket.send_text(event_str)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 continue
             ## GUARDRAIL: run on transcription events in provider_config path too
             if (
@@ -484,14 +531,7 @@
             ):
                 self.store_message(raw_response)
                 await self.websocket.send_text(raw_response)
-                await self._send_to_backend(
-                    json.dumps(
-                        {
-                            "type": "session.update",
-                            "session": {"turn_detection": {"create_response": False}},
-                        }
-                    )
-                )
+                await self._send_to_backend(self._make_disable_auto_response_message())
                 return True
 
             if (
@@ -542,8 +582,21 @@
                         continue
                     ## LOGGING
                     self.store_message(raw_response)
-                    await self.websocket.send_text(raw_response)
 
+                    # If the client opted into beta protocol, translate GA event
+                    # names/shapes back to the beta equivalents before forwarding.
+                    if self._client_wants_beta:
+                        try:
+                            event_dict = json.loads(raw_response)
+                            translated = self._translate_event_to_beta(event_dict)
+                            if translated is None:
+                                continue  # drop GA-only events (e.g. conversation.item.done)
+                            await self.websocket.send_text(json.dumps(translated))
+                        except Exception:
+                            await self.websocket.send_text(raw_response)
+                    else:
+                        await self.websocket.send_text(raw_response)
+
         except websockets.exceptions.ConnectionClosed as e:  # type: ignore
             verbose_logger.exception(
                 f"Connection closed in backend to client send messages - {e}"
@@ -553,6 +606,183 @@
         finally:
             await self.log_messages()
 
+    @staticmethod
+    def _detect_beta_header(websocket: Any) -> bool:
+        """Return True if the client sent 'OpenAI-Beta: realtime=v1'.
+
+        Checks the raw ASGI scope headers so it works for both FastAPI WebSocket
+        objects and any test doubles that expose a .scope dict.
+        """
+        try:
+            headers = websocket.scope.get("headers", [])
+            for name, value in headers:
+                if isinstance(name, bytes):
+                    name = name.decode("latin-1")
+                if isinstance(value, bytes):
+                    value = value.decode("latin-1")
+                if name.lower() == "openai-beta" and "realtime=v1" in value.lower():
+                    return True
+        except Exception:
+            pass
+        return False
+
+    @staticmethod
+    def _remap_beta_session_to_ga(session: dict) -> dict:
+        """
+        Convert a beta-style session.update payload to the GA nested schema.
+
+        Beta → GA field mappings
+        ─────────────────────────────────────────────────────────────────────
+        session.type                    (inject "realtime" if absent)
+        session.modalities              → session.output_modalities
+        session.voice                   → session.audio.output.voice
+        session.input_audio_format      → session.audio.input.format  (with type/rate)
+        session.output_audio_format     → session.audio.output.format (with type/rate)
+        session.turn_detection          → session.audio.input.turn_detection
+        session.input_audio_transcription → session.audio.input.transcription
+        ─────────────────────────────────────────────────────────────────────
+        Fields not in the mapping (instructions, tools, etc.) are passed through.
+        GA clients that already use the nested shape are unaffected.
+        """
+        # Work on a shallow copy so we don't mutate the caller's dict
+        session = dict(session)
+
+        # 1. Ensure session.type is present
+        if "type" not in session:
+            session["type"] = "realtime"
+
+        # 2. Rename modalities → output_modalities and normalise combinations.
+        # Beta allowed ["audio", "text"] together; GA only supports ["audio"] or
+        # ["text"] as single-element lists. When both are present we prefer
+        # ["audio"] because audio mode already delivers transcripts via events.
+        if "modalities" in session:
+            mods = session.pop("modalities")
+            if "output_modalities" not in session:
+                mods_set = {m.lower() for m in (mods or [])}
+                if "audio" in mods_set:
+                    session["output_modalities"] = ["audio"]
+                elif "text" in mods_set:
+                    session["output_modalities"] = ["text"]
+
+        # 3-7. Lift flat audio fields into the nested audio object
+        audio: Dict[str, Any] = {}
+        inp: Dict[str, Any] = {}
+        out: Dict[str, Any] = {}
+
+        # voice → audio.output.voice
+        if "voice" in session:
+            out["voice"] = session.pop("voice")
+
+        # input_audio_format → audio.input.format
+        if "input_audio_format" in session:
+            raw = session.pop("input_audio_format")
+            inp["format"] = (
+                RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+                if isinstance(raw, str)
+                else raw
+            )
+
+        # output_audio_format → audio.output.format
+        if "output_audio_format" in session:
+            raw = session.pop("output_audio_format")
+            out["format"] = (
+                RealTimeStreaming._AUDIO_FORMAT_MAP.get(raw, raw)
+                if isinstance(raw, str)
+                else raw
+            )
+
+        # turn_detection → audio.input.turn_detection
+        if "turn_detection" in session:
+            inp["turn_detection"] = session.pop("turn_detection")
+
+        # input_audio_transcription → audio.input.transcription
+        if "input_audio_transcription" in session:
+            inp["transcription"] = session.pop("input_audio_transcription")
+
+        if inp:
+            audio["input"] = inp
+        if out:
+            audio["output"] = out
+
+        if audio:
+            # Merge with any existing GA-style `audio` block the client already set,
+            # letting the remapped values take precedence within each sub-key.
+            existing = session.get("audio") or {}
+            for sub_key, sub_val in audio.items():
+                if (
+                    sub_key in existing
+                    and isinstance(existing[sub_key], dict)
+                    and isinstance(sub_val, dict)
+                ):
+                    existing[sub_key] = {**existing[sub_key], **sub_val}
+                else:
+                    existing[sub_key] = sub_val
+            session["audio"] = existing
+
+        return session
+
+    @staticmethod
+    def _translate_event_to_beta(event: dict) -> Optional[dict]:
+        """Translate a single GA event dict to its beta equivalent.
+
+        Returns None if the event should be dropped entirely (e.g. the GA-only
+        conversation.item.done has no beta counterpart).
+        Returns the (possibly mutated copy of the) event otherwise.
+        """
+        event_type = event.get("type", "")
+
+        # conversation.item.done has no beta equivalent — the client already
+        # received conversation.item.created (translated from .added).
+        if event_type == "conversation.item.done":
+            return None
+
+        # Shallow-copy so we don't mutate the stored message
+        translated = dict(event)
+
+        # Rename the type field
+        if event_type in RealTimeStreaming._GA_TO_BETA_EVENT_TYPES:
+            translated["type"] = RealTimeStreaming._GA_TO_BETA_EVENT_TYPES[event_type]
+
+        # Fix content block types inside items (response.done output list,
+        # conversation.item.created item content, etc.)
+        if "item" in translated and isinstance(translated["item"], dict):
+            translated["item"] = RealTimeStreaming._translate_item_content_types(
+                dict(translated["item"])
+            )
+        if "response" in translated and isinstance(translated["response"], dict):
+            resp = dict(translated["response"])
+            if "output" in resp and isinstance(resp["output"], list):
+                resp["output"] = [
+                    (
+                        RealTimeStreaming._translate_item_content_types(dict(o))
+                        if isinstance(o, dict)
+                        else o
+                    )
+                    for o in resp["output"]
+                ]
+            translated["response"] = resp
+
+        return translated
+
+    @staticmethod
+    def _translate_item_content_types(item: dict) -> dict:
+        """Replace GA content type names with beta names inside a single item."""
+        if "content" not in item or not isinstance(item["content"], list):
+            return item
+        new_content = []
+        for block in item["content"]:
+            if (
+                isinstance(block, dict)
+                and block.get("type") in RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES
+            ):
+                block = dict(block)
+                block["type"] = RealTimeStreaming._GA_TO_BETA_CONTENT_TYPES[
+                    block["type"]
+                ]
+            new_content.append(block)
+        item["content"] = new_content
+        return item
+
     async def client_ack_messages(self):
         try:
             while True:
@@ -594,6 +824,19 @@
                         self._pending_guardrail_message = None
                         continue
 
+                    # GA compatibility: remap beta-style session fields only when
+                    # the upstream is in GA mode. Beta upstreams expect the flat
+                    # session shape unchanged.
+                    if (
+                        msg_type == "session.update"
+                        and not self._backend_uses_beta_protocol
+                    ):
+                        session = msg_obj.get("session", {})
+                        if isinstance(session, dict):
+                            session = self._remap_beta_session_to_ga(session)
+                            msg_obj["session"] = session
+                            message = json.dumps(msg_obj)
+
                 except (json.JSONDecodeError, AttributeError):
                     pass
 
@@ -627,3 +870,8 @@
                     await forward_task
                 except asyncio.CancelledError:
                     pass
+
+
+def client_sent_openai_beta_realtime_header(websocket: Any) -> bool:
+    """True when the client WebSocket includes ``OpenAI-Beta: realtime=v1``."""
+    return RealTimeStreaming._detect_beta_header(websocket)

diff --git a/litellm/llms/azure/realtime/handler.py b/litellm/llms/azure/realtime/handler.py
--- a/litellm/llms/azure/realtime/handler.py
+++ b/litellm/llms/azure/realtime/handler.py
@@ -89,9 +89,10 @@
 
         if api_base is None:
             raise ValueError("api_base is required for Azure OpenAI calls")
-        if api_version is None and (
+        backend_uses_beta_protocol = (
             realtime_protocol is None or realtime_protocol.upper() not in ("GA", "V1")
-        ):
+        )
+        if api_version is None and backend_uses_beta_protocol:
             raise ValueError("api_version is required for Azure OpenAI calls")
 
         url = self._construct_url(
@@ -114,6 +115,7 @@
                     logging_obj,
                     user_api_key_dict=user_api_key_dict,
                     request_data={"litellm_metadata": litellm_metadata or {}},
+                    backend_uses_beta_protocol=backend_uses_beta_protocol,
                 )
                 await realtime_streaming.bidirectional_forward()
 

diff --git a/litellm/llms/custom_httpx/llm_http_handler.py b/litellm/llms/custom_httpx/llm_http_handler.py
--- a/litellm/llms/custom_httpx/llm_http_handler.py
+++ b/litellm/llms/custom_httpx/llm_http_handler.py
@@ -5255,7 +5255,6 @@
             headers = {
                 "Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json",
-                "OpenAI-Beta": "realtime=v1",
             }
 
         if extra_headers:

diff --git a/litellm/llms/openai/realtime/handler.py b/litellm/llms/openai/realtime/handler.py
--- a/litellm/llms/openai/realtime/handler.py
+++ b/litellm/llms/openai/realtime/handler.py
@@ -6,12 +6,15 @@
 
 from typing import Any, Optional, cast
 
-from litellm._logging import _redact_string
+from litellm._logging import _redact_string, verbose_logger
 from litellm.constants import REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES
 from litellm.types.realtime import RealtimeQueryParams
 
 from ....litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
-from ....litellm_core_utils.realtime_streaming import RealTimeStreaming
+from ....litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from ....llms.custom_httpx.http_handler import get_shared_realtime_ssl_context
 from ..openai import OpenAIChatCompletion
 
@@ -33,21 +36,24 @@
         """
         return "https://api.openai.com/"
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
-        Get additional headers beyond Authorization.
-        Override this in subclasses to customize headers (e.g., remove OpenAI-Beta).
+        Headers for the upstream OpenAI Realtime WebSocket.
 
-        Args:
-            api_key: API key for authentication
-
-        Returns:
-            Dictionary of additional headers
+        When the client sent ``OpenAI-Beta: realtime=v1`` on the proxy WebSocket,
+        ``openai_beta_realtime`` is True and the same header is forwarded upstream
+        so the legacy beta API is used. GA clients omit that header on the client
+        connection and must send GA-shaped ``session.update`` payloads.
         """
-        return {
-            "Authorization": f"Bearer {api_key}",
-            "OpenAI-Beta": "realtime=v1",
-        }
+        headers: dict = {"Authorization": f"Bearer {api_key}"}
+        if openai_beta_realtime:
+            headers["OpenAI-Beta"] = "realtime=v1"
+        return headers
 
     def _get_ssl_config(self, url: str) -> Any:
         """
@@ -120,8 +126,16 @@
             # Get provider-specific SSL configuration
             ssl_config = self._get_ssl_config(url)
 
-            # Get provider-specific headers
-            headers = self._get_additional_headers(api_key)
+            openai_beta_realtime = client_sent_openai_beta_realtime_header(websocket)
+            if not openai_beta_realtime:
+                verbose_logger.debug(
+                    "OpenAI Realtime: connecting with GA protocol (no OpenAI-Beta header). "
+                    "If your client expects beta event names, add 'OpenAI-Beta: realtime=v1' "
+                    "to the WebSocket headers sent to the LiteLLM proxy."
+                )
+            headers = self._get_additional_headers(
+                api_key, openai_beta_realtime=openai_beta_realtime
+            )
 
             # Log a masked request preview consistent with other endpoints.
             logging_obj.pre_call(

diff --git a/litellm/llms/xai/realtime/handler.py b/litellm/llms/xai/realtime/handler.py
--- a/litellm/llms/xai/realtime/handler.py
+++ b/litellm/llms/xai/realtime/handler.py
@@ -28,7 +28,12 @@
         """xAI uses a different API base URL."""
         return XAI_API_BASE
 
-    def _get_additional_headers(self, api_key: str) -> dict:
+    def _get_additional_headers(
+        self,
+        api_key: str,
+        *,
+        openai_beta_realtime: bool = False,
+    ) -> dict:
         """
         xAI does NOT require the OpenAI-Beta header.
         Only send Authorization header.

diff --git a/litellm/types/llms/openai.py b/litellm/types/llms/openai.py
--- a/litellm/types/llms/openai.py
+++ b/litellm/types/llms/openai.py
@@ -1878,14 +1878,24 @@
 
 class OpenAIRealtimeStreamResponseOutputItemContent(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, used for 'input_audio' content types"""
+    """Base64-encoded audio bytes, used for 'input_audio' / 'audio' / 'output_audio' content types"""
     id: str
     """The ID of the previous conversation item for reference"""
     text: str
-    """The text content, used for 'input_text' and 'text' content types"""
+    """The text content, used for 'input_text' / 'text' / 'output_text' content types"""
     transcript: str
-    """The transcript content, used for 'input_audio' content types"""
-    type: Literal["input_audio", "input_text", "text", "item_reference", "audio"]
+    """The transcript content, used for 'input_audio' / 'audio' content types"""
+    type: Literal[
+        "input_audio",
+        "input_text",
+        # Beta assistant content types
+        "text",
+        "audio",
+        "item_reference",
+        # GA assistant content types (aligns with Responses API)
+        "output_text",
+        "output_audio",
+    ]
     """The type of content"""
 
 
@@ -1945,23 +1955,46 @@
 
 
 class OpenAIRealtimeConversationItemCreated(TypedDict, total=False):
+    """Beta: single event emitted when a conversation item is created."""
+
     type: Required[Literal["conversation.item.created"]]
     item: OpenAIRealtimeStreamResponseOutputItem
     event_id: str
-    previous_item_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
 
 
+class OpenAIRealtimeConversationItemAdded(TypedDict, total=False):
+    """GA: emitted immediately when a conversation item is added (replaces .created)."""
+
+    type: Required[Literal["conversation.item.added"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
+class OpenAIRealtimeConversationItemDone(TypedDict, total=False):
+    """GA: emitted when a conversation item is fully complete (e.g. transcription done)."""
+
+    type: Required[Literal["conversation.item.done"]]
+    item: OpenAIRealtimeStreamResponseOutputItem
+    event_id: str
+    previous_item_id: Optional[str]  # None when this is the first item
+
+
 class OpenAIRealtimeResponseContentPart(TypedDict, total=False):
     audio: str
-    """Base64-encoded audio bytes, if type is 'audio'"""
+    """Base64-encoded audio bytes, if type is 'audio' or 'output_audio'"""
 
     text: str
-    """The text content, if type is 'text'"""
+    """The text content, if type is 'text' or 'output_text'"""
 
     transcript: str
-    """The transcript content, if type is 'audio'"""
+    """The transcript content, if type is 'audio' or 'output_audio'"""
 
-    type: Literal["audio", "text"]
+    type: Union[
+        Literal["audio", "text"],  # beta
+        Literal["output_audio", "output_text"],  # GA
+    ]
     """The type of content"""
 
 
@@ -1982,7 +2015,14 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Union[Literal["response.text.delta"], Literal["response.audio.delta"]]
+    type: Union[
+        Literal["response.text.delta"],
+        Literal["response.audio.delta"],
+        # GA renamed events
+        Literal["response.output_text.delta"],
+        Literal["response.output_audio.delta"],
+        Literal["response.output_audio_transcript.delta"],
+    ]
 
 
 class OpenAIRealtimeResponseTextDone(TypedDict):
@@ -1992,7 +2032,10 @@
     output_index: int
     response_id: str
     text: str
-    type: Literal["response.text.done"]
+    type: Union[
+        Literal["response.text.done"],
+        Literal["response.output_text.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeResponseAudioDone(TypedDict):
@@ -2001,7 +2044,11 @@
     item_id: str
     output_index: int
     response_id: str
-    type: Literal["response.audio.done"]
+    type: Union[
+        Literal["response.audio.done"],
+        Literal["response.output_audio.done"],  # GA rename
+        Literal["response.output_audio_transcript.done"],  # GA rename
+    ]
 
 
 class OpenAIRealtimeContentPartDone(TypedDict):
@@ -2046,10 +2093,18 @@
 
 class OpenAIRealtimeEventTypes(Enum):
     SESSION_CREATED = "session.created"
+    # Beta delta event names
     RESPONSE_TEXT_DELTA = "response.text.delta"
     RESPONSE_AUDIO_DELTA = "response.audio.delta"
     RESPONSE_TEXT_DONE = "response.text.done"
     RESPONSE_AUDIO_DONE = "response.audio.done"
+    # GA renamed delta event names
+    RESPONSE_OUTPUT_TEXT_DELTA = "response.output_text.delta"
+    RESPONSE_OUTPUT_AUDIO_DELTA = "response.output_audio.delta"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DELTA = "response.output_audio_transcript.delta"
+    RESPONSE_OUTPUT_TEXT_DONE = "response.output_text.done"
+    RESPONSE_OUTPUT_AUDIO_DONE = "response.output_audio.done"
+    RESPONSE_OUTPUT_AUDIO_TRANSCRIPT_DONE = "response.output_audio_transcript.done"
     RESPONSE_DONE = "response.done"
     RESPONSE_OUTPUT_ITEM_ADDED = "response.output_item.added"
     RESPONSE_CONTENT_PART_ADDED = "response.content_part.added"
@@ -2060,7 +2115,11 @@
     OpenAIRealtimeStreamSessionEvents,
     OpenAIRealtimeStreamResponseOutputItemAdded,
     OpenAIRealtimeResponseContentPartAdded,
+    # Beta conversation item event
     OpenAIRealtimeConversationItemCreated,
+    # GA conversation item events
+    OpenAIRealtimeConversationItemAdded,
+    OpenAIRealtimeConversationItemDone,
     OpenAIRealtimeConversationCreated,
     OpenAIRealtimeResponseDelta,
     OpenAIRealtimeResponseTextDone,

diff --git a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
--- a/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
+++ b/tests/test_litellm/litellm_core_utils/test_realtime_streaming.py
@@ -13,7 +13,10 @@
 )  # Adds the parent directory to the system path
 
 from litellm.integrations.custom_guardrail import CustomGuardrail
-from litellm.litellm_core_utils.realtime_streaming import RealTimeStreaming
+from litellm.litellm_core_utils.realtime_streaming import (
+    RealTimeStreaming,
+    client_sent_openai_beta_realtime_header,
+)
 from litellm.types.guardrails import GuardrailEventHooks
 from litellm.types.llms.openai import (
     OpenAIRealtimeStreamResponseBaseObject,
@@ -80,6 +83,175 @@
     assert len(streaming.messages) == 2  # Should not store the new message
 
 
+def test_remap_beta_session_to_ga_normalizes_modalities_and_audio():
+    out = RealTimeStreaming._remap_beta_session_to_ga(
+        {"modalities": ["audio", "text"], "voice": "alloy"}
+    )
+    assert out["type"] == "realtime"
+    assert out["output_modalities"] == ["audio"]
+    assert out["audio"]["output"]["voice"] == "alloy"
+
+
+def test_remap_beta_session_to_ga_preserves_ga_audio_format_dicts():
+    input_format = {"type": "audio/pcm", "rate": 24000}
+    output_format = {"type": "audio/G711-ulaw", "rate": 8000}
+
+    out = RealTimeStreaming._remap_beta_session_to_ga(
+        {
+            "input_audio_format": input_format,
+            "output_audio_format": output_format,
+        }
+    )
+
+    assert out["audio"]["input"]["format"] == input_format
+    assert out["audio"]["output"]["format"] == output_format
+
+
+def test_make_disable_auto_response_message_produces_ga_shape():
+    """_make_disable_auto_response_message must produce a GA-shaped session.update.
+
+    The GA Realtime API requires:
+      - session.type = "realtime"
+      - turn_detection nested at session.audio.input.turn_detection
+    The old beta-style flat ``session.turn_detection`` is rejected by GA upstreams.
+    """
+    websocket = MagicMock()
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert (
+        session.get("type") == "realtime"
+    ), "GA session.update must include session.type='realtime'"
+    # turn_detection must NOT be at the flat beta location
+    assert (
+        "turn_detection" not in session
+    ), "turn_detection must not be at the top-level session (beta shape); use audio.input"
+    # turn_detection must be nested under audio.input
+    assert session["audio"]["input"]["turn_detection"]["create_response"] is False
+
+
+def test_make_disable_auto_response_message_produces_beta_shape_for_beta_clients():
+    websocket = MagicMock()
+    websocket.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    backend_ws = MagicMock()
+    logging_obj = MagicMock()
+    streaming = RealTimeStreaming(websocket, backend_ws, logging_obj)
+
+    raw = streaming._make_disable_auto_response_message()
+    msg = json.loads(raw)
+
+    assert msg["type"] == "session.update"
+    session = msg["session"]
+    assert session == {"turn_detection": {"create_response": False}}
+
+
+@pytest.mark.asyncio
+async def test_client_ack_messages_keeps_beta_session_shape_for_beta_clients():
+    client_ws = MagicMock()
+    client_ws.scope = {"headers": [(b"openai-beta", b"realtime=v1")]}
+    session_update = json.dumps(
+        {
+            "type": "session.update",
+            "session": {
+                "modalities": ["audio", "text"],
+                "voice": "alloy",
+                "turn_detection": {"create_response": False},
+            },
+        }
+    )
+    client_ws.receive_text = AsyncMock(
+        side_effect=[
+            session_update,
... diff truncated: showing 800 of 1111 lines

You can send follow-ups to the cloud agent here.

Comment thread litellm/litellm_core_utils/realtime_streaming.py
@mateo-berri

Copy link
Copy Markdown
Collaborator

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 2cf367f. Configure here.

@mateo-berri

Copy link
Copy Markdown
Collaborator

@greptileai

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ci tests not passing

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

@mateo-berri https://app.circleci.com/pipelines/github/BerriAI/litellm/76398. I can see that CI is itself failing from litellm_internal_staging. THis is not because of the changes in this PR

@Sameerlite Sameerlite requested a review from mateo-berri May 5, 2026 03:51
@mateo-berri

Copy link
Copy Markdown
Collaborator

OK let's wait for staging to stabilize and rebase

@codecov

codecov Bot commented May 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.24551% with 38 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/litellm_core_utils/realtime_streaming.py 69.60% 38 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Sameerlite

Copy link
Copy Markdown
Collaborator Author

@mateo-berri Passing all tests now

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@mateo-berri mateo-berri merged commit b8635bb into litellm_internal_staging May 5, 2026
115 checks passed
@mateo-berri mateo-berri deleted the litellm_openai_realtime_ga branch May 5, 2026 23:49
Sameerlite pushed a commit that referenced this pull request Jun 11, 2026
…me JSON work on OpenAI realtime relay (#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in #27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
pghuge-cloudwiz pushed a commit to cloudwizio/litellm that referenced this pull request Jun 11, 2026
…per-frame JSON work on OpenAI realtime relay (BerriAI#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in BerriAI#27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
michaelxer pushed a commit to michaelxer/litellm that referenced this pull request Jun 17, 2026
…per-frame JSON work on OpenAI realtime relay (BerriAI#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in BerriAI#27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
michaelxer pushed a commit to michaelxer/litellm that referenced this pull request Jun 17, 2026
…per-frame JSON work on OpenAI realtime relay (BerriAI#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in BerriAI#27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
koladefaj pushed a commit to koladefaj/litellm that referenced this pull request Jun 17, 2026
…per-frame JSON work on OpenAI realtime relay (BerriAI#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in BerriAI#27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
factnn pushed a commit to factnn/litellm that referenced this pull request Jun 18, 2026
…per-frame JSON work on OpenAI realtime relay (BerriAI#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in BerriAI#27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
…rriAI#27110)

* feat(realtime): OpenAI Realtime GA support and beta compatibility

- Normalize beta-style session.update to GA for upstream OpenAI; optional GA→beta
  event translation when client sends OpenAI-Beta: realtime=v1
- Default upstream WebSocket without OpenAI-Beta; forward header when client opts in
- Extend OpenAI realtime types for GA event names and conversation item shapes
- Relax LiteLLMRealtimeStreamLoggingObject.results to List[Any] for GA events
- Update proxy client_secrets fallback to omit beta header; dashboard RealtimePlayground
- Add unit tests for remap, translation, and beta header helper

Co-authored-by: Cursor <[email protected]>

* fix results

* fix greptile

* Fix mypy issues

* Remove unused class constants _GA_TEXT_DELTA_TYPES and _GA_AUDIO_DELTA_TYPES

These frozensets were defined as class-level constants in realtime_streaming.py
but never referenced anywhere in the codebase. Removing dead code.

Co-authored-by: Sameer Kankute <[email protected]>

* fix(realtime): use GA-shaped session.update in guardrail injections

The guardrail VAD injection code sent a beta-style session.update with a
flat turn_detection field:

  {"session": {"turn_detection": {"create_response": false}}}

When the upstream OpenAI backend operates in GA mode (no OpenAI-Beta
header forwarded), it requires the nested GA shape:

  {"session": {"type": "realtime", "audio": {"input": {"turn_detection": {"create_response": false}}}}}

The _remap_beta_session_to_ga helper was only applied to client-
originated session.update messages in client_ack_messages. Internally-
generated session.updates (sent via _send_to_backend) in two paths:
  - _handle_raw_backend_message (raw/no provider_config path, line 518)
  - backend_to_client_send_messages provider_config path (line 481)
bypassed the remap, so GA upstreams ignored or rejected them, breaking
audio transcription guardrails for all non-beta clients.

Fix: add _make_disable_auto_response_message() helper that always emits
the correct GA-shaped session.update, and replace both injection sites
with it.

Update existing tests to assert the GA nested shape instead of the old
flat beta shape, and add a new unit test for the helper itself.

Co-authored-by: Sameer Kankute <[email protected]>

* Log realtime session type

* Fix beta realtime session payloads

* Fix realtime audio format remapping edge case

* Fix Azure realtime beta session shape

---------

Co-authored-by: Cursor <[email protected]>
Co-authored-by: Sameer Kankute <[email protected]>
Co-authored-by: mateo-berri <[email protected]>
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
…per-frame JSON work on OpenAI realtime relay (BerriAI#30142)

* perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay

The GA realtime support added in BerriAI#27110 made backend_to_client_send_messages
parse every backend frame up to three times for beta clients (OpenAI-Beta:
realtime=v1), build a discarded Pydantic object per frame for logging, and
re-serialize even frames that need no translation. For high-frequency
response.output_audio.delta frames carrying multi-KB base64 payloads, that
serialized CPU work on the hottest relay path drove the latency regression
between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2.

This parses each frame once via _parse_backend_event and threads the dict into
_handle_raw_backend_message, store_message, and _translate_event_to_beta;
short-circuits store_message before the Pydantic build for events not in the
logged set; returns the original event unchanged from _translate_event_to_beta
when no rename applies so the raw frame is forwarded without re-serialization;
and only json.dumps when the type is actually renamed.

* fix(realtime): widen store_message type hint to accept plain dict

The parse-once refactor passes the dict produced by _parse_backend_event into
store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents
(a union of TypedDicts), which mypy does not consider compatible with a plain
dict. Add dict to the accepted union; the body already handles it.

---------

Co-authored-by: Miguel Armenta <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: gpt-realtime-1.5 ( OpenAI) [Bug]: gpt-realtime (GA) fails with OpenAI-Beta: realtime=v1 header

4 participants