Skip to content

[BUG]: [playground] OpenAI streaming keys tool-call fragments by enumerate() position instead of index — parallel tool calls garbled, id-less fragments raise KeyError #16941

Open

Description

@BlueX888

Where do you use Phoenix

Self-hosted

What version of Phoenix are you using?

main @ d0dc9e5ff0a33a1962da6aae83e6f56ac72c444f (__version__ = "20.21.0", src/phoenix/version.py). Python 3.12.14, openai 3.29.0 (repo pins openai>=3.1.0,<4, pyproject.toml:122), macOS 26.6.2 (arm64).

What happened?

In the playground, the OpenAI streaming path keys streamed tool-call fragments by their position inside the current delta (enumerate) instead of by the fragment's own index field. Parallel tool calls get garbled arguments, and a first fragment that arrives without an id crashes the whole playground run.

In OpenAICompatibleClient._chat_completion_create (src/phoenix/server/api/helpers/playground_clients.py:775-781):

if (tool_calls := delta.tool_calls) is not None:
    for tool_call_index, tool_call in enumerate(tool_calls):   # position inside THIS delta
        tool_call_id = (
            tool_call.id
            if tool_call.id is not None
            else tool_call_ids[tool_call_index]                # KeyError if never seen
        )
        tool_call_ids[tool_call_index] = tool_call_id

In OpenAI's streaming protocol a delta normally carries a fragment for only one tool call, so the enumerate() position is almost always 0 no matter which call the fragment belongs to; delta.tool_calls[i].index is the field that identifies the call, and id is only sent on the first fragment of each call. Two consequences:

  1. Parallel tool calls get garbled arguments. Argument fragments for index 0 are attributed to whichever call id last landed at map key 0. Reassembled by id — which is what both the frontend and the span do — one call ends up with empty arguments and the other gets both calls' fragments concatenated into invalid JSON.
  2. A first fragment without an id raises KeyError: 0. The SDK types ChoiceDeltaToolCall.id as Optional[str], so a provider that starts a tool call without sending id kills the entire playground run with the bare error message 0.

Minimal reproduction (offline, no API key — synthetic ChatCompletionChunk deltas fed through _chat_completion_create):

import asyncio
from contextlib import asynccontextmanager
from typing import Any, AsyncIterator, Sequence

from phoenix.db.types.model_provider import LLMClientFactory
from phoenix.server.api.helpers.playground_clients import (
    OpenAIChatCompletionsClient,
    ToolCallChunk,
)


class FakeSpan:
    def set_attribute(self, *a: Any, **k: Any) -> None:
        pass

    def set_attributes(self, *a: Any, **k: Any) -> None:
        pass


def chunk(tool_calls: list[dict[str, Any]]) -> Any:
    from openai.types.chat import ChatCompletionChunk
    from openai.types.chat.chat_completion_chunk import (
        Choice,
        ChoiceDelta,
        ChoiceDeltaToolCall,
        ChoiceDeltaToolCallFunction,
    )

    return ChatCompletionChunk(
        id="chatcmpl-x",
        choices=[
            Choice(
                delta=ChoiceDelta(
                    tool_calls=[
                        ChoiceDeltaToolCall(
                            index=tc.get("index"),
                            id=tc.get("id"),
                            function=ChoiceDeltaToolCallFunction(
                                name=tc["function"].get("name"),
                                arguments=tc["function"].get("arguments"),
                            ),
                        )
                        for tc in tool_calls
                    ]
                ),
                finish_reason=None,
                index=0,
            )
        ],
        created=1700000000,
        model="gpt-4o",
        object="chat.completion.chunk",
    )


def make_client(stream_chunks: Sequence[Any]) -> OpenAIChatCompletionsClient:
    @asynccontextmanager
    async def create_client() -> Any:
        class S:
            _client = None

        yield S()

    client = OpenAIChatCompletionsClient(
        client_factory=LLMClientFactory(create=create_client, rate_limit_key=("openai",)),
        model_name="gpt-4o",
        provider="openai",
    )

    async def fake_stream(**kwargs: Any) -> Any:
        async def gen() -> AsyncIterator[Any]:
            for c in stream_chunks:
                yield c

        return gen()

    client._openai_chat_completion_create_stream = fake_stream  # type: ignore[method-assign]
    return client


async def run(label: str, stream_chunks: Sequence[Any]) -> None:
    print(f"=== {label} ===")
    client = make_client(stream_chunks)
    per_id: dict[str, str] = {}
    try:
        async for c in client._chat_completion_create(
            messages=[{"role": "user", "content": "weather and time in Paris?"}],
            tools=None,
            response_format=None,
            invocation_parameters=None,
            span=FakeSpan(),
            stream_model_output=True,
        ):
            assert isinstance(c, ToolCallChunk)
            per_id.setdefault(c.id, "")
            per_id[c.id] += c.function.arguments
        for tid, args in per_id.items():  # reassembled by id, as the frontend/span do
            print(f"  reassembled id={tid!r} arguments={args!r}")
    except Exception as e:
        print(f"  CRASH: {type(e).__name__}: {e}")
    print()


async def main() -> None:
    await run(
        "Scenario 1: interleaved parallel tool calls (ids first, then args)",
        [
            chunk([{"index": 0, "id": "call_A", "function": {"name": "get_weather"}}]),
            chunk([{"index": 1, "id": "call_B", "function": {"name": "get_time"}}]),
            chunk([{"index": 0, "function": {"arguments": '{"city": "Paris"}'}}]),
            chunk([{"index": 1, "function": {"arguments": '{"tz": "CET"}'}}]),
        ],
    )
    print("  expected: call_A -> '{\"city\": \"Paris\"}'  call_B -> '{\"tz\": \"CET\"}'\n")

    await run(
        "Scenario 1b: same, id deltas arriving in reverse order",
        [
            chunk([{"index": 1, "id": "call_B", "function": {"name": "get_time"}}]),
            chunk([{"index": 0, "id": "call_A", "function": {"name": "get_weather"}}]),
            chunk([{"index": 0, "function": {"arguments": '{"city": "Paris"}'}}]),
            chunk([{"index": 1, "function": {"arguments": '{"tz": "CET"}'}}]),
        ],
    )

    await run(
        "Scenario 2: first fragment arrives without an id (id: Optional[str])",
        [chunk([{"index": 0, "function": {"name": "get_weather", "arguments": "{}"}}])],
    )


if __name__ == "__main__":
    asyncio.run(main())

Actual output (verbatim at the commit above):

=== Scenario 1: interleaved parallel tool calls (ids first, then args) ===
  reassembled id='call_A' arguments=''
  reassembled id='call_B' arguments='{"city": "Paris"}{"tz": "CET"}'

  expected: call_A -> '{"city": "Paris"}'  call_B -> '{"tz": "CET"}'

=== Scenario 1b: same, id deltas arriving in reverse order ===
  reassembled id='call_B' arguments=''
  reassembled id='call_A' arguments='{"city": "Paris"}{"tz": "CET"}'

=== Scenario 2: first fragment arrives without an id (id: Optional[str]) ===
  CRASH: KeyError: 0

The trace is corrupted too: _llm_output_messages (playground_clients.py:4268) reassembles from the same id-keyed map, so for Scenario 1 the span records

llm.output_messages.0.message.tool_calls.0.tool_call.function.name = 'get_weather'
llm.output_messages.0.message.tool_calls.0.tool_call.id = 'call_A'
llm.output_messages.0.message.tool_calls.1.tool_call.function.arguments = '{"city": "Paris"}{"tz": "CET"}'
llm.output_messages.0.message.tool_calls.1.tool_call.function.name = 'get_time'
llm.output_messages.0.message.tool_calls.1.tool_call.id = 'call_B'

i.e. call_A has no arguments at all and call_B carries invalid concatenated JSON.

Expected behavior: fragments should be keyed by delta.tool_calls[i].index — the field OpenAI's streaming protocol defines for exactly this purpose (the openai SDK requires ChoiceDeltaToolCall.index: int while id is Optional[str], so index is the reliable key and id is the optional one). A fragment that has no id and no recorded id for its index should get a synthesized id instead of raising KeyError.

This is also how the sibling branches in the same file already handle it:

  • Bedrock keys active tool calls by contentBlockIndex ("Get the actual index"), playground_clients.py:1801-1840.
  • The Gemini branch synthesizes stable ids precisely because "The frontend merges streamed tool-call chunks by id", playground_clients.py:2843-2853.
  • The frontend merge-by-id with argument concatenation is addRepetitionPartialToolCall in js/app/src/store/playground/playgroundStore.tsx:1303-1323.

Root cause: src/phoenix/server/api/helpers/playground_clients.py:775 — enumerate(tool_calls) yields the fragment's position within the current delta, which is unrelated to which tool call the fragment belongs to, and that position is then used both to look up and to store the id (:779-781). The map key should be the fragment's own index (falling back to the enumerate position when absent), and a missing id should synthesize one rather than raise.

This affects every playground client inheriting OpenAICompatibleClient whenever the model streams tool calls: OpenAI (playground_clients.py:1973), Azure (:2056), DeepSeek, xAI, Ollama, Cerebras, Fireworks, Groq, Moonshot, MiniMax, Perplexity, Together, ZAI and Meta (:1274-1491) — reached from every streamed playground chat completion (src/phoenix/server/api/subscriptions.py:75 → chat_completion_create at :105).

Additional information

Activity

  1. BlueX888 commented on Oct 12, 2026

    @BlueX888
    Author

    Reproduced this. Root cause: src/phoenix/server/api/helpers/playground_clients.py:775 - enumerate(tool_calls) keyed the tool_call_ids map by the fragment's position within the current delta instead of by the fragment's own index field, so parallel tool-call fragments collided on position 0 and a first fragment without an id raised KeyError: 0

    Minimal reproduction:

    offline repro script from the issue body: synthetic ChatCompletionChunk tool-call deltas streamed through OpenAICompatibleClient._chat_completion_create with stream_model_output=True (Scenario 1: ids for index 0/1, then interleaved argument fragments; Scenario 2: first fragment without an id)
    

    Observed output:

    === Scenario 1: interleaved parallel tool calls (ids first, then args) ===
      reassembled id='call_A' arguments=''
      reassembled id='call_B' arguments='{"city": "Paris"}{"tz": "CET"}'
    === Scenario 2: first fragment arrives without an id (id: Optional[str]) ===
      CRASH: KeyError: 0
    

    Happy to open a PR with this fix, or to be assigned if you prefer to take it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageissues that need triage

    Type

    No type

    Projects

    • Status
      📘 Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions