Skip to content

Tool results silently dropped when provider returns composite tool_call ids (call_xxx|fc_yyy) — model never sees output and confabulates #63000

Description

@angersbach

Summary

When the active provider returns composite tool_call ids of the form call_xxx|fc_yyy (OpenAI-compatible gateways that bridge Chat Completions to the Responses API encode the function_call item id after a |), every tool result is silently dropped before the follow-up LLM call. The tool executes (side effects happen on the host), but the model only ever receives the sanitizer stub:

[Result unavailable — see context summary above]

The model then either reports the result as unavailable or — far worse — confabulates a plausible-looking tool output (in our case it consistently "quoted" a stale number from its hindsight memory as the raw output of a grep -c it never saw). This is a dangerous silent failure: the agent's answers look verified but are invented.

Environment

  • Hermes 0.16.0 (v2026.6.5-519-g5a4297a11), bug confirmed still present on current main (2026-07-11, v2026.7.7.2) by code inspection — see line refs below
  • Provider: custom: provider → LiteLLM proxy → OpenAI-compatible gateway that bridges Chat Completions to the Responses API (api_mode: chat_completions)
  • Platform: gateway / api_server; Linux, Python 3.12

Root cause (verified end-to-end with a logging proxy between Hermes and the provider)

The composite id is split in one place but not the other, and the strict id matching in the repair pass then eats the result:

  1. Assistant side (short id): agent/chat_completion_helpers.py splits the raw id via _split_responses_tool_id and persists the assistant tool_calls entry with id/call_id = short call_xxx (the fc_yyy part goes to response_item_id).
  2. Tool-result side (raw composite id): agent/tool_executor.py (line 1656 on current main) builds the result message with the raw SDK object id:
    tool_message = make_tool_result_message(function_name, _tool_content, tool_call.id)
    tool_call_id = call_xxx|fc_yyy.
  3. Repair pass drops it: agent/agent_runtime_helpers.pyrepair_message_sequence pass 1 collects known_tool_ids from the assistant tool_calls dicts (short ids) and exact-matches msg["tool_call_id"] against them. The composite id never matches → the fresh tool result is classified as a stray orphan and removed.
  4. Sanitizer stubs it: sanitize_api_messages then sees an assistant tool_call with no matching result and injects the [Result unavailable — see context summary above] stub, which is what the model receives.

The same raw-id append also happens on the error paths in agent/conversation_loop.py (invalid tool name / invalid JSON / execution error), so those messages are dropped identically.

Reproduction

Any OpenAI-compatible upstream whose tool_calls[].id differs from the internally normalized call_id triggers it — a |-composite id is the concrete case:

  1. Configure a custom_providers entry pointing at a gateway that returns ids like call_abc|fc_def (e.g. a Chat-Completions→Responses bridge behind LiteLLM).
  2. Ask the agent to run any tool, e.g. terminal with grep -c foo somefile | tee /tmp/out.txt.
  3. /tmp/out.txt contains the real output (tool ran), but the outbound follow-up request contains the stub instead of the result (easily verified with a local logging proxy as base_url), and the model answers from priors.

Note the provider itself is fine: replaying the identical two-turn exchange manually against the same gateway (both with the composite id and with only the call_ part echoed back, streaming and non-streaming) works correctly. The drop happens entirely inside Hermes between tool execution and the next API call.

Suggested fix

Normalize at the source so the result message always carries the same id shape as the persisted assistant message, e.g. in make_tool_result_message:

if isinstance(tool_call_id, str) and "|" in tool_call_id:
    tool_call_id = tool_call_id.split("|", 1)[0]

plus (defense in depth) tolerate both variants in repair_message_sequence pass 1 and in sanitize_api_messages. We are running exactly this 3-point patch locally and it fully restores tool round-trips; short ids are accepted end-to-end by the bridge (response_item_id already carries the fc_ part separately for the Codex adapter).

Possibly related

#55626 reports the same symptom ([Result unavailable] for tools that demonstrably executed, fresh session, OpenRouter) with no root cause identified — a provider-specific id-shape mismatch in the same exact-match pipeline would explain its "some tools work, some never do" pattern.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointsweeper:risk-session-stateSweeper risk: may lose/corrupt/mis-associate session or context statetype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions