Skip to content

Feature: per-agent realtime voice in voice-call plugin #78190

Description

@legonhilltech-jpg

Summary

When using the voice-call plugin with OpenAI Realtime (bidirectional voice), all calls use a single global voice from realtime.providers.openai.voice. In a multi-agent setup where different agents handle different callers (via the callback transfer pattern), each agent should be able to have its own distinct OpenAI Realtime voice.

Current Behavior

realtime-handler creates every bridge session with the same providerConfig:

const session = createRealtimeVoiceBridgeSession({
    provider: this.realtimeProvider,
    providerConfig: this.providerConfig,  // same for ALL calls
    instructions: this.config.instructions,
    ...
});

All calls get the same voice regardless of which agent is handling the call.

Proposed Solution

Add a realtime.agentVoices config key that maps agentId → OpenAI voice name:

{
  "realtime": {
    "providers": {
      "openai": {
        "voice": "coral",  // default voice
        "model": "gpt-4o-realtime-preview"
      }
    },
    "agentVoices": {
      "main": "coral",
      "family-assistant": "ballad",
      "personal-assistant": "shimmer",
      "relationship-assistant": "echo",
      "research-assistant": "ash",
      "coder-assistant": "sage"
    }
  }
}

The handleCall() method already has access to agentId (line ~3004 in runtime-entry: const agentId = effectiveConfig.agentId ?? "main"). The fix would shallow-clone providerConfig and override voice when an agent-specific voice is configured:

const agentId = effectiveConfig.agentId ?? "main";
const agentVoice = this.config.agentVoices?.[agentId];
const callProviderConfig = agentVoice
    ? { ...this.providerConfig, voice: agentVoice }
    : this.providerConfig;

This covers:

  • Inbound calls: agent resolved from numbers[dialedNumber].agentId or default
  • Outbound calls: agentId from call metadata (already passed through)
  • Callback transfers: transfer_to_agent tool sets agentId on the outbound call

Use Case

Multi-agent voice system where ASANTE (receptionist, coral voice) answers all inbound calls, identifies the caller, and transfers to the appropriate agent via callback. Each agent should answer with their own distinct voice — Amma (family, ballad), Moneypenny (personal, shimmer), etc. This gives callers an immediate sense of who they are talking to.

Environment

  • OpenClaw: 2026.5.4
  • voice-call plugin: 2026.5.4
  • Realtime provider: OpenAI (gpt-4o-realtime-preview)
  • 7 agents, single Twilio number, callback transfer pattern

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low-priority cleanup, docs, polish, ergonomics, or speculative work.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions