-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Feature: per-agent realtime voice in voice-call plugin #78190
Copy link
Copy link
Open
Labels
P3Low-priority cleanup, docs, polish, ergonomics, or speculative work.Low-priority cleanup, docs, polish, ergonomics, or speculative work.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P3Low-priority cleanup, docs, polish, ergonomics, or speculative work.Low-priority cleanup, docs, polish, ergonomics, or speculative work.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
Summary
When using the voice-call plugin with OpenAI Realtime (bidirectional voice), all calls use a single global voice from
realtime.providers.openai.voice. In a multi-agent setup where different agents handle different callers (via the callback transfer pattern), each agent should be able to have its own distinct OpenAI Realtime voice.Current Behavior
realtime-handlercreates every bridge session with the sameproviderConfig:All calls get the same voice regardless of which agent is handling the call.
Proposed Solution
Add a
realtime.agentVoicesconfig key that mapsagentId→ OpenAI voice name:{ "realtime": { "providers": { "openai": { "voice": "coral", // default voice "model": "gpt-4o-realtime-preview" } }, "agentVoices": { "main": "coral", "family-assistant": "ballad", "personal-assistant": "shimmer", "relationship-assistant": "echo", "research-assistant": "ash", "coder-assistant": "sage" } } }The
handleCall()method already has access toagentId(line ~3004 inruntime-entry:const agentId = effectiveConfig.agentId ?? "main"). The fix would shallow-cloneproviderConfigand overridevoicewhen an agent-specific voice is configured:This covers:
numbers[dialedNumber].agentIdor defaultagentIdfrom call metadata (already passed through)transfer_to_agenttool setsagentIdon the outbound callUse Case
Multi-agent voice system where ASANTE (receptionist, coral voice) answers all inbound calls, identifies the caller, and transfers to the appropriate agent via callback. Each agent should answer with their own distinct voice — Amma (family, ballad), Moneypenny (personal, shimmer), etc. This gives callers an immediate sense of who they are talking to.
Environment