Summary

Adds the Microsoft Teams (msteams) provider to the voice-call extension — inbound voice + inbound video vision, group/meeting awareness, and outbound "call me back" — over an HMAC-authenticated WebSocket endpoint. Works in 1:1 and group/meeting calls, on both voice-call modes (streaming: STT → agent → TTS, and realtime: speech-to-speech).

This is a refreshed, standalone superset of the earlier Teams PRs (see Relationship below) and is intended to replace them. It also carries the CVI rendering drivers — emotion expression cues, the viseme lip-shape spike, and assistant visual sharing (show_to_caller → the bot shows an image on its tile) — plus realtime continuous vision (sendImage, a small core plugin-sdk addition). The companion .NET worker renders; this PR sends the drivers.

✨ The big picture: a Conversational Video Interface (CVI) for OpenClaw

This PR is the foundation for AI Video Agents — bringing an interactive, face-to-face video interface to OpenClaw. The Conversational Video Interface (CVI) is the end-to-end pipeline for face-to-face AI, and its three pillars map directly onto what lands here:

Together these turn a Teams call into a true two-way video conversation with the agent: it watches what you show, talks with you in real time, and appears as an animated face. This is the centerpiece — perception and dialogue land in this PR; the rendering pillar runs in the companion media worker (source to be shared — see below), and richer continuous face-to-face video is the roadmap.

What's in this PR

Voice: msteams provider, caller allowlist (safe-by-default, closed unless matched), streaming + realtime paths, recording-status gate before any media-derived data is processed.

Inbound video (see): video.frame ingestion (buffered per source), the look_at_screen tool (realtime) + automatic per-turn frame attachment (streaming, so the agent is visually aware each turn), recording-gated and rate-limited. The agent sees the caller's camera and screen-share (VBSS). A per-call vision spend cap (msteams.maxVisionPerMinute, default 30, 0 = unlimited) bounds vision-model cost.

Group / meeting calls:

• Per-participant attribution — each video.frame carries the participant it belongs to (participantId / participantName); the streaming path labels the attached image and look_at_screen names the participant, so the vision model can reason about who is showing what.
• Speak only when addressed — without a gate, the assistant would reply to every utterance in a meeting and talk over participants. In a group call (2+ humans) it now stays silent until addressed by name — e.g. "Hey Aria, …" / any configured wakePhrases (case-insensitive, boundary-aware) — mirroring the chat channel's group @mention gate. Once addressed, a short follow-up window (followUpWindowMs) lets the back-and-forth continue without repeating the name, then it goes quiet again. 1:1 calls respond to everything (gate off). Deterministic on the streaming path (shouldRespondToGroupTurn keyed on the live participants count); instruction-based on realtime.

Realtime delegation & background tasks: in realtime (speech-to-speech) mode the model handles small talk itself and delegates real work to the OpenClaw agent — openclaw_agent_consult for quick in-line answers (a short "working on it" filler covers longer agent runs), and (under realtime.toolPolicy: "owner") openclaw_agent_task to run a long job in the background. A background result is delivered two ways: a Teams chat message (deliverVia:"message", the default) or an outbound voice call-back (deliverVia:"call"). Both are gated on recording status. If the caller has no Teams chat target (anonymous), a background task is refused up front rather than acking a delivery that can't happen.

Outbound (call me back): initiateCall places a Teams call via the provider's authenticated endpoint (SSRF-guarded); on the call-back the agent speaks the result, then hangs up (the result-delivery instruction is hardened so it states the result, not a generic greeting). If the callee doesn't answer, the bot leaves that same result as a Teams voicemail — which Teams' own transcription then emails to them — so the answer reaches the caller even when the call-back is missed. A configurable no-answer timeout (msteams.outbound.answerTimeoutMs, default 120s) finalizes a call that never connects, and timed-out call ids auto-expire so the dedupe set can't grow unbounded. tenantId inherited from channels.msteams.

Avatar expression cues: a cheap, dependency-free lexical heuristic (expression.ts, no extra model call) infers a coarse emotion — neutral / happy / sad / surprised — from the assistant's reply text and sends it to the worker as an additive expression message, on both the streaming (playTts) and realtime (assistant transcript, cued early + self-correcting) paths. The avatar shapes its mouth (smile / frown / round "O") accordingly. Best-effort and cosmetic: a send failure never disrupts the call, audio is untouched, and an older worker ignores the message.

Avatar viseme lip-shape (spike): an additive speech.marks message carries a viseme timeline so the avatar mouth changes shape per sound (wide "ee", round "oo", closed "m/b/p"), blended over the worker's RMS openness. The spike derives marks from the reply text (estimateVisemes); the full pass uses real ElevenLabs /with-timestamps character timing (visemesFromAlignment, landed) instead of an even spread. Streaming path only; falls back to RMS-only when absent.

Assistant visual sharing (show_to_caller): the bot can show the caller an image on its video tile mid-call. A realtime tool show_to_caller runs the agent (so its screenshot / image-generation tools are available) to produce one image; consultRealtimeVoiceAgent returns the produced trusted-local file path (remote URLs excluded — no SSRF surface), and the bridge reads it (image-type + 4 MB cap) and sends an additive display.image message the worker renders fullscreen for a few seconds, then returns to the avatar. Recording-gated like the other agent tools.

Realtime continuous vision (sendImage): a new RealtimeVoiceBridge.sendImage (a small core plugin-sdk addition + the openai/azure realtime provider) pushes the latest changed inbound frame into the realtime session as an ambient input_image conversation item with no forced response, every ~6 s, budget-capped and recording-gated — so the realtime model is continuously visually aware, not just on look_at_screen.

Sessions & memory: caller identity (Teams AAD object id, else a per-call-unique id) keys the session, so per-caller memory works and distinct anonymous callers never collide; sessions end cleanly on socket close.

Code quality / refactoring

Shared types & single-source defaults:

• Shared MsteamsVideoFrame type (msteams-video-frame.ts) — the provider and the realtime path reference one type instead of a structural duplicate (type-only import, no runtime cycle).
• Single gate-defaults resolver — resolveGroupCallGateConfig is the one source of the group-call defaults, used by the config layer, runtime wiring, and provider (no triplicated ?? true / ?? [...] / ?? 12000).
• One PCM sample-rate constant — MSTEAMS_PCM_SAMPLE_RATE_HZ (16 kHz) defined once; provider/realtime/tts alias it.

Extracted shared helpers (kill copy-paste in the provider + realtime paths):

• createStreamingCallState — one STT-session + per-call-state + connect/flush/fail factory shared by the inbound and outbound session-start paths.
• runMsteamsConsult — one consult invocation (model + thinking-level resolution + consultRealtimeVoiceAgent) shared by the consult / look_at_screen / background-task handlers.
• readArgText (in utils.ts) — shared tool-arg reader (replaced readArgString / readConsultArgText).
• describeMsteamsVideoFrameOwner — one place builds the "<name>'s camera/shared screen" vision-attribution string for both streaming and realtime.
• clearTimedOutOutbound helper + auto-expiry — one place forgets a timed-out outbound id and cancels its (unref'd) expiry timer; called from late-connect, session-end, and teardown.

Correctness/robustness:

• Outbound realtime vision wired — getLatestFrame now passed on call-backs too (was inbound-only), so look_at_screen works if the callee shares video.
• Validated TTS merge (TtsConfigSchema.parse, not a bare cast) and surfaced audio.frame send failures (streamPcmFrames try/catch → stop on WS failure) instead of silently succeeding.
• Pure, unit-tested gate core (group-call-gate.ts: isAddressed, shouldRespondToGroupTurn).

Relationship to #89724, #91107, #91176

• feat(voice-call): add Microsoft Teams voice provider (OpenClawTeamsBridge) #89724 = voice-only msteams provider.
• feat(voice-call): Microsoft Teams provider (msteams) — inbound voice + video vision #91107 = voice + inbound video vision.
• feat(voice-call): Microsoft Teams provider (msteams) — voice + inbound video + outbound call-back #91176 = voice + inbound video + outbound calling.
• This PR = the above + group-call attribution & speak-only-when-addressed gate + outbound timeout hardening, refreshed and standalone — intended to supersede all three. (They overlap; pick this one.)

Safety & compliance

• Recording gate: no audio-derived data and no video frame is processed before the call's recording status is active.
• Allowlist: inbound rejected unless the caller matches; default posture is closed.
• SSRF-guarded outbound place-call; HMAC-signed WS upgrade; shared secret via secretref.

Real behavior proof

• Behavior or issue addressed: Inbound Teams voice + video (look_at_screen) in 1:1 and group calls, the group "speak only when addressed" gate, a Teams-chat task callback, and an outbound voice call-back.

• Real environment tested: Live 1:1 and 3-party group Microsoft Teams calls against a running OpenClaw gateway (realtime path, vision-capable model). Real calls, not mocks.

• Exact steps or command run after this patch:

  1. pnpm openclaw gateway run with the msteams provider + realtime + outbound enabled.
  2. 1:1: shared a screen / turned on camera, asked "what's on my screen / can you see me?".
  3. Group (user + colleague + assistant): shared a screen, then turned the camera on, and asked the assistant to describe what it saw.
  4. From Teams chat / mid-call asked for a result "in the background, then call me back".

• Evidence (copied live output, redacted):

  be-seen avatar tile:         the bot joined the call as a video participant publishing its avatar
                                (the OpenClaw lobster) — a live Teams call shows the rendered avatar tile
                                alongside the caller's camera (rendering pillar; the companion worker
                                renders the NV12 tile, this PR sends the expression/viseme/display drivers).
  look_at_screen (screen):     "Your screen is showing the OpenClaw website in Chrome — the OpenClaw
                                logo, the tagline 'The AI that actually does things,' and a Quick Start
                                install section with a one-line command."
  look_at_screen (after I switched the shared window to GitHub):
                               "Now I see the GitHub repository page for openclaw/openclaw in Chrome —
                                the repo file list and stats." (proves the scene-change fix: switching
                                the shared content is now detected, not stuck on the previous page.)
  look_at_screen (camera):     "I see you holding a phone in front of your face, wearing a light-coloured
                                shirt; a modern office with desks and bright ceiling lights behind you."
  "Hey assistant" gate:        addressing the bot by its wake phrase ("Hey assistant, …") triggered a
                                response; the speak-only-when-addressed gate works on a live call.
  outbound "call me back":     the bot placed a Teams VOICE call that STATED THE RESULT, e.g. "here's the
                                Dubai time you asked for — it's 10:36 AM on Tuesday, June 9" (not a hello).
  unanswered call-back:        when the callee didn't pick up, the call went to Teams voicemail and the
                                bot left the SAME result; Teams' own voicemail transcription then emailed
                                it to the callee (audio.mp3 + transcript) — Microsoft platform behaviour,
                                not our code, but it confirms the call-back speaks the result.
  

• Observed result: On a live call the agent described the shared screen, correctly tracked a content switch (openclaw.ai → the openclaw GitHub page), described the camera, responded to the "Hey assistant" wake phrase, and stated the resolved result on the outbound call-back (incl. to voicemail when unanswered). The avatar renders as a live video tile.

• Attached evidence (redacted, from the live call):

  1. Be-seen avatar — the bot publishes its avatar as a live video tile

  2. Outbound "call me back" — the bot rings the caller on Teams

  3. Result delivery even to voicemail — Teams transcribes the call-back's spoken result

  4. Screen recording of the live call

  5. show_to_caller — the bot screenshots a page and displays it on its own video tile mid-call

• Agent transcripts (from the gateway logs, redacted):

  Screen share: "Your screen is showing the OpenClaw website in Chrome — the OpenClaw logo, the tagline 'The AI that actually does things,' and a Quick Start install section."

  After I switched the shared window to GitHub: "Now I see the GitHub repository page for openclaw/openclaw in Chrome — the repo file list and stats." (the scene-change fix: switching the shared content is detected, not stuck on the previous page.)

  Camera: "I see you holding a phone in front of your face, wearing a light-coloured shirt; a modern office with desks and bright ceiling lights behind you."

  Outbound call-back: "Here's the current local time in Dubai you asked for — it's 10:36 AM on Tuesday, June 9." (the call-back states the resolved result, not a generic hello.)

• show_to_caller display path (gateway + worker logs, redacted): on a live call the bot browsed to the OpenClaw GitHub page, screenshotted it, and rendered it on its own video tile.

  Gateway: voice-realtime-show:<callId> consult ran, produced the screenshot file, and sent display.image.

  Worker: [display] showing image for 5000 ms — the screenshot was rendered fullscreen on the bot's video tile, then it returned to the avatar.

• What was not tested: None — all paths in this PR were exercised on live Microsoft Teams calls (including show_to_caller, proven above).

• Proof limitations or environment constraints: Evidence is copied live output, redacted to avoid publishing screen/camera frames, call identifiers, and account details.

Config (example)

"voice-call": { "config": {
  "msteams": {
    "port": 8443, "path": "/voice/msteams/stream",
    "sharedSecret": "<secretref>", "requireRecordingStatus": true,
    "maxVisionPerMinute": 30,
    "outbound": { "enabled": true, "workerBaseUrl": "<authenticated endpoint>", "answerTimeoutMs": 120000 },
    "groupCall": { "requireAddress": true, "wakePhrases": ["assistant", "<bot name>"], "followUpWindowMs": 12000 }
  },
  "realtime": { "enabled": true }
} }

tenantId is read from channels.msteams.tenantId. groupCall only affects calls with 2+ humans; 1:1 always responds. maxVisionPerMinute (default 30, 0 = unlimited) caps the per-call vision spend across all three consumers — look_at_screen, streaming auto-attach, and the realtime ambient sendImage push.

Companion media worker & follow-ups

• Companion media worker — source to be shared. The avatar render + inbound-media bridge runs in a .NET Windows media worker (Microsoft Graph Communications + Microsoft.Skype.Bots.Media, Windows-only) that pairs with this provider over the HMAC WebSocket bridge. It renders the be-seen avatar — RMS lip-sync, subtle head bob/sway, eye-blink, attention gaze toward the active speaker, emotion-shaped and viseme-shaped mouths (driven by this PR's expression / speech.marks senders), and the display.image tile (show_to_caller) — and does adaptive inbound-video sampling (caller-speech + scene-change → faster sampling) with static-frame dedup. Per the maintainer's suggestion the worker source will be shared; candidate home is openclaw/openclaw (vendored subdirectory) or a dedicated openclaw/openclaw-windows-node repo. The bridge protocol is already versioned and additive, so either layout works — it's a repo-layout / release-ownership decision, not a code one.
• Continuous face-to-face CVI (the masterpiece direction): always-on two-way video — streaming perception (not just sampled frames), sub-second dialogue, and continuous photorealistic/animated rendering driven by viseme timing — built on the perception + dialogue foundation in this PR.
• Assistant visual sharing — PiP overlay (the fullscreen path ships in this PR; a picture-in-picture inset over the avatar is the follow-up).
• Viseme full pass — the spike renders end-to-end with text-estimated marks; swapping in ElevenLabs /with-timestamps real timing needs the TTS-path threading (optional speechMarks on the telephony result → playTts), additive/opt-in so other channels are untouched.
• Optional dominant-speaker camera selection (opt-in).
• Outbound call-back result-delivery robustness (the call-back connects; delivering the resolved result is being hardened — instruction tightened, deterministic delivery is a follow-up).