feat(voice-call): Microsoft Teams provider (msteams) β voice + inbound video + outbound call-back#91176
Closed
ahenawy wants to merge 34 commits into
Closed
feat(voice-call): Microsoft Teams provider (msteams) β voice + inbound video + outbound call-back#91176ahenawy wants to merge 34 commits into
ahenawy wants to merge 34 commits into
Conversation
Adds a 5th voice-call provider, `msteams`, that bridges Microsoft Teams call
audio into the existing voice-call machinery (CallManager, realtime
transcription, response generation, telephony TTS) via an external Windows
worker over an HMAC-authenticated WebSocket (inbound-only, PCM 16 kHz).
Transport & security:
- HMAC-SHA256 authenticated WebSocket upgrade (x-openclawteamsbridge-timestamp /
-signature over `${ts}.${callId}`), constant-time compare, replay window.
- Binds loopback by default (msteams.bindAddress); 64 KB max payload; total +
per-IP connection caps; pre-start idle reaper; callId integrity check.
- sharedSecret is SecretRef-compatible (registered in the credential matrix).
Two modes:
- Agent (streaming): caller audio -> realtime STT -> OpenClaw agent (tools) ->
TTS. Early-audio buffered until STT connects; STT connect failure tears the
call down; runtime fails fast if streaming is enabled without a usable STT.
- Realtime (speech-to-speech): caller bridged to a realtime voice provider, which
delegates real work to the OpenClaw agent via openclaw_agent_consult (inline)
and openclaw_agent_task (owner-policy background task that delivers the result
to the caller's Teams chat). 16k<->24k resampling; barge-in.
Recording-status gate (Microsoft Media Access API): media-derived data is not
processed/persisted before Teams recording status is active. The streaming path
drops raw audio + transcripts; the realtime path refuses consult/task and does
not retain pre-recording transcripts. requireRecordingStatus (default true) is
operator-overridable.
setup/status exempts msteams from the public-webhook-exposure check (it uses the
bridge listener, not the webhook plane). Docs: voice-call README provider
section. Restores packages/plugin-sdk/src/testing.ts to its canonical single
re-export (an upstream docs pass had tripped the package-contract guardrail).
Tests cover the transport (HMAC/caps/recording protocol), provider audio loop,
realtime consult/async + recording gate, runtime fail-fast, and webhook-exposure
exemption.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦o option Addresses the two author-side review findings on the inbound provider: - [P1] Teams allowlists now match AAD callers. isAllowlistedCaller matches by phone number (digits) OR by exact caller id (case-insensitive), so a Teams aadId is no longer phone-normalized into a non-matching value. The msteams paths (realtime isInboundCallAllowed + streaming shouldAcceptInbound) pass the raw caller id; carrier paths still match by phone. Tests cover AAD-id matching. - [P2] Wire the documented realtime echo option. realtime.suppressInputDuringPlayback is now a real config field (schema + manifest + threaded into the msteams realtime deps), matching the README; it was previously documented but only an internal default. oxlint/oxfmt/tsgo prod+test clean; allowlist + webhook + config suites green; base-config-schema unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes the allowlist fix. Runtime matching already accepted AAD caller ids, but config + manifest validation still required E.164 for every allowFrom entry, so `inboundPolicy: "allowlist"` was unusable for Teams (a GUID failed validation before runtime ever saw it). allowFrom entries now accept an E.164 phone number OR a Teams AAD object id (GUID) in both the Zod schema and the plugin manifest; README documents the AAD-id form. Carrier providers are unchanged (E.164). Tests cover the schema accepting phone + GUID (either case) and rejecting non-E.164/non-GUID entries. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Addresses the reviewer's inbound-posture decision: msteams previously defaulted inboundPolicy to "open", accepting every authenticated Teams caller out of the box. It now defaults to an allowlist-oriented policy β with an empty allowFrom no caller is accepted until the operator opts callers in (the allowlist accepts AAD object ids) or sets inboundPolicy: "open" explicitly. No unsafe default; explicit policies are honored unchanged. README + tests updated. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
[P3] Keep manifest provider metadata in sync β the provider enum includes "msteams" but the human-readable description still only listed Twilio/Telnyx/ Plivo. Mention Microsoft Teams voice so the manifest metadata matches. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
[P1/P2] The public voice-call docs described only Twilio/Telnyx/Plivo/mock, so the new msteams provider shipped undocumented. Add it to the summary and provider list and add a "Microsoft Teams voice (msteams)" section covering the external Windows-worker WebSocket bridge, streaming vs realtime modes, the safe allowlist default + AAD-id allowFrom, the recording-status gate, and the HMAC/loopback security model, linking to the extension README for the full protocol. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦ty in session keys Two source-review defects: - Socket-close teardown: a started Teams session that dropped without an explicit session.end frame (worker crash, network loss, hangup) only released local connection tracking β onSessionEnd never fired, so the provider/manager call record leaked until the stale-call reaper. The ws "close" handler now delivers onSessionEnd (reason "socket-closed") for a started-but-not-ended session, guarded against double-delivery when an explicit session.end preceded it. - Session-key caller identity: when a Teams caller has no aadId, `from` fell back to the shared literal "teams", so every anonymous caller collided into one session key (cross-caller memory bleed under sessionScope: per-phone) and would match an allowlist as a generic caller. Fall back to a per-call-unique `teams:<callId>` instead; the AAD id is still preferred when present. Tests cover abrupt-close onSessionEnd, no double-fire after session.end, no fire before session.start, and the per-call-unique session key. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦arget A realtime openclaw_agent_task delivers its result to the caller's Teams chat (message tool -> user:<aadId>). When the bridge session has no caller.aadId (anonymous caller), handleAsyncTask still acked "I'll message you on Teams" and ran the task β but runAsyncTask had no delivery target, so the promised result was silently dropped. Refuse the task up front when caller.aadId is absent, returning a message that tells the model to offer to answer on the call instead. Regression test covers the no-aadId refusal (no false delivery ack). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦rd Teams worker docs - The two new socket-close tests used `(resolve) => setTimeout(resolve, 100)`, whose arrow body implicitly returns the timer handle β oxlint no-promise-executor-return. Use a block body so nothing is returned. This was failing both check-lint and check-additional-extension-bundled (lint:extensions:bundled). - Docs: describe the Teams bridge as a "Windows-side bridge worker" rather than "an external Windows worker (not part of this repo)". Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦ posture Reviewer follow-up: the setup examples showed `inboundPolicy: "open"` as the headline, which undercuts the safe default. The README and public-docs snippets now use `inboundPolicy: "allowlist"` with a sample AAD `allowFrom` entry, and note `"open"` only as the explicit accept-all alternative. Matches the code/ schema default (allowlist) and the now-corrected PR description. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
OpenClaw side of inbound vision (the worker already sends video.frame). The Teams bridge now parses the `video.frame` message (camera / screen-share JPEG), and the provider buffers the latest frame per source β recording-gated (Media Access API, parity with audio/transcripts) and path-agnostic (streaming + realtime) β exposing `getLatestVideoFrame()` for an upcoming "look" tool to attach to a vision model. Also raises the inbound WS payload cap 64 KB -> 2 MB: video frames are far larger than audio, and the sender is the HMAC-authenticated worker, so the looser bound only affects the trusted path. Tests cover video.frame ingestion and the new cap. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦he caller's screen/camera
Completes the inbound-vision loop (PR3). A realtime `look_at_screen` tool lets the
model answer visual questions ("what's on my screen?", "read this error"):
- pulls the latest buffered frame via the provider's getLatestVideoFrame() (per-call
accessor threaded into the realtime deps; screen-share preferred, then camera),
- runs the OpenClaw agent over it as an image (consultRealtimeVoiceAgent now forwards
an optional `images: ImageContent[]` to runEmbeddedAgent β a vision-capable run),
- speaks back a concise description; recording-gated like consult/transcripts.
Exposed only when the agent runtime + a frame source are available; returns a clear
"camera/share not on yet" message when no frame has arrived. Tests cover the
no-frame path and that the latest frame is passed through as an image.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦rding is active Adds the missing recording-gate coverage for video (parity with the audio/transcript gate tests): a video.frame before recording is active is not buffered, and frames after recording.status=active are. setup() now returns the provider so the test can assert getLatestVideoFrame(). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The realtime path has look_at_screen; the streaming path runs the agent per turn, so attach the latest shared video frame to that run (recording-gated via getLatestVideoFrame) β the streaming agent can now answer "what's on my screen?" in-turn. generateVoiceResponse gains an optional images param forwarded to the run. Test covers the frame riding along on a streaming turn. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦ged frame Bounds vision-model cost: look_at_screen (realtime) caches its answer keyed by the exact frame bytes and returns it instantly for repeat asks on an unchanged screen, re-running only when the frame changes. The streaming path likewise attaches a frame to the agent turn only when it changed since the last turn. Test covers the cache hit + re-run-on-change. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦assertion, unshadow provider)
β¦orrelate)
Lets OpenClaw place an outbound Teams call ("call me when the task is done")
through the external worker's HMAC HTTP endpoint, reusing the inbound streaming
media bridge for the conversation/notification.
- config: msteams.outbound { enabled, workerBaseUrl, tenantId }.
- MsteamsProvider.initiateCall: signs HMAC over `${ts}.${userObjectId}` and
POSTs {workerBaseUrl}/api/calls/place; tracks the returned callId in
pendingOutbound; returns it as providerCallId. (Realtime outbound deferred β
Phase 1 rides the streaming path.)
- handleSessionStart: when session.callId matches a pending outbound placement,
attach to the existing CallRecord and emit call.answered (skipping inbound
policy / call.initiated) so the manager speaks the result / starts the convo.
- manager/outbound: exempt msteams from the fromNumber guard (synthetic "from").
- media-stream: accept + expose the new session.start `direction` field.
- Tests: initiateCall disabled-guard + signed POST happy-path.
Worker side (separate repo): session.start now carries `direction:"outbound"`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
(cherry picked from commit 223ee6b)
β¦ia:"call"
- Realtime outbound: initiateCall no longer requires the streaming path. When a
placed call's media WS attaches under realtime mode, the speech-to-speech model
opens the call by speaking the delivered result (greeting), then converses.
The CallRecord is transitioned to answered without the manager re-speaking, and
session.end finalizes it (outboundRealtimeInternalIds).
- InitiateCallInput.message threads the result through manager -> provider so the
realtime greeting can speak it.
- openclaw_agent_task gains deliverVia ("message" | "call"): "call" makes the
background agent place a Teams call back to the caller (voice_call initiate_call,
notify mode) instead of sending a Teams chat message. Ack line adapts.
- Endpoint path updated to POST {workerBaseUrl}/api/calls (was /api/calls/place).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
(cherry picked from commit 19c0abd)
β¦lan) + tests
Fixes the UX bug where the realtime callback opened by restating the task in
future tense ("I'll find the Dubai time and call you back") instead of speaking
the finished result:
- Callback greeting now frames it as an OUTBOUND callback delivering an
already-completed result, instructing the model to state it directly and not
say it will look it up / call back.
- deliverVia:"call" instruction now tells the background agent to complete the
task FIRST and pass the FINAL ANSWER as the message (with an example), not a
description of what it is about to do.
Regression tests:
- realtime: openclaw_agent_task with deliverVia:"call" acks with a call-back line.
- provider: an outbound call placed via the worker (mocked /api/calls) attaches
to the existing CallRecord on session.start (correlation by callId), not a new
inbound call. (setup() gained an outbound option.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
(cherry picked from commit 3816b4e)
Enables the Teams CHAT agent to place an outbound voice call back to the user: - msteams channel: expose the inbound sender's Azure AD object id explicitly as `MsteamsAadObjectId` in the inbound context metadata (it was only implicitly the sender.id, conflated with the Bot Framework service id). - voice_call tool: clarify the initiate_call `to` description β for the msteams provider it's the target user's AAD object id (`user:<aadObjectId>`), e.g. the inbound chat sender's MsteamsAadObjectId β so the agent can call a chat user back. With this, "find X and call me when you're done" sent as a Teams chat lets the agent run the task then invoke voice_call initiate_call to the caller's aadObjectId. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> (cherry picked from commit 7f1a3b3)
Wires the "call me back" behavior in automatically (no system-prompt edit): the voice_call tool description now tells the agent that, to call a user back with a result (e.g. a Teams chat user saying "call me when you find the answer"), it should complete the task then use initiate_call with mode "notify", message = the final answer, and (for msteams) to = "user:<aadObjectId>" sourced from the inbound message's MsteamsAadObjectId. Combined with the explicit MsteamsAadObjectId metadata, the chat -> outbound-call flow no longer relies on system-prompt tuning. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> (cherry picked from commit 5baa313)
β¦rve-caught-error)
β¦t timeout (no leak)
β¦tenantId (no duplication)
β¦, surface audio.frame send failures
β¦w audio.frame send failure, guard late answer after no-answer timeout, typed channel tenantId, provider-neutral tool desc, drop dead re-export
β¦ms.outbound.answerTimeoutMs), default 120s with rationale
β¦d type (tsgo fix)
In a group call the worker subscribes one camera tile at a time, but the frame gave no hint whose it was β so a vision model could see a face yet not know who was speaking. Thread the participant through end to end: - video.frame schema + onVideoFrame callback gain optional participantId / participantName (back-compatible: absent for 1:1, guests, or older workers). - Shared MsteamsVideoFrame now also carries `source` (camera|screenshare) so a buffered frame is self-describing. - Provider buffers the identity and, on the streaming path, prefixes the turn with "[Attached image is <name>'s camera/shared screen]" when known. - Realtime look_at_screen names the participant in the consult surface text.
Mirror the chat channel's group @mention gate for Teams calls: in a 1:1 call the
bot replies to everything; in a group/meeting it stays quiet until addressed by
name. A call has no structured @mention, so "addressed" is matched from the
transcript against configurable wake phrases, with a short follow-up window so a
back-and-forth doesn't need the bot's name every turn.
- group-call-gate.ts: pure, unit-tested core β isAddressed (boundary-aware),
shouldRespondToGroupTurn (group + follow-up window), and a single
resolveGroupCallGateConfig() defaults resolver reused by every caller.
- config: msteams.groupCall { requireAddress, wakePhrases, followUpWindowMs }.
- media-stream: `participants` message (human count) -> onParticipants.
- Streaming path: deterministic gate in handleTranscript (count-based); the
utterance is still recorded so the agent has heard prior turns once addressed.
- Realtime path: the bridge exposes no response-suppress hook, so the gate is
applied as a model instruction (stay silent unless addressed by name).
The timedOutOutbound Set remembered a placed call's id so a late callee answer isn't mistaken for a fresh inbound call, but it was only cleared on a late connect or session end β a call that timed out and never connected late stayed forever, so the Set could grow unbounded. - finalizeUnansweredOutbound now arms a 5-minute self-expiry per entry (unref'd so it never keeps the process alive). - clearTimedOutOutbound() forgets the id and cancels its expiry timer; used by the late-connect path and session-end. - stop() cancels all outstanding outbound + expiry timers so none fire after teardown.
Author
|
Consolidating the Microsoft Teams CVI voice/video work into a single refreshed PR (rebased onto current upstream); the foundation PR #91438 stays open. Closing as superseded - its content/summary is carried into the consolidated PR. |
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 19, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 20, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 20, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 21, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 22, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 23, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 23, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 23, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 23, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 23, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 24, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
ahenawy
added a commit
to ahenawy/openclaw
that referenced
this pull request
Jun 25, 2026
Consolidated PR: the full Microsoft Teams integration for OpenClaw, rebased onto current upstream/main as one branch. Supersedes the closed PRs openclaw#89724, openclaw#91176, openclaw#92081 (content carried into the PR body); coexists with the foundation PR openclaw#91438. Includes: msteams voice provider (streaming + realtime), inbound video vision (look_at_screen + streaming), CVI rendering drivers (expression/viseme/show_to_caller/ display.image + realtime sendImage), group attribution + speak-only-when-addressed gate, outbound call-back, chat channel + governance, meeting minutes, roster/bilingual/DTMF, plus the latest reliability fixes (group-meeting wake gate, minutes recipient, screen-share). Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Microsoft Teams (
msteams) provider to thevoice-callextension β inbound voice + inbound video vision and outbound "call me back" β over an HMAC-authenticated WebSocket endpoint. Works in 1:1 calls and group/meeting calls, on both existing voice-call modes (streaming: STT β agent β TTS, and realtime: speech-to-speech).What's in this PR
Voice:
msteamsprovider, caller allowlist (safe-by-default), streaming + realtime paths.Inbound video (see):
video.frameingestion (buffered per source), thelook_at_screentool (realtime) + automatic frame attachment (streaming), recording-gated and rate-limited. The agent sees the caller's camera and screen-share (VBSS) and can answer visual questions about either.Group / meeting calls:
video.framecarries the participant it belongs to (participantId/participantName): the active speaker for a camera frame, the sharer for a screen-share. The streaming path labels the attached image ([Attached image is <name>'s camera/shared screen]) andlook_at_screennames the participant, so the vision model can reason about who is showing what in a meeting.wakePhrases), mirroring the chat channel's group @mention gate; a short follow-up window keeps a natural back-and-forth going without repeating the name. 1:1 calls respond to everything. Deterministic on the streaming path; instruction-based on the realtime path.Outbound (call me back):
initiateCallplaces a Teams call via the provider's authenticated endpoint;openclaw_agent_task deliverVia:"call"; the agent speaks the result then hangs up. A configurable no-answer timeout (msteams.outbound.answerTimeoutMs, default 120s) finalizes a call that never connects, so nothing leaks.tenantIdis inherited fromchannels.msteams.Relationship to #89724 and #91107
msteamsprovider.main(option a).Safety & compliance
Real behavior proof
look_at_screen) in 1:1 and group calls, Teams-chat task callback, and an outbound voice call-back to a Teams user.pnpm openclaw gateway runwith themsteamsprovider + realtime + outbound enabled.Config (example)
tenantIdis read fromchannels.msteams.tenantId.groupCallonly affects calls with 2+ humans; 1:1 calls always respond.Out-of-scope follow-ups