feat(voice-call): Microsoft Teams provider (msteams) β inbound voice + video vision#91107
feat(voice-call): Microsoft Teams provider (msteams) β inbound voice + video vision#91107ahenawy wants to merge 17 commits into
Conversation
|
Thanks for the context here. I swept through the related work, and this is now duplicate or superseded. Close because the useful Teams voice/video work is now owned by the newer clean, proof-sufficient replacement at #91438; keeping this branch open would split review across overlapping provider/config/security surfaces. Canonical path: Close this branch and continue Teams voice/video review on #91438, with later stacked work handled only after that foundation is accepted or rejected. So Iβm closing this here and keeping the remaining discussion on #91438. Review detailsBest possible solution: Close this branch and continue Teams voice/video review on #91438, with later stacked work handled only after that foundation is accepted or rejected. Do we have a high-confidence way to reproduce the issue? Not applicable: this is a feature PR, not a broken existing behavior report. The PR body supplies real Teams-call proof for the proposed behavior, but no bug reproduction is required for this close decision. Is this the best way to solve the issue? No: this branch is no longer the best landing path because the newer replacement PR is a clean, proof-sufficient superset and is already the foundation for later stacked Teams work. Security review: Security review cleared: No separate supply-chain or credential-handling defect was identified in this read-only supersession pass; the security-sensitive listener and recording-gate review should continue on the canonical PR. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against b1caba590631. |
Adds a 5th voice-call provider, `msteams`, that bridges Microsoft Teams call
audio into the existing voice-call machinery (CallManager, realtime
transcription, response generation, telephony TTS) via an external Windows
worker over an HMAC-authenticated WebSocket (inbound-only, PCM 16 kHz).
Transport & security:
- HMAC-SHA256 authenticated WebSocket upgrade (x-openclawteamsbridge-timestamp /
-signature over `${ts}.${callId}`), constant-time compare, replay window.
- Binds loopback by default (msteams.bindAddress); 64 KB max payload; total +
per-IP connection caps; pre-start idle reaper; callId integrity check.
- sharedSecret is SecretRef-compatible (registered in the credential matrix).
Two modes:
- Agent (streaming): caller audio -> realtime STT -> OpenClaw agent (tools) ->
TTS. Early-audio buffered until STT connects; STT connect failure tears the
call down; runtime fails fast if streaming is enabled without a usable STT.
- Realtime (speech-to-speech): caller bridged to a realtime voice provider, which
delegates real work to the OpenClaw agent via openclaw_agent_consult (inline)
and openclaw_agent_task (owner-policy background task that delivers the result
to the caller's Teams chat). 16k<->24k resampling; barge-in.
Recording-status gate (Microsoft Media Access API): media-derived data is not
processed/persisted before Teams recording status is active. The streaming path
drops raw audio + transcripts; the realtime path refuses consult/task and does
not retain pre-recording transcripts. requireRecordingStatus (default true) is
operator-overridable.
setup/status exempts msteams from the public-webhook-exposure check (it uses the
bridge listener, not the webhook plane). Docs: voice-call README provider
section. Restores packages/plugin-sdk/src/testing.ts to its canonical single
re-export (an upstream docs pass had tripped the package-contract guardrail).
Tests cover the transport (HMAC/caps/recording protocol), provider audio loop,
realtime consult/async + recording gate, runtime fail-fast, and webhook-exposure
exemption.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦o option Addresses the two author-side review findings on the inbound provider: - [P1] Teams allowlists now match AAD callers. isAllowlistedCaller matches by phone number (digits) OR by exact caller id (case-insensitive), so a Teams aadId is no longer phone-normalized into a non-matching value. The msteams paths (realtime isInboundCallAllowed + streaming shouldAcceptInbound) pass the raw caller id; carrier paths still match by phone. Tests cover AAD-id matching. - [P2] Wire the documented realtime echo option. realtime.suppressInputDuringPlayback is now a real config field (schema + manifest + threaded into the msteams realtime deps), matching the README; it was previously documented but only an internal default. oxlint/oxfmt/tsgo prod+test clean; allowlist + webhook + config suites green; base-config-schema unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes the allowlist fix. Runtime matching already accepted AAD caller ids, but config + manifest validation still required E.164 for every allowFrom entry, so `inboundPolicy: "allowlist"` was unusable for Teams (a GUID failed validation before runtime ever saw it). allowFrom entries now accept an E.164 phone number OR a Teams AAD object id (GUID) in both the Zod schema and the plugin manifest; README documents the AAD-id form. Carrier providers are unchanged (E.164). Tests cover the schema accepting phone + GUID (either case) and rejecting non-E.164/non-GUID entries. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Addresses the reviewer's inbound-posture decision: msteams previously defaulted inboundPolicy to "open", accepting every authenticated Teams caller out of the box. It now defaults to an allowlist-oriented policy β with an empty allowFrom no caller is accepted until the operator opts callers in (the allowlist accepts AAD object ids) or sets inboundPolicy: "open" explicitly. No unsafe default; explicit policies are honored unchanged. README + tests updated. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
[P3] Keep manifest provider metadata in sync β the provider enum includes "msteams" but the human-readable description still only listed Twilio/Telnyx/ Plivo. Mention Microsoft Teams voice so the manifest metadata matches. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
[P1/P2] The public voice-call docs described only Twilio/Telnyx/Plivo/mock, so the new msteams provider shipped undocumented. Add it to the summary and provider list and add a "Microsoft Teams voice (msteams)" section covering the external Windows-worker WebSocket bridge, streaming vs realtime modes, the safe allowlist default + AAD-id allowFrom, the recording-status gate, and the HMAC/loopback security model, linking to the extension README for the full protocol. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦ty in session keys Two source-review defects: - Socket-close teardown: a started Teams session that dropped without an explicit session.end frame (worker crash, network loss, hangup) only released local connection tracking β onSessionEnd never fired, so the provider/manager call record leaked until the stale-call reaper. The ws "close" handler now delivers onSessionEnd (reason "socket-closed") for a started-but-not-ended session, guarded against double-delivery when an explicit session.end preceded it. - Session-key caller identity: when a Teams caller has no aadId, `from` fell back to the shared literal "teams", so every anonymous caller collided into one session key (cross-caller memory bleed under sessionScope: per-phone) and would match an allowlist as a generic caller. Fall back to a per-call-unique `teams:<callId>` instead; the AAD id is still preferred when present. Tests cover abrupt-close onSessionEnd, no double-fire after session.end, no fire before session.start, and the per-call-unique session key. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦arget A realtime openclaw_agent_task delivers its result to the caller's Teams chat (message tool -> user:<aadId>). When the bridge session has no caller.aadId (anonymous caller), handleAsyncTask still acked "I'll message you on Teams" and ran the task β but runAsyncTask had no delivery target, so the promised result was silently dropped. Refuse the task up front when caller.aadId is absent, returning a message that tells the model to offer to answer on the call instead. Regression test covers the no-aadId refusal (no false delivery ack). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦rd Teams worker docs - The two new socket-close tests used `(resolve) => setTimeout(resolve, 100)`, whose arrow body implicitly returns the timer handle β oxlint no-promise-executor-return. Use a block body so nothing is returned. This was failing both check-lint and check-additional-extension-bundled (lint:extensions:bundled). - Docs: describe the Teams bridge as a "Windows-side bridge worker" rather than "an external Windows worker (not part of this repo)". Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦ posture Reviewer follow-up: the setup examples showed `inboundPolicy: "open"` as the headline, which undercuts the safe default. The README and public-docs snippets now use `inboundPolicy: "allowlist"` with a sample AAD `allowFrom` entry, and note `"open"` only as the explicit accept-all alternative. Matches the code/ schema default (allowlist) and the now-corrected PR description. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
OpenClaw side of inbound vision (the worker already sends video.frame). The Teams bridge now parses the `video.frame` message (camera / screen-share JPEG), and the provider buffers the latest frame per source β recording-gated (Media Access API, parity with audio/transcripts) and path-agnostic (streaming + realtime) β exposing `getLatestVideoFrame()` for an upcoming "look" tool to attach to a vision model. Also raises the inbound WS payload cap 64 KB -> 2 MB: video frames are far larger than audio, and the sender is the HMAC-authenticated worker, so the looser bound only affects the trusted path. Tests cover video.frame ingestion and the new cap. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦he caller's screen/camera
Completes the inbound-vision loop (PR3). A realtime `look_at_screen` tool lets the
model answer visual questions ("what's on my screen?", "read this error"):
- pulls the latest buffered frame via the provider's getLatestVideoFrame() (per-call
accessor threaded into the realtime deps; screen-share preferred, then camera),
- runs the OpenClaw agent over it as an image (consultRealtimeVoiceAgent now forwards
an optional `images: ImageContent[]` to runEmbeddedAgent β a vision-capable run),
- speaks back a concise description; recording-gated like consult/transcripts.
Exposed only when the agent runtime + a frame source are available; returns a clear
"camera/share not on yet" message when no frame has arrived. Tests cover the
no-frame path and that the latest frame is passed through as an image.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦rding is active Adds the missing recording-gate coverage for video (parity with the audio/transcript gate tests): a video.frame before recording is active is not buffered, and frames after recording.status=active are. setup() now returns the provider so the test can assert getLatestVideoFrame(). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The realtime path has look_at_screen; the streaming path runs the agent per turn, so attach the latest shared video frame to that run (recording-gated via getLatestVideoFrame) β the streaming agent can now answer "what's on my screen?" in-turn. generateVoiceResponse gains an optional images param forwarded to the run. Test covers the frame riding along on a streaming turn. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦ged frame Bounds vision-model cost: look_at_screen (realtime) caches its answer keyed by the exact frame bytes and returns it instantly for repeat asks on an unchanged screen, re-running only when the frame changes. The streaming path likewise attaches a frame to the agent turn only when it changed since the last turn. Test covers the cache hit + re-run-on-change. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
β¦assertion, unshadow provider)
b15bfd1 to
94fff78
Compare
|
ClawSweeper applied the proposed close for this PR.
|
Summary
Adds a Microsoft Teams (
msteams) provider to thevoice-callextension: an inbound voice assistant for 1:1 Teams calls, plus optional inbound video vision so the agent can see the caller's camera or screen-share and answer questions about it.Both existing voice-call interaction modes are supported:
What's in this PR
Voice (the
msteamsprovider):msteamsprovider that exposes an HMAC-authenticated WebSocket endpoint for per-call audio.Video / vision (new here):
video.frameingestion on the same authenticated endpoint; the provider buffers the latest frame per source (camera / screen-share).look_at_screentool (realtime) + automatic frame attachment (streaming): the agent runs a vision-capable model over the latest frame and answers in line.Relationship to #89724
#89724 proposes the voice-only
msteamsprovider. This PR is a standalone superset β the same voice provider plus inbound video/vision. If maintainers prefer to land voice first, #89724 is the voice-only subset; otherwise this PR delivers both together. They overlap β pick one.Safety & compliance
Real behavior proof
look_at_screen, and hand a long-running ask off as a background task that replies on Microsoft Teams.pnpm openclaw gateway run --forceto start the gateway with themsteamsprovider and realtime enabled.Tests and validation
Config (example)
Out-of-scope follow-ups