Skip to content

feat(voice-call): Microsoft Teams provider (msteams) β€” inbound voice + video vision#91107

Closed
ahenawy wants to merge 17 commits into
openclaw:mainfrom
ahenawy:feat/msteams-inbound-video
Closed

feat(voice-call): Microsoft Teams provider (msteams) β€” inbound voice + video vision#91107
ahenawy wants to merge 17 commits into
openclaw:mainfrom
ahenawy:feat/msteams-inbound-video

Conversation

@ahenawy

@ahenawy ahenawy commented Jun 7, 2026

Copy link
Copy Markdown

Summary

Adds a Microsoft Teams (msteams) provider to the voice-call extension: an inbound voice assistant for 1:1 Teams calls, plus optional inbound video vision so the agent can see the caller's camera or screen-share and answer questions about it.

Both existing voice-call interaction modes are supported:

  • Streaming (STT β†’ agent β†’ TTS)
  • Realtime (speech-to-speech)

What's in this PR

Voice (the msteams provider):

  • New msteams provider that exposes an HMAC-authenticated WebSocket endpoint for per-call audio.
  • Caller allowlist, safe-by-default (never open; matches Teams AAD object ids / UPNs).
  • Streaming + realtime paths; background-task delegation that messages the caller back on Microsoft Teams when long-running work finishes.
  • Config schema, plugin-manifest entry, and docs.

Video / vision (new here):

  • Inbound video.frame ingestion on the same authenticated endpoint; the provider buffers the latest frame per source (camera / screen-share).
  • look_at_screen tool (realtime) + automatic frame attachment (streaming): the agent runs a vision-capable model over the latest frame and answers in line.
  • Rate-limited (skips re-running on an unchanged frame) and recording-gated (no frame is processed before recording is active).

Relationship to #89724

#89724 proposes the voice-only msteams provider. This PR is a standalone superset β€” the same voice provider plus inbound video/vision. If maintainers prefer to land voice first, #89724 is the voice-only subset; otherwise this PR delivers both together. They overlap β€” pick one.

Safety & compliance

  • Recording gate: no audio-derived data and no video frame is processed/persisted before the call's recording status is active.
  • Allowlist: inbound is rejected unless the caller matches the configured allowlist; default posture is closed.
  • HMAC-signed upgrade; shared secret supplied via secretref.

Real behavior proof

  • Behavior or issue addressed: Inbound Microsoft Teams voice + video β€” the agent can answer questions about the caller's shared screen via look_at_screen, and hand a long-running ask off as a background task that replies on Microsoft Teams.
  • Real environment tested: A live 1:1 Microsoft Teams call against a running OpenClaw gateway (realtime speech path + a vision-capable model). A real call, not a mock.
  • Exact steps or command run after this patch:
    1. pnpm openclaw gateway run --force to start the gateway with the msteams provider and realtime enabled.
    2. Placed a 1:1 Microsoft Teams call to the assistant and shared a screen.
    3. Asked aloud: "what's on my screen?"
    4. Then asked: "what's the current Dubai time β€” check it in the background and message me." and ended the call.
  • Evidence after fix (copied live output / redacted runtime log):
    look_at_screen β€” spoken answer to the caller:
      "A Chrome window open to the LibreChat docs Quick Start page; center cards
       Docker, npm, Railway; left nav Quick Start, Deploy, Configuration,
       Services, Infrastructure."
    
    background task β€” Microsoft Teams chat message received after the call:
      1) "checking the current time in Dubai, I'll send you the result shortly"
      2) "the current time in Dubai is 11:12 AM, Sunday 7 June 2026 (GST)"
    
  • Observed result after fix: The agent described the live shared screen accurately and in line, and the background task delivered the Dubai time as a Microsoft Teams chat message after the call ended. Both matched expectations.
  • What was not tested: Multi-party (group-call) per-participant camera selection.
  • Proof limitations or environment constraints: Evidence is copied live output and redacted to avoid publishing the shared-screen image, call identifiers, and account details.

Tests and validation

  • Unit coverage across provider, media-stream, realtime, config, allowlist, and webhook-exposure (supplemental to the live proof above).

Config (example)

"voice-call": {
  "config": {
    "msteams": {
      "port": 8443,
      "path": "/voice/msteams/stream",
      "sharedSecret": "<secretref>",
      "requireRecordingStatus": true
    },
    "realtime": { "enabled": true }
  }
}

Out-of-scope follow-ups

  • Inbound camera is 1:1; screen-share (VBSS) is source-addressed and works in meetings. Per-participant camera selection in group calls is out of scope here.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation channel: voice-call Channel integration: voice-call size: XL triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 7, 2026
@clawsweeper

clawsweeper Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close because the useful Teams voice/video work is now owned by the newer clean, proof-sufficient replacement at #91438; keeping this branch open would split review across overlapping provider/config/security surfaces.

Canonical path: Close this branch and continue Teams voice/video review on #91438, with later stacked work handled only after that foundation is accepted or rejected.

So I’m closing this here and keeping the remaining discussion on #91438.

Review details

Best possible solution:

Close this branch and continue Teams voice/video review on #91438, with later stacked work handled only after that foundation is accepted or rejected.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a feature PR, not a broken existing behavior report. The PR body supplies real Teams-call proof for the proposed behavior, but no bug reproduction is required for this close decision.

Is this the best way to solve the issue?

No: this branch is no longer the best landing path because the newer replacement PR is a clean, proof-sufficient superset and is already the foundation for later stacked Teams work.

Security review:

Security review cleared: No separate supply-chain or credential-handling defect was identified in this read-only supersession pass; the security-sensitive listener and recording-gate review should continue on the canonical PR.

AGENTS.md: found and applied where relevant.

What I checked:

  • Repository policy applied: Root and scoped policy were read; provider routing, plugin config/defaults, SecretRef, media listener, and session-state changes are compatibility-sensitive under the review policy. (AGENTS.md:1, b1caba590631)
  • Current main has no Teams voice-call provider: Current main's voice-call provider enum still lists only telnyx, twilio, plivo, and mock, so this PR is not already implemented on main. (extensions/voice-call/src/types.ts:9, b1caba590631)
  • Current main loader has no msteams runtime: Current main's runtime lazy-provider module list includes Telnyx/Twilio/Plivo/Mock but no msteams provider loader. (extensions/voice-call/src/runtime.ts:59, b1caba590631)
  • This PR's central surface: Live PR metadata shows this branch is open, mergeable, and changes the voice-call docs, config, runtime, provider, media-stream, realtime, tests, SecretRef matrix, and talk consult image forwarding for the msteams voice/video provider. (94fff7885364)
  • Canonical replacement is viable: The newer PR at feat(voice-call): Microsoft Teams provider β€” CVI voice/video callsΒ #91438 is open, mergeable clean, labeled proof sufficient and ready for maintainer look, and its body explicitly says it is a refreshed standalone superset intended to supersede this PR plus the earlier Teams voice PRs. (02f7aedd4c25)
  • Superset file comparison: The canonical replacement touches every central msteams voice-call file from this PR and adds the group-call, outbound, SDK, OpenAI realtime, and hardening surfaces that maintainers should review together. (02f7aedd4c25)

Likely related people:

  • steipete: Recent GitHub history shows repeated work on voice-call config, runtime, plugin manifest, and session/config refactors that define the owner boundary this PR expands. (role: feature-history owner; confidence: high; commits: 39a931f1bfe0, e4aab1419ab8, 827b0de0ce74; files: extensions/voice-call/src/config.ts, extensions/voice-call/src/runtime.ts, extensions/voice-call/openclaw.plugin.json)
  • dynamite-bud: Introduced Telnyx Media Streaming and shared realtime audio plumbing adjacent to the Teams media-stream provider shape. (role: adjacent provider/realtime contributor; confidence: medium; commits: bb1b075118e5; files: extensions/voice-call/src/runtime.ts, extensions/voice-call/src/config.ts)
  • scoootscooob: Recent voice-call realtime Meet agent work touched delegated realtime consult/session behavior that this PR extends for Teams. (role: adjacent realtime voice contributor; confidence: medium; commits: 79dd65e20889; files: extensions/voice-call/src/runtime.ts, extensions/voice-call/openclaw.plugin.json)
  • VACInc: Added realtime consult overrides in the same talk/voice consult path that this PR extends with image forwarding. (role: adjacent consult contributor; confidence: medium; commits: cfb0c34ff6bd; files: src/talk/agent-consult-runtime.ts, extensions/voice-call/src/runtime.ts)

Codex review notes: model internal, reasoning high; reviewed against b1caba590631.

@clawsweeper clawsweeper Bot added rating: πŸ§‚ unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: πŸ“£ needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jun 7, 2026
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 7, 2026
@ahenawy
ahenawy marked this pull request as ready for review June 7, 2026 07:35
@ahenawy
ahenawy requested a review from a team as a code owner June 7, 2026 07:35
@ahenawy ahenawy changed the title voice-call: Microsoft Teams provider (msteams) β€” inbound voice + video vision feat(voice-call): Microsoft Teams provider (msteams) β€” inbound voice + video vision Jun 7, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: πŸ§‚ unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: πŸ§‚ unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: πŸ“£ needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 7, 2026
ahenawy and others added 8 commits June 7, 2026 08:12
Adds a 5th voice-call provider, `msteams`, that bridges Microsoft Teams call
audio into the existing voice-call machinery (CallManager, realtime
transcription, response generation, telephony TTS) via an external Windows
worker over an HMAC-authenticated WebSocket (inbound-only, PCM 16 kHz).

Transport & security:
- HMAC-SHA256 authenticated WebSocket upgrade (x-openclawteamsbridge-timestamp /
  -signature over `${ts}.${callId}`), constant-time compare, replay window.
- Binds loopback by default (msteams.bindAddress); 64 KB max payload; total +
  per-IP connection caps; pre-start idle reaper; callId integrity check.
- sharedSecret is SecretRef-compatible (registered in the credential matrix).

Two modes:
- Agent (streaming): caller audio -> realtime STT -> OpenClaw agent (tools) ->
  TTS. Early-audio buffered until STT connects; STT connect failure tears the
  call down; runtime fails fast if streaming is enabled without a usable STT.
- Realtime (speech-to-speech): caller bridged to a realtime voice provider, which
  delegates real work to the OpenClaw agent via openclaw_agent_consult (inline)
  and openclaw_agent_task (owner-policy background task that delivers the result
  to the caller's Teams chat). 16k<->24k resampling; barge-in.

Recording-status gate (Microsoft Media Access API): media-derived data is not
processed/persisted before Teams recording status is active. The streaming path
drops raw audio + transcripts; the realtime path refuses consult/task and does
not retain pre-recording transcripts. requireRecordingStatus (default true) is
operator-overridable.

setup/status exempts msteams from the public-webhook-exposure check (it uses the
bridge listener, not the webhook plane). Docs: voice-call README provider
section. Restores packages/plugin-sdk/src/testing.ts to its canonical single
re-export (an upstream docs pass had tripped the package-contract guardrail).

Tests cover the transport (HMAC/caps/recording protocol), provider audio loop,
realtime consult/async + recording gate, runtime fail-fast, and webhook-exposure
exemption.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…o option

Addresses the two author-side review findings on the inbound provider:

- [P1] Teams allowlists now match AAD callers. isAllowlistedCaller matches by
  phone number (digits) OR by exact caller id (case-insensitive), so a Teams
  aadId is no longer phone-normalized into a non-matching value. The msteams
  paths (realtime isInboundCallAllowed + streaming shouldAcceptInbound) pass the
  raw caller id; carrier paths still match by phone. Tests cover AAD-id matching.
- [P2] Wire the documented realtime echo option. realtime.suppressInputDuringPlayback
  is now a real config field (schema + manifest + threaded into the msteams
  realtime deps), matching the README; it was previously documented but only an
  internal default.

oxlint/oxfmt/tsgo prod+test clean; allowlist + webhook + config suites green;
base-config-schema unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes the allowlist fix. Runtime matching already accepted AAD caller ids,
but config + manifest validation still required E.164 for every allowFrom entry,
so `inboundPolicy: "allowlist"` was unusable for Teams (a GUID failed validation
before runtime ever saw it). allowFrom entries now accept an E.164 phone number
OR a Teams AAD object id (GUID) in both the Zod schema and the plugin manifest;
README documents the AAD-id form. Carrier providers are unchanged (E.164).

Tests cover the schema accepting phone + GUID (either case) and rejecting
non-E.164/non-GUID entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Addresses the reviewer's inbound-posture decision: msteams previously defaulted
inboundPolicy to "open", accepting every authenticated Teams caller out of the
box. It now defaults to an allowlist-oriented policy β€” with an empty allowFrom no
caller is accepted until the operator opts callers in (the allowlist accepts AAD
object ids) or sets inboundPolicy: "open" explicitly. No unsafe default; explicit
policies are honored unchanged. README + tests updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
[P3] Keep manifest provider metadata in sync β€” the provider enum includes
"msteams" but the human-readable description still only listed Twilio/Telnyx/
Plivo. Mention Microsoft Teams voice so the manifest metadata matches.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
[P1/P2] The public voice-call docs described only Twilio/Telnyx/Plivo/mock, so
the new msteams provider shipped undocumented. Add it to the summary and
provider list and add a "Microsoft Teams voice (msteams)" section covering the
external Windows-worker WebSocket bridge, streaming vs realtime modes, the safe
allowlist default + AAD-id allowFrom, the recording-status gate, and the
HMAC/loopback security model, linking to the extension README for the full
protocol.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ty in session keys

Two source-review defects:

- Socket-close teardown: a started Teams session that dropped without an explicit
  session.end frame (worker crash, network loss, hangup) only released local
  connection tracking β€” onSessionEnd never fired, so the provider/manager call
  record leaked until the stale-call reaper. The ws "close" handler now delivers
  onSessionEnd (reason "socket-closed") for a started-but-not-ended session,
  guarded against double-delivery when an explicit session.end preceded it.

- Session-key caller identity: when a Teams caller has no aadId, `from` fell back
  to the shared literal "teams", so every anonymous caller collided into one
  session key (cross-caller memory bleed under sessionScope: per-phone) and would
  match an allowlist as a generic caller. Fall back to a per-call-unique
  `teams:<callId>` instead; the AAD id is still preferred when present.

Tests cover abrupt-close onSessionEnd, no double-fire after session.end, no fire
before session.start, and the per-call-unique session key.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…arget

A realtime openclaw_agent_task delivers its result to the caller's Teams chat
(message tool -> user:<aadId>). When the bridge session has no caller.aadId
(anonymous caller), handleAsyncTask still acked "I'll message you on Teams" and
ran the task β€” but runAsyncTask had no delivery target, so the promised result
was silently dropped. Refuse the task up front when caller.aadId is absent,
returning a message that tells the model to offer to answer on the call instead.
Regression test covers the no-aadId refusal (no false delivery ack).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ahenawy and others added 9 commits June 7, 2026 08:12
…rd Teams worker docs

- The two new socket-close tests used `(resolve) => setTimeout(resolve, 100)`,
  whose arrow body implicitly returns the timer handle β€” oxlint
  no-promise-executor-return. Use a block body so nothing is returned. This was
  failing both check-lint and check-additional-extension-bundled
  (lint:extensions:bundled).
- Docs: describe the Teams bridge as a "Windows-side bridge worker" rather than
  "an external Windows worker (not part of this repo)".

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… posture

Reviewer follow-up: the setup examples showed `inboundPolicy: "open"` as the
headline, which undercuts the safe default. The README and public-docs snippets
now use `inboundPolicy: "allowlist"` with a sample AAD `allowFrom` entry, and
note `"open"` only as the explicit accept-all alternative. Matches the code/
schema default (allowlist) and the now-corrected PR description.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
OpenClaw side of inbound vision (the worker already sends video.frame). The Teams
bridge now parses the `video.frame` message (camera / screen-share JPEG), and the
provider buffers the latest frame per source β€” recording-gated (Media Access API,
parity with audio/transcripts) and path-agnostic (streaming + realtime) β€” exposing
`getLatestVideoFrame()` for an upcoming "look" tool to attach to a vision model.

Also raises the inbound WS payload cap 64 KB -> 2 MB: video frames are far larger
than audio, and the sender is the HMAC-authenticated worker, so the looser bound
only affects the trusted path. Tests cover video.frame ingestion and the new cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…he caller's screen/camera

Completes the inbound-vision loop (PR3). A realtime `look_at_screen` tool lets the
model answer visual questions ("what's on my screen?", "read this error"):
- pulls the latest buffered frame via the provider's getLatestVideoFrame() (per-call
  accessor threaded into the realtime deps; screen-share preferred, then camera),
- runs the OpenClaw agent over it as an image (consultRealtimeVoiceAgent now forwards
  an optional `images: ImageContent[]` to runEmbeddedAgent β€” a vision-capable run),
- speaks back a concise description; recording-gated like consult/transcripts.

Exposed only when the agent runtime + a frame source are available; returns a clear
"camera/share not on yet" message when no frame has arrived. Tests cover the
no-frame path and that the latest frame is passed through as an image.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…rding is active

Adds the missing recording-gate coverage for video (parity with the audio/transcript
gate tests): a video.frame before recording is active is not buffered, and frames
after recording.status=active are. setup() now returns the provider so the test can
assert getLatestVideoFrame().

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The realtime path has look_at_screen; the streaming path runs the agent per turn, so
attach the latest shared video frame to that run (recording-gated via
getLatestVideoFrame) β€” the streaming agent can now answer "what's on my screen?"
in-turn. generateVoiceResponse gains an optional images param forwarded to the run.
Test covers the frame riding along on a streaming turn.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ged frame

Bounds vision-model cost: look_at_screen (realtime) caches its answer keyed by the
exact frame bytes and returns it instantly for repeat asks on an unchanged screen,
re-running only when the frame changes. The streaming path likewise attaches a frame
to the agent turn only when it changed since the last turn. Test covers the cache
hit + re-run-on-change.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ahenawy
ahenawy force-pushed the feat/msteams-inbound-video branch from b15bfd1 to 94fff78 Compare June 7, 2026 08:15
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 7, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. and removed rating: πŸ§‚ unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 7, 2026
@clawsweeper clawsweeper Bot added status: πŸ‘€ ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 14, 2026
@clawsweeper

clawsweeper Bot commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this Jun 14, 2026
@ahenawy
ahenawy deleted the feat/msteams-inbound-video branch July 24, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: voice-call Channel integration: voice-call docs Improvements or additions to documentation merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XL status: πŸ‘€ ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant