Skip to content

feat: add NVIDIA Nemotron Speech ASR and TTS#112078

Open
jmayank1511 wants to merge 3 commits into
openclaw:mainfrom
jmayank1511:nemotron-speech
Open

feat: add NVIDIA Nemotron Speech ASR and TTS#112078
jmayank1511 wants to merge 3 commits into
openclaw:mainfrom
jmayank1511:nemotron-speech

Conversation

@jmayank1511

@jmayank1511 jmayank1511 commented Jul 21, 2026

Copy link
Copy Markdown

What Problem This Solves

NVIDIA users could configure NVIDIA-hosted chat models, but could not reuse the same NVIDIA credential for OpenClaw's batch speech-to-text and text-to-speech workflows. This prevented inbound audio from using Parakeet and outbound spoken replies from using Magpie through the NVIDIA plugin.

Why This Change Was Made

This extends the existing NVIDIA plugin with one-shot HTTP Parakeet ASR and Magpie TTS providers. It follows OpenClaw's media and speech provider contracts, reuses shared NVIDIA authentication, normalizes unsupported inbound formats through trusted FFmpeg discovery, and deliberately does not register realtime or streaming speech providers.

User Impact

Users can now:

  • Transcribe inbound audio with NVIDIA Parakeet TDT, with automatic CTC 1.1B fallback.
  • Generate spoken replies with Magpie Multilingual and its configurable voice, language, sample rate, dictionary, and customization fields.
  • Reuse the NVIDIA credential already configured for chat models.
  • Point ASR or TTS at compatible self-hosted NVIDIA NIM HTTP endpoints.

Documentation now covers setup, supported formats, customization, hosted endpoint overrides, and provider capabilities.

Evidence

Focused NVIDIA speech tests pass:

Test Files  2 passed (2)
Tests      12 passed (12)

Command:

./node_modules/.bin/vitest run \
  extensions/nvidia/nvidia-speech-http.runtime.test.ts \
  extensions/nvidia/speech-provider.test.ts

The branch was also checked with git diff --check upstream/main...HEAD.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation extensions: nvidia extensions: tts-local-cli plugin: azure-speech Azure Speech plugin size: L labels Jul 21, 2026
@BradGroux BradGroux mentioned this pull request Jul 21, 2026
Draft
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jul 23, 2026
@clawsweeper

clawsweeper Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 21, 2026, 1:43 AM ET / 05:43 UTC.

Summary
The branch adds NVIDIA Parakeet batch ASR and Magpie batch TTS to the NVIDIA plugin, registering media and speech providers and documenting their setup and options.

PR surface: Source +498, Tests +344, Docs +152. Total +994 across 13 files.

Reproducibility: yes. from source: an existing configuration with NVIDIA auth and automatic media audio enabled can become eligible for the newly registered NVIDIA audio provider through its default model and auto-priority. The PR does not include a live upgrade-path demonstration confirming that this routing change is intended.

Review metrics: 1 noteworthy metric.

  • Automatic provider routing: 1 new audio auto-priority registration. A saved NVIDIA credential can now affect automatic batch transcription selection for existing users, not only newly configured speech users.

Stored data model
Persistent data-model change detected: serialized state: extensions/nvidia/nvidia-speech-http.runtime.test.ts, serialized state: extensions/nvidia/nvidia-speech-http.runtime.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Resolve whether existing NVIDIA credentials may auto-enable inbound ASR, then cover the selected upgrade behavior.
  • [P1] Add redacted real behavior proof for one Parakeet transcription and one Magpie synthesis request; redact API keys, endpoints, and other private data.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body provides focused mock-based test results and a whitespace check, but no redacted live request, terminal output, recording, or runtime log showing successful ASR/TTS against NVIDIA or a compatible NIM endpoint after the change. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] Merging the automatic audio registration can cause existing installations with an NVIDIA chat credential to start selecting NVIDIA for inbound transcription, changing provider routing and sending audio to a new remote service without an explicit speech-selection upgrade path.

Maintainer options:

  1. Make ASR activation explicit (recommended)
    Remove or gate NVIDIA's automatic audio priority so pre-existing NVIDIA credentials do not change transcription routing until users select NVIDIA audio.
  2. Accept the new automatic default
    Approve credential-only NVIDIA ASR selection after documenting the upgrade and data-routing change and adding fresh-install and upgrade evidence.

Next step before merge

  • [P1] A maintainer must choose the intended upgrade behavior for existing NVIDIA credentials before a mechanical repair can be safely scoped.

Maintainer decision needed

  • Question: Should an existing NVIDIA chat credential automatically make NVIDIA Parakeet eligible for batch inbound-audio transcription, or should users explicitly select NVIDIA in audio configuration?
  • Rationale: The proposed autoPriority makes a product-default and external-data-routing choice for existing authenticated users; source tests cannot establish whether that compatibility change is intended.
  • Likely owner: steipete — This is a provider-routing and default-behavior decision spanning core discovery and a bundled provider plugin.
  • Options:
    • Require explicit NVIDIA audio selection (recommended): Register the provider without automatic audio eligibility, and enable Parakeet only when users configure NVIDIA under tools.media.audio.models.
    • Approve credential-only automatic selection: Keep the auto-priority, but document the upgrade behavior and provide real proof that fresh and existing NVIDIA-authenticated configurations route as intended.

Security
Cleared: The reviewed diff adds scoped HTTPS provider adapters and does not add dependency, workflow, secret-permission, or package-execution changes; the routing/privacy consequence of automatic credential reuse is tracked as a compatibility concern.

Review findings

  • [P1] Gate automatic NVIDIA transcription behind explicit audio selection — extensions/nvidia/audio-transcription-provider.ts:8-10
Review details

Best possible solution:

Keep NVIDIA ASR/TTS in the existing NVIDIA plugin, but make automatic transcription opt-in through explicit audio configuration unless maintainers deliberately approve credential-only auto-selection and its privacy, routing, and upgrade behavior.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: an existing configuration with NVIDIA auth and automatic media audio enabled can become eligible for the newly registered NVIDIA audio provider through its default model and auto-priority. The PR does not include a live upgrade-path demonstration confirming that this routing change is intended.

Is this the best way to solve the issue?

No, not yet. Reusing the existing NVIDIA plugin and credential is the right ownership boundary, but automatic ASR selection needs an explicit compatibility decision or an opt-in configuration path before it is the safest solution.

Full review comments:

  • [P1] Gate automatic NVIDIA transcription behind explicit audio selection — extensions/nvidia/audio-transcription-provider.ts:8-10
    Registering this provider with a default audio model and autoPriority makes any existing NVIDIA-authenticated profile eligible for automatic inbound transcription. That silently changes provider routing and remote audio handling for users who only configured NVIDIA chat; require an explicit audio selection, or obtain and document a deliberate upgrade decision with coverage for that path.
    Confidence: 0.93

Overall correctness: patch is incorrect
Overall confidence: 0.9

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against b521a85d85f6.

Label changes

Label changes:

  • add P1: The branch can change a working inbound-audio workflow for existing NVIDIA-authenticated installs at runtime.
  • add merge-risk: 🚨 compatibility: The new automatic provider registration changes upgrade-time defaults and fallback selection without an explicit existing-user opt-in.
  • add merge-risk: 🚨 auth-provider: NVIDIA credential resolution now participates in a new media-provider routing path.
  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body provides focused mock-based test results and a whitespace check, but no redacted live request, terminal output, recording, or runtime log showing successful ASR/TTS against NVIDIA or a compatible NIM endpoint after the change. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Label justifications:

  • P1: The branch can change a working inbound-audio workflow for existing NVIDIA-authenticated installs at runtime.
  • merge-risk: 🚨 compatibility: The new automatic provider registration changes upgrade-time defaults and fallback selection without an explicit existing-user opt-in.
  • merge-risk: 🚨 auth-provider: NVIDIA credential resolution now participates in a new media-provider routing path.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body provides focused mock-based test results and a whitespace check, but no redacted live request, terminal output, recording, or runtime log showing successful ASR/TTS against NVIDIA or a compatible NIM endpoint after the change. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +498, Tests +344, Docs +152. Total +994 across 13 files.

View PR surface stats
Area Files Added Removed Net
Source 6 498 0 +498
Tests 2 344 0 +344
Docs 5 162 10 +152
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 13 1004 10 +994

What I checked:

  • Automatic routing change: The new media-understanding provider advertises an audio default model and an auto-priority, while the accompanying audio documentation adds NVIDIA to the configured-provider priority order; existing NVIDIA-authenticated profiles can therefore become eligible for automatic inbound transcription. (extensions/nvidia/audio-transcription-provider.ts:8, 51f1b6064b75)
  • Existing credential reuse: The PR description and NVIDIA documentation explicitly state that the shared NVIDIA credential is reused for Parakeet ASR and Magpie TTS, so this is not limited to newly created dedicated speech credentials. Public docs: docs/providers/nvidia.md. (docs/providers/nvidia.md:36, 51f1b6064b75)
  • Repository policy: The root AGENTS.md treats provider routing, defaults, config loading, and fallback behavior as compatibility-sensitive, and requires explicit upgrade safety for behavior that changes existing users' configured flows. (AGENTS.md:1, b521a85d85f6)
  • Focused tests are mock-based: The submitted proof reports two focused Vitest files and 12 passing tests; the added runtime test mocks provider HTTP helpers and transcoding, so it does not demonstrate an after-fix request against an NVIDIA or compatible NIM endpoint. (extensions/nvidia/nvidia-speech-http.runtime.test.ts:1, 51f1b6064b75)

Likely related people:

  • steipete: The routing question crosses the existing provider-auth, media-provider, and bundled-plugin boundary and needs repository-level product direction rather than attribution to this external PR author. (role: likely product and provider-boundary decision owner; confidence: low; files: extensions/nvidia/index.ts, extensions/nvidia/audio-transcription-provider.ts, docs/nodes/audio.md)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docs Improvements or additions to documentation extensions: nvidia extensions: tts-local-cli merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. plugin: azure-speech Azure Speech plugin rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: L status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jmayank1511