Skip to content

Explicit audio models in tools.media.audio.models are ignored, falls back to active model (Claude) #7573

Description

@nezovskii

Description

Explicit audio transcription models configured in tools.media.audio.models or tools.media.models (with capabilities: ["audio"]) are ignored. Instead, audio is passed directly to the active chat model (Claude Opus 4.5 in our case), resulting in expensive multimodal transcription instead of using dedicated Whisper APIs.

Steps to Reproduce

  1. Configure Groq Whisper in openclaw.json:
{
  "env": {
    "vars": {
      "GROQ_API_KEY": "gsk_..."
    }
  },
  "tools": {
    "media": {
      "models": [
        {
          "provider": "groq",
          "model": "whisper-large-v3-turbo",
          "capabilities": ["audio"]
        }
      ],
      "audio": {
        "enabled": true,
        "timeoutSeconds": 200,
        "models": [
          {
            "provider": "groq",
            "model": "whisper-large-v3-turbo"
          }
        ]
      }
    }
  }
}
  1. Restart gateway
  2. Send voice message via WhatsApp (no native transcription)
  3. Check Groq Console - no requests appear
  4. Check Anthropic Console - requests appear (audio processed by Claude)

Expected Behavior

  • Explicit tools.media.audio.models should be used for audio transcription
  • Groq Whisper API should receive transcription requests
  • Audio should NOT fall back to active model when explicit models are configured

Actual Behavior

  • Explicit models are ignored
  • resolveModelEntries() appears to return entries, but they are not used
  • Audio is passed to Claude as multimodal attachment
  • Claude processes audio (expensive)

Investigation Notes

  • Verified Groq API key works: curl https://api.groq.com/openai/v1/models -H "Authorization: Bearer $GROQ_API_KEY" returns whisper models
  • groqProvider is registered in /src/media-understanding/providers/index.ts with capabilities: ["audio"]
  • anthropicProvider does NOT have transcribeAudio method, only describeImage
  • Config validation passes (valid: true)
  • Tested with both audio.models and shared models with capabilities: ["audio"] - same result

Environment

  • OpenClaw version: 2026.2.1
  • OS: macOS (arm64)
  • Active model: anthropic/claude-opus-4-5
  • Channels tested: WhatsApp, Telegram

Workaround

Currently none that preserves transcription. Disabling audio.enabled stops transcription entirely.

Related Code

  • src/media-understanding/runner.ts - resolveModelEntries(), runCapability()
  • src/media-understanding/resolve.ts - resolveModelEntries()
  • src/media-understanding/providers/groq/index.ts - Groq provider definition

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions