azure_speech_recognition_services #

Speech-to-text for Flutter with Azure AI Speech, using Microsoft's official Speech SDKs (Android 1.53, iOS 1.52, JavaScript 1.53).

The API is stream-based: partial and final results, session state, mic level and typed errors all arrive as events. On top of that you get continuous recognition, language auto-detect, phrase lists, word timestamps, speaker diarization, pronunciation assessment and token auth with automatic refresh.

_{Screenshots are rendered from the example app with real fonts. The recognition events in them are scripted, because the screenshots are made without an Azure key.}

Features #

• Streams. results, finalResults, audioLevel, errors, and the full events stream as a sealed class for exhaustive switch. There is also a ValueListenable session state.
• Single-shot (recognizeOnce) and continuous recognition. Both can stop gracefully (the last phrase is still delivered) or cancel at once.
• Language identification across up to 4 languages at start, or up to 10 in continuous mode, where speakers can switch language mid-session.
• Phrase lists with weight, to bias recognition towards names and jargon.
• Word timestamps, the detailed n-best output (confidence, lexical, ITN), dictation mode, profanity masking, and Custom Speech endpoint IDs.
• Silence tuning: segmentation silence timeout, semantic segmentation, and initial and end silence timeouts.
• Speaker diarization (Guest-1, Guest-2, ...) through Azure's ConversationTranscriber.
• Pronunciation assessment: accuracy, fluency, completeness and prosody scores, plus per-word and per-phoneme scores with miscue detection.
• Stop phrases. Say "stop listening" to end a continuous session.
• Auth: a key and region, short-lived tokens refreshed for you inside the live recognizer, custom endpoints and hosts (containers, private links), and Microsoft Entra ID.
• Typed errors: AzureSpeechException carries a SpeechErrorCode mapped from Azure's cancellation codes, such as authenticationFailure, tooManyRequests and connectionFailure, plus permissionDenied and microphoneUnavailable.
• Built-in microphone permission check and request. No extra package needed.
• Mic level meter. The plugin captures 16 kHz mono PCM itself, in the service's native format, so no resampling happens downstream.
• prepare() pre-opens the connection, so the first word comes back sooner.
• Safe lifecycle. Calls are serialized, every event is stamped with its session, dispose() releases everything, and the mic is released on every exit path, including hot restart.

Platform support #

macOS, Windows and Linux aren't supported yet.

How each platform was verified for 1.0.0:

• Android: the example APK was built with AGP 8.11 and with AGP 9.1.
• Web: the example was built and run in Chromium. The JS SDK loaded, a session opened a WebSocket to Azure, and the failure with a test key came back as a typed error.
• iOS: the Swift was checked against the real 1.52.0 SDK headers, but it hasn't been compiled. No Mac was available. If you hit a problem on iOS, please open an issue.

Install #

dependencies:
  azure_speech_recognition_services: ^1.0.0

You need an Azure AI Speech resource for its key and region (for example westeurope).

Android #

• minSdk must be 26 or higher (android/app/build.gradle.kts).
• The RECORD_AUDIO and INTERNET permissions are merged in from the plugin's manifest automatically.
• R8/ProGuard keep rules for the Speech SDK ship with the plugin.

iOS #

Add the usage description to ios/Runner/Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>The microphone is used to transcribe your speech.</string>

The deployment target must be iOS 13 or higher. The plugin pulls in MicrosoftCognitiveServicesSpeech-iOS through CocoaPods. By default it sets the audio session to play-and-record (ducking other audio) while listening, and deactivates it afterwards. If your app manages AVAudioSession itself, pass manageAudioSession: false.

Web #

• Microphone access needs HTTPS; localhost also works.
• The Azure Speech JS SDK (565 KB) is loaded on first use from jsDelivr. It is pinned to 1.53.0 and checked with Subresource Integrity. If your Content Security Policy is strict, allow https://cdn.jsdelivr.net in script-src and wss://*.speech.microsoft.com in connect-src.
• To self-host, either add the bundle with a <script> tag (the plugin then uses window.SpeechSDK), or set AzureSpeechWeb.sdkUrl and sdkIntegrity.
• Code in the browser is public, so use token auth (below) in production.
• A wrong key shows up as connectionFailure on web, not authenticationFailure. Browsers hide the HTTP 401 behind WebSocket close code 1006.

Quick start #

import 'package:azure_speech_recognition_services/azure_speech_recognition_services.dart';

final speech = AzureSpeech(
  credentials: const AzureCredentials.subscription(key: myKey, region: 'westeurope'),
  config: const RecognitionConfig(languages: ['en-US']),
);

// One phrase.
final result = await speech.recognizeOnce();
print(result.text);

// Continuous, with live partial results.
speech.results.listen((r) {
  if (r.isFinal) {
    transcript.add(r.text);
  } else {
    partial = r.text; // changes as the user speaks
  }
});
speech.audioLevel.listen((level) => meter = level); // 0..1, ~20 Hz
speech.errors.listen((e) => showError(e.code, e.message));

await speech.startContinuous();
// ...
await speech.stop(); // the last phrase is still delivered
await speech.dispose();

Permission is asked for automatically the first time a session starts. To ask earlier, call requestPermission(), which returns a MicrophonePermission. Set autoRequestPermission: false to handle it yourself.

All events in one place #

speech.events.listen((event) {
  switch (event) {
    case RecognitionEvent(:final result) when result.isFinal:
      print('${result.speakerId ?? ''} [${result.language}] ${result.text}');
    case RecognitionEvent(:final result):
      print('… ${result.text}');
    case AudioLevelEvent(:final level):
      meter.value = level;
    case SpeechErrorEvent(:final error, fatal: true):
      print('failed: ${error.code}');
    case SessionStartedEvent() || SessionStoppedEvent():
    case SpeechStartDetectedEvent() || SpeechEndDetectedEvent():
    case SpeechErrorEvent():
  }
});

ValueListenableBuilder(
  valueListenable: speech.state, // idle, starting, listening, stopping
  builder: (context, state, _) => MicButton(listening: state == SpeechState.listening),
);

Authentication #

A key is fine for prototypes. Don't ship a key in a production app.

const AzureCredentials.subscription(key: key, region: 'westeurope')

Tokens are the right choice for production. Your backend exchanges the key for a 10-minute token:

POST https://<region>.api.cognitive.microsoft.com/sts/v1.0/issueToken
Ocp-Apim-Subscription-Key: <key>

The plugin fetches a token when a session starts. It fetches a new one every refreshInterval (9 minutes by default) and pushes it into the running recognizer, so long sessions don't drop.

AzureCredentials.token(
  region: 'westeurope',
  tokenProvider: () async => (await http.get(Uri.parse('https://api.example.com/speech-token'))).body,
)

For Microsoft Entra ID, return aad#<resourceId>#<entraAccessToken> from the provider.

Custom endpoints and containers:

AzureCredentials.endpoint(Uri.parse('wss://my-speech.cognitiveservices.azure.com/stt/speech/universal/v2'), key: key)
AzureCredentials.host(Uri.parse('ws://localhost:5000')) // Speech container

Configuration #

Pass a RecognitionConfig to AzureSpeech as the default, or to a single call. It is checked before anything is sent: invalid values throw AzureSpeechException(invalidConfiguration).

const RecognitionConfig(
  languages: ['en-US', 'de-DE', 'fr-FR'],      // 2+ → language auto-detect
  languageIdMode: LanguageIdMode.continuous,   // re-detect for every phrase
  phrases: ['Contoso', 'Jessie', 'Rehaan'],    // bias towards these words
  phraseWeight: 1.5,
  wordTimestamps: true,
  profanity: Profanity.removed,
  segmentationSilenceTimeout: Duration(milliseconds: 600),
  stopPhrases: ['stop listening'],
)

Pronunciation assessment #

final result = await speech.recognizeOnce(
  config: const RecognitionConfig(
    pronunciationAssessment: PronunciationAssessmentConfig(
      referenceText: 'The quick brown fox jumps over the lazy dog',
      granularity: PronunciationGranularity.phoneme,
      enableMiscue: true,   // flags omitted and inserted words
      enableProsody: true,  // en-US only
    ),
  ),
);
final scores = result.pronunciation!;
print('${scores.pronunciation} ${scores.accuracy} ${scores.fluency} ${scores.prosody}');
for (final w in result.words) {
  print('${w.text}: ${w.accuracyScore} ${w.errorType}'); // e.g. "Mispronunciation"
}

Errors #

Every failure is an AzureSpeechException, with a code, a message, the Azure cancellation reason, and isTransient (true when a retry may help).

When starting fails for a local reason (configuration, permission or mic), the error is thrown from the call. Errors that Azure reports after the session has started arrive on errors and end the session. recognizeOnce() throws both kinds.

Performance #

• Native threads. The main thread never blocks. On Android and iOS, all session work runs on one serial control thread, and SDK callbacks only post small event lists.
• Compact events. Partials and level updates (~20/s) go over the channel as short positional lists, not maps. The detailed JSON is only sent when a feature needs it.
• Audio capture. The plugin records 16 kHz, 16-bit mono PCM in 50 ms chunks into a reused buffer, which is the format the service expects.
• prepare() opens the WebSocket before the user taps the mic. On Android, the next matching start reuses that recognizer.
• Finalizing on stop. stop() ends the audio stream first, so the service finalizes the last phrase straight away instead of waiting for silence.

Testing your app #

The platform is pluggable. Swap in a fake and drive events through the real wire codec:

class FakeSpeech extends AzureSpeechPlatform with MockPlatformInterfaceMixin { ... }

AzureSpeechPlatform.instance = FakeSpeech();
// or: AzureSpeech(credentials: ..., platform: FakeSpeech())

See example/test/support/fake_platform.dart for a complete fake.

Migrating from 0.0.x #

The 0.0.x API (AzureSpeechRecognitionServices with callback setters) still works. It is deprecated and now runs on the new engine, so it gets the bug fixes too. See MIGRATION.md for the mapping from old calls to new ones.

Example #

example/ is a complete app: a live transcript with partial results, speakers and detected languages, a mic level meter, a settings sheet, and a pronunciation scorecard. Run it with your own key:

cd example
flutter run --dart-define=AZURE_KEY=<your-key> --dart-define=AZURE_REGION=westeurope

License #

MIT. The Azure Speech SDK is licensed separately by Microsoft under the Speech SDK license.