Oliver Yu

Embed2ASR

Speech recognition from a frozen EmbeddingGemma 2 audio encoder and a small GRU-CTC reader, measured against Whisper large-v3.

Solo project.

Diagram: audio goes into the frozen EmbeddingGemma 2 audio encoder, its pre-pooling last_hidden_state goes into a trainable GRU-CTC reader, and the reader outputs the transcript. The original pooled-embedding path is not used.
The encoder is never updated. The reader reads the states before pooling; the original pooled-embedding path is not used.

The problem

EmbeddingGemma 2’s audio path pools the encoder output into one vector per clip. That is enough for retrieval and it throws the time axis away. The question was whether the states before pooling still carry enough to transcribe speech, with the encoder untouched.

Embed2ASR is that experiment as code. The audio encoder stays frozen, and only a small reader on top is trained. One CTC output of 4,996 units covers Chinese characters and English BPE pieces, so Mandarin, English and code-switched speech share an output layer and need no language tag.

What it does

  • Reader. LayerNorm and a linear projection, three residual Conv1d blocks, a 2-layer BiGRU and a linear layer onto the CTC units. 9,655,172 trainable parameters on top of a 304,824,608-parameter frozen encoder. Training runs on cached encoder features, so the encoder runs once per utterance.
  • Speed. On one MI300X, same 200 clips, batch 1: real-time factor 0.0138 against 0.0576 for Whisper large-v3, which is 4.16× faster, with peak allocated GPU memory (torch, encoder included) of 0.75 GiB against 3.20 GiB. Cold start is slower, 7.9 s against 6.0 s.
  • Accuracy. It is ahead of Whisper large-v3 on the two parliament test sets and on the ASCEND mixed Mandarin-English sentences. Whisper is more accurate on 11 of the 14 test subsets, and the biggest relative gap is English: 24.5% WER against 4.5% on FLEURS en-US.
  • Where a frozen encoder stops. More data helped less and less, and even on sentences it had seen many times in training, the reader’s error stayed close to its dev error. I read that as the limit of the frozen encoder’s representation.
Bar charts: Embed2ASR-v1 against Whisper large-v3 on one MI300X. Real-time factor at two batch sizes, and peak GPU memory. Lower is better.
Speed and memory on one MI300X, lower is better. The conditions for each bar are in docs/benchmarks.md.
Bar chart of error rates on held-out test sets for Embed2ASR greedy, Embed2ASR with a language model, and Whisper large-v3. Whisper is lower on most sets.
Error rate per test set, lower is better. The ASCEND and ML2021_ASR_ST rows are the mixed-language subsets.

Evidence and limits

Every number above comes from the repository.

  • 9.66M counts trainable parameters only; inference runs the whole model, about 314.5M parameters.
  • The speed numbers come from one AMD GPU at batch 1. Reports from CUDA, Apple silicon or CPU are welcome; the repository has a guide.
  • The three sets where it leads are in-domain. On ML2021_ASR_ST, the only out-of-domain set, Whisper is clearly ahead: 29.5% MER against 16.0%.
  • Offline only, with no streaming. The training data is not part of the release.

The rest, with the evidence for each limit, is in docs/limitations.md and docs/benchmarks.md.

TingMix, the follow-up

TingMix adds LoRA on the encoder, a 22.2M reader, attention rescoring and a streaming API, and retrains the character LM. On Common Voice zh-TW, TingMix-Large reaches 12.02% CER against 15.62% here, and LibriSpeech clean goes from 8.99% to 7.05% WER. Those gaps include the decoder change. Comparing greedy to greedy, an earlier candidate only got Common Voice to 14.39%. Against the Whisper numbers from the Embed2ASR benchmark, it leads on Parliament A (17.10% vs 22.73% CER), though Parliament is in its training data and Whisper ran greedy with no LM. It trails on Common Voice (12.02% vs 9.60%) and four of the five English sets it reports; AMI IHM ties. TingMix-Large streams by re-decoding a buffer and needs a GPU for real time. It is a larger model with no matching speed measurement, so the speed numbers on this page do not carry over.