Repository navigation
fix(audio): preserve empty ASR transcripts when scoring WER - #1722
Merged
Yunnglin merged 1 commit intoSep 9, 2026
Merged
Conversation
Yunnglin
approved these changes
Sep 9, 2026
Yunnglin
left a comment
Yunnglin
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. The fix correctly preserves a valid empty ASR transcript and falls back only for None. The evaluation-version bump and regression coverage for field precedence, malformed payloads, and HTTP errors are appropriate; targeted tests and CI are green.
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An ASR response containing
{"text": ""}is currently treated as missing text. In Seed-TTS-Eval, the resulting exception is caught and recorded as WER=0, although an empty transcript against a nonempty reference should have WER=1.Preserve empty primary transcripts and consult fallback fields only when the primary value is None. Add adapter-level coverage, field-precedence tests and error controls. Bump Seed-TTS-Eval's evaluation version to v1.1 because the scoring behavior changes.
Validation: the same expanded suite produces 4 failures and 24 passes with the baseline metric implementation, and 28 passes with the fix. The regressions cover adapter scoring, metric scoring, and empty primary text being overwritten by alternate fields. Missing/null fields, non-object JSON responses, and HTTP errors remain errors at the metric boundary. Applicable pre-commit hooks and
git diff --checkpass.The scope is successful
/audio/transcriptionsresponses with string text fields. HTTP responses are mocked; no production incident or live ASR response is claimed. The adapter's general error-to-zero fallback and empty Responses-protocol outputs are separate unresolved issues.