Skip to content

Using the whisper initial_prompt field for custom words#1035

Merged
cjpais merged 1 commit intocjpais:mainfrom
jmcampanini:jcamp/whisper-initial-prompt
Mar 14, 2026
Merged

Using the whisper initial_prompt field for custom words#1035
cjpais merged 1 commit intocjpais:mainfrom
jmcampanini:jcamp/whisper-initial-prompt

Conversation

@jmcampanini
Copy link
Copy Markdown
Contributor

Whisper supports an initial_prompt parameter that biases the model's vocabulary during transcription. For Whisper models, we now join the user's custom words list into a comma-separated string and pass it as the initial_prompt. This guides the model to prefer those spellings during decoding rather than relying solely on fuzzy post-correction.

For non-Whisper engines (Parakeet, Moonshine, SenseVoice, GigaAM), the existing apply_custom_words post-correction continues to apply since those engines don't support prompt-based vocabulary biasing.

Before Submitting This PR

Please confirm you have done the following:

If this is a feature or change that was previously closed/rejected:

  • I have explained in the description below why this should be reconsidered
  • I have gathered community feedback (link to discussion below)

Human Written Description

From the discussion:

I've encountered an issue with custom vocabulary/custom words in Handy. The current bigram replacement mechanism can be overly aggressive - it sometimes replaces words that shouldn't be replaced. However, removing those entries from the custom words list means the desired words never appear at all.

I know that Whisper supports an "initial prompt" concept, which biases the model toward producing specific words without forcefully replacing output. While looking through the codebase, I noticed there appears to be a way to use custom words to generate an initial prompt, and I wanted to ask whether a pull request for this would be welcome.

Related Issues/Discussions

Fixes #
Discussion: #1009

Community Feedback

Testing

No unit tests for this. I built and used this on my machine. The use case I was working on was "CLI" and "call" - that was fun to see in transcripts once I noticed it :)

Screenshots/Videos (if applicable)

AI Assistance

  • No AI was used in this PR
  • AI was used (please describe below)

If AI was used:

  • Tools used: claude-code opus 4.6
  • How extensively: I knew of the capability from prior research and came up with the idea of using the existing vocabulary as the prompt instead of replacement. I then used claude-code to see if the fix was small and implement it. I then reviewed it, but I admit I have not written rust before, so my human review may be insufficient.

…rection

Whisper supports an `initial_prompt` parameter that biases the model's
vocabulary during transcription. For Whisper models, we now join the
user's custom words list into a comma-separated string and pass it as
the initial_prompt. This guides the model to prefer those spellings
during decoding rather than relying solely on fuzzy post-correction.

For non-Whisper engines (Parakeet, Moonshine, SenseVoice, GigaAM),
the existing apply_custom_words post-correction continues to apply
since those engines don't support prompt-based vocabulary biasing.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
@cjpais
Copy link
Copy Markdown
Owner

cjpais commented Mar 14, 2026

This looks good to me. Thank you. Approved and merged.

@cjpais cjpais merged commit 85a8ed7 into cjpais:main Mar 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants