You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
should not skip text normalization when the text is just an emoji #353
Feature Request: Enable Text Normalization also for text with special character only
Summary
Currently, text normalization is only applied for English language codes (en-us, en-gb, etc.). However, text normalization should run for all languages since special characters and emoji sequences can appear in any language and need to be processed consistently.
Current Behavior
iflang_codein ["a", "b", "en-us", "en-gb"]:
# Custom phonemes + normalizationprocessed_text=CUSTOM_PHONEMES.sub(...)
processed_text=normalize_text(processed_text, normalization_options)
else:
logger.info("Skipping text normalization as it is only supported for english")
this approach will not normalize correctly in this case:
10:14:18 AM | DEBUG | text_processor:164 | Split raw text into 1 parts by pause tags.
10:14:18 AM | INFO | text_processor:188 | Skipping text normalization as it is only supported for english
10:14:18 AM | DEBUG | text_processor:65 | Total processing took 0.21ms for chunk: '😊"'
is it possible to always normalize the non-english language in a way that special characters and emojis are removed?
The main reason is that normalizing for languages that the normalization was not designed for can cause unintended side effects. I will look into only normalizing emojis though
Activity
fireblade2534 commentedon Jun 27, 2025
The main reason is that normalizing for languages that the normalization was not designed for can cause unintended side effects. I will look into only normalizing emojis though
Merge pull request #529 from remsky/feat/normalization-followups
feat(normalizer): opt-in remove_emoji, 400 before streaming on unspea…
feat(normalizer): caps_normalization option, spoken phone numbers, he…
chore(release): 0.6.0 [skip ci]