fix(i18n): CLDR-plural-aware translate-locales + backfill 360 missing variants#3908
Conversation
Before: the script diffed each locale against EN at the flat-key level, so the only plural variants ever generated were the two EN itself defines (_one, _other). Locales whose CLDR plural rules require additional categories (_few/_many in Slavic; _zero/_two/_few/_many in Arabic; _many in Romance for fractional counts) silently shipped with their non-_one/_other count values falling back to the English string. Now: per non-EN locale, the script consults `Intl.PluralRules(loc).resolvedOptions().pluralCategories` to discover the required CLDR categories, fans out each EN pluralized base (any `<base>_one` + `<base>_other` pair) into one expected key per category, and adds the missing ones to the translation batch. The prompt grew one rule (#8) telling the model that suffixes name the CLDR category whose morphology it should inflect — Claude already knows CLDR semantics so no per-locale lookup table is needed. Helpers added: - getPluralCategories(loc): wraps Intl.PluralRules; defaults to ['one','other'] on any unknown tag for safety. - findPluralBases(enFlat): returns the bases where EN has both _one and _other (the only shape EN can express); the same set is reused for every locale. - expectedKeysForLocale(enFlat, pluralBases, categories): the per-locale expected-key set used both by the missing-key detector in the main loop and by the post-write completeness scan. Net behaviour on a locale that's already correct: zero change (the existing _one/_other keys satisfy expected and `missing` stays empty). On a fresh locale or any locale that's missing a CLDR category, only the absent keys are filled — existing translations are never overwritten (same as before).
Re-ran the now-plural-aware translate-locales.mjs against all 20
non-EN locales. 360 missing keys total:
- 6 baseline keys added to en.json since the last full backfill,
propagated to every locale:
components.cii.methodologyLink + _methodologyLink_translatorNote
components.diseaseOutbreaks.methodologyNote
components.fsi.cissStale
components.progressCharts.fallbackBadge + fallbackTooltip
- CLDR plural variants filled per locale's required categories:
ar (zero/one/two/few/many/other): +64 plural keys
cs (one/few/many/other): +32 plural keys
pl (one/few/many/other): +32 plural keys
ru (one/few/many/other): +32 plural keys
ro (one/few/other): +16 plural keys
es/fr/it/pt (one/many/other): +16 each (decimal counts)
Verified post-write: pure additions, zero existing keys modified
across all 20 files (validated with set-diff of every flattened
key/value vs HEAD). Spot-checked the tricky morphology:
ru sources: один=источник, few=источника, many=источников ✓
pl sources: один=źródło, few=źródła, many=źródeł ✓
cs sources: один=zdroj, few=zdroje, many=zdrojů ✓
ar sources: dual=مصدران (two), pl=مصادر, sg-after-large=مصدر ✓
ro sources: one=sursă, few/other=surse ✓
User-visible effect: counts like 2, 3, 4 in Czech/Polish/Russian/
Romanian/Croatian and 2/0/3-10 in Arabic now render in the actual
locale instead of falling through to English via i18next's
fallbackLng. Closes the gap that PR #3887 surfaced for Croatian.
Known quality note (not fixed here — flagged for native review):
the Arabic `designations` plurals mix two valid Arabic roots
(tasmiya / ta'yīn) across categories. Each form is correct MSA on
its own; the inconsistency is cosmetic.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR makes
Confidence Score: 4/5Safe to merge — all 360 additions are pure data, no existing translations were modified, and the tooling change is a net improvement over the previous flat-diff approach. The data changes are well-contained additive JSON. The script has two small gaps: getPluralCategories returns undefined rather than the intended fallback when pluralCategories is absent from resolvedOptions(), and validateTranslation does not enforce that URLs in new methodologyLink values survive translation intact. Neither affects the already-committed locale data, but both are worth fixing before the next backfill run. scripts/translate-locales.mjs (fallback logic and validation gap), src/locales/fr.json (capitalisation inconsistency in the two new _many time strings) Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[en.json flat keys] --> B[findPluralBases\ndetect _one + _other pairs]
B --> C{For each target locale}
C --> D[getPluralCategories\nIntl.PluralRules locale\n.resolvedOptions\n.pluralCategories]
D --> E[expectedKeysForLocale\nexpand plural bases per CLDR categories]
E --> F{diff: expected vs locale file}
F -->|missing keys| G[translateBatch\nClaude Haiku 4.5]
G --> H[validateTranslation\ntokens + HTML tags]
H -->|valid| I[setNested → write locale JSON]
H -->|invalid| J[reject + count]
F -->|complete| K[skip locale]
I --> L[post-write re-scan\nconfirm 0 still-missing]
|
Reviewer (PR #3908 P2/P3) caught two real issues: bg.json — fallbackBadge contained 'živоте серии', a Latin-Cyrillic mixed-script word ('živ' Latin + 'оте' Cyrillic). Now uses standard Bulgarian 'серии на живо' (live series). Also fixed the same key's fallbackTooltip where 'семе' (plant-seed) was a misread of the tech-sense 'data seed' — replaced with 'Живите данни' (live data). th.json — fallbackBadge / fallbackTooltip both contained 'สดใจ' ('cheerful' — literal: fresh-hearted), an artefact where the model appended 'ใจ' (heart) to 'สด' (fresh/live). Replaced with the unambiguous Thai-tech loanword 'เรียลไทม์' (real-time). Programmatic sweep for the bg-style mistake (adjacent Latin/non-Latin chars within a single word) across the full 6-baseline-keys × 20- locales matrix surfaced only the bg one; ko 'ECB가' tripped the scanner but is just an acronym + Korean particle (false positive). Thai 'สดใจ' is a semantic error within Thai script — not detectable by mixed-script heuristics; relied on the reviewer's read. Prompt rule #8 (P3): the previous version said _few is "2-4" and _many is "5+" for Slavic locales, which silently glosses over the CLDR teen-case exceptions (e.g. Russian/Croatian _few requires n%100 != 12-14). Reworded to explicitly tell the model to follow CLDR rules exactly and NOT use the simplified rules of thumb. The code path already delegates to Intl.PluralRules for category selection — this just stops the prompt from biasing the model toward incorrect teen-case morphology in future runs.
Three tooling fixes + two data fixes in response to Greptile review.
Tooling (scripts/translate-locales.mjs):
1. getPluralCategories null guard. The previous try/catch only caught
constructor throws (RangeError on unknown locale tag), not the case
where `resolvedOptions().pluralCategories` itself is absent (older
Node where the property predates the spec). That `undefined` then
crashed the next `for (const cat of categories)` mid-run with a
TypeError. Now uses `?? ['one','other']` to cover both paths.
2. validateTranslation URL preservation. The model previously could
silently paraphrase or drop URLs/paths embedded in translation
sources (e.g. the methodologyLink `/docs/methodology/cii-risk-scores`)
and the validator wouldn't catch it. Added a URL/path multiset check
alongside the existing interpolation-token and HTML-tag checks. The
path pattern requires the first segment after `/` to start with a
letter so it doesn't fire on number fractions (`50/100`) or
interpolation tokens (`{{count}}/{{total}}`). Verified against 6
cases (preserve/drop/rewrite path; ignore fraction/interpolation;
match https URL).
3. Skip leading-underscore "private" keys. Convention: any dotted-path
segment starting with `_` is a translator-instruction key meant to
stay in English so translators reading the raw files can understand
it (e.g. `_methodologyLink_translatorNote` is a TODO note about its
sibling `methodologyLink`). Sending those through the model produced
the visible mistranslations in ar/ja/pt/th in the previous run.
Added `isPrivateKey()` filter at the expected-keys layer so private
keys never enter the missing-batch or the post-write coverage scan.
Data:
4. Reverted `components.cii._methodologyLink_translatorNote` to the
English source in ar.json, ja.json, pt.json, th.json (the 4 of 20
locales that the model translated; the other 16 already kept EN).
5. Aligned time-ago plural variants to their `_other` form across cs,
es, fr, ro (22 strings total). The previous backfill chose more
idiomatic prepositional forms (`před {{count}}h` / `hace {{count}}h`
/ `Il y a {{count}}h` / `acum {{count}}h`) while the pre-existing
`_one`/`_other` use postpositional forms (`{{count}}h zpět` /
`{{count}}h atrás` / `il y a {{count}}h` / `{{count}}h în urmă`).
i18next picks exactly one variant per render, so users would have
seen mixed-style timestamps depending on count. Greptile flagged
only the fr capitalization symptom (`Il y a` vs `il y a`); the
project-wide scan I ran surfaced the same wider issue in cs/es/ro
so this fix covers all four locales.
Note on radiationWatch/thermalEscalation time-ago strings in fr.json:
the pre-existing `_one`/`_other` values are still English ("{{count}}h
ago") rather than French — a separate pre-existing translation gap not
in scope of this PR. Aligning `_many` to match `_other` keeps the
consistency rule but inherits the English text for those 5 keys;
fixing the root French translation there should be a follow-up.
Static-analysis sweep against literal t() calls, dynamic key construction,
pluralization variants, and substring use in lookup tables / property-access
scripts. 178 keys had no reference in src/, scripts/, server/, shared/, api/,
public/, or tests/, plus 13 stale extras that existed only in non-English
translations (8 components.economic.* + panels.{liveYouTube,pinnedWebcams}
universally, 5 more under deckgl.layers / modals.countryIntel in nl/pt/sv).
translate-locales.mjs would otherwise re-add them on next backfill.
Coverage: all 22 locales (en + 20 originals + hu added in koala73#3857).
Preserved: shell.offline{Title,Message,Retry} (read by
scripts/sync-offline-translations.mjs into public/offline.html) and the
components.cii._methodologyLink_translatorNote sidecar.
Test-contract: drop 'createWithAi' and 'changeAccent' from REQUIRED_KEYS in
tests/widget-builder.test.mjs — both keys lost their last production
reference in 80c3751 and 47f0dd1 respectively; the contract was stale.
Verification: all 22 JSON files validate; widget-builder test 203/203;
sync-offline runs clean (no public/offline.html diff); all 21 locales
have exact en-parity except CLDR plural-cohort siblings (signals_few/_many/
_zero etc) added by koala73#3908, which are legitimate.
Co-authored-by: Elie Habib <[email protected]>
Static-analysis sweep against literal t() calls, dynamic key construction,
pluralization variants, and substring use in lookup tables / property-access
scripts. 178 keys had no reference in src/, scripts/, server/, shared/, api/,
public/, or tests/, plus 13 stale extras that existed only in non-English
translations (8 components.economic.* + panels.{liveYouTube,pinnedWebcams}
universally, 5 more under deckgl.layers / modals.countryIntel in nl/pt/sv).
translate-locales.mjs would otherwise re-add them on next backfill.
Coverage: all 22 locales (en + 20 originals + hu added in #3857).
Preserved: shell.offline{Title,Message,Retry} (read by
scripts/sync-offline-translations.mjs into public/offline.html) and the
components.cii._methodologyLink_translatorNote sidecar.
Test-contract: drop 'createWithAi' and 'changeAccent' from REQUIRED_KEYS in
tests/widget-builder.test.mjs — both keys lost their last production
reference in 80c3751 and 47f0dd1 respectively; the contract was stale.
Verification: all 22 JSON files validate; widget-builder test 203/203;
sync-offline runs clean (no public/offline.html diff); all 21 locales
have exact en-parity except CLDR plural-cohort siblings (signals_few/_many/
_zero etc) added by #3908, which are legitimate.
Co-authored-by: Elie Habib <[email protected]>
Summary
Follow-up to PR #3887. That PR added Croatian and the reviewer caught that hr.json was missing CLDR
_fewplural forms — i18next was falling through to English for counts 2/3/4. The same gap existed project-wide: cs, pl, ru, ro, ar, and even Romance locales (_manyfor fractional counts) had only_one/_other.Two parts:
1. Tooling:
scripts/translate-locales.mjsis now CLDR-plural-awareIntl.PluralRules(loc).resolvedOptions().pluralCategoriesto discover required categories (no hand-maintained per-locale table).<base>_one+<base>_other), fans out the expected key set per the locale's CLDR categories._zero/_one/_two/_few/_many/_otherare CLDR variants of the same noun; inflect morphology accordingly.Future pluralized en.json additions will auto-generate correct plural variants for every locale on the next backfill run.
2. Data: 360 missing keys backfilled
Result of running the now-plural-aware script against all 20 non-EN locales:
The
+6baseline for every locale is 6 keys added to en.json since the last full backfill that hadn't propagated:components.cii.methodologyLink(+ its_methodologyLink_translatorNotesibling)components.diseaseOutbreaks.methodologyNotecomponents.fsi.cissStalecomponents.progressCharts.fallbackBadge(+fallbackTooltip)Safety verification
Set-diff of every flattened key/value vs HEAD across all 20 files:
ru sources:_one=источник(nom sg) /_few=источника(gen sg paucal) /_many=источников(gen pl) ✓pl sources:_one=źródło/_few=źródła/_many=źródeł✓cs sources:_one=zdroj/_few=zdroje/_many=zdrojů✓ar sources:_two=مصدران(dual) /_few=مصادر/_many=مصدر(MSA singular after large numerals) ✓ro sources:_one=sursă/_few=surse✓Known quality note (not fixed here)
Arabic
designationsplurals mix two valid MSA roots —_one=تسمية(tasmiya) but_few/_many=تعيينات(ta'yīn). Each form is correct on its own; the inconsistency across categories is cosmetic and flagged for a native-speaker review pass, not auto-fixable.Test plan