Skip to content

fix(i18n): CLDR-plural-aware translate-locales + backfill 360 missing variants#3908

Merged
koala73 merged 4 commits into
mainfrom
fix/cldr-plurals
May 25, 2026
Merged

fix(i18n): CLDR-plural-aware translate-locales + backfill 360 missing variants#3908
koala73 merged 4 commits into
mainfrom
fix/cldr-plurals

Conversation

@koala73

@koala73 koala73 commented May 25, 2026

Copy link
Copy Markdown
Owner

Summary

Follow-up to PR #3887. That PR added Croatian and the reviewer caught that hr.json was missing CLDR _few plural forms — i18next was falling through to English for counts 2/3/4. The same gap existed project-wide: cs, pl, ru, ro, ar, and even Romance locales (_many for fractional counts) had only _one/_other.

Two parts:

1. Tooling: scripts/translate-locales.mjs is now CLDR-plural-aware

  • Per non-EN locale, consults Intl.PluralRules(loc).resolvedOptions().pluralCategories to discover required categories (no hand-maintained per-locale table).
  • For each EN pluralized base (<base>_one+<base>_other), fans out the expected key set per the locale's CLDR categories.
  • Translation prompt got one rule: keys ending in _zero/_one/_two/_few/_many/_other are CLDR variants of the same noun; inflect morphology accordingly.

Future pluralized en.json additions will auto-generate correct plural variants for every locale on the next backfill run.

2. Data: 360 missing keys backfilled

Result of running the now-plural-aware script against all 20 non-EN locales:

Locale CLDR categories Added
ar zero/one/two/few/many/other +70
cs one/few/many/other +38
pl one/few/many/other +38
ru one/few/many/other +38
es / fr / it / pt one/many/other +22 each
ro one/few/other +22
bg / de / el / nl / sv / tr one/other +6 each
ja / ko / th / vi / zh other +6 each
hr / hu already complete 0

The +6 baseline for every locale is 6 keys added to en.json since the last full backfill that hadn't propagated:

  • components.cii.methodologyLink (+ its _methodologyLink_translatorNote sibling)
  • components.diseaseOutbreaks.methodologyNote
  • components.fsi.cissStale
  • components.progressCharts.fallbackBadge (+ fallbackTooltip)

Safety verification

Set-diff of every flattened key/value vs HEAD across all 20 files:

  • +360 additions, 0 modifications, 0 removals. Existing translations not touched.
  • Spot-checked morphology on the hardest locales:
    • ru sources: _one=источник (nom sg) / _few=источника (gen sg paucal) / _many=источников (gen pl) ✓
    • pl sources: _one=źródło / _few=źródła / _many=źródeł
    • cs sources: _one=zdroj / _few=zdroje / _many=zdrojů
    • ar sources: _two=مصدران (dual) / _few=مصادر / _many=مصدر (MSA singular after large numerals) ✓
    • ro sources: _one=sursă / _few=surse

Known quality note (not fixed here)

Arabic designations plurals mix two valid MSA roots — _one=تسمية (tasmiya) but _few/_many=تعيينات (ta'yīn). Each form is correct on its own; the inconsistency across categories is cosmetic and flagged for a native-speaker review pass, not auto-fixable.

Test plan

  • Dry-run reported expected per-locale gaps before the live run
  • Live run: 360 translated, 0 rejected, 0 still-missing-after-run
  • Set-diff against HEAD: pure additions, zero modifications
  • Pre-push pipeline (typecheck, edge tests, lint, version sync) — green
  • Reviewer spot-check on at least one native-speaker locale (any of ar/cs/pl/ru/ro)

koala73 added 2 commits May 25, 2026 21:46
Before: the script diffed each locale against EN at the flat-key level,
so the only plural variants ever generated were the two EN itself
defines (_one, _other). Locales whose CLDR plural rules require
additional categories (_few/_many in Slavic; _zero/_two/_few/_many in
Arabic; _many in Romance for fractional counts) silently shipped with
their non-_one/_other count values falling back to the English string.

Now: per non-EN locale, the script consults
`Intl.PluralRules(loc).resolvedOptions().pluralCategories` to discover
the required CLDR categories, fans out each EN pluralized base
(any `<base>_one` + `<base>_other` pair) into one expected key per
category, and adds the missing ones to the translation batch. The
prompt grew one rule (#8) telling the model that suffixes name the
CLDR category whose morphology it should inflect — Claude already
knows CLDR semantics so no per-locale lookup table is needed.

Helpers added:
- getPluralCategories(loc): wraps Intl.PluralRules; defaults to
  ['one','other'] on any unknown tag for safety.
- findPluralBases(enFlat): returns the bases where EN has both _one
  and _other (the only shape EN can express); the same set is reused
  for every locale.
- expectedKeysForLocale(enFlat, pluralBases, categories): the
  per-locale expected-key set used both by the missing-key detector
  in the main loop and by the post-write completeness scan.

Net behaviour on a locale that's already correct: zero change (the
existing _one/_other keys satisfy expected and `missing` stays
empty). On a fresh locale or any locale that's missing a CLDR
category, only the absent keys are filled — existing translations
are never overwritten (same as before).
Re-ran the now-plural-aware translate-locales.mjs against all 20
non-EN locales. 360 missing keys total:

- 6 baseline keys added to en.json since the last full backfill,
  propagated to every locale:
    components.cii.methodologyLink + _methodologyLink_translatorNote
    components.diseaseOutbreaks.methodologyNote
    components.fsi.cissStale
    components.progressCharts.fallbackBadge + fallbackTooltip

- CLDR plural variants filled per locale's required categories:
    ar (zero/one/two/few/many/other):  +64 plural keys
    cs (one/few/many/other):           +32 plural keys
    pl (one/few/many/other):           +32 plural keys
    ru (one/few/many/other):           +32 plural keys
    ro (one/few/other):                +16 plural keys
    es/fr/it/pt (one/many/other):      +16 each (decimal counts)

Verified post-write: pure additions, zero existing keys modified
across all 20 files (validated with set-diff of every flattened
key/value vs HEAD). Spot-checked the tricky morphology:

  ru sources: один=источник, few=источника, many=источников ✓
  pl sources: один=źródło, few=źródła, many=źródeł ✓
  cs sources: один=zdroj, few=zdroje, many=zdrojů ✓
  ar sources: dual=مصدران (two), pl=مصادر, sg-after-large=مصدر ✓
  ro sources: one=sursă, few/other=surse ✓

User-visible effect: counts like 2, 3, 4 in Czech/Polish/Russian/
Romanian/Croatian and 2/0/3-10 in Arabic now render in the actual
locale instead of falling through to English via i18next's
fallbackLng. Closes the gap that PR #3887 surfaced for Croatian.

Known quality note (not fixed here — flagged for native review):
the Arabic `designations` plurals mix two valid Arabic roots
(tasmiya / ta'yīn) across categories. Each form is correct MSA on
its own; the inconsistency is cosmetic.
@vercel

vercel Bot commented May 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 25, 2026 6:16pm

Request Review

@greptile-apps

greptile-apps Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes translate-locales.mjs CLDR-plural-aware by using Intl.PluralRules.resolvedOptions().pluralCategories to fan out the correct plural key set per locale, then applies it to backfill 360 missing translation variants across all 20 non-English locales.

  • Tooling: Three new helpers (getPluralCategories, findPluralBases, expectedKeysForLocale) replace the previous flat EN-key diff with a locale-specific expected key set; the translation prompt gains a rule instructing the model to inflect morphology per CLDR suffix.
  • Data: Pure additions across all locale files — zero modifications to existing values. Arabic received the full six-category expansion (zero/one/two/few/many/other); Slavic locales (cs, pl, ru) got one/few/many/other; Romance locales (es, fr, it, pt) got one/many/other; CJK/SEA locales only needed the 6 new EN baseline keys propagated.
  • One style nit in fr.json: the generated hoursAgo_many/daysAgo_many strings use a capitalised "Il y a" inconsistent with the lowercase "il y a" in all other French time variants.

Confidence Score: 4/5

Safe to merge — all 360 additions are pure data, no existing translations were modified, and the tooling change is a net improvement over the previous flat-diff approach.

The data changes are well-contained additive JSON. The script has two small gaps: getPluralCategories returns undefined rather than the intended fallback when pluralCategories is absent from resolvedOptions(), and validateTranslation does not enforce that URLs in new methodologyLink values survive translation intact. Neither affects the already-committed locale data, but both are worth fixing before the next backfill run.

scripts/translate-locales.mjs (fallback logic and validation gap), src/locales/fr.json (capitalisation inconsistency in the two new _many time strings)

Important Files Changed

Filename Overview
scripts/translate-locales.mjs Added CLDR-plural-aware logic via getPluralCategories, findPluralBases, and expectedKeysForLocale; fallback doesn't guard against undefined pluralCategories, and validateTranslation has no URL-preservation check.
src/locales/ar.json +70 Arabic plural forms backfilled (zero/one/two/few/many/other); _methodologyLink_translatorNote developer comment was translated to Arabic unlike all other locales where it stays in English.
src/locales/fr.json +22 French plural forms added; hoursAgo_many and daysAgo_many use capitalised 'Il y a' inconsistent with lowercase 'il y a' in existing _one/_other variants.
src/locales/ru.json +38 Russian plural variants added (one/few/many/other); morphology spot-check matches PR description.
src/locales/cs.json +38 Czech plural variants added (one/few/many/other); word-order difference in new forms is consistent with Czech idiom.
src/locales/pl.json +38 Polish plural variants added (one/few/many/other); morphology and suffixes look correct.
src/locales/ro.json +22 Romanian plural forms added (one/few/other); sursă/surse morphology matches PR spot-check.
src/locales/es.json +22 Spanish plural forms added (one/many/other); watch_many uses a different lexical choice but both are acceptable Spanish options.
src/locales/it.json +22 Italian plural forms added (one/many/other); morphology looks correct.
src/locales/pt.json +22 Portuguese (Brazil) plural forms added (one/many/other); translations look correct.
src/locales/de.json +6 German keys added (one/other); only the 6 new EN baseline keys needed backfilling.
src/locales/bg.json +6 Bulgarian keys added (one/other); only baseline keys needed backfilling.
src/locales/el.json +6 Greek keys added (one/other); only baseline keys needed backfilling.
src/locales/nl.json +6 Dutch keys added (one/other); only baseline keys needed backfilling.
src/locales/sv.json +6 Swedish keys added (one/other); only baseline keys needed backfilling.
src/locales/tr.json +6 Turkish keys added (one/other); only baseline keys needed backfilling.
src/locales/ja.json +6 Japanese keys added; CLDR other-only locale, no plural expansion needed.
src/locales/ko.json +6 Korean keys added; CLDR other-only locale, no plural expansion needed.
src/locales/th.json +6 Thai keys added; CLDR other-only locale, no plural expansion needed.
src/locales/vi.json +6 Vietnamese keys added; CLDR other-only locale, no plural expansion needed.
src/locales/zh.json +6 Simplified Chinese keys added; CLDR other-only locale, no plural expansion needed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[en.json flat keys] --> B[findPluralBases\ndetect _one + _other pairs]
    B --> C{For each target locale}
    C --> D[getPluralCategories\nIntl.PluralRules locale\n.resolvedOptions\n.pluralCategories]
    D --> E[expectedKeysForLocale\nexpand plural bases per CLDR categories]
    E --> F{diff: expected vs locale file}
    F -->|missing keys| G[translateBatch\nClaude Haiku 4.5]
    G --> H[validateTranslation\ntokens + HTML tags]
    H -->|valid| I[setNested → write locale JSON]
    H -->|invalid| J[reject + count]
    F -->|complete| K[skip locale]
    I --> L[post-write re-scan\nconfirm 0 still-missing]
Loading

Comments Outside Diff (2)

  1. scripts/translate-locales.mjs, line 194-198 (link)

    P2 validateTranslation checks that {{...}} interpolation tokens and HTML tags are preserved, but it has no corresponding check for URLs. The newly-added methodologyLink values embed a path like /docs/methodology/cii-risk-scores, and if the model rewrites or omits it the function still returns true. A URL regex similar to the token regex would catch this class of translation drift.

  2. src/locales/fr.json, line 1994-2002 (link)

    P2 The new _many forms use a capital "I" in "Il y a" whereas all existing _one and _other forms use lowercase "il y a". i18next selects exactly one plural variant per render, so users will see an inconsistently-cased string depending on the count (the _many category in French fires for large compacted numbers).

Reviews (1): Last reviewed commit: "fix(i18n): backfill missing CLDR plural ..." | Re-trigger Greptile

Comment thread scripts/translate-locales.mjs
Reviewer (PR #3908 P2/P3) caught two real issues:

bg.json — fallbackBadge contained 'živоте серии', a Latin-Cyrillic
mixed-script word ('živ' Latin + 'оте' Cyrillic). Now uses standard
Bulgarian 'серии на живо' (live series). Also fixed the same key's
fallbackTooltip where 'семе' (plant-seed) was a misread of the
tech-sense 'data seed' — replaced with 'Живите данни' (live data).

th.json — fallbackBadge / fallbackTooltip both contained 'สดใจ'
('cheerful' — literal: fresh-hearted), an artefact where the model
appended 'ใจ' (heart) to 'สด' (fresh/live). Replaced with the
unambiguous Thai-tech loanword 'เรียลไทม์' (real-time).

Programmatic sweep for the bg-style mistake (adjacent Latin/non-Latin
chars within a single word) across the full 6-baseline-keys × 20-
locales matrix surfaced only the bg one; ko 'ECB가' tripped the
scanner but is just an acronym + Korean particle (false positive).
Thai 'สดใจ' is a semantic error within Thai script — not detectable
by mixed-script heuristics; relied on the reviewer's read.

Prompt rule #8 (P3): the previous version said _few is "2-4" and
_many is "5+" for Slavic locales, which silently glosses over the
CLDR teen-case exceptions (e.g. Russian/Croatian _few requires
n%100 != 12-14). Reworded to explicitly tell the model to follow
CLDR rules exactly and NOT use the simplified rules of thumb. The
code path already delegates to Intl.PluralRules for category
selection — this just stops the prompt from biasing the model
toward incorrect teen-case morphology in future runs.
Three tooling fixes + two data fixes in response to Greptile review.

Tooling (scripts/translate-locales.mjs):

1. getPluralCategories null guard. The previous try/catch only caught
   constructor throws (RangeError on unknown locale tag), not the case
   where `resolvedOptions().pluralCategories` itself is absent (older
   Node where the property predates the spec). That `undefined` then
   crashed the next `for (const cat of categories)` mid-run with a
   TypeError. Now uses `?? ['one','other']` to cover both paths.

2. validateTranslation URL preservation. The model previously could
   silently paraphrase or drop URLs/paths embedded in translation
   sources (e.g. the methodologyLink `/docs/methodology/cii-risk-scores`)
   and the validator wouldn't catch it. Added a URL/path multiset check
   alongside the existing interpolation-token and HTML-tag checks. The
   path pattern requires the first segment after `/` to start with a
   letter so it doesn't fire on number fractions (`50/100`) or
   interpolation tokens (`{{count}}/{{total}}`). Verified against 6
   cases (preserve/drop/rewrite path; ignore fraction/interpolation;
   match https URL).

3. Skip leading-underscore "private" keys. Convention: any dotted-path
   segment starting with `_` is a translator-instruction key meant to
   stay in English so translators reading the raw files can understand
   it (e.g. `_methodologyLink_translatorNote` is a TODO note about its
   sibling `methodologyLink`). Sending those through the model produced
   the visible mistranslations in ar/ja/pt/th in the previous run.
   Added `isPrivateKey()` filter at the expected-keys layer so private
   keys never enter the missing-batch or the post-write coverage scan.

Data:

4. Reverted `components.cii._methodologyLink_translatorNote` to the
   English source in ar.json, ja.json, pt.json, th.json (the 4 of 20
   locales that the model translated; the other 16 already kept EN).

5. Aligned time-ago plural variants to their `_other` form across cs,
   es, fr, ro (22 strings total). The previous backfill chose more
   idiomatic prepositional forms (`před {{count}}h` / `hace {{count}}h`
   / `Il y a {{count}}h` / `acum {{count}}h`) while the pre-existing
   `_one`/`_other` use postpositional forms (`{{count}}h zpět` /
   `{{count}}h atrás` / `il y a {{count}}h` / `{{count}}h în urmă`).
   i18next picks exactly one variant per render, so users would have
   seen mixed-style timestamps depending on count. Greptile flagged
   only the fr capitalization symptom (`Il y a` vs `il y a`); the
   project-wide scan I ran surfaced the same wider issue in cs/es/ro
   so this fix covers all four locales.

Note on radiationWatch/thermalEscalation time-ago strings in fr.json:
the pre-existing `_one`/`_other` values are still English ("{{count}}h
ago") rather than French — a separate pre-existing translation gap not
in scope of this PR. Aligning `_many` to match `_other` keeps the
consistency rule but inherits the English text for those 5 keys;
fixing the root French translation there should be a follow-up.
@koala73
koala73 merged commit 39b237b into main May 25, 2026
12 checks passed
@koala73
koala73 deleted the fix/cldr-plurals branch May 25, 2026 18:23
koala73 pushed a commit to shubham1206agra/worldmonitor that referenced this pull request May 26, 2026
Static-analysis sweep against literal t() calls, dynamic key construction,
pluralization variants, and substring use in lookup tables / property-access
scripts. 178 keys had no reference in src/, scripts/, server/, shared/, api/,
public/, or tests/, plus 13 stale extras that existed only in non-English
translations (8 components.economic.* + panels.{liveYouTube,pinnedWebcams}
universally, 5 more under deckgl.layers / modals.countryIntel in nl/pt/sv).
translate-locales.mjs would otherwise re-add them on next backfill.

Coverage: all 22 locales (en + 20 originals + hu added in koala73#3857).

Preserved: shell.offline{Title,Message,Retry} (read by
scripts/sync-offline-translations.mjs into public/offline.html) and the
components.cii._methodologyLink_translatorNote sidecar.

Test-contract: drop 'createWithAi' and 'changeAccent' from REQUIRED_KEYS in
tests/widget-builder.test.mjs — both keys lost their last production
reference in 80c3751 and 47f0dd1 respectively; the contract was stale.

Verification: all 22 JSON files validate; widget-builder test 203/203;
sync-offline runs clean (no public/offline.html diff); all 21 locales
have exact en-parity except CLDR plural-cohort siblings (signals_few/_many/
_zero etc) added by koala73#3908, which are legitimate.

Co-authored-by: Elie Habib <[email protected]>
koala73 pushed a commit that referenced this pull request May 26, 2026
Static-analysis sweep against literal t() calls, dynamic key construction,
pluralization variants, and substring use in lookup tables / property-access
scripts. 178 keys had no reference in src/, scripts/, server/, shared/, api/,
public/, or tests/, plus 13 stale extras that existed only in non-English
translations (8 components.economic.* + panels.{liveYouTube,pinnedWebcams}
universally, 5 more under deckgl.layers / modals.countryIntel in nl/pt/sv).
translate-locales.mjs would otherwise re-add them on next backfill.

Coverage: all 22 locales (en + 20 originals + hu added in #3857).

Preserved: shell.offline{Title,Message,Retry} (read by
scripts/sync-offline-translations.mjs into public/offline.html) and the
components.cii._methodologyLink_translatorNote sidecar.

Test-contract: drop 'createWithAi' and 'changeAccent' from REQUIRED_KEYS in
tests/widget-builder.test.mjs — both keys lost their last production
reference in 80c3751 and 47f0dd1 respectively; the contract was stale.

Verification: all 22 JSON files validate; widget-builder test 203/203;
sync-offline runs clean (no public/offline.html diff); all 21 locales
have exact en-parity except CLDR plural-cohort siblings (signals_few/_many/
_zero etc) added by #3908, which are legitimate.

Co-authored-by: Elie Habib <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant