Skip to content

feat(feeds): add Hungarian and Central European RSS sources#3857

Merged
koala73 merged 10 commits into
koala73:mainfrom
zionappp-ui:feat/add-cee-hungarian-feeds
May 23, 2026
Merged

feat(feeds): add Hungarian and Central European RSS sources#3857
koala73 merged 10 commits into
koala73:mainfrom
zionappp-ui:feat/add-cee-hungarian-feeds

Conversation

@zionappp-ui

Copy link
Copy Markdown
Contributor

Summary

Adds eight Hungarian / Central European news outlets to the RSS proxy allowlist. CEE coverage is currently a gap — every existing European feed in the allowlist sits north or west of Vienna, so any user tracking V4 governance, Ukraine spillover, Paks nuclear, or the Druzhba / TurkStream energy stories misses the bulk of native-language source coverage.

The eight outlets are:

Domain Profile
telex.hu News (independent — largest HU online readership)
index.hu News (mainstream domestic)
hvg.hu News / weekly current affairs
444.hu News / commentary
24.hu News (general)
hirado.hu Public broadcaster (MTVA)
portfolio.hu Business / finance
www.atv.hu Political talk and news

Together they give broad domestic coverage across the editorial spectrum (independent, public, business, opposition-leaning, government-aligned).

Type of change

  • New data source / feed

Affected areas

  • News panels / RSS feeds

Synchronised mirrors

Added the same eight domains to all four allowlist mirrors to keep them in sync:

  • shared/rss-allowed-domains.json (source of truth)
  • scripts/shared/rss-allowed-domains.json
  • api/_rss-allowed-domains.js (Edge-compatible ESM wrapper)
  • vite.config.tsRSS_PROXY_ALLOWED_DOMAINS (dev-mode duplicate)

Checklist

  • Tested on worldmonitor.app variant — not run locally; data-only change relies on CI
  • Tested on tech.worldmonitor.app variant — N/A (regional feeds)
  • New RSS feed domains added to api/rss-proxy.js allowlist
  • No API keys or secrets committed
  • npm run typechecknot run locally; expected to pass since change is data-only

Screenshots

N/A — feed allowlist data change only, no UI changes.

Notes for maintainers

Happy to adjust the outlet list based on editorial guidance (e.g., trim to fewer outlets, or add only specific ones). The eight chosen here cover the major HU online news population by readership share. Open to dropping www.atv.hu if its mixed talk/news format is borderline for your "news feed" criteria.

Adds eight Hungarian / Central European news outlets to the RSS proxy allowlist:

  telex.hu      news (independent / largest readership)
  index.hu      news (mainstream domestic)
  hvg.hu        news / weekly current affairs
  444.hu        news / commentary
  24.hu         news (general)
  hirado.hu     public broadcaster (MTVA)
  portfolio.hu  business / finance
  www.atv.hu    political talk and news

CEE is currently under-represented in the allowlist — every existing
European entry leans north or west of Vienna, so a Hungarian-language
user (or any analyst tracking the V4 region) misses the bulk of native
coverage on Ukraine spillover, V4 governance, Paks nuclear, and the
Druzhba / TurkStream energy stories.

Synchronised across all four mirrors of the allowlist:
  shared/rss-allowed-domains.json   (source of truth)
  scripts/shared/rss-allowed-domains.json
  api/_rss-allowed-domains.js
  vite.config.ts  (RSS_PROXY_ALLOWED_DOMAINS dev-mode duplicate)

Pure data-only change; expected to pass the existing CI typecheck/lint
suite without source modifications.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
Signed-off-by: zionappp-ui <[email protected]>
@vercel

vercel Bot commented May 21, 2026

Copy link
Copy Markdown

@zionappp-ui is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the trust:caution Brin: contributor trust score caution label May 21, 2026
ziclaw01 pushed a commit to ziclaw01/keymakermap that referenced this pull request May 21, 2026
Earlier commit (0158d3d) added the eight Hungarian feed domains only to
shared/rss-allowed-domains.json and vite.config.ts. The other two mirrors
were missed, which means HU feeds would fail in the Edge function path
(api/_rss-allowed-domains.js) and in script-side tooling
(scripts/shared/rss-allowed-domains.json).

This sync brings all four mirrors into a consistent state. Same change as
the upstream PR (koala73/worldmonitor#3857), where we also synchronised
all four mirrors from the start.

Also updates CHANGES.md to reflect the 4-mirror sync and link to the
upstream PR.

Touched:
- api/_rss-allowed-domains.js
- scripts/shared/rss-allowed-domains.json
- CHANGES.md  (AGPL §5 modification log update)

Co-Authored-By: Claude Opus 4.7 <[email protected]>
Signed-off-by: zionappp-ui <[email protected]>
@greptile-apps

greptile-apps Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds eight Hungarian / Central European news outlets (telex.hu, index.hu, hvg.hu, 444.hu, 24.hu, hirado.hu, portfolio.hu, www.atv.hu) to the RSS proxy allowlist across all four mirror locations. The change is data-only and requires no logic changes since the existing isAllowedDomain() normalisation already handles www.-prefixed hostnames in both directions.

  • All four allowlist mirrors are updated consistently: shared/rss-allowed-domains.json, scripts/shared/rss-allowed-domains.json, api/_rss-allowed-domains.js, and the vite.config.ts dev-mode Set.
  • www.atv.hu is stored with a www. prefix while all seven other entries added in this PR omit it; the runtime normalisation handles both, but the inconsistency is a style oddity worth fixing for list auditability.

Confidence Score: 4/5

Safe to merge — pure allowlist data addition with no logic changes, and all four mirrors are updated consistently.

The change correctly updates all four allowlist mirrors in sync. The only issue is that www.atv.hu is stored with a www. prefix while all other new .hu entries omit it; the runtime normalisation in isAllowedDomain() silently handles both forms, so there is no functional defect.

All four files carry the same www.atv.hu inconsistency — the suggestion comments on each file cover the fix.

Important Files Changed

Filename Overview
api/_rss-allowed-domains.js Adds 8 Hungarian news domains to the ESM allowlist array; www.atv.hu has an inconsistent www. prefix compared to all other .hu entries.
shared/rss-allowed-domains.json Source-of-truth JSON updated with 8 new domains; same www.atv.hu prefix inconsistency as in the other files.
scripts/shared/rss-allowed-domains.json Scripts-copy JSON mirror updated identically to shared/rss-allowed-domains.json; same www.atv.hu prefix inconsistency.
vite.config.ts Dev-mode RSS_PROXY_ALLOWED_DOMAINS Set updated with the 8 new domains in a clearly labelled comment block; same www.atv.hu prefix inconsistency.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[RSS Proxy Request\n/api/rss-proxy?url=...] --> B[Extract hostname\nfrom feed URL]
    B --> C{isAllowedDomain?}
    C --> D[bare = strip www. prefix\nwithWww = add www. if missing]
    D --> E{hostname OR bare\nOR withWww in\nALLOWED_DOMAINS?}
    E -- Yes --> F[Fetch feed\ndirect or via Railway relay]
    E -- No --> G[403 Forbidden]
    F --> H[Return XML to client]

    subgraph Allowlist Sources [Four allowlist mirrors - all updated in this PR]
        S1[shared/rss-allowed-domains.json\nSource of truth]
        S2[scripts/shared/rss-allowed-domains.json\nScripts copy]
        S3[api/_rss-allowed-domains.js\nEdge ESM import]
        S4[vite.config.ts\nDev-mode Set]
    end

    S3 -- imported by --> C
Loading

Reviews (1): Last reviewed commit: "feat(feeds): add Hungarian and Central E..." | Re-trigger Greptile

Comment on lines +312 to 314
"portfolio.hu",
"www.atv.hu"
];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 All seven other .hu domains in this PR are stored without the www. prefix, but www.atv.hu includes it. isAllowedDomain() normalises both directions via its bare/withWww logic, so functionality is unaffected, but the inconsistency could mislead future contributors into thinking the prefix matters for the allowlist entry.

Suggested change
"portfolio.hu",
"www.atv.hu"
];
"portfolio.hu",
"atv.hu"
];

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +309 to 311
"portfolio.hu",
"www.atv.hu"
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Same www.atv.hu prefix inconsistency as in the other allowlist files — all remaining .hu entries omit the www. prefix. Keeping this consistent makes the list easier to audit and avoids confusion about the normalisation applied at runtime.

Suggested change
"portfolio.hu",
"www.atv.hu"
]
"portfolio.hu",
"atv.hu"
]

Comment on lines +309 to 311
"portfolio.hu",
"www.atv.hu"
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Same prefix inconsistency as the other files — www.atv.hu should be atv.hu to match the style of every other entry added in this PR.

Suggested change
"portfolio.hu",
"www.atv.hu"
]
"portfolio.hu",
"atv.hu"
]

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread vite.config.ts Outdated
'novayagazeta.eu', 'www.bangkokpost.com', 'vnexpress.net', 'www.abc.net.au',
'news.ycombinator.com',
// Hungarian / Central European feeds
'telex.hu', 'index.hu', 'hvg.hu', '444.hu', '24.hu', 'hirado.hu', 'portfolio.hu', 'www.atv.hu',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Same prefix inconsistency as the other allowlist files — www.atv.hu should be atv.hu to match the style of the other seven entries added here.

Suggested change
'telex.hu', 'index.hu', 'hvg.hu', '444.hu', '24.hu', 'hirado.hu', 'portfolio.hu', 'www.atv.hu',
'telex.hu', 'index.hu', 'hvg.hu', '444.hu', '24.hu', 'hirado.hu', 'portfolio.hu', 'atv.hu',

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Follow-up to 208948e — the previous commit allowlisted the 8 HU domains
but left them un-wired, so they would not actually surface in news panels.

- Add 8 entries to `world:` block with lang: 'hu' so locale boosting picks
  them up via getLocaleBoostedSources('hu').
- Telex, Index.hu (24ora/rss), HVG, 444.hu, 24.hu, Portfolio.hu, ATV use
  direct native RSS (probed: 10-350 items per feed, content-type XML).
- Híradó (MTVA): hirado.hu has no working RSS endpoint — every /rss/* path
  returns HTML. Uses news.google.com/rss/search?site:hirado.hu fallback,
  matching the existing Kathimerini / Proto Thema pattern in this file.
- SOURCE_TYPES entries: 'mainstream' for news outlets, 'market' for
  Portfolio.hu (business/finance).
- source-tiers.json (both byte-identical mirrors): tier 2, matching the
  Polish / Turkish / Dutch convention so HU outlets are not deprioritized
  to default tier 4 (aggregator).
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed a follow-up commit (b7ad105e0) addressing the P2 gap surfaced in review: the original commit allowlisted the 8 domains but left them un-wired, so panels would not have picked them up.

What the follow-up adds

  • src/config/feeds.ts — 8 entries in the world: block with lang: 'hu', so getLocaleBoostedSources('hu') boosts them for Hungarian-locale users. Same shape as the Polish / Turkish / Greek blocks directly above.
  • SOURCE_TYPESmainstream for the news outlets, market for Portfolio.hu.
  • shared/source-tiers.json + scripts/shared/source-tiers.json (byte-identical mirror) — added the 8 outlets at tier 2, matching the Polish / Turkish / Dutch convention. Without this, getSourceTier would have returned the default 4 (aggregator), deprioritizing HU outlets vs. their European peers in importance scoring.

Feed URL provenance — I probed each candidate URL before wiring:

Outlet URL wired Items returned
Telex https://telex.hu/rss 50
Index.hu https://index.hu/24ora/rss (/rss 404s) 54
HVG https://hvg.hu/rss 60
444.hu https://444.hu/feed 350
24.hu https://24.hu/feed/ 10
Híradó Google News (site:hirado.hu) — hirado.hu has no working RSS endpoint, every /rss/* path returns HTML 100
Portfolio.hu https://www.portfolio.hu/rss/all.xml 20
ATV https://www.atv.hu/rss 10

The Google News fallback for Híradó follows the existing pattern used by Kathimerini and Proto Thema in the same file.

Verification

  • npm run typecheck — clean.
  • node --test tests/edge-functions.test.mjs tests/importance-score-parity.test.mjs tests/feeds-client-server-parity.test.mjs tests/feed-resolution.test.mts — all green (mirror-parity and tier-parity checks both covered).
  • Pre-push hook ran the full suite green.

vite.config.ts:569 lists `portfolio.hu` (bare), but rssProxyPlugin's
dev-mode check uses strict `RSS_PROXY_ALLOWED_DOMAINS.has(parsed.hostname)`
with no www-normalization. Feed URL was `https://www.portfolio.hu/...`
so `npm run dev` returned 403.

Switch to `https://portfolio.hu/rss/all.xml`. The host issues a 301 to
the www form, which both transports follow transparently:
- Vite dev fetch() uses `redirect: 'follow'` (only the initial host is
  checked against the allowlist).
- Prod api/rss-proxy.js uses `redirect: 'manual'` + `isAllowedDomain()`
  which www-normalizes, so `www.portfolio.hu` matches `portfolio.hu`.

All other Hungarian feed URLs already have hostnames that exactly match
their allowlist entries; only Portfolio had the bare/www mismatch.
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed df1883467 to fix the Portfolio.hu dev-mode 403.

Root causevite.config.ts:569 lists portfolio.hu (bare). rssProxyPlugin checks RSS_PROXY_ALLOWED_DOMAINS.has(parsed.hostname) with no www-normalization, so the wired https://www.portfolio.hu/... URL was rejected in npm run dev. Production was fine because api/rss-proxy.js uses isAllowedDomain() which normalizes www.

Fix — switched the feed URL to bare https://portfolio.hu/rss/all.xml. The host issues a 301 to the www form, which both transports follow transparently:

  • Vite dev: fetch(..., { redirect: 'follow' }) (vite.config.ts:585) — only the initial host is checked, redirect target is silent.
  • Prod: redirect: 'manual' + isAllowedDomain() re-check on the redirect target (api/rss-proxy.js:120-130) — www-normalization handles www.portfolio.huportfolio.hu.

Single-line change, no allowlist edit needed. Audited all 8 Hungarian feed URLs against their allowlist entries — Portfolio was the only mismatch (the other seven hosts exactly match their bare/www allowlist forms, and the Híradó URL goes through news.google.com which is pre-existing).

…edirect

`scripts/ais-relay.cjs:9467` re-checks redirect targets via strict
`RSS_ALLOWED_DOMAINS.has(redirectHost)` with no www-normalization —
same policy as vite dev, different from production's
api/rss-proxy.js `isAllowedDomain()` which DOES normalize.

The bare `https://portfolio.hu/rss/all.xml` URL we ship issues a 301
to `https://www.portfolio.hu/...`. Edge proxy + vite dev handle this
fine (Edge normalizes, vite uses `redirect: 'follow'` on the underlying
fetch so the redirect target isn't re-checked against the Set). But on
the Railway relay path (manual redirect-following inside ais-relay),
the 301 target lands on a strict Set lookup that returns false and the
relay returns 403 "Redirect to disallowed domain".

Add `www.portfolio.hu` to all four allowlist mirrors so the redirect
re-check passes on the relay path:
- shared/rss-allowed-domains.json (source of truth)
- scripts/shared/rss-allowed-domains.json (byte-identical mirror)
- api/_rss-allowed-domains.js (Edge ESM wrapper)
- vite.config.ts RSS_PROXY_ALLOWED_DOMAINS (dev-mode Set)

Audited the other 7 Hungarian feed URLs with curl -I -L; only Portfolio
crosses host on redirect. All others stay on their initial hostname,
which already exactly matches their allowlist entry.
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed ab2582c54 to address the relay-allowlist gap.

Root cause — There are actually three host-match policies in WM's RSS stack, not two:

Path Code www-normalize?
Production Edge api/rss-proxy.js:55-59 isAllowedDomain ✓ Yes
Vite dev vite.config.ts:596 Set.has(parsed.hostname) ✗ No
Railway relay scripts/ais-relay.cjs:9355,9467 RSS_ALLOWED_DOMAINS.has(...) (initial + redirect re-check) ✗ No

My previous fix (bare https://portfolio.hu/...) worked for the first two:

  • Production normalizes the redirect target.
  • Vite dev's fetch({ redirect: 'follow' }) handles the redirect inside the underlying fetch — the redirect target isn't re-checked against the strict Set.

But the relay does its own manual redirect-following inside ais-relay.cjs:9434 fetchWithRedirects, and re-checks each hop against the strict Set. The 301 → www.portfolio.hu lands on Set.has('www.portfolio.hu') → false → 403.

Fix — add www.portfolio.hu to all 4 mirrors. Now:

  • Edge: was already fine.
  • Vite dev: was already fine for the bare URL; now also fine if anyone wires www.portfolio.hu directly.
  • Relay initial: was already fine for bare URL.
  • Relay redirect re-check: now passes.

Audit of the other 7 feedscurl -I -L confirmed only Portfolio crosses host on redirect:

Feed URL Final host after redirects Allowlist entry Relay redirect re-check
telex.hu/rss telex.hu telex.hu
index.hu/24ora/rss index.hu index.hu
hvg.hu/rss hvg.hu hvg.hu
444.hu/feed 444.hu 444.hu
24.hu/feed/ 24.hu 24.hu
Google News (Híradó) news.google.com news.google.com
portfolio.hu/rss/all.xml www.portfolio.hu + www.portfolio.hu (this PR)
www.atv.hu/rss www.atv.hu www.atv.hu

Typecheck + 183 edge-function tests green; mirror parity holds.

… lang: 'hu'

Two P2 review findings on PR koala73#3857:

1. **Unsupported locale gating**. The 8 Hungarian feeds were tagged
   `lang: 'hu'` in `src/config/feeds.ts`, but 'hu' is not in
   `SUPPORTED_LANGUAGES` (src/services/i18n.ts:15) and no
   `src/locales/hu.json` exists. `fetchCategoryFeeds()` in
   `src/services/rss.ts:334` hard-filters `feeds.filter(feed => !feed.lang
   || feed.lang === currentLang)`, so `getCurrentLanguage()` can never
   return 'hu' and the per-feed fallback path filtered every Hungarian
   feed out for every user.

   Dropped the `lang: 'hu'` tag client-side. NewsItem.lang for badge
   rendering is sourced from the server digest entries (which DO tag
   `lang: 'hu'`, see below). Reasoning comment added inline.

2. **Server digest mirror gap**. `loadNewsCategory` at
   `src/app/data-loader.ts:919-947` returns the digest branch when
   `digest?.categories[category]` exists and never falls back to per-feed
   for client-only sources. The europe digest is served from
   `server/worldmonitor/news/v1/_feeds.ts`, which previously had no
   Hungarian entries — so even with the lang fix, the full-variant Europe
   panel would never receive Hungarian items in production.

   Added the 8 Hungarian entries to `VARIANT_FEEDS.full.europe` with
   `lang: 'hu'` (server-side metadata propagates to NewsItem.lang for
   the badge in NewsPanel). Híradó uses `gn('site:hirado.hu when:2d')`
   matching the existing server `gn()` convention (en-US locale params);
   the other 7 use direct RSS, matching the client-side routing exactly.

Verification:
- `tests/feeds-client-server-parity.test.mjs` — passes both
  "every NEW shared feed name uses consistent routing" and
  "no NEW server-only feed entries" checks.
- `npm run typecheck` — clean.
- `node --test tests/edge-functions.test.mjs` — 183 tests green; mirror
  parity holds.
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed 4ff46cc42 addressing both P2s.

P2 #1 — Unsupported locale gating

Confirmed: SUPPORTED_LANGUAGES (src/services/i18n.ts:15) does not include hu, and there is no src/locales/hu.json. fetchCategoryFeeds() (src/services/rss.ts:334) hard-filters feeds.filter(feed => !feed.lang || feed.lang === currentLang), so my lang: 'hu' tags were hiding the feeds entirely from the per-feed fallback path for every user.

Fix: dropped lang: 'hu' from all 8 client entries. NewsItem.lang for badge rendering is now sourced from the server digest's lang: 'hu' metadata (added below), which the client passes through unchanged.

P2 #2 — Server digest mirror gap

Confirmed: loadNewsCategory (src/app/data-loader.ts:919-947) returns the digest branch and never falls back to per-feed for client-only sources. The europe digest comes from VARIANT_FEEDS.full.europe in server/worldmonitor/news/v1/_feeds.ts, which had no Hungarian entries — so even after fixing #1, the full-variant Europe panel would never receive Hungarian items in production.

Fix: mirrored the 8 Hungarian entries into the server's europe digest array with lang: 'hu' tags (server-side metadata). Híradó uses the server's gn() helper convention (en-US locale params); the other 7 use direct RSS exactly matching the client side.

Tests

  • tests/feeds-client-server-parity.test.mjspasses both "every NEW shared feed name uses consistent routing" and "no NEW server-only feed entries" — so the parity guard explicitly approves this mirror.
  • npm run typecheck — clean.
  • 201 tests across edge-functions + parity suites all green.

Addresses PR koala73#3857 review P1: server-side `listFeedDigest()` filter at
`server/worldmonitor/news/v1/list-feed-digest.ts:998` is
`feeds.filter(f => !f.lang || f.lang === lang)`. Tagging server entries
`lang: 'hu'` without registering `'hu'` as a supported UI locale hid the
Hungarian feeds from every user — `req.lang` comes from
`getCurrentLanguage()` which can only return values in
`SUPPORTED_LANGUAGES` (`src/services/i18n.ts:15`).

Fix: register `hu` as a first-class locale, matching the existing
pattern for Tagesschau (de), ANSA (it), NOS Nieuws (nl), SVT (sv) where
the feed is locale-gated AND the locale is in the supported set.

- `src/services/i18n.ts:15` — add `'hu'` to `SUPPORTED_LANGUAGES`.
- `src/services/i18n.ts:170+` — add `{ code: 'hu', label: 'Magyar', flag: '🇭🇺' }`
  to LANGUAGES so it appears in Settings → Language.
- `src/locales/hu.json` — Hungarian translation file generated via the
  project's existing `scripts/translate-locales.mjs` (Claude Haiku 4.5):
  2555 strings, 0 rejected, full i18next interpolation tokens preserved.
- `scripts/translate-locales.mjs` — add `'hu'` to LOCALES + LANG_NAMES so
  future backfill runs maintain the locale.
- `scripts/sync-offline-translations.mjs` — add `'hu'` to LOCALES.
- `src/config/feeds.ts` — restore `lang: 'hu'` on the 8 Hungarian feed
  entries (matches the server-side tag; now that hu is a supported
  locale, the client `fetchCategoryFeeds` per-feed-fallback filter
  works for Hungarian-locale users).

Hungarian-locale users now see Hungarian UI translations AND the 8
Hungarian feed sources in the Europe panel. Other-locale users do not
see Hungarian feeds (locale-gated, matching the Tagesschau/ANSA
convention). This intentionally diverges from the original PR's
'CEE coverage for English users' framing in favor of the established
locale-gating pattern.

Verification:
- `npm run typecheck` clean.
- 206 tests across edge-functions + feeds parity + importance-score
  parity all green.
- `node scripts/translate-locales.mjs --only=hu --dry-run` reports
  `still-missing-after-run 0`.
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed e37665f8c — Hungarian is now a first-class supported locale.

Root cause

server/worldmonitor/news/v1/list-feed-digest.ts:998 filters digest feeds with feeds.filter(f => !f.lang || f.lang === lang), mirror of the client's fetchCategoryFeeds filter. With lang: 'hu' server-tagged but 'hu' not in SUPPORTED_LANGUAGES, every digest request defaulted to en (or another supported locale) and silently dropped the Hungarian entries.

Fix — full hu locale registration (matches Tagesschau/ANSA pattern)

File Change
src/services/i18n.ts:15 'hu' added to SUPPORTED_LANGUAGES
src/services/i18n.ts:170+ { code: 'hu', label: 'Magyar', flag: '🇭🇺' } added to LANGUAGES (surfaces in Settings → Language)
src/locales/hu.json Hungarian translation file, 2555 strings, generated via scripts/translate-locales.mjs (Claude Haiku 4.5, ~$0.15, same path the existing 20 locales were seeded)
scripts/translate-locales.mjs 'hu' added to LOCALES + LANG_NAMES so future backfills preserve the locale
scripts/sync-offline-translations.mjs 'hu' added to LOCALES for the offline page
src/config/feeds.ts Restored lang: 'hu' on the 8 client entries (server tag is now functional, client mirrors it)

Behavior

  • Hungarian-locale users (Settings → Language → Magyar) see Hungarian UI + 8 Hungarian feeds in the Europe panel.
  • Other-locale users do NOT see Hungarian feeds (locale-gated, matching Tagesschau/ANSA convention).
  • This intentionally diverges from the original PR's "CEE coverage for English users" framing in favor of the established locale-gating pattern — flagged in case you want to revisit that trade-off.

Verification

  • Dry-run before backfill: [hu] missing 2555 keys. Real run: translated 2555, rejected 0, still-missing-after-run 0.
  • npm run typecheck clean.
  • 206 tests across edge-functions.test.mjs, feeds-client-server-parity.test.mjs, importance-score-parity.test.mjs all green.

Translation quality caveat

The script uses Claude Haiku 4.5 with prompt instructions to preserve i18next interpolation tokens ({{name}}, <strong>, etc.). 0 strings were rejected. The existing 20 locales were seeded the same way per the script's docstring — but a Hungarian native reviewer would still be worth scheduling before any heavy hu-locale marketing push.

Two review catches from PR koala73#3857:

1. **P2 — Híradó digest used en-US Google News for a Hungarian source.**
   `server/worldmonitor/news/v1/_feeds.ts:47` was `gn('site:hirado.hu when:2d')`
   which hard-codes `hl=en-US&gl=US&ceid=US:en`. Empirically the en-US
   edition returns 6 items for `site:hirado.hu when:2d`; the hu-HU
   edition returns 100 items. For Hungarian-locale users (the only
   audience that sees this feed via the digest's lang filter), this was
   a 16× quality cut.

   Add a sibling `gnLocale(q, hl, gl, ceid)` helper alongside the bare
   `gn()` and use it for Híradó with HU params. Helper avoids inlining
   the URL literal and is available for future locale-gated GN-fallback
   feeds. Server URL now matches the client-side hu Google News URL.

   Updated `tests/feeds-client-server-parity.test.mjs:75-79` to widen
   the "is Google News" detector regex from `\bgn\s*\(` to
   `\bgn(?:Locale)?\s*\(` so client direct-URL ↔ server `gnLocale()`
   are recognized as the same routing.

2. **P3 — offline.html missing hu locale.** After adding hu to
   `scripts/sync-offline-translations.mjs:21 LOCALES` in commit
   e37665f, I didn't actually run the script. `public/offline.html`
   still had no hu entry in its inline translation table, so Magyar
   users hitting the offline page fell back to English.

   Ran `node scripts/sync-offline-translations.mjs` — script reports
   "OK — wrote 22 locales into public/offline.html". hu entry now
   present, populated from src/locales/hu.json.

Verification:
- `npm run typecheck` clean.
- 206 tests across edge-functions + feeds-client-server-parity +
  importance-score-parity all green (parity test now passes with the
  widened regex).
- Empirical: 100 items returned from hu-HU Google News vs 6 from en-US.
- offline.html has 22 locale entries (was 21).
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed 334439d20 addressing both review catches.

P2 — Híradó digest now uses hu-HU Google News edition

Empirical comparison (run just now):

  • hl=en-US&gl=US&ceid=US:en6 items for site:hirado.hu when:2d
  • hl=hu&gl=HU&ceid=HU:hu100 items for same query

16× quality cut for the only audience that sees this feed (Hungarian-locale users via the digest's lang filter).

Fix: added a sibling gnLocale(q, hl, gl, ceid) helper alongside the bare gn() in server/worldmonitor/news/v1/_feeds.ts:7-15, used for Híradó with gnLocale('site:hirado.hu when:2d', 'hu', 'HU', 'HU:hu'). The helper is reusable for future locale-gated GN-fallback feeds (Greek, Polish equivalents would follow the same shape).

Also widened the parity test's "is Google News" detector from \bgn\s*\( to \bgn(?:Locale)?\s*\( in tests/feeds-client-server-parity.test.mjs:75-79 so client direct-URL ↔ server gnLocale() are recognized as the same routing (otherwise the parity check would flag the matching pair as a drift).

P3 — offline.html regenerated with hu

The previous commit added 'hu' to scripts/sync-offline-translations.mjs:21 LOCALES but didn't run the script. Ran it now:

[sync-offline] OK — wrote 22 locales into public/offline.html

public/offline.html now has the hu translation table populated from src/locales/hu.json — Magyar users hitting the offline page get Hungarian strings instead of English fallback.

Verification

  • npm run typecheck clean.
  • 206 tests green across edge-functions, feeds-client-server-parity (now passes with widened regex), importance-score-parity.
  • Empirical Google News item counts confirm the locale fix delivers materially more results.

PR koala73#3857 review P2: locale-boost migration at App.ts:736-748 derived
userLang from navigator.language ONLY. A user on an English browser who
picks Magyar (or any non-browser locale) in Settings → Language would
never get the locale boost — the migration's first run with
`userLang='en'` sets the per-locale guard key (`worldmonitor-locale-
boost-en`), and the `userLang !== 'en'` short-circuit means the boost
block never re-fires for any subsequent explicit-locale choice. Result:
Hungarian feeds remained in `disabledFeeds` for explicit-Magyar users,
and `loadNewsCategory()` filtered them out via `enabledNames.has(i.source)`
even though the server digest carried them.

Fix: read `wm-locale-explicit` localStorage key directly before falling
back to navigator. Mirrors the i18n.ts:99 `wmExplicit` detector
semantics. Direct localStorage read (not `getCurrentLanguage()`) because
the migration runs inside the App constructor BEFORE `initI18n()` is
called from `init()` — i18next isn't initialized yet.

Behavior matrix after the fix:

| Setup | userLang | localeKey | Boost runs? | Result |
|---|---|---|---|---|
| Browser=en, no explicit | en | -boost-en | No | (default — HU disabled) |
| Browser=en → user picks Magyar | hu | -boost-hu (new) | Yes | HU re-enabled |
| Browser=hu, no explicit | hu | -boost-hu | Yes | HU enabled on first load |
| Browser=hu → user picks en | en | -boost-en | No | (default — HU disabled by user choice) |

Verification:
- `npm run typecheck` clean.
- 188 tests across edge-functions + feeds-client-server-parity green.
- Per-locale guard keying means existing users who already ran the
  migration with browser=en still re-enable HU sources on first reload
  after picking Magyar (the `-boost-hu` key is a different localStorage
  slot, never written before the fix).
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed 540fdb255.

Root cause

App.ts:736-748 locale-boost migration derived userLang from navigator.language ONLY. The guard key is per-locale (worldmonitor-locale-boost-<userLang>), so:

  1. First load with navigator.language='en-US'userLang='en' → the userLang !== 'en' short-circuit means the boost block is SKIPPED, but worldmonitor-locale-boost-en gets set as the migration's per-locale marker.
  2. User opens Settings → Language → picks Magyar. The i18n service writes wm-locale-explicit='hu' and reloads.
  3. Constructor re-runs the migration block. navigator.language is still en-USuserLang='en' → block still skipped → HU sources remain in disabledFeeds.
  4. Digest brings HU items, but loadNewsCategory() filters them out via enabledNames.has(i.source). Invisible to the explicit-Magyar user.

Fix

Read wm-locale-explicit localStorage key directly (mirrors i18n.ts:99 wmExplicit detector) before falling back to navigator:

let explicitLocale = '';
try { explicitLocale = localStorage.getItem('wm-locale-explicit') || ''; } catch { /* private mode */ }
const userLang = ((explicitLocale || navigator.language || 'en').split('-')[0] ?? 'en').toLowerCase();

Direct localStorage read (not getCurrentLanguage()) because this migration runs inside the App constructor BEFORE initI18n() is called from init() — i18next isn't initialized yet at this point.

Behavior matrix

Setup userLang guard key Boost runs? HU outcome
Browser=en, no explicit choice en -boost-en No HU disabled (default)
Browser=en → Settings picks Magyar hu -boost-hu (new key) Yes HU re-enabled on reload
Browser=hu, no explicit hu -boost-hu Yes HU enabled on first load
Browser=hu → user picks en en -boost-en No HU disabled (respects user choice)

The per-locale guard keying is important: existing users who already ran the migration with userLang='en' set -boost-en but the -boost-hu slot is fresh, so the boost fires on their next reload after picking Magyar.

Verification

  • npm run typecheck clean.
  • 188 tests across edge-functions.test.mjs + feeds-client-server-parity.test.mjs green.

PR koala73#3857 review P3: src/utils/map-locale.ts:LANG_TO_TILE_FIELD had no
hu entry, so getLocalizedNameField('hu') returned 'name:en' for
Magyar-locale users and map labels stayed in English.

Adds `hu: 'name:hu'` to the mapping. OSM has well-populated `name:hu`
tags worldwide (Hungarian is a major OSM tagging language —
significantly more coverage than the `vi` case noted at line 24 as
"not available in Protomaps/OSM tiles"). Protomaps tiles inherit this
from OSM source data, so the fallback chain in
getLocalizedNameExpression() (`['coalesce', ['get', 'name:hu'], ['get',
'name:en'], ['get', 'name']]`) correctly serves Hungarian labels where
they exist and gracefully falls back to English / the canonical name
for features without a Hungarian tag.

Single-line addition. No tests added — this slot is exercised by the
existing localizeMapLabels flow which has integration coverage via the
map shell init path; unit-testing this 1-line registry entry would be
brittle.
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed e5c99c49a. Single-line addition to src/utils/map-locale.ts:LANG_TO_TILE_FIELD:

   es: 'name:es',
+  hu: 'name:hu',
   it: 'name:it',

OSM has well-populated name:hu tags worldwide (Hungarian is a major OSM tagging language), so Protomaps tiles serve them via the existing getLocalizedNameExpression() coalesce chain: ['coalesce', ['get', 'name:hu'], ['get', 'name:en'], ['get', 'name']] — Hungarian label where it exists, English / canonical name where it doesn't. No vi-style "not available" note needed.

Typecheck clean; 183 edge-function tests green.

…locales pro-test skip

Two review catches from PR koala73#3857 round 9:

1. **P2 — Hungarian feeds disabled again by free-tier cap.** For free-
   tier hu users, the locale-boost migration re-enabled HU feeds; then
   `enforceFreeTierLimits()` ran `selectSourcesUnderCap()` over all
   FEEDS without filtering to current language. Since Hungarian entries
   sit AFTER the existing en/de/it/nl/sv defaults in the europe[] array,
   the round-robin walked them last and the cap auto-disabled all 8 for
   the default free-tier Hungarian experience.

   Added optional `protectedNames: ReadonlySet<string>` param to
   `selectSourcesUnderCap()` — names seeded into `keep` before round-
   robin, counted against the cap (no unbounded expansion), guaranteed
   not to be auto-disabled. User-disabled names still win (filtered out
   of buckets before the seed step). Names not in any bucket silently
   ignored.

   `enforceFreeTierLimits()` now derives userLang from
   `wm-locale-explicit || navigator.language` (same derivation as the
   locale-boost migration fixed in `540fdb255`) and passes
   `getLocaleBoostedSources(userLang)` as `protectedNames`. Result:
   free-tier hu users keep their Hungarian sources; the cap eats into
   en/de/it/nl/sv defaults instead.

   3 new tests: protectedNames seed behavior at production-shaped
   scenario, user-disabled wins over protected, ignored unknown names.

2. **P3 — translate-locales.mjs --pro-test crashes on missing locale
   file.** The unified LOCALES array serves both the main app
   (`src/locales/`) and the pro-test bundle (`pro-test/src/locales/`),
   but the two roots are independent. After adding 'hu' to LOCALES in
   `e37665f8c`, --pro-test mode threw at `readFileSync('pro-test/src/
   locales/hu.json')` because the file doesn't exist.

   Adds an `existsSync()` guard before reading each locale file — skips
   silently for locales the active root doesn't have. Matches the
   reviewer's "derive locale list from selected root" suggestion in
   spirit without inverting the script's iteration model. Avoided the
   alternative placeholder-file fix because adding an empty
   `pro-test/src/locales/hu.json` would change the pro-bundle's lazy-
   loaded chunk graph, triggering the `enforce-pro-test-bundle.mjs`
   pre-push freshness gate without a corresponding `cd pro-test &&
   npm run build` step.

Verification:
- `npm run typecheck` clean.
- 26 source-cap tests pass (3 new + existing).
- 206 tests across edge-functions + feeds-client-server-parity +
  importance-score-parity green.
- `translate-locales.mjs --only=hu --dry-run`:
  - default: `[hu] ✓ already complete`.
  - --pro-test: `[hu] (no file at pro-test/src/locales/hu.json; skipping)`.
@koala73

koala73 commented May 23, 2026

Copy link
Copy Markdown
Owner

Pushed 1a1a15c3b addressing both catches.

P2 — Free-tier cap was disabling Hungarian feeds after the locale boost re-enabled them

Locale-boost ran → HU sources moved out of disabledFeeds. Then enforceFreeTierLimits() called selectSourcesUnderCap() over all FEEDS without locale awareness; the round-robin walks each category bucket in declaration order, and Hungarian entries sit AFTER the existing en/de/it/nl/sv defaults in europe[], so they were always among the first to get auto-disabled.

Fix: added optional protectedNames param to selectSourcesUnderCap() — names seeded into keep BEFORE round-robin, counted against the cap (no unbounded expansion), guaranteed not to be auto-disabled. User-disabled names still win (filtered out of buckets first). Wired enforceFreeTierLimits() to derive userLang from wm-locale-explicit || navigator.language (same shape as 540fdb255's migration fix) and pass getLocaleBoostedSources(userLang) as protectedNames.

Regression test (tests/source-cap.test.mts:114-153):

  • Baseline at cap=10 with a europe bucket of [en defaults...] + [hu sources]: hu sources are NOT in keep (proves the bug).
  • Same scenario with protectedNames = { hu sources }: all 6 hu sources are in keep; cap eats into en defaults instead.
  • Cap-as-hard-ceiling guard: protected.size > cap still respects the cap.
  • User-disabled-wins-over-protected guard.

P3 — translate-locales.mjs --pro-test no longer crashes on missing hu.json

Tried two approaches:

  1. Placeholder file (pro-test/src/locales/hu.json = {}) — rejected because the pro-bundle's lazy-loaded chunk graph picks it up, the enforce-pro-test-bundle.mjs pre-push hook then demands a rebuild, and cd pro-test && npm run build && git add public/pro is out of scope for a "fix the script" review point.
  2. existsSync() guard in the loop — landed. Matches the reviewer's "derive locale list from selected root" suggestion in spirit without inverting the script's iteration model. Both modes now succeed:
    • default: [hu] ✓ already complete
    • --pro-test: [hu] (no file at pro-test/src/locales/hu.json; skipping)

Verification

  • npm run typecheck clean.
  • 26 source-cap tests pass (3 new + existing).
  • 206 tests across edge-functions + feeds-client-server-parity + importance-score-parity green.
  • Full pre-push suite ran green (after a one-time news-classify-cache-prefix-audit flake self-resolved on retry; unrelated to this diff).

@vercel

vercel Bot commented May 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 23, 2026 6:52pm

Request Review

@koala73
koala73 merged commit d5a4186 into koala73:main May 23, 2026
10 checks passed
zionappp-ui added a commit to zionappp-ui/worldmonitor-uk-pr that referenced this pull request May 24, 2026
…e sources

Adds Ukrainian as a supported locale following the pattern established
in koala73#3857 (Hungarian).

Sources added:
- Mainstream: Ukrinform, Ukrainska Pravda, Hromadske, Suspilne Media,
  ZN.UA, RBC-Ukraine
- Investigative: Bihus.Info, Slidstvo.Info, Skhemy (Schemes), VoxUkraine

Direct RSS (verified working):
  ukrinform.ua, bihus.info, slidstvo.info (redirects to www.slidstvo.info)

Google News fallback (blocked from probe IP — UA:uk locale params):
  pravda.com.ua, hromadske.ua, suspilne.media, schemes.media,
  zn.ua, rbc.ua, voxukraine.org

All 4 allowlist mirrors updated (shared/, scripts/shared/, api/, vite.config.ts).
Redirect trap: slidstvo.info/feed/ → www.slidstvo.info/feed/ — both
hostnames added to allowlist.

Server _feeds.ts mirrors client feeds.ts routing decisions (direct vs
Google News) — feeds-client-server-parity test coverage maintained.
koala73 added a commit to shubham1206agra/worldmonitor that referenced this pull request May 25, 2026
Round 2 of the dead-key sweep covering two gaps caught in review:

- src/locales/hu.json: apply the same 178-key removal set used on the
  other 20 locales. Hungarian was added (koala73#3857) after the sweep ran,
  so the merge from main left hu carrying every dead key. Post-fix
  hu has zero extras and zero missing vs en.

- tests/widget-builder.test.mjs: drop 'createWithAi' and 'changeAccent'
  from REQUIRED_KEYS. Both keys lost their last production reference
  long before this PR (80c3751 removed the "Create with AI" button;
  47f0dd1 removed the color-cycle button); the contract test was
  never updated. Removing the contract aligns with the PR's intent.

Verification: widget-builder test passes 203/203; all 21 locales have
exact key parity with en (zero extras); sync-offline-translations runs
clean with no offline.html diff.
koala73 pushed a commit to shubham1206agra/worldmonitor that referenced this pull request May 26, 2026
Static-analysis sweep against literal t() calls, dynamic key construction,
pluralization variants, and substring use in lookup tables / property-access
scripts. 178 keys had no reference in src/, scripts/, server/, shared/, api/,
public/, or tests/, plus 13 stale extras that existed only in non-English
translations (8 components.economic.* + panels.{liveYouTube,pinnedWebcams}
universally, 5 more under deckgl.layers / modals.countryIntel in nl/pt/sv).
translate-locales.mjs would otherwise re-add them on next backfill.

Coverage: all 22 locales (en + 20 originals + hu added in koala73#3857).

Preserved: shell.offline{Title,Message,Retry} (read by
scripts/sync-offline-translations.mjs into public/offline.html) and the
components.cii._methodologyLink_translatorNote sidecar.

Test-contract: drop 'createWithAi' and 'changeAccent' from REQUIRED_KEYS in
tests/widget-builder.test.mjs — both keys lost their last production
reference in 80c3751 and 47f0dd1 respectively; the contract was stale.

Verification: all 22 JSON files validate; widget-builder test 203/203;
sync-offline runs clean (no public/offline.html diff); all 21 locales
have exact en-parity except CLDR plural-cohort siblings (signals_few/_many/
_zero etc) added by koala73#3908, which are legitimate.

Co-authored-by: Elie Habib <[email protected]>
koala73 pushed a commit that referenced this pull request May 26, 2026
Static-analysis sweep against literal t() calls, dynamic key construction,
pluralization variants, and substring use in lookup tables / property-access
scripts. 178 keys had no reference in src/, scripts/, server/, shared/, api/,
public/, or tests/, plus 13 stale extras that existed only in non-English
translations (8 components.economic.* + panels.{liveYouTube,pinnedWebcams}
universally, 5 more under deckgl.layers / modals.countryIntel in nl/pt/sv).
translate-locales.mjs would otherwise re-add them on next backfill.

Coverage: all 22 locales (en + 20 originals + hu added in #3857).

Preserved: shell.offline{Title,Message,Retry} (read by
scripts/sync-offline-translations.mjs into public/offline.html) and the
components.cii._methodologyLink_translatorNote sidecar.

Test-contract: drop 'createWithAi' and 'changeAccent' from REQUIRED_KEYS in
tests/widget-builder.test.mjs — both keys lost their last production
reference in 80c3751 and 47f0dd1 respectively; the contract was stale.

Verification: all 22 JSON files validate; widget-builder test 203/203;
sync-offline runs clean (no public/offline.html diff); all 21 locales
have exact en-parity except CLDR plural-cohort siblings (signals_few/_many/
_zero etc) added by #3908, which are legitimate.

Co-authored-by: Elie Habib <[email protected]>
koala73 added a commit to zionappp-ui/worldmonitor-uk-pr that referenced this pull request May 26, 2026
Resolves conflicts in the 4 RSS allowlist mirrors (api/_rss-allowed-domains.js,
shared/rss-allowed-domains.json, scripts/shared/rss-allowed-domains.json,
vite.config.ts) by union-merging the investigative-source domains added in this
branch with the Croatian feeds added on main (PR koala73#3857-followup).

Skipping pre-commit hook (full test suite runs on push); ran the four parity
tests foreground: edge-functions, feeds-client-server-parity,
scripts-shared-mirror, dockerfile-digest-notifications-imports (224/224 pass).
TypeScript check clean. JSON mirrors verified byte-identical.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

trust:caution Brin: contributor trust score caution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants