docs(plan): Middle East source gap audit by koala73 · Pull Request #3226 · koala73/worldmonitor

koala73 · 2026-04-20T07:19:00Z

Summary

Audits WorldMonitor's ME source coverage (src/config/feeds.ts, 523 feeds) against an external reference-class ME conflict monitor with 152 declared sources across 10 regions. Captures the gaps as a prioritized inventory and proposes eight focused follow-up PRs.

Headline gaps (all currently missing as direct feeds):

All GCC state wires (WAM, SPA, QNA, KUNA, BNA).
All Iranian state media (IRNA, Tasnim, Mehr, ISNA, Tehran Times, Press TV, Khamenei EN).
All Israeli domestic press (Times of Israel, Jerusalem Post, Ynetnews, i24, Globes, Calcalist, Mako, Kan, Maariv, Walla, IDF Spokesperson, PMO, MoFA, Knesset) — only Haaretz today, via a Google News wrapper.
All Gulf Arabic dailies across six GCC states.
US Gov direct RSS is collapsed into coarse Google News wrappers; granular per-agency feeds (White House sub-feeds, State Dept Near East Bureau, CENTCOM, Travel Advisories) are missing.
NOTAM coverage is FAA-only; ME civil aviation authorities not represented.

What's in this PR

plans/middle-east-source-gap-audit.md — 8 priority buckets, per-source effort tier (S/M/L), caveats (cloud-IP blocks, RTL rendering, editorial slant tagging), and proposed follow-up PRs.
External reference URL parked in docs/internal/ (gitignored) per the no-competitive-disclosure rule. This commit / PR references no competitor by name or URL.

This is inventory only — no feed entries added. Each follow-up PR will verify its own URLs, handle per-source blocks, and add tests.

Test plan

Markdown lint clean (48 files).
npm run typecheck + npm run typecheck:api clean.
npm run version:check OK.
Reviewer sanity-checks the priority ordering and confirms no false-positive gaps (i.e. sources we actually do have that I missed).

Post-Deploy Monitoring & Validation

No additional operational monitoring required: docs-only change, no code or runtime impact. Follow-up PRs that actually add feeds will each carry their own monitoring plan.

🤖 Generated with Claude Opus 4.7 (1M context) via Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) [email protected]

Audits WorldMonitor's current ME source coverage (src/config/feeds.ts, 523 feeds) against an external reference-class ME conflict monitor with 152 declared sources across 10 regions. Headline findings: - Zero direct RSS from any GCC state wire (WAM, SPA, QNA, KUNA, BNA). - Zero direct Iranian state-media feeds (IRNA, Tasnim, Mehr, ISNA, Tehran Times, Press TV, Khamenei EN). - Zero direct Israeli domestic-press feeds (Times of Israel, Jerusalem Post, Ynet, i24, Globes, IDF Spokesperson, PMO, MoFA, Knesset) — only Haaretz via a Google News wrapper. - Zero direct Gulf Arabic dailies across all six GCC states. - US Gov coverage is coarse Google News wrappers; granular per-agency feeds (White House sub-feeds, State Dept Near East Bureau, CENTCOM, Travel Advisories) are available and missing. - NOTAM coverage is FAA-only; civil aviation authorities in the ME region are not represented. Plan scopes eight priority buckets and proposes eight separate follow-up PRs so each batch stays small, testable, and independently revertable. The external reference URL is stashed privately in docs/internal/ per the no-competitive-disclosure rule and is not referenced in this plan or commit. Implementation-neutral: this PR ships the inventory only. Each follow-up PR will verify its own feed URLs + handle geo/UA blocks + write tests. Gates: md lint clean (48 files), typecheck clean (src + api), version sync OK.

vercel · 2026-04-20T07:19:07Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored		Apr 20, 2026 7:19am

greptile-apps · 2026-04-20T07:20:44Z

Greptile Summary

This PR adds a planning document (plans/middle-east-source-gap-audit.md) auditing WorldMonitor's Middle East feed coverage against an external 152-source reference. It identifies ~80+ missing direct feeds across 8 priority buckets and scopes eight follow-up PRs; no code or runtime changes are included.

Confidence Score: 5/5

Docs-only change with no runtime impact; safe to merge.

All findings are P2 style suggestions in a planning markdown file. No code is changed, no feeds are added, and the document explicitly defers URL verification and testing to follow-up PRs.

No files require special attention beyond the three minor documentation inconsistencies noted inline.

Important Files Changed

Filename	Overview
plans/middle-east-source-gap-audit.md	New planning document auditing ME source coverage gaps across 8 priority buckets; docs-only, no runtime impact. Minor issues: duplicate gov.il domain for PMO/MoFA rows, Priority 7 has no scoped follow-up PR, and an unexplained Washington Post mention in caveats.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ME Source Gap Audit\nplans/middle-east-source-gap-audit.md] --> P1[P1: GCC + Iran State Wires\nWAM · SPA · QNA · KUNA · BNA\nIRNA · Tasnim · Mehr · ISNA · Tehran Times]
    A --> P2[P2: Israeli Domestic Media\nToI · JPost · Ynet · i24 · Globes\nIDF · PMO · MoFA · Knesset]
    A --> P3[P3: Gulf Arabic Dailies\n~25 publications · 5 GCC states]
    A --> P4[P4: Gulf English Dailies\n~13 publications]
    A --> P5[P5: US Gov Fine-Grained\nCENTCOM · State Near East\nTravel Advisories · OFAC]
    A --> P6[P6: City/Ministry Comms\nDubai MO · Abu Dhabi MO · RTA\nGCC MoFA + MoD]
    A --> P7[P7: International Press\nAsharq Al-Awsat · FT ME\nSky News Arabia]
    A --> P8[P8 Spike: NOTAM Upgrade\nNotamify vs EUROCONTROL relay]
    P1 --> PR1[feat: direct state wires ~10 feeds]
    P2 --> PR2[feat: Israeli domestic EN ~8 feeds]
    P2 --> PR3[feat: Israeli domestic HE ~7 feeds]
    P3 --> PR4[feat: Gulf Arabic dailies ~25 feeds]
    P4 --> PR5[feat: Gulf English dailies ~13 feeds]
    P5 --> PR6[feat: US gov fine-grained ~10 feeds]
    P6 --> PR7[feat: city/ministry comms ~10 feeds]
    P8 --> PR8[spike: NOTAM coverage upgrade]

_{Reviews (1): Last reviewed commit: "docs(plan): Middle East source gap audit" | Re-trigger Greptile}

greptile-apps · 2026-04-20T07:20:48Z

+| Israeli Prime Minister's Office | gov.il | EN + HE | S |
+| Israeli MoFA | gov.il | EN + HE | S |


greptile-apps · 2026-04-20T07:20:49Z

+## Proposed follow-up PRs
+
+Each a separate PR, each a few feeds → fast merge, isolated blast radius.
+
+1. **feat(feeds): direct state wires** — adds P1 sources (WAM/SPA/QNA/KUNA/BNA/IRNA/Tasnim/Mehr/ISNA/Tehran Times). Replaces `{Iran International}` Google-News wrapper with direct IRNA/Tasnim/Mehr. ~10 feeds.
+2. **feat(feeds): Israeli domestic EN** — P2 English half (Times of Israel, Jerusalem Post, Ynetnews, i24, Globes, IDF Spokesperson RSS, PMO, MoFA). ~8 feeds.
+3. **feat(feeds): Israeli domestic Hebrew + Knesset** — P2 Hebrew (Ynet HE, Mako, Channel 13, Kan, Maariv, Walla) + Knesset scrape. ~7 feeds.
+4. **feat(feeds): Gulf Arabic dailies** — P3 batched per country. ~25 feeds across 5 PRs or 1 batch.
+5. **feat(feeds): Gulf English dailies** — P4 batch. ~13 feeds.
+6. **feat(feeds): US gov fine-grained** — P5. Replace coarse wrappers with direct sub-feeds; add CENTCOM + Travel Advisories. ~10 feeds.
+7. **feat(feeds): city/ministry comms** — P6. ~10 feeds.
+8. **spike: NOTAM coverage upgrade** — P8. Decision doc + cost estimate for Notamify vs. self-built EUROCONTROL relay.


Priority 7 sources have no corresponding follow-up PR

The plan documents a Priority 7 bucket (Asharq Al-Awsat, FT Middle East, Sky News Arabia, NHK/TASS/DW/Politico/Fox News verification) but the "Proposed follow-up PRs" list jumps straight from P6 (item #7 city/ministry comms) to P8 (item #8 NOTAM spike), leaving Priority 7 without a scoped PR. Consider either adding a feat(feeds): international ME press entry or explicitly noting that Priority 7 sources will be folded into another PR.

greptile-apps · 2026-04-20T07:20:50Z

+
+## Caveats / implementation notes
+
+- Several publishers block cloud IPs (Washington Post, Israel Hayom, Arabian Business, Press TV). Route through our existing Decodo proxy (see `scripts/decodo/`) and per-source UA overrides per `feedback_jrc_waf_chrome_ua_returns_html.md`.


Washington Post mentioned in caveats but absent from the gap inventory

The caveats list Washington Post among publishers that block cloud IPs, but WaPo does not appear in any of the Priority 1–8 gap tables. If the intent is to add a WaPo feed as part of this audit's follow-ups, it should appear in the inventory (probably Priority 7). If it's just a copy-paste artifact from prior docs, remove it to avoid confusing implementers who will grep this file for guidance.

Four direct-RSS sources verified from a clean IP and absent everywhere in the repo (src/config/feeds.ts, scripts/seed-*, ais-relay.cjs, RSS allowlist). Closes the highest-ROI Iran / Israel domestic-press gap from the ME source audit (PR #3226) with zero infra changes. - IRNA https://en.irna.ir/rss - Mehr News https://en.mehrnews.com/rss - Jerusalem Post https://www.jpost.com/rss/rssfeedsheadlines.aspx - Ynetnews https://www.ynetnews.com/Integration/StoryRss3089.xml Propaganda-risk metadata: - IRNA + Mehr tagged high / Iran state-affiliated (join Press TV). - JPost + Ynetnews tagged low with knownBiases for transparency. RSS allowlist updated in all three mirrors (shared/, scripts/shared/, api/_rss-allowed-domains.js) per the byte-identical mirror contract enforced by tests/edge-functions.test.mjs. Deferred (separate PRs): - Times of Israel: already in allowlist; was removed from feeds for cloud-IP 403. Needs Decodo routing. - IDF Spokesperson: idf.il has no direct RSS endpoint; needs scraper. - Tasnim / Press TV RSS / Israel Hayom: known cloud-IP blocks. - WAM / SPA / KUNA / QNA / BNA: public RSS endpoints are dead; sites migrated to SPAs or gate with 403. Plan doc (PR #3226) overstated the gap: it audited only feeds.ts and missed that travel advisories + US Embassy alerts are already covered by scripts/seed-security-advisories.mjs. NOTAM claim in that doc is also wrong: we use ICAO's global NOTAM API, not FAA.

…3236) * feat(feeds): add IRNA, Mehr, Jerusalem Post, Ynetnews to middleeast Four direct-RSS sources verified from a clean IP and absent everywhere in the repo (src/config/feeds.ts, scripts/seed-*, ais-relay.cjs, RSS allowlist). Closes the highest-ROI Iran / Israel domestic-press gap from the ME source audit (PR #3226) with zero infra changes. - IRNA https://en.irna.ir/rss - Mehr News https://en.mehrnews.com/rss - Jerusalem Post https://www.jpost.com/rss/rssfeedsheadlines.aspx - Ynetnews https://www.ynetnews.com/Integration/StoryRss3089.xml Propaganda-risk metadata: - IRNA + Mehr tagged high / Iran state-affiliated (join Press TV). - JPost + Ynetnews tagged low with knownBiases for transparency. RSS allowlist updated in all three mirrors (shared/, scripts/shared/, api/_rss-allowed-domains.js) per the byte-identical mirror contract enforced by tests/edge-functions.test.mjs. Deferred (separate PRs): - Times of Israel: already in allowlist; was removed from feeds for cloud-IP 403. Needs Decodo routing. - IDF Spokesperson: idf.il has no direct RSS endpoint; needs scraper. - Tasnim / Press TV RSS / Israel Hayom: known cloud-IP blocks. - WAM / SPA / KUNA / QNA / BNA: public RSS endpoints are dead; sites migrated to SPAs or gate with 403. Plan doc (PR #3226) overstated the gap: it audited only feeds.ts and missed that travel advisories + US Embassy alerts are already covered by scripts/seed-security-advisories.mjs. NOTAM claim in that doc is also wrong: we use ICAO's global NOTAM API, not FAA. * fix(feeds): enable IRNA, Mehr, Jerusalem Post, Ynetnews by default Reviewer on #3236 flagged that adding the four new ME feeds to FULL_FEEDS.middleeast alone leaves them disabled on first run, because App.ts:661 persists computeDefaultDisabledSources() output derived from DEFAULT_ENABLED_SOURCES. Users would have to manually re-enable via Settings > Sources, defeating the purpose of broadening the default ME mix. Add the four new sources to DEFAULT_ENABLED_SOURCES.middleeast so they ship on by default. Placement keeps them adjacent to their peers (IRNA / Mehr with the other Iran sources, JPost / Ynetnews after Haaretz). Risk/slant tags already in SOURCE_PROPAGANDA_RISK ensure downstream digest dedup + summarization weights them correctly. * style(feeds): move JPost + Ynetnews under Low-risk section header Greptile on #3236 flagged that both entries are risk: 'low' but were inserted above the `// Low risk - Independent with editorial standards` comment header, making the section boundary misleading for future contributors. Shift them under the header where they belong. No runtime change; cosmetic ordering only.

greptile-apps Bot reviewed Apr 20, 2026

View reviewed changes

koala73 mentioned this pull request Apr 20, 2026

feat(feeds): add IRNA, Mehr, Jerusalem Post, Ynetnews to middleeast #3236

Merged

6 tasks

koala73 closed this Apr 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(plan): Middle East source gap audit#3226

docs(plan): Middle East source gap audit#3226
koala73 wants to merge 1 commit into
mainfrom
feat/me-source-gap-audit

koala73 commented Apr 20, 2026

Uh oh!

vercel Bot commented Apr 20, 2026

Uh oh!

greptile-apps Bot commented Apr 20, 2026

Uh oh!

greptile-apps Bot Apr 20, 2026

Uh oh!

greptile-apps Bot Apr 20, 2026

Uh oh!

greptile-apps Bot Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		\| Israeli Prime Minister's Office \| gov.il \| EN + HE \| S \|
		\| Israeli MoFA \| gov.il \| EN + HE \| S \|


		## Caveats / implementation notes

		- Several publishers block cloud IPs (Washington Post, Israel Hayom, Arabian Business, Press TV). Route through our existing Decodo proxy (see `scripts/decodo/`) and per-source UA overrides per `feedback_jrc_waf_chrome_ua_returns_html.md`.

Conversation

koala73 commented Apr 20, 2026

Summary

What's in this PR

Test plan

Post-Deploy Monitoring & Validation

Uh oh!

vercel Bot commented Apr 20, 2026

Uh oh!

greptile-apps Bot commented Apr 20, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

greptile-apps Bot Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant