@shubham1206agra is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

@koala73 This is how I came up with themes which can give maximum throughput.

GDELT GKG GeoJSON v1 — Positive-tone yield by theme

Probed via QUERY=<theme>&TIMESPAN=1440 (24h window, ~99% of features carry urltone).
Probe date: 2026-05-21, ~24h windows during US business hours.

Selected for production POSITIVE_QUERIES (6 themes)

Balances absolute positive volume vs. precision; covers the 6 classifier categories:

Greptile Summary

This PR fixes two GDELT GKG seeders that were sending free-text keyword queries the v1 API doesn't index well, replacing them with single-theme fan-out calls (one request per theme tag), fixing QUERY/MAXROWS param casing, and adding cross-call URL dedup so a multi-theme article can't inflate location counts.

• scripts/ais-relay.cjs: 3 OR-queries → 6 theme tags; adds a urltone > 5 gate to keep only positive-tone articles; seenUrls dedup is correctly scoped across the sequential calls.
• scripts/seed-unrest-events.mjs: 1 OR-query → 3 theme tags (PROTEST, STRIKE, VIOLENT_UNREST); introduces a shared locationMap and seenUrls across themes; adds partial-failure tolerance — but the console.log after the fan-out loop references features, now a block-scoped variable inside the loop, causing a ReferenceError on every successful run.

Confidence Score: 3/5

The unrest seeder will throw a ReferenceError on every successful run, silently discarding all aggregated events — this path needs a fix before merging.

The features variable is declared with const inside the fan-out loop body in seed-unrest-events.mjs, making it block-scoped to each iteration. The console.log statement that follows the loop still references features.length; as an ES module the file runs in strict mode, so every successful call to fetchGdeltEvents ends with a ReferenceError: features is not defined instead of returning the aggregated events. The positive-events changes in ais-relay.cjs are clean.

scripts/seed-unrest-events.mjs — the post-loop console.log references a block-scoped variable from the loop

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant S as Seeder
    participant G as GDELT GKG v1 API
    participant R as Redis / Store

    Note over S: seed-unrest-events.mjs
    loop for each theme in [PROTEST, STRIKE, VIOLENT_UNREST]
        S->>G: "GET ?QUERY=THEME&MAXROWS=2500"
        G-->>S: GeoJSON features
        S->>S: skip if fUrl in seenUrls
        S->>S: merge into shared locationMap
        S->>S: add fUrl to seenUrls
        Note over S: wait 5.5s (rate limit)
    end
    S->>S: "filter count >= 5"
    S->>R: store aggregated events

    Note over S: ais-relay.cjs (positive events)
    loop for each theme in [SOC_INNOVATION, EDUCATION, MEDICAL, TOURISM, ...]
        S->>G: "GET ?QUERY=THEME&MAXROWS=500"
        G-->>S: GeoJSON features
        S->>S: skip if url in seenUrls
        S->>S: "skip if urltone <= 5"
        S->>S: merge into local locationMap
        S->>S: add url to seenUrls
        Note over S: wait 5.5s (rate limit)
    end
    S->>S: "filter count >= 3, dedup by name"
    S->>R: store positive events

_{Reviews (1): Last reviewed commit: "fix(seeders): switch GDELT GKG queries t..." | Re-trigger Greptile}

I have tried switching to https://blog.gdeltproject.org/gdelt-geo-2-0-api-debuts/, but the API is 404'ing for me.

Code Review — switch GDELT GKG queries to single-theme fan-out + URL dedup

The core diagnosis (param casing + single-theme constraint) is sound and the partial-failure handling is a genuine improvement. But the PR ships non-trivial new logic with zero test changes, and one existing locked contract is broken.

🔴 Blocking

• Breaks existing test tests/seed-unrest-gdelt-fetch.test.mjs (case Fetch AIS signals from API instead of hardcoded data #6).
  That test mocks _proxyFetcher to throw once then succeed, and asserts calls === 2. With 3-theme fan-out the fetcher is now invoked 4 times (1 throw + 3 successful theme calls), so assert.equal(calls, 2, 'should retry exactly once after the 522') fails. The test header explicitly says it "locks the behavioural contract" — this is CI breakage. The test must be updated to the fan-out model.

• No test coverage for any of the new behavior. Both changed files are production scripts. Nothing exercises: the QUERY/MAXROWS casing fix, theme fan-out + locationMap merge, cross-call seenUrls dedup, the anyThemeSucceeded partial-failure path (all-fail → throw, one-succeeds → proceed), or the positive urltone gate. At minimum add: all-themes-fail → throws with lastError; one-theme-fails → still aggregates; same-URL across two themes counted once.

🟡 Should fix

• URL-only dedup collapses multi-location articles. GKG v1 GeoJSON emits one feature per (article, location) mention (hence the original existing.count++ and the "individual mentions, not aggregated counts" comment). seenUrls keys on properties.url alone, so an article mentioning Cairo + Alexandria now contributes to only the first location — and this fires within a single theme call, not just across calls. The intended fix (don't double-count an article carrying multiple theme tags) needs a composite key:

  const dedupKey = url ? `${url}|${key}` : null;   // key = `${lat.toFixed(1)}:${lon.toFixed(1)}`

  As written, the change also silently shifts count semantics from "total mentions at a location" to "distinct articles at a location," lowering values against the count < 5 floor and the count > 20 HIGH-confidence threshold in the unrest seeder — fewer events qualify. If that re-tune is intentional, call it out; otherwise the composite key restores prior behavior.

• Dedup field mismatch with extractGdeltSourceUrls. Dedup checks only feature.properties?.url, but extractGdeltSourceUrls recognizes 7 URL fields (source_url, document_url, article_url, …). A feature exposing only source_url escapes dedup entirely. Test Fetch AIS signals from API instead of hardcoded data #6's own mock has features[1] carrying just source_url, so the inconsistency is already latent in the fixture. Use the same URL-resolution logic for the dedup key as for source extraction.

• Positive urltone > 5 gate is aggressive. GDELT urltone is roughly −10..+10; strictly > 5 is a small slice, and themes like MEDICAL/EDUCATION skew neutral-to-negative. Combined with the theme switch, the positive-events panel could go sparse or empty. Verify live volume before merge and consider a lower threshold (e.g. > 2).

🟢 Minor / notes

• Runtime grows: positive seeder is now 6 themes × 5.5s spacing (~+28s) plus per-call proxy timeouts/retries; unrest +11s. Fine for a cron seeder, but worth noting if either has a wall-clock budget.
• Good: seenUrls.add is placed after coordinate validation in both files, so invalid features don't poison the dedup set.
• The load-bearing claim — v1 GKG requires uppercase QUERY/MAXROWS and ignored the old lowercase params — is plausible and matches GDELT's case-sensitive v1 APIs, but it's the entire premise of the fix. A one-line note on before/after live response volume would close that loop.

Verdict

Right root cause and reasonable structure, but not mergeable as-is: it breaks an existing test and adds substantial untested logic. Update seed-unrest-gdelt-fetch.test.mjs for fan-out, add coverage for dedup + partial-failure, and fix the URL-only dedup key (composite url|location) before merge.

@koala73 Thanks for the review. But can you confirm if the positive themes are fine with you?

Let me address #3853 (comment) and I will let you know when this is ready

A one-line note on before/after live response volume would close that loop.

@koala73 Please see the following table for confirmation

@koala73 This is ready for 2nd round of review

Re-review (after fan-out/dedup updates)

Thanks for the revisions — the earlier blocking items are resolved. Verified by checking out the PR head and running node --test tests/seed-unrest-gdelt-fetch.test.mjs → 11/11 pass.

✅ Resolved

• Composite dedup key — both seeders now key on ${url}|${latlonBucket}, so a multi-location article still contributes to each location. Test Add mobile warning popup for desktop-optimized experience #10 explicitly proves Cairo + Alexandria both reach 13 (URL-only dedup would leave Alexandria at 12).
• Multi-field URL resolution — positive uses gkgFeatureUrl(), unrest uses extractGdeltSourceUrls(...)[0]; test Add WebGL LOD clustering for map layers #11 confirms a source_url-only feature is still deduped.
• Tone threshold lowered 5 → 2 — reasonable.
• Test coverage — the broken test Fetch AIS signals from API instead of hardcoded data #6 is fixed (calls === 4) and tests Add power grid stress & blackout alert layer #7–Add WebGL LOD clustering for map layers #11 cover all-fail, fan-out merge, partial-failure tolerance, composite dedup, and multi-field URL. Good, meaningful tests.

🟡 Should fix — new CI-time regression

The inter-theme rate-limit delay at seed-unrest-events.mjs:279 is not injectable:

if (i > 0) await new Promise((r) => setTimeout(r, 5_500)); // GDELT rate limit

The proxy-retry path has _sleep/_jitter seams, but this new fan-out delay has none. Result: each of the 6 fetchGdeltEvents tests sleeps a real 2 × 5.5 s = 11 s — I measured tests #6–#11 at ~11 000 ms each, ~66 s of pure sleep added to every CI run. Add a _themeDelayMs opt (default 5500) — or route the delay through the existing _sleep seam — so tests can pass 0. The positive seeder's 5×5.5 s loop has the same untestable delay.

🟢 Minor / nits

• Positive uses gkgFeatureUrl() (raw first-match of 7 fields); unrest uses extractGdeltSourceUrls(...)[0] (normalized). Both correct internally, but a single shared helper would avoid drift.
• fetchGdeltGeoPositive in ais-relay.cjs got the same dedup/tone/URL logic but has zero test coverage — ais-relay.cjs exports nothing, so it's structurally untestable. Not blocking, but the positive path is now unverified; consider extracting the GKG aggregation into a shared tested module.
• Stale assertion messages in test Fetch AIS signals from API instead of hardcoded data #6: assert.equal(calls, 4, 'should retry exactly once after the 522') (4 isn't intuitively "once") and 'Fifteen mentions' (11 after dedup).
• const url = gkgFeatureUrl(...) — double space after =.

Verdict

Core logic is correct and now well-tested on the unrest side. Recommend adding the inter-theme delay test seam before merge (66 s of CI sleep is a real cost the PR's own tests incur); the rest are nits.

@koala73 Thanks for the review. I have made the changes

@koala73 This is ready for review

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	May 26, 2026 9:34am