fix(intelligence): publish CII risk scoring methodology + de-dup country constants (#3725)#3780
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR publishes the previously opaque CII scoring methodology by adding
Confidence Score: 3/5The methodology publication and new proto fields are well-structured, but the unchanged Redis cache keys mean that a deployed instance may serve cached responses missing the newly guaranteed fields for up to an hour after rollout. The core correctness concern is the stale Redis cache: both server/worldmonitor/intelligence/v1/get-risk-scores.ts — the Important Files Changed
Sequence DiagramsequenceDiagram
participant Client
participant Handler as getRiskScores handler
participant Cache as Redis Cache
participant ACLED
participant Aux as Auxiliary Sources
Client->>Handler: GET /api/intelligence/risk-scores
Handler->>Cache: getCachedJson('risk:scores:sebuf:v1')
alt cache hit (up to 600s TTL)
Cache-->>Handler: cached GetRiskScoresResponse (may lack methodologyVersion/eventMultiplier if written pre-deploy)
Handler-->>Client: response (fields may be proto-default empty-string / 0.0)
else cache miss
Handler->>ACLED: fetchACLEDEvents()
Handler->>Aux: fetchAuxiliarySources()
ACLED-->>Handler: acledEvents[]
Aux-->>Handler: AuxiliarySources
Handler->>Handler: computeCIIScores() — sets methodologyVersion + eventMultiplier
Handler->>Handler: computeStrategicRisks()
Handler->>Cache: setCachedJson('risk:scores:sebuf:v1', result, 600)
Handler->>Cache: setCachedJson('risk:scores:sebuf:stale:v1', result, 3600)
Handler-->>Client: fresh GetRiskScoresResponse with methodologyVersion and eventMultiplier
end
|
| * Rationale: gives the most-affected country full weight, lets the next four | ||
| * contribute decreasing influence, and reaches zero weight at position 6 so | ||
| * the slice size of 5 is naturally bounded. |
There was a problem hiding this comment.
Contradictory claim about when the positional weight reaches zero
The comment says "reaches zero weight at position 6" but at position 6 (0-based), weight = 1 − 6 × 0.15 = 0.10, not zero. The weight only turns negative at position 7 (−0.05). The sibling STRATEGIC_RISK_TOP_N JSDoc in the same file correctly states "position 6 hits weight=0.10 (still contributing)", so the two comments directly contradict each other.
| * Rationale: gives the most-affected country full weight, lets the next four | |
| * contribute decreasing influence, and reaches zero weight at position 6 so | |
| * the slice size of 5 is naturally bounded. | |
| * Rationale: gives the most-affected country full weight, lets the next four | |
| * contribute decreasing influence, and reaches weight 0.10 at position 6 — | |
| * still non-trivial, which is why STRATEGIC_RISK_TOP_N caps the window at 5 | |
| * to keep the roll-up focused on the very top of the dashboard. |
…ghten drift test, split tooltip key for i18n, bound event_multiplier Review-requested follow-ups for #3725 (PR #3780): - Bump CII risk-scores Redis cache prefix `risk:scores:sebuf:v1` → `risk:scores:sebuf:stale:v2` across every reader (get-risk-scores writer, get-country-risk, chat-analyst-context, brief-story-context, _shared/ cache-keys, api/bootstrap, api/health, api/mcp, regional-snapshot/*, seed-cross-source-signals, seed-forecasts, plus the two tests that hardcode these keys). The proto change in #3725 added REQUIRED methodologyVersion + eventMultiplier; pre-existing v1 payloads no longer satisfy the shape. - Split the methodology link into its own i18n key `components.cii.methodologyLink` so the transparency disclosure ships in all 21 locales immediately without holding the feature for translation. Revert `components.cii.infoTooltip` in `src/locales/en.json` to the pre-#3725 wording. Add the new key to all 20 non-en locale files with the English value as stop-gap; translators can localize asynchronously. CIIPanel.ts concatenates both keys for render. Embed a sibling `_methodologyLink_translatorNote` in en.json with a translation-pipeline TODO. - Tighten the methodology-doc drift guard (`tests/cii-scoring.test.mts`) — previously checked only that a row exists for each code; now parses the numeric columns and asserts every baseline + multiplier in the doc matches `BASELINE_RISK[code]` and `EVENT_MULTIPLIER[code]` exactly. Catches silent drift between code and doc. Required exporting `BASELINE_RISK` / `EVENT_MULTIPLIER` from `get-risk-scores.ts`. - Add `lte = 10` upper bound on `CiiScore.event_multiplier` in `proto/.../intelligence.proto` (de-facto editorial max is ~3.0; 10 gives ~3.3x headroom while still catching obviously-wrong values). Refreshed openapi schemas via `make generate`. - File issue #3789 to track the deferred drift reconciliation (7 countries: AF, EG, IQ, JP, KR, LB, QA). Link it from the methodology doc. Verification: - `make generate` succeeds with the new proto bound (openapi schemas carry maximum: 10). - `npm run typecheck` and `npm run typecheck:api` clean. - `npm run test:data` 9108 passing (was 9107; +1 from the stricter drift test). - `npm run lint` no NEW warnings on touched files (pre-existing complexity warnings on `computeCIIScores`, `buildOilStocksCover`, etc. are unchanged).
…try constants (#3725, closes #3722) The /api/intelligence/risk-scores endpoint emits a per-country Composite Instability Index whose editorial weights (baselineRisk, eventMultiplier, the positional-decay step, scale-floor, and scale-factor) were undisclosed magic numbers — see #3725 (and strict subset #3722). API consumers had no way to detect drift or audit the formula. This change: - Adds CiiScore.methodology_version (string) and CiiScore.event_multiplier (double) to the proto so every score carries its provenance on the wire (regenerated via `make generate`). - Extracts the strategic-risk roll-up coefficients (positional decay, scale floor, scale factor, top-N) into a named module server/worldmonitor/intelligence/v1/_risk-config.ts with a change-protocol header. - Publishes the methodology in docs/methodology/cii-risk-scores.md including the per-country baseline + multiplier table, the U/C/S/I component meanings, and the strategic-risk band derivation. - Documents the drift between server BASELINE_RISK / EVENT_MULTIPLIER and the frontend CURATED_COUNTRIES table (12 cells across 7 codes) rather than silently reconciling either side — server values remain authoritative on the wire. - Adds an `Includes per-country baseline + event multiplier — see /docs/methodology/cii-risk-scores for the published table` sentence to the cii infoTooltip locale (en.json only — translation team can catch up). - Extends tests/cii-scoring.test.mts (24 -> 35 tests) with disclosure-field assertions, strategic-risk roll-up coverage (floor/cap, all-zero case, empty input), and a doc-drift guard that fails loudly when CURATED_COUNTRIES or TIER1_COUNTRIES gains a code not listed in the methodology table. Verification: `npm run typecheck`, `npm run typecheck:api`, `npm run test:data` (9107 passing). Lint clean on changed files (pre-existing complexity warning on computeCIIScores is unchanged).
…ghten drift test, split tooltip key for i18n, bound event_multiplier Review-requested follow-ups for #3725 (PR #3780): - Bump CII risk-scores Redis cache prefix `risk:scores:sebuf:v1` → `risk:scores:sebuf:stale:v2` across every reader (get-risk-scores writer, get-country-risk, chat-analyst-context, brief-story-context, _shared/ cache-keys, api/bootstrap, api/health, api/mcp, regional-snapshot/*, seed-cross-source-signals, seed-forecasts, plus the two tests that hardcode these keys). The proto change in #3725 added REQUIRED methodologyVersion + eventMultiplier; pre-existing v1 payloads no longer satisfy the shape. - Split the methodology link into its own i18n key `components.cii.methodologyLink` so the transparency disclosure ships in all 21 locales immediately without holding the feature for translation. Revert `components.cii.infoTooltip` in `src/locales/en.json` to the pre-#3725 wording. Add the new key to all 20 non-en locale files with the English value as stop-gap; translators can localize asynchronously. CIIPanel.ts concatenates both keys for render. Embed a sibling `_methodologyLink_translatorNote` in en.json with a translation-pipeline TODO. - Tighten the methodology-doc drift guard (`tests/cii-scoring.test.mts`) — previously checked only that a row exists for each code; now parses the numeric columns and asserts every baseline + multiplier in the doc matches `BASELINE_RISK[code]` and `EVENT_MULTIPLIER[code]` exactly. Catches silent drift between code and doc. Required exporting `BASELINE_RISK` / `EVENT_MULTIPLIER` from `get-risk-scores.ts`. - Add `lte = 10` upper bound on `CiiScore.event_multiplier` in `proto/.../intelligence.proto` (de-facto editorial max is ~3.0; 10 gives ~3.3x headroom while still catching obviously-wrong values). Refreshed openapi schemas via `make generate`. - File issue #3789 to track the deferred drift reconciliation (7 countries: AF, EG, IQ, JP, KR, LB, QA). Link it from the methodology doc. Verification: - `make generate` succeeds with the new proto bound (openapi schemas carry maximum: 10). - `npm run typecheck` and `npm run typecheck:api` clean. - `npm run test:data` 9108 passing (was 9107; +1 from the stricter drift test). - `npm run lint` no NEW warnings on touched files (pre-existing complexity warnings on `computeCIIScores`, `buildOilStocksCover`, etc. are unchanged).
f564e98 to
66a5518
Compare
…ator pipeline picks it up (#3796) PR #3780 (CII methodology link tooltip) shipped a new i18n key `components.cii.methodologyLink` in en.json and pre-populated the same English string into all 20 non-EN locale files as a stop-gap. Discovered during PR #3793 review: scripts/translate-locales.mjs:175 detects missing keys via `!(k in flat)`. Pre-populating the key makes every non-EN locale appear "already complete", so the translator script will never re-translate them — the English copy would be stranded in 20 locales forever. Fix: remove `components.cii.methodologyLink` from the 20 non-EN locale files (leaving en.json untouched). On the next backfill, the translator pipeline will see the key as missing and translate it properly. Same class of bug surfaced in #3793; this is the retroactive cleanup for the already-merged #3780 keys. Refs: #3780, #3793
…ng false positives (#3791) (#3793) * fix(disease-outbreaks): disclose alert-level classifier + fix substring false positives (#3791) The disease outbreaks panel labels each item with an ALERT / WARNING / WATCH badge derived from `detectAlertLevel` in `scripts/_disease-outbreaks-helpers.mjs`. Previously the function used naked `text.includes(...)` substring checks with no methodology disclosure to users — same defect class as #3725 (CII), at the smaller scale appropriate for a 6-keyword classifier. Changes - Extract `ALERT_KEYWORDS` and `WARNING_KEYWORDS` as frozen exports + bump to word-boundary regex matching. Prior substring matching could fire "epidemic" inside "antiepidemic" and "spread" inside "widespread", silently over-promoting items. Both regressions are now covered by tests. - Add a methodology header comment + `ALERT_LEVEL_METHODOLOGY_VERSION = 'v1'` so callers can pin a version and the change protocol is in-code. - Publish `docs/methodology/disease-alert-level.md` with the full keyword table, inputs, known limitations, and a change protocol (mirrors the pattern established by docs/methodology/cii-risk-scores.md in #3780). - Add `components.diseaseOutbreaks.methodologyNote` to en.json + 20 sibling locales (English stop-gap) so the disclosure ships in every locale on day one — same approach as #3780 review round 2 (i18next does not fall back on stale values for present keys). - Render the new key in `DiseaseOutbreaksPanel`'s infoTooltip after the existing copy. - Tests (+9 cases in `tests/disease-outbreaks-seed.test.mjs`): cover each keyword class, substring-false-positive regression guards, case-insensitive matching, title+desc concatenation, null/undefined inputs, alert-beats- warning precedence, frozen-keyword invariant, and version-string format. Closes #3791 * fix(disease-outbreaks): address review #3793 — restore translator pipeline visibility, ship version field, rename to avoid ALERT_KEYWORDS collision Restores the translator pipeline's visibility into the methodologyNote key: the prior approach pre-populated EN text in all 20 non-EN locales, which made scripts/translate-locales.mjs treat every locale as "already complete" and permanently skip translation. Now only en.json carries the key; the next translator run will fill in the missing locales naturally. Plumbs ALERT_LEVEL_METHODOLOGY_VERSION onto the published payload as `alertLevelMethodologyVersion`. The constant existed but nothing consumed it, so the change protocol's "bump the version" step had no observable effect. The version field now flows through diseasePublishTransform and lands on the canonical Redis key, giving clients a real signal when the editorial keyword set changes. Renames ALERT_KEYWORDS → DISEASE_ALERT_KEYWORDS and WARNING_KEYWORDS → DISEASE_WARNING_KEYWORDS to avoid name collision with the unrelated geopolitical ALERT_KEYWORDS export in src/config/feeds.ts (war/invasion/ nuclear). Exports the precompiled DISEASE_ALERT_RE and DISEASE_WARNING_RE regexes so future callers don't reach for text.includes(kw) and silently re-introduce the #3791 substring bug. Drops the brittle line-range pointer in the methodology doc in favor of the symbol name detectAlertLevel — symbols don't rot. Tests: 28/28 pass (was 26 — added version-field plumbing test and exported-regex word-boundary test). * fix(disease-outbreaks): #3793 round 3 — add alertLevelMethodologyVersion to proto contract (P2 review finding) The seeded JSON payload already carries alertLevelMethodologyVersion (round 2), but it was not declared on the proto contract — so generated clients and OpenAPI consumers couldn't see it. This closes the gap: - proto: add `string alert_level_methodology_version = 3` on ListDiseaseOutbreaksResponse with a comment pointing at the methodology doc. - generate: regenerated client/server TS + HealthService + worldmonitor OpenAPI specs via `make generate` (no hand-edits). - handler: server/worldmonitor/health/v1/list-disease-outbreaks.ts now reads the field off the cached payload and falls back to 'v1' so payloads cached before the field was published still satisfy the proto contract during the deploy → next-seed-publish window (transitional read tolerance). - src/services/disease-outbreaks.ts emptyOutbreaks constant gets the same 'v1' fallback so the offline/empty path satisfies the new required field. - tests/list-disease-outbreaks-handler.test.mjs (new): asserts the proto + generated bindings declare the field, plus behavioural coverage that the handler echoes the field when present and falls back to 'v1' when absent.
…ift callout (#3816) * docs(mintlify): publish CII + disease-alert methodology pages with drift callout Promotes the two methodology docs from PRs #3780 (CII risk scores) and #3793 (disease alert level) from plain markdown into the Mintlify documentation site so they render in the public docs nav under "Intelligence & Analysis". - Convert `docs/methodology/cii-risk-scores.md` → `.mdx` with Mintlify frontmatter and a prominent `<Warning>` callout at the top linking issue #3789, so API consumers know the server-side `BASELINE_RISK` / `EVENT_MULTIPLIER` tables are authoritative over the frontend `CURATED_COUNTRIES` for the 7 drifted countries (AF, EG, IQ, JP, KR, LB, QA) until reconciliation lands. - Convert `docs/methodology/disease-alert-level.md` → `.mdx` with Mintlify frontmatter. No drift callout — this one has no known drift. - Wire both pages into `docs/docs.json` after the existing methodology entries in the "Intelligence & Analysis" group. - Add a one-line cross-link from `docs/algorithms.mdx` (existing CII section) to the new methodology page. - Update `tests/cii-scoring.test.mts` to read `cii-risk-scores.mdx` (the four `resolve(..., 'cii-risk-scores.md')` paths); test logic and the drift/version assertions are unchanged. No production code is touched. The proto and server-side comments that still reference the `.md` path are left as-is per scope; the public URL slug (`/methodology/cii-risk-scores`) is unchanged and is already what `src/locales/en.json` and other consumers link to. * docs(comments): update .md → .mdx references in proto, source, and test comments Follow-on to the .md→.mdx rename in this PR: chase down all in-source doc-path references so they point at the actual file that ships in Mintlify. Touches: - proto headers (regenerated OpenAPI .yaml/.json picked up the rename) - _risk-config.ts, get-risk-scores.ts header comments - _disease-outbreaks-helpers.mjs, seed-disease-outbreaks.mjs comments - cii-scoring.test.mts inline comments + assertion messages No behavioural change — only doc-path strings inside comments and error messages.
Summary
/api/intelligence/risk-scoresemits per-country Composite Instability Index (CII) scores whose editorial weights —baselineRisk,eventMultiplier, and the strategic-risk roll-up coefficients — were undisclosed magic numbers living next to each other inget-risk-scores.ts. There was no way for an API consumer to detect coefficient drift between deploys or audit which values produced a given score.This PR closes the disclosure gap and folds in #3722 (strict subset — also asked for methodology publication).
Changes
CiiScore.methodology_version(string) andCiiScore.event_multiplier(double) so every score carries its provenance on the wire. Regenerated viamake generate.server/worldmonitor/intelligence/v1/_risk-config.tsnow ownsCII_FORMULA_VERSION, the strategic-risk positional decay (0.15), scale floor (15), scale factor (0.7), and top-N window (5), each with a rationale comment and a "bump the version + update the doc + CHANGELOG" change protocol header.docs/methodology/cii-risk-scores.mdcovers the four CII components (U/C/S/I) qualitatively, lists the per-countrybaselineRisk/eventMultipliertable, documents the strategic-risk band derivation, and explicitly flags drift between the server tables and the frontendCURATED_COUNTRIEStable.BASELINE_RISK/EVENT_MULTIPLIERare a near-copy ofCURATED_COUNTRIESinsrc/config/countries.ts, but they have drifted in 12 cells across 7 codes (AF, EG, IQ, JP, KR, LB, QA). Server values remain authoritative on the wire (we did not silently reconcile either side); the divergence is now visible in the methodology doc with an explicitDrift vs CURATED_COUNTRIEScolumn. If a reviewer wants the server to adopt frontend values (or vice versa), that is a follow-up — call it out and bumpCII_FORMULA_VERSION.components.cii.infoTooltiplocale string insrc/locales/en.jsonnow points to/docs/methodology/cii-risk-scores. Onlyen.jsonis updated; the translation team will pick up the other locales in their normal cadence.tests/cii-scoring.test.mtsfrom 24 → 35 tests:CiiScorecarriesmethodologyVersion === CII_FORMULA_VERSIONand a finiteeventMultiplier > 0staticBaselinematchesCURATED_COUNTRIESfor the 8 representative no-drift codescomputeStrategicRisksoutput is in[STRATEGIC_RISK_SCALE_FLOOR=15, 100]and equals exactly15when all top-N CII scores are 0[1.00, 0.85, 0.70, 0.55, 0.40]docs/methodology/cii-risk-scores.mdmust list everyCURATED_COUNTRIESand everyTIER1_COUNTRIEScode, and must reference the currentCII_FORMULA_VERSIONstring — catches "added a country but forgot the doc" and "changed the version but forgot the doc"Closes #3725
Closes #3722 (duplicate — #3725 is a strict superset)
Test plan
npm run typecheck— cleannpm run typecheck:api— cleannpm run test:data— 9107 / 9107 passing (24 → 35 incii-scoring.test.mts)make generate— proto regen clean, new fields visible insrc/generated/server/.../service_server.tsnoExcessiveCognitiveComplexityoncomputeCIIScoresis unchanged)Review round 2 —
f564e98Folded in CHANGES-REQUESTED items from the round-1 review (no blockers, all should-fixes):
v1→v2—risk:scores:sebuf:v1andrisk:scores:sebuf:stale:v1now:v2across every reader/writer (15 source files + 2 test files). The proto change adds REQUIREDmethodologyVersion+eventMultiplier; old:v1payloads in Redis violated the new shape on read. Seed-meta key (seed-meta:risk:scores:sebuf) is intentionally unchanged — that's freshness tracking, not payload.event_multiplierupper bound — added(buf.validate.field).double.lte = 10onproto/.../intelligence.proto. De-facto editorial max is ~3.0 (KP); 10 gives ~3.3x headroom while still catching obviously-wrong values. Refreshed openapi schemas viamake generate(maximum: 10visible indocs/api/IntelligenceService.openapi.{yaml,json}).components.cii.methodologyLink(sibling toinfoTooltip).en.jsoninfoTooltipreverted to pre-Undisclosed algorithmic bias: hardcoded country event multipliers (up to 10x) and arbitrary formula coefficients in CII risk scores #3725 wording; the new key shipped to all 21 locale files (English text used as stop-gap in the 20 non-en locales — translators can localize independently of the long-form tooltip).CIIPanel.tsconcatenates both at render time. Sibling_methodologyLink_translatorNoteinen.jsonflags the new key for translation pipelines (JSON has no comment syntax).tests/cii-scoring.test.mtspreviously checked only row existence (\|\s${code}\s\|). New test parses each row's numeric columns and assertsBASELINE_RISK[code]+EVENT_MULTIPLIER[code]match the methodology doc exactly. Required exporting both tables fromget-risk-scores.ts. Adds 1 test (9107 → 9108).baselineRisk/eventMultiplierdrift (AF, EG, IQ, JP, KR, LB, QA). Linked fromdocs/methodology/cii-risk-scores.md§ 2.Verification (round 2)
make generate— clean, openapi reflectsmaximum: 10oneventMultipliernpm run typecheck— cleannpm run typecheck:api— cleannpm run test:data— 9108 / 9108 passing (+1 from new drift-guard test)npm run lint— no NEW warnings on touched files (pre-existing complexity warnings oncomputeCIIScores,buildOilStocksCover,useOptionalChainlints unchanged)