Weekly review deep-analysis pipeline: full-document enrichment, all 8 article types, 112 new articles in 14 languages#435
Conversation
…logging, fix 7 failing tests - Use client.searchDocuments() with date range as primary data source (aligns with REQUIRED_TOOLS['search_dokument'] and test mocking) - Wrap supplementary fetchers in Promise.resolve().then() so synchronous TypeErrors (e.g. method not found on mock) become rejected promises that .catch() can handle correctly - Add console.error() logging in all supplementary catch blocks (fixes review comment about silent error swallowing) - Fall back to supplementary type-specific results only when searchDocuments returns empty - Add search_dokument to generateSources() list Co-authored-by: pethers <[email protected]>
…IA historical context - Extend RawDocument with fullText, fullContent, contentFetched, speeches fields - Add CIAContext interface (party performance, coalition stability, voting patterns) - Extend ArticleContentData with optional ciaContext - weekly-review.ts: 5-step pipeline — search_dokument (IDs) → get_dokument_innehall per doc (full text, concurrency 3) → search_anforanden (speeches) → typed metadata fetchers → load CIA static JSON from data/cia-exports/ - generateGenericContent: document content is primary (full text excerpt, policy significance, chamber speeches per document). CIA data is secondary footnote only where concretely informative: party-specific motion pass rate (when known), and razor-thin majority warning (≤2 seats) — never a banner or primary framing - Remove CIA strategic context banner; replace with per-document <small> footnotes - Render document types in significance order: propositions → committee reports → motions - Test mock: add fetchDocumentDetails + searchSpeeches stubs (all 1586 tests pass) Co-authored-by: pethers <[email protected]>
…g + speeches key Co-authored-by: pethers <[email protected]>
Co-authored-by: pethers <[email protected]>
- week-ahead: add searchDocuments+searchSpeeches; show Legislative Agenda section - month-ahead: remove early-return on 0 events; add searchDocuments fallback - monthly-review: apply full 5-step enrichment pipeline (search→enrich→speeches→CIA) - breaking: auto-detect significant votes/docs; call generateBreakingNews - data-transformers: extend WeekAheadData with documents?; render in content - weekly-review: export loadCIAContext/enrichWithFullText/attachSpeeches helpers - tests: add new mock methods; fix REQUIRED_TOOLS assertions; all 1588 pass Co-authored-by: pethers <[email protected]>
…rticles
- parseMotionAuthorParty(): parse author/party from Swedish motion text (fixes Unknown)
- cleanMotionText(): strip Swedish boilerplate ("Förslag till riksdagsbeslut...")
- propSummaryFromOrgan(): ministry-specific framing for 9 known ministries
- generateEnhancedSummary(): use cleanMotionText for motions, propSummaryFromOrgan for props
- generateMotionsContent(): fallback to parseMotionAuthorParty when intressent_namn empty
- generatePolicySignificance(): doc-type-specific second sentence (mot/bet/prop)
- generateDocumentIntelligenceAnalysis(): cleanMotionText before extractKeyPassage
- Regenerated news/index*.html (531 articles in 14 langs)
- Regenerated data/news-articles.json (517 articles, 72 for 2026-02-22)
- Regenerated sitemap.xml (585 URLs covering all 7 article types for Feb22)
- All 1588 tests pass; CodeQL 0 alerts
Co-authored-by: pethers <[email protected]>
github-actions
bot
added
html-css
github-actions
bot
added
testing
…CIA spam, generate 112 articles in 14 languages Co-authored-by: pethers <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR expands the news generation pipeline to enrich weekly/monthly review content with full-document text and additional MCP cross-references (speeches/questions), updates the MCP client’s HTTP transport behavior, and regenerates site indexes/metadata for the newly produced multi-language articles.
Changes:
- Added full-document enrichment + CIA-context loading helpers to the weekly review pipeline and reused them in monthly review.
- Expanded week-ahead and month-ahead generators with additional MCP data sources and fallbacks.
- Updated MCPClient transport (fetch → HTTPS fallback) and added new MCP tool wrappers (written questions/interpellations), then regenerated news metadata/index pages.
Reviewed changes
Copilot reviewed 50 out of 127 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/news-types/weekly-review.test.ts | Extends weekly-review mocks for document details + speeches. |
| tests/news-types/week-ahead.test.ts | Updates week-ahead mock shape / REQUIRED_TOOLS assertions. |
| tests/news-types/monthly-review.test.ts | Extends monthly-review mocks and REQUIRED_TOOLS expectations. |
| tests/news-types/month-ahead.test.ts | Adds searchDocuments mocking and REQUIRED_TOOLS expectation for fallback. |
| scripts/news-types/weekly-review.ts | Implements full-text enrichment, speech attachment, CIA context loading. |
| scripts/news-types/week-ahead.ts | Adds supplementary documents/speeches/questions/interpellations to week-ahead output. |
| scripts/news-types/monthly-review.ts | Reuses weekly-review enrichment pipeline for monthly-review generation. |
| scripts/news-types/month-ahead.ts | Adds searchDocuments fallback when calendar events are empty. |
| scripts/mcp-client.ts | Adds HTTPS fallback transport + question/interpellation fetchers + speech response key fix. |
| scripts/generate-news-enhanced.ts | Expands generation to include breaking news and enriches week-ahead inputs/sources. |
| news/metadata/last-generation.json | Updates generation run metadata schema/values. |
| news/metadata/batch-status.json | Updates batch timestamp. |
| news/index.html | Regenerated English news index + JSON-LD item list. |
| news/index_sv.html | Regenerated Swedish news index + JSON-LD item list. |
| news/index_zh.html | Regenerated Chinese news index + JSON-LD item list. |
| news/index_no.html | Regenerated Norwegian news index + JSON-LD item list. |
| news/index_nl.html | Regenerated Dutch news index + JSON-LD item list. |
| news/index_ko.html | Regenerated Korean news index + JSON-LD item list. |
| news/index_ja.html | Regenerated Japanese news index + JSON-LD item list. |
| news/index_he.html | Regenerated Hebrew news index + JSON-LD item list. |
| news/index_fr.html | Regenerated French news index + JSON-LD item list. |
| news/index_fi.html | Regenerated Finnish news index + JSON-LD item list. |
| news/index_es.html | Regenerated Spanish news index + JSON-LD item list. |
| news/index_de.html | Regenerated German news index + JSON-LD item list. |
| news/index_da.html | Regenerated Danish news index + JSON-LD item list. |
| news/index_ar.html | Regenerated Arabic news index + JSON-LD item list. |
| const [recentReports, recentPropositions, recentMotions] = await Promise.all([ | ||
| Promise.resolve().then(() => client.fetchCommitteeReports({ from: fromStr, to: toStr, limit: 30 })) | ||
| .catch((err: unknown) => { console.error('Failed to fetch committee reports:', err); return [] as unknown[]; }), | ||
| Promise.resolve().then(() => client.fetchPropositions({ from: fromStr, to: toStr, limit: 20 })) | ||
| .catch((err: unknown) => { console.error('Failed to fetch propositions:', err); return [] as unknown[]; }), | ||
| Promise.resolve().then(() => client.fetchMotions({ from: fromStr, to: toStr, limit: 20 })) | ||
| .catch((err: unknown) => { console.error('Failed to fetch motions:', err); return [] as unknown[]; }), |
There was a problem hiding this comment.
fetchCommitteeReports / fetchPropositions / fetchMotions are being called with an object ({ from, to, limit }), but MCPClient defines these methods as positional args (limit, rm, organ). As written, the request payload will send a non-numeric limit and omit rm, likely breaking the MCP call. Update the calls to match the client method signature (or update MCPClient to accept the new param object consistently).
| fetchCommitteeReports: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; | ||
| fetchPropositions: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; | ||
| fetchMotions: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; | ||
| fetchDocumentDetails: Mock<(dokId: string, full?: boolean) => Promise<Record<string, unknown>>>; | ||
| searchSpeeches: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; |
There was a problem hiding this comment.
The MockMCPClientShape in this test models fetchCommitteeReports/fetchPropositions/fetchMotions as taking a params object, which doesn’t match the real MCPClient API (positional args). This mismatch can let tests pass while production code breaks; align the mock method signatures with scripts/mcp-client.ts.
| interface MockMCPClientShape { | ||
| fetchCalendarEvents: Mock<(start: string, end: string) => Promise<CalendarEvent[]>>; | ||
| searchDocuments: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; | ||
| searchSpeeches: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; | ||
| } |
There was a problem hiding this comment.
generateWeekAhead now calls fetchWrittenQuestions() and fetchInterpellations(), but the MCP client mock used in this test doesn’t define them. The mocked MCPClient will throw at runtime (TypeError: client.fetchWrittenQuestions is not a function). Add these methods to MockMCPClientShape and mockClientInstance, and initialize their mockResolvedValue(s) in beforeEach.
| // ── Step 4: fetch speeches from the period ───────────────────────────── | ||
| console.log(' 🔄 Step 4 — Fetching speeches from the period...'); | ||
| const speeches = await Promise.resolve() | ||
| .then(() => client.searchSpeeches({ rm: '2025/26', limit: 100 }) as Promise<unknown[]>) | ||
| .catch((err: unknown) => { console.error('Failed to fetch speeches:', err); return [] as unknown[]; }); |
There was a problem hiding this comment.
Step 4 claims to fetch speeches “from the period”, but the MCP query only sets rm/limit and omits the weekly fromStr/toStr date range. This can pull unrelated speeches and inflate processing; pass the same date bounds used for documents to searchSpeeches (if supported by SearchSpeechesParams).
| const eventGrid = transformCalendarToEventGrid(events as Record<string, unknown>[], lang); | ||
| const content = generateArticleContent({ events, highlights: [] }, 'week-ahead', lang); | ||
| const watchPoints = extractWatchPoints({ events }, lang); | ||
| const metadata = generateMetadata({ events }, 'week-ahead', lang); | ||
| const weekData = { events, documents, questions, interpellations, highlights: [] as Array<{title:string;description:string}> }; | ||
| const content = generateArticleContent(weekData, 'week-ahead', lang); | ||
| const watchPoints = extractWatchPoints({ events, documents }, lang); | ||
| const metadata = generateMetadata({ events, documents }, 'week-ahead', lang); | ||
| const readTime = calculateReadTime(content); | ||
| const sources = generateSources(['get_calendar_events']); | ||
| const sources = generateSources(['get_calendar_events', 'search_dokument', 'get_fragor', 'get_interpellationer']); | ||
|
|
There was a problem hiding this comment.
generateWeekAhead records a search_anforanden MCP call, but the sources list passed into the article omits it. This makes the rendered “Sources” section incomplete/misleading; include search_anforanden in the generateSources(...) input (and consider adding it to crossReferences.sources too for consistency).
| crossReferences: { | ||
| event: `${events.length} events over ${daysAhead} days`, | ||
| sources: ['calendar_events'] | ||
| } | ||
| event: events.length > 0 | ||
| ? `${events.length} events over ${daysAhead} days` | ||
| : `${documents.length} upcoming documents`, | ||
| sources: ['calendar_events'], | ||
| }, |
There was a problem hiding this comment.
When the calendar is empty and the generator falls back to search_dokument, crossReferences.sources still reports only calendar_events. That misrepresents which data sources were used; add the document source (e.g., search_dokument) when documents are used so cross-references remain accurate.
scripts/generate-news-enhanced.ts
Outdated
| const content: string = generateArticleContent(weekData, 'week-ahead', lang); | ||
| const watchPoints = extractWatchPoints({ events: events as Parameters<typeof transformCalendarToEventGrid>[0], documents }, lang); | ||
| const metadata = generateMetadata({ events: events as Parameters<typeof transformCalendarToEventGrid>[0], documents }, 'week-ahead', lang); | ||
| const readTime: string = calculateReadTime(content); | ||
| const sources: string[] = generateSources(['get_calendar_events']); | ||
| const sources: string[] = generateSources(['get_calendar_events', 'get_fragor', 'get_interpellationer']); |
There was a problem hiding this comment.
This week-ahead generator calls searchDocuments(...) earlier in the pipeline, but the sources list here omits that tool. For source transparency (and to match the actual MCP calls), include search_dokument in the generateSources(...) input.
|
@copilot apply changes based on the comments in this thread |
…ency, date-scoped speeches
- monthly-review.ts: fetchCommitteeReports/fetchPropositions/fetchMotions now use
positional args (limit, rm) to match MCPClient API (was: object {from,to,limit})
- monthly-review.test.ts: align mock signatures for the three fetchers to match
real MCPClient positional args (limit?, rm?, organ?)
- week-ahead.test.ts: add fetchWrittenQuestions + fetchInterpellations to
MockMCPClientShape, mockClientInstance, and beforeEach reinit
- weekly-review.ts: pass from/to date range to searchSpeeches so only speeches
from the review period are fetched
- week-ahead.ts generateWeekAhead: add search_anforanden to generateSources()
- generate-news-enhanced.ts generateWeekAhead: add search_dokument to generateSources()
- month-ahead.ts: crossReferences.sources is now dynamic — includes search_dokument
when falling back to documents (calendar empty)
All 1588 tests pass
Co-authored-by: pethers <[email protected]>
Applied all 7 fixes in edc5a54:
All 1588 tests pass. |
| // Step 2: typed metadata fetchers (robust: errors → empty []) | ||
| console.log(' 🔄 Step 2 — Fetching typed metadata (reports, propositions, motions)...'); | ||
| const [recentReports, recentPropositions, recentMotions] = await Promise.all([ | ||
| Promise.resolve().then(() => client.fetchCommitteeReports(30, '2025/26')) | ||
| .catch((err: unknown) => { console.error('Failed to fetch committee reports:', err); return [] as unknown[]; }), | ||
| Promise.resolve().then(() => client.fetchPropositions(20, '2025/26')) | ||
| .catch((err: unknown) => { console.error('Failed to fetch propositions:', err); return [] as unknown[]; }), | ||
| Promise.resolve().then(() => client.fetchMotions(20, '2025/26')) | ||
| .catch((err: unknown) => { console.error('Failed to fetch motions:', err); return [] as unknown[]; }), | ||
| ]); |
There was a problem hiding this comment.
monthly-review calls fetchCommitteeReports/fetchPropositions/fetchMotions but never records corresponding MCPCallRecord entries (e.g. tool: 'get_betankanden' / 'get_propositioner' / 'get_motioner'). This reduces traceability and can break downstream cross-reference/source reporting; push those tool records to mcpCalls (as weekly-review does) when the calls succeed.
| const sources: string[] = generateSources([ | ||
| 'search_dokument', | ||
| 'get_dokument_innehall', | ||
| 'search_anforanden', | ||
| 'get_betankanden', | ||
| 'get_propositioner', | ||
| 'get_motioner', | ||
| ]); |
There was a problem hiding this comment.
REQUIRED_TOOLS still lists only 'search_dokument', but generateWeeklyReview now uses additional MCP tools (get_dokument_innehall, search_anforanden, get_betankanden, get_propositioner, get_motioner). Update REQUIRED_TOOLS to reflect the actual tool usage so validation/docs don’t drift.
|
|
||
| signal.addEventListener('abort', () => req.destroy(new Error('Request aborted'))); | ||
| req.on('error', reject); |
There was a problem hiding this comment.
nodeHttpsPost only listens for future abort events; if the AbortSignal is already aborted, the request will still be created and only fail later. Check signal.aborted up front and reject/throw immediately to avoid unnecessary sockets and to match fetch() behavior more closely.
| interface MockMCPClientShape { | ||
| searchDocuments: Mock<(params: Record<string, unknown>) => Promise<SearchDocument[]>>; | ||
| fetchDocumentDetails: Mock<(dokId: string) => Promise<Record<string, unknown>>>; | ||
| searchSpeeches: Mock<(params: Record<string, unknown>) => Promise<unknown[]>>; |
There was a problem hiding this comment.
Mock type for fetchDocumentDetails doesn’t match the MCPClient API/usage (the implementation calls fetchDocumentDetails(dokId, true)). Update the mock signature to accept the optional boolean arg so TypeScript reflects the real contract and the test won’t drift from production code.
| crossReferences: { | ||
| event: `${(events as unknown[]).length} events`, | ||
| sources: ['calendar_events'] | ||
| } | ||
| event: `${(events as unknown[]).length} events, ${documents.length} docs, ${questions.length} questions, ${interpellations.length} interpellations`, | ||
| sources: ['calendar_events', 'search_dokument', 'get_fragor', 'get_interpellationer'], | ||
| }, |
There was a problem hiding this comment.
crossReferences.sources omits 'search_anforanden' even though this generator calls searchSpeeches() and records an MCP call with tool 'search_anforanden'. This makes the reported sources inconsistent with actual data used; include 'search_anforanden' (and keep naming consistent with the tool identifiers you record in mcpCalls).
| * Required MCP tools for week-ahead articles. | ||
| * Updated 2026-02-22: also calls get_fragor and get_interpellationer | ||
| * as parliamentary-questions context sources when calendar is sparse. | ||
| */ | ||
| export const REQUIRED_TOOLS: readonly string[] = [ |
There was a problem hiding this comment.
The REQUIRED_TOOLS docstring says questions/interpellations are fetched only when the calendar is sparse, but the implementation always fetches get_fragor/get_interpellationer (and also search_dokument/search_anforanden). Either make these calls conditional on event scarcity as described, or update the comment to match the current behavior.
| // Step 2: typed metadata fetchers (robust: errors → empty []) | ||
| console.log(' 🔄 Step 2 — Fetching typed metadata (reports, propositions, motions)...'); | ||
| const [recentReports, recentPropositions, recentMotions] = await Promise.all([ | ||
| Promise.resolve().then(() => client.fetchCommitteeReports(30, '2025/26')) | ||
| .catch((err: unknown) => { console.error('Failed to fetch committee reports:', err); return [] as unknown[]; }), | ||
| Promise.resolve().then(() => client.fetchPropositions(20, '2025/26')) | ||
| .catch((err: unknown) => { console.error('Failed to fetch propositions:', err); return [] as unknown[]; }), | ||
| Promise.resolve().then(() => client.fetchMotions(20, '2025/26')) | ||
| .catch((err: unknown) => { console.error('Failed to fetch motions:', err); return [] as unknown[]; }), | ||
| ]); |
There was a problem hiding this comment.
Typed metadata results (fetchCommitteeReports/fetchPropositions/fetchMotions) are merged into documents without filtering to the [fromStr,toStr] lookback window. Since these endpoints can return items outside the last 30 days, this can skew the monthly review scope; filter these lists by date (similar to weekly-review’s filterRecent) before merging.
| { | ||
| "title": "Mes adelante: Perspectiva parlamentaria", | ||
| "date": "2026-02-22", | ||
| "type": "analysis", | ||
| "slug": "2026-02-22-month-ahead-es.html", | ||
| "lang": "es", |
There was a problem hiding this comment.
This month-ahead entry is categorized as type 'analysis', but month-ahead articles are rendered as 'prospective' in the generators/templates. If the index type drives filtering/styling, this will misclassify the article; adjust the index generation logic (likely classifyArticleType) to treat month-ahead as prospective and regenerate indexes.
| { | ||
| "title": "월간 전망: 의회 전망", | ||
| "date": "2026-02-22", | ||
| "type": "analysis", | ||
| "slug": "2026-02-22-month-ahead-ko.html", | ||
| "lang": "ko", |
There was a problem hiding this comment.
This month-ahead entry is categorized as type 'analysis', but month-ahead articles are rendered as 'prospective' in the generators/templates. If the index type drives filtering/styling, this will misclassify the article; adjust the index generation logic (likely classifyArticleType) to treat month-ahead as prospective and regenerate indexes.
|
@copilot apply changes based on the comments in this thread |
…, type correctness - weekly-review.ts: REQUIRED_TOOLS now lists all 6 actual tools used (search_dokument, get_dokument_innehall, search_anforanden, get_betankanden, get_propositioner, get_motioner) - mcp-client.ts: nodeHttpsPost checks signal.aborted up front and rejects immediately instead of creating a socket for an already-cancelled request - tests/weekly-review.test.ts: fetchDocumentDetails mock signature updated to (dokId: string, includeFullText?: boolean) matching real MCPClient API - week-ahead.ts: REQUIRED_TOOLS comment updated (all tools always invoked, not conditional on calendar scarcity); crossReferences.sources now includes 'search_anforanden' for full transparency - monthly-review.ts: push MCPCallRecord entries for get_betankanden / get_propositioner / get_motioner after typed fetchers complete; filter typed results by [fromStr..toStr] lookback window via filterByDate() before merging so out-of-period documents cannot skew the monthly scope - generate-news-indexes.ts: classifyArticleType now treats month-ahead filenames as 'prospective' (was incorrectly falling through to 'analysis') - Regenerated news/index*.html — all month-ahead articles now show type:'prospective' (571 articles, 14 languages) - All 1588 tests pass Co-authored-by: pethers <[email protected]>
Applied all 9 fixes in fc61576:
All 1588 tests pass. |
pethers
merged commit 383e447
into
news-generation/weekly-review-2026-02-22-25732d3df5a3d00f
Rewrites the weekly review from a document-list generator into a true parliamentary intelligence pipeline. Every document is fetched in full, analysed by policy domain, and cross-referenced with chamber speeches — CIA static data used only as secondary historical footnote, never as primary framing.
Pipeline architecture
weekly-review.ts— 5-step enrichmentsearch_dokument→ IDs + types onlyget_dokument_innehallper doc (concurrency 3) → full text (never the rawtextXML dump)search_anforanden→ chamber speeches attached per document, date-scoped to the review periodget_betankanden,get_propositioner,get_motioner) with MCPCallRecord traceabilitydata/cia-exports/) loaded as secondary contextgenerateGenericContent— document-primary renderinggeneratePolicySignificance()+ attached speechesmajorityMargin ≤ 2All 8 article types now generate daily content
weekly-reviewmonthly-reviewweek-aheadget_fragor+get_interpellationer+search_anforanden(always)month-aheadcrossReferences.sourcesbreakingContent quality fixes
parseMotionAuthorParty()— regex-extracts author/party from Swedish motion boilerplate (av NAME (PARTY)) fixing "Unknown (Unknown)" attributionscleanMotionText()— strips"Förslag till riksdagsbeslut"preamble before summary extractionpropSummaryFromOrgan()— ministry-specific framing for 9 ministriesgenerateDocumentIntelligenceAnalysis()— normalisesprop→proposition,bet→report,mot→motion; appliescleanMotionTextbeforeextractKeyPassage; CIA per-document coalition spam removedfetchWrittenQuestions/fetchInterpellationsadded toMCPClientMCPClient.nodeHttpsPost()— checkssignal.abortedup front before creating a socket (matchesfetch()behaviour)searchSpeeches()response key fixed:anforanden(notspeeches)performPost()falls back tohttps.requestSource transparency fixes
weekly-review.tsREQUIRED_TOOLSupdated to all 6 tools actually used:search_dokument,get_dokument_innehall,search_anforanden,get_betankanden,get_propositioner,get_motionerweek-ahead.tsREQUIRED_TOOLScomment corrected (all tools always invoked);crossReferences.sourcesincludessearch_anforandenmonth-ahead.tscrossReferences.sourcesis dynamic — includessearch_dokumentwhen falling back to documentsmonthly-review.tspushesMCPCallRecordentries for typed fetchers; filters results to the[fromStr..toStr]lookback window before merginggenerate-news-indexes.tsclassifyArticleTypenow correctly mapsmonth-aheadfilenames to typeprospectiveGenerated output
112 new HTML articles (8 article types × 14 languages) for 2026-02-22 — week-ahead, monthly-review, and month-ahead generated for the first time.
news/index*.htmlregenerated with correctprospectivetype for all month-ahead entries.data/news-articles.json(557 articles) andsitemap.xml(625 URLs) regenerated.🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.