Hindsight

Sealed before the outcome. Graded after.

Multi-source financial intelligence pipeline. Ingests global financial news from free sources, deduplicates it into events, scores market impact with grounded LLM analysis, and measures its own accuracy against labelled historical outcomes.

The design commitment is that every claim the system makes is traceable to a source URL, and every accuracy number it reports comes from a harness you can run yourself.

Status

Stage	State
Canonical event schema	Done
Source adapters (RSS, GDELT, regulators)	Done — 14/15 live
Live source verification	Done
Corpus quality audit + normalisation	Done — measured, see below
Dedup + clustering	Done — measured, see below
Cross-publisher similarity metric	Done — measured, see below
Impact scoring + reasoning	Done — working end to end
Trend aggregation + market mood	Done
Accuracy tracking vs real price moves	Done — accumulating
Web app (six views, charts, search, watchlist)	Done — static, self-contained
Event persistence across runs	Done — measured, see below
Deployment (GitHub Pages, scheduled)	Done — see "Deploying it"
HTTP API (FastAPI, read-only)	Done — see "The platform"
Next.js frontend (6 routes, interactive charts)	Done

Measuring whether any of it is right

uv run python -m scripts.scorecard record   # seal today's calls
uv run python -m scripts.scorecard grade    # tomorrow: fill in outcomes
uv run python -m scripts.scorecard report   # accuracy vs baseline

Direction only — never magnitude. Predicting the size of a move is a different and much harder problem, and claiming to do it invites a question the system cannot answer.

Four properties keep the number honest:

Calls are sealed before the outcome exists. record writes the prediction with the timestamp it was made at; grade comes back later and fills in only the outcome columns. The insert is ON CONFLICT DO NOTHING, never DO UPDATE, so a call cannot be quietly improved once the answer is visible. That property has its own test.

The baseline is printed as prominently as the score. Always guessing the majority direction is the number to beat. On the first run that baseline was 83%, so a model scoring 80% would look respectable while having learned nothing.

Every figure carries a Wilson interval. 62% over 40 calls spans roughly 46–76% — consistent with a coin. The report refuses a verdict below n=30 and says NO EDGE DEMONSTRATED whenever the interval still contains the baseline, however good the point estimate looks.

Retrospective runs are quarantined. backfill scores past news so the harness can be exercised without waiting days, but those calls are stored with mode='retrospective' and never enter the headline figure — the model may have met those outcomes in training, so it can look prescient without predicting anything. all_graded() defaults to live-only, so a caller who forgets to filter gets the defensible number rather than the flattering one.

The report also splits accuracy by stated confidence, by corroboration, and by conviction. Each split tests an assumption the product makes: if high confidence is no better than low, the label is decoration and should not be shown to users; if multi-source events are not called better than single-source ones, the corroboration weighting in the trend view is unearned.

Sign conventions are explicit because getting one wrong inverts a result silently rather than raising: INR trades as USD/INR, Bonds as the 10-year yield, so both carry a -1.

29 of 34 vocabulary targets are gradable, including Indian sectors against real NSE sector indices — ^NSEBANK, ^CNXIT, ^CNXPHARMA and six more. These are published indices, not proxies assembled for the occasion, and sector calls were previously discarded unrecorded, which capped how fast the sample could grow. Sample size is the binding constraint on knowing anything here, so this is the single largest lever available.

Five targets stay ungradable on purpose. Telecom, Defence and Aviation have no NSE index yfinance serves; ^CNXMEDIA is Media, and mapping Telecom onto it would grade every telecom call against the wrong basket while the scorecard looked perfectly healthy. Emerging Markets and Middle East are too diffuse to pin to one instrument. An ungradable call is better than a confidently mis-graded one — the second kind corrupts the number the whole project rests on, and leaves no trace that it did.

Because "Banking" with no country is the same unfalsifiable claim "Equities" was, the prompt states that sector targets mean Indian listed sectors and tells the model to score a country market instead when a story moves a sector elsewhere.

Running it on a schedule

Accuracy only means something at a few hundred calls, which needs the pipeline running unattended. Three workflows in .github/workflows/:

Workflow	When	What
record.yml	03:30, 11:00, 20:30 UTC, weekdays	Fetch, score, seal new calls
grade.yml	22:00 UTC, weekdays	Fill in settled outcomes, print the report
test.yml	every push	Lint, tests, prediction-log integrity

The published page was showing July. record.yml rebuilt the site with --snapshot, which reads fixtures/latest.json — the frozen regression corpus committed to the repo. The step's own comment claimed it was "reusing the snapshot the record step just fetched", but record never writes one, so the page rendered July's news under a generated-at stamp of today, and a trailing || true meant nothing ever failed. It now builds from live data.

record.yml also carries the event store between runs as a cache. Without it every scheduled run starts from an empty database, sees one instant, and reports stories as single-sourced that several outlets covered — the failure the persistence layer exists to prevent, reintroduced by the deployment.

To turn it on: push the repo to GitHub and add GROQ_API_KEY under Settings → Secrets → Actions. Optionally set NC_CONTACT_EMAIL as a variable (SEC EDGAR requires it once that source is enabled). There is also scripts/run_local_schedule.sh for plain cron, though a laptop that sleeps overnight leaves holes in the record — and those holes cluster on exactly the days you were away, which is not random missingness.

The recording times are spread across sessions on purpose. Sampling only the pre-open would produce an accuracy figure that really describes "how well the model reads morning headlines" while presenting itself as a general one.

The log lives in git, not just in a database. data/predictions.jsonl is append-only and committed by the bot; the SQLite file is a working copy rebuilt from it each run. This is not a workaround for ephemeral CI runners — it is a stronger integrity guarantee than the database gives on its own. Git timestamps every prediction independently of this program, so "the call existed before its outcome did" is evidenced by a commit rather than asserted by the code that benefits from it.

scripts/verify_log.py runs on every push and rejects a corrupted record: malformed lines, duplicate prediction IDs, scores outside the scale, timestamps in the future, and above all a call whose graded_at precedes its made_at — which would mean the outcome was known first and would invalidate everything downstream of it.

Deploying it

The page is a single self-contained HTML file with no build step, so anything that serves static files will host it. GitHub Pages is wired up already.

1. Push the repository to GitHub.

2. Add the API key. Settings → Secrets and variables → Actions → New repository secret, named GROQ_API_KEY. Optionally add NC_CONTACT_EMAIL as a variable — SEC EDGAR requires a real contact address in the User-Agent and will IP-ban without one, so it is needed only if that source is enabled.

3. Turn on Pages. Settings → Pages → Source → GitHub Actions. Not "Deploy from a branch": pages.yml builds the Next.js frontend as a static export and uploads it as an artefact. It sets NEXT_BASE_PATH to the repo name because a project site is served from /<repo>/ — on a user site or a custom domain, set that to an empty string.

4. Run it once. Actions → Record predictions → Run workflow. That fetches live news, seals today's calls, rebuilds the page and commits it; the commit then triggers pages.yml, which publishes. The URL appears on the workflow's summary and under Settings → Pages.

After that it is unattended — three recording runs and one grading run per weekday.

What deploys, and what does not

workflow	trigger	does
test.yml	every push	backend lint + tests + log integrity; frontend types, lint, audit, build
record.yml	03:30, 11:00, 20:30 UTC weekdays	fetch, score, seal, rebuild the page, commit
grade.yml	22:00 UTC weekdays	fill in settled outcomes, print the report
pages.yml	a commit touching frontend/ or the payload	build and publish the frontend

Publishing is deliberately a separate workflow from recording. A scoring run that fails must not take the published page down with it, and a Pages outage must not stop predictions being sealed — those two things have different failure costs and should not share a job.

The Pages job builds the frontend but never rebuilds the data — that is backend/site/app-data.json, which record.yml produces and commits. So the deploy needs no API key, runs in a couple of minutes, and cannot change a number; it can only change how one is displayed.

For the same reason the page build inside record.yml is continue-on-error: true rather than || true. The sealed predictions are written before it and are the irreplaceable part, so the commit has to happen even if the page build dies. But || true also made a broken build look identical to a working one, which is exactly how the page came to serve July news under today's date. continue-on-error keeps the job alive and marks the run red, so the failure is both survivable and visible.

Hosting it somewhere else

backend/site/ is two files and depends on nothing:

cd backend
uv run python -m scripts.build_site --top 10   # or --snapshot for no network
python3 -m http.server -d site 8000            # then open localhost:8000

Upload those two files anywhere — Netlify, Vercel, S3, a university web folder. There is no server, no database and no API call at view time: the page is rendered ahead of time and ships as markup, which is also why it survives having JavaScript switched off.

Nothing secret reaches the page. .env is git-ignored, the API key is only ever read by the build, and site/data.json is the same data the HTML already shows. The one thing to keep in mind is that backend/data/predictions.jsonl is committed on purpose — it is the public record, and its git timestamps are what evidence that a call existed before its outcome did.

The one command

export GROQ_API_KEY=...
uv run python -m scripts.brief             # live news, ingested into the store
uv run python -m scripts.brief --snapshot  # saved corpus, no network
uv run python -m scripts.brief --no-store  # cluster this batch only, persist nothing

Fetches ~650 headlines from 14 sources, cleans them, folds them into the events already on record, and asks an LLM what each one moves and why. The events it scores come from the store, not from the fetch, so a story two outlets filed three hours apart is one corroborated event rather than two single-source ones. --no-store keeps the old batch-only path for comparison. A real run:

1. Collecting news    fetched from 14/15 sources, 656 articles
2. Cleaning           521 kept, 135 dropped
3. Grouping           507 events, 9 confirmed by >1 publisher
4. Analysing top 8    1 call, 4,008 tokens, 7.0s

#1  Houthis impose Saudi naval blockade, opening new front in US-Iran war
    ▲ +3  Crude Oil   Blockade threatens Saudi exports, creating supply risk
    ▲ +2  Gold        Higher geopolitical risk drives investors to safe havens
    ▲ +1  USD         Risk aversion lifts demand for the dollar

    Why:  Houthis impose a naval blockade on Saudi vessels.
        ↓ Could disrupt oil shipments from Saudi Arabia and the wider Gulf.
        ↓ Markets price in a potential supply shortage, raising crude.
        ↓ Increased risk prompts a flight to safety, boosting gold and USD.

    Caveat: If the blockade is short-lived, price moves may be muted.
    Geopolitical · confidence MEDIUM · impact 6 · bloomberg.com, reuters.com

Today's trends
  INR       +5.6 ▲  moderate   2 mentions
  Crude Oil +3.4 ▲  moderate   3 mentions
  USD       -3.0 ▼  moderate   4 mentions

  Market mood: MIXED (+1.9) — no clear rotation between risk and havens

Cost of that run: ~4,000 tokens against a 14,400-request/day free tier. Batching events into one call rather than one call each is what keeps it there.

Running it

This project standardises on uv run. The editable-install .pth is not honoured by the Anaconda-derived interpreter the venv builds on, so .venv/bin/python script.py fails intermittently with ModuleNotFoundError. uv run and pytest's pythonpath setting both work reliably.

cd backend
uv venv --python 3.13
uv pip install -e ".[dev]"

uv run python -m scripts.verify_sources    # probe every feed live
uv run python -m scripts.snapshot          # freeze a corpus to fixtures/
uv run python -m scripts.audit_quality     # measure corpus quality
uv run python -m scripts.measure_normalize # before/after effect of cleaning
uv run python -m scripts.measure_dedup     # clustering precision/recall
uv run python -m scripts.measure_corroboration  # batch vs stored events
uv run python -m scripts.measure_similarity     # what the clusterer should compare
uv run pytest -q                           # 287 tests, no network, ~7s

What the sources actually are

Verified live, July 2026. The honest version of the source list differs substantially from what a naive plan assumes:

• MoneyControl's RSS is abandoned. Every path under /rss/ returns HTTP 200 with well-formed XML whose newest entry is ~817 days old (MCtopnews.xml is ~9.8 years stale). It is reached through the Google News index instead.
• Reuters and Bloomberg have no usable public feed. They are reached via site-scoped Google News search, which is an index of their reporting, not a licensed feed — so both are tiered AGGREGATOR and must be labelled that way in any UI.
• GDELT rate-limits far harder than it documents. Its 429 body asks for one request per 5 seconds; measured behaviour is a multi-minute lockout once tripped. A single paced request still 429'd 150 seconds after a violation, and only recovered at ~5 minutes. GDELT requests are therefore never retried — retrying is what converts a transient 429 into an outage.
• Business Standard was blocking our own User-Agent. It answered 403 to every request while serving a browser UA the identical 200. The default UA carries a contact email because SEC EDGAR requires one and IP-bans for ten minutes without it; Business Standard's bot filter reads that address as a scraper signature. The override is per-feed, not global — dropping the contact UA everywhere would fix one feed and earn a ban on sec.gov, which is the more expensive of the two failures. A test pins that only bs:markets carries it, because a blanket override is exactly the kind of change that goes unnoticed.
• NewsAPI.org is unusable for anything deployed: 24-hour article delay, localhost-only CORS, production use prohibited on the free tier.
• X/Twitter has had no free tier since February 2026.

Measured corpus quality

There is no model yet, so "accuracy" at this stage means: is the corpus handed to the scoring stage actually correct? Measured on a frozen 652-article live snapshot (fixtures/latest.json, committed as the regression corpus).

Check	Before	After cleaning
Aggregator suffix in title	46.0%	0.0%
Quote / profile / template pages	9.7%	0.0%
HTML markup in summary	49.1%	0.0%
Items older than 30 days	3.5%	0.0%
Titles too short to cluster	2.8%	0.0%
Off-topic (sport, entertainment)	2.6%	0.0%
Publisher/URL attribution mismatch	0.0%	0.0%
Future-dated timestamps	0.0%	0.0%
Missing published_at	3.1%	3.7%
Non-resolvable (aggregator) URL	46.0%	46.3%

652 articles in, 474 kept (72.7% retention) — dropped as boilerplate (63), stale filings (63), ticker stubs (35) and off-topic (17). Four of the survivors share a canonical URL with another, so 470 distinct articles reach clustering.

The two largest junk families were invisible until the corpus was inspected rather than sampled: Bloomberg executive profile pages (Kamlesh Soni, Jaiprakash Power Ventures Ltd: Profile and Biography) at 8.4% of all articles, and sport coverage that cleared the feed-level finance clause at 4.9%. Both drop rules were checked by hand — all 17 off-topic drops were genuine match reports, while the six sports-business stories in the corpus (FIFA's $9bn windfall, prediction-market volumes) were correctly kept.

Suffix stripping is precise, not just effective. Google News appends - Reuters to every headline, which would defeat title-based dedup. The rule matches against the article's own publisher rather than stripping after the last dash, because real headlines contain dashes ("Fed holds rates - what it means for the rupee"). Across all 300 stripped titles the removed text was 10, 16, 19 or 34 characters — exactly publisher names, and the 34 is a stacked - Moneycontrol - Moneycontrol.com. Zero headline content was cut. The corpus-level regression test asserts this rather than trusting it.

Google News URLs cannot be resolved server-side. 46% of the corpus arrives through Google's index, and those links have no HTTP redirect, no URL embedded in the HTML, and a token that does not decode. They resolve fine in a browser, so they stay usable as a user-facing citation, but anything needing to fetch the article must treat them as unavailable — is_citable() marks them.

Measured clustering accuracy

Clustering is deterministic — IDF-weighted token overlap, no LLM. Tuned against 28 hand-labelled pairs (fixtures/dedup_labels.json) whose negatives are drawn from the template families where lexical similarity actually fails, plus corpus-level checks on the live snapshot.

Headline-only model: precision 1.00, recall 0.73, F1 0.84, measured on those 28 pairs. That number is about the headline component only, and taking it for the whole thing is what hid a much worse figure — see "The clusterer was measuring one thing and grading another" below, where the shipped configuration turns out to sit at recall 0.35 against cross-publisher pairs scored the way production actually scores them. With the blended metric the live corpus forms 21 multi-article clusters, all correct on inspection, 18 of them spanning more than one publisher.

Three findings shaped the design, each one contradicting the previous plan:

1. One threshold cannot work. Cross-publisher pairs are linearly separable (same [0.208, 1.000], different [0.000, 0.050]). Same-publisher pairs are not separable at all (overlap of 0.517), because one outlet emits many headlines on a rigid skeleton — RBI's (<bank category> – Income Recognition, Asset Classification, Provisioning) Directions, Business Standard's <company> net profit rises <x>% in the June quarter. Those score higher against each other (0.53–0.72) than genuine cross-publisher rewrites of the same story do (0.18–0.78). So: cross-publisher threshold 0.30, same-publisher 0.92, plus a date-conflict veto for RBI's daily releases.

2. Pairwise precision of 1.00 still produced a 208-article cluster. Union-find single linkage merges A and C whenever some B is similar to both, so every borderline pair becomes a bridge and errors compound transitively. Pairwise metrics are blind to this. Replaced with complete linkage — an article joins a cluster only if it clears the threshold against every member — and the largest cluster on real data dropped from 208 to 2.

3. The evaluation itself was wrong before the model was. The first threshold was picked with IDF computed over the 56 documents in the label file. Under production IDF, 13% of random unrelated pairs cleared it (measured over 20,000 sampled pairs); at 0.30 that falls to 0.16%. An evaluation has to run under deployment conditions or it measures nothing.

Along the way this surfaced a data defect affecting everything downstream: 49% of articles carried raw HTML in their summary — Google News anchor tags, RBI <table> layouts. That markup is near-identical within a publisher, acting as a similarity floor that pushed two unrelated RBI circulars to 0.851 versus 0.694 on their actual text. It would also have gone straight into every LLM prompt.

Events that outlive the run

Clustering each fetch in isolation gets the corroboration count wrong, and corroboration is load-bearing: it feeds the confidence model, it weights the trend view, and a scorecard slice tests whether multi-source events are called better than single-source ones.

The reason is a property of the news, not of the code. A fetch sees one instant, but coverage of one story arrives over hours — Reuters files at 09:40, the Economic Times picks it up at 11:15, Mint runs its own version after lunch. Clustered per batch those are three runs, three events, three claims of "one publisher" about a story three publishers ran.

uv run python -m scripts.measure_corroboration
uv run python -m scripts.measure_corroboration --interval 1 --depth 4 --trace

The frozen corpus carries a real timestamp per article, so a schedule can be replayed over it rather than waited for. 387 half-hourly fetches, each seeing the last two hours — so the average article is offered 3.6 times, because that is what feeds do and re-offering is the case idempotence has to survive. Both arms cluster the same 533 articles at the same thresholds, differing only in whether events survive the run:

	batch-only	incremental
events formed	528	511
multi-publisher events	2	18
corroboration rate	0.4%	3.5%
most publishers on one event	2	3

Eighteen rather than two, and it reaches the ceiling exactly: whole-corpus clustering over the same articles also finds 18. Persistence recovers all of what batch-only discards, rather than most of it.

The stability matters as much as the size. Rerun at a one-hour cadence with a four-hour feed depth and the incremental arm still finds 18, while the batch number moves around — the batch figure is measuring how often two outlets happened to file inside the same window, which is a fact about the cron schedule rather than about the news. That is exactly the contamination that would have shown up in the scorecard's corroboration slice as if it were a property of journalism.

Idempotence has two guards, not one. article_id hashes the canonical URL, so re-inserting a re-offered article is a no-op. That alone is not enough. recent_events only looks back 48 hours, so an article a feed is still listing a week later arrives at the clusterer looking new — and because an event id seeds from the first article of its cluster, re-clustering a second or third member mints a different event and repoints the article at it, leaving the original still counting a publisher it no longer has. Articles already bound to an event are therefore excluded regardless of window. Twenty overlapping re-ingests of the same three articles is a test.

IDF has to come from the corpus, not the batch. This was found by measuring rather than by reading: incremental ingestion scored 11 corroborated events where whole-corpus clustering found 13. A scheduled run commonly carries a single article — the median simulated batch here is exactly one — and cluster_incremental was deriving token weights from whatever was in front of it. IDF over one document is not a weighting, and the 0.30 threshold was calibrated under corpus-wide IDF, so the shipped path was applying a measured threshold to a differently-computed score.

This is the same mistake the clustering work already documents once, in the evaluation: the first threshold was tuned with IDF over the 56 documents in the label file, and under real corpus IDF 13% of unrelated pairs cleared it. The second time it was in production rather than in the harness. IDF is now built from every stored article in a 14-day window, which is a tokenise pass, not more comparisons.

The scheduled deployment needs the store to survive the runner. CI runners start empty, so without carrying the database between runs the whole exercise would work locally and quietly buy nothing where it matters — three runs a day, each seeing one instant. record.yml caches it, keyed per run with a restore-keys fallback so a miss takes the most recent cache instead of nothing.

That is deliberately weaker than how predictions are stored, and the difference is a judgement about what each thing is. predictions.jsonl is evidence and lives in git, where a commit timestamp proves independently that a call existed before its outcome. The event store is a working copy: only the last 48 hours affect clustering, and a lost cache costs corroboration on a few runs rather than any part of the record. It is not free — an evicted cache means a cold start, with the thinner IDF that implies — but it is recoverable, and nothing downstream of it is.

One wrinkle the cache forced into the open: under WAL, a copy of hindsight.db taken mid-run has no tables in it at all, because every commit is still in the -wal sidecar. SQLite folds it in when the last connection closes cleanly, so the normal path is safe, but a killed job is not — the cache carries the sidecars too.

The clusterer was measuring one thing and grading another

No event in the corpus had ever been backed by more than two publishers. With ten publishers filing finance news over seven days that looked like the news rather than the code, so it got measured.

Two articles, Reuters and the Economic Times, running the same story under a token-identical headline:

Rupee weakens to two-month low as oil climbs; state-run banks cushion
Rupee weakens to two-month low as oil climbs; state-run banks cushion

  headline similarity     1.000
  title+summary similarity 0.193      threshold 0.30 -> rejected

Nine tokens each, identical. The Reuters copy arrives through Google News with a 77-character stub where its summary should be; the Economic Times copy carries 375 characters of real body text. Jaccard divides by the union, so those unmatchable summary tokens sink the score — and summary length is a property of the delivery channel, not of the story. 54% of this corpus arrives without a usable summary, so the penalty lands hardest on precisely the pairs corroboration is made of: two publishers, two channels, one story.

Underneath it was an evaluation/production mismatch, the same species of error this project already caught once. fixtures/dedup_labels.json stores headlines only, so measure_dedup.py has only ever scored titles — while production compares title + summary. The threshold was tuned on one function and applied to a different one. The sweep reported recall 0.73; measured against cross-publisher pairs that carry their summaries, the shipped configuration was at recall 0.35 — 30 of 46 genuine duplicates missed.

uv run python -m scripts.measure_similarity --show-errors

Fixing it needed labels the old file could not provide, so fixtures/dedup_pairs_xpub.json is 72 hand-labelled cross-publisher pairs referenced by URL, so the full article — summary included — is recoverable from the snapshot. Candidates were drawn by headline similarity ≥ 0.18 within the window, which is the whole region where the decision is actually made, and the negatives are weighted toward the families where lexical similarity fails: fifteen pairs of different IPOs sharing an identical headline template.

Selection is by most recall at precision 1.00, not best F1. F1 treats the two errors as interchangeable and they are not — a false merge is a visible wrong claim that also corrupts the corroboration count, while a missed merge just leaves two events where one would be tidier. Maximising F1 here picks thresholds at precision 0.67.

Each metric is shown at its own best threshold — the lowest one that still holds precision 1.00 on the labelled pairs:

compares	threshold	recall	random pairs admitted
title + summary	0.27	0.39	0.02%
headline only	0.44	0.39	0.02%
max(title, full)	0.44	0.39	0.06%
mean(title, full)	0.29	0.52	0.03%

(the previously shipped configuration — title+summary at 0.30 — scored recall 0.35 on these pairs)

What actually ships is mean(title, full) at 0.30, not 0.29. At 0.28 precision breaks (one false merge), so 0.29 is a single step above a cliff, and 0.30 costs 0.02 of recall to stand clear of it — recall 0.50 rather than 0.52, still no false merges. That is the same reasoning that moved the same-publisher threshold to 0.92: a threshold wants open space around it, not a local optimum found by a sweep on 72 pairs.

Headline-only is not the answer, which is why the metric is a blend. Indian outlets run heavy headline templates — <Company> IPO: price band set at X; issue opens July 23 — and the body is the only text distinguishing two different IPOs filed the same morning. Averaging keeps the headline decisive when the summary is missing and lets the body separate template families when it is present.

One hypothesis was tested and rejected. Same-story pairs ought to share a company name where template pairs share only template words, so a rarest-shared-token gate should separate them. It does not: peak shared IDF runs 4.76–6.06 for same pairs and 4.59–6.06 for different ones. A rare token like "Farnborough" is shared just as readily by two different stories about the same airshow.

Corpus-level inspection, because pairwise metrics are not allowed the last word here — a model with pairwise precision 1.00 once produced a 208-article cluster. All 21 multi-article clusters on the live snapshot are correct on inspection, the largest holds 3 articles, and Lohia Corp and Indo-MIM stayed in separate events. The corpus gains its first three-publisher story: Samsung Biologics' $1.8bn PolyPeptide bid, run by Bloomberg, Reuters and CNBC.

The margins are asymmetric and worth stating plainly. Two different IPOs sit 0.11 below the threshold; the same IPO from two outlets clears it by 0.02. Recall is what is tight here, not precision.

Spending the token budget on questions already answered

With events persisting, record scores the best-corroborated events in the 48-hour window. The same events are still there on the next run, and the run after that.

Re-scoring one cannot produce anything. Prediction identity is (event, target, mode) and the insert is ON CONFLICT DO NOTHING, so the first call is the call — by design, and that design is what makes the accuracy figure trustworthy. A repeat spends tokens to write nothing.

The cost is not the interesting part. The shortlist is ordered by corroboration, so the best-covered events reoccupy it every single run and crowd out events that have never been scored at all. Replaying the frozen corpus at the real three-runs-a-day cadence:

	model calls	distinct events scored	wasted
re-score everything	432	162	62%
skip already scored	168	168	0%

More stories covered, for 61% fewer calls. That matters more than the money: sample size is the binding constraint on every accuracy claim this project makes, and two thirds of the budget was being spent re-asking questions whose answers were already sealed.

events.scored_at records it, written after the model returns so a run that dies mid-request leaves the event for the next one. --rescore overrides.

The regression corpus had rotted

Three corpus-level tests were failing, and the reason was the calendar.

fixtures/latest.json is the committed evidence behind every measured claim about normalisation and clustering. Normalisation drops anything older than MAX_AGE, measured against the wall clock. The snapshot was captured in July; once it aged past that, every article in it was dropped as stale, retention measured 0%, and the corpus quality audit reported the entire corpus as ancient.

The failure was not that the tests broke. It is that they broke looking like a code regression — retention outside its band, no clusters formed — while nothing about the pipeline had changed. A regression corpus that decays with time is worse than no regression corpus, because its failures are indistinguishable from real ones and the natural response is to widen the threshold band until it goes quiet.

The age cutoff is relative to the run, not to today: the question is whether an article was stale when it was fetched. normalize() and audit() now take the clock explicitly, and hindsight/snapshot.py keeps captured_at attached to the articles it loads, because six call sites had each been parsing the snapshot by hand and dropping it. Replays pass the capture time; live runs take the default and behave exactly as before.

Design notes

Provenance is never collapsed. RawArticle is immutable and keeps its source, URL and fetch time. An Event is a cluster of articles, so "three sources reported this" stays literally true and corroboration counts distinct publishers rather than articles — one outlet filing five updates is one piece of evidence, not five.

Staleness is per-source. A newswire quiet for two days is broken; the Federal Reserve quiet for two days is an ordinary week. A single global threshold trains everyone to ignore the signal.

Sources fail independently. Source.fetch never raises. One dead feed returning [] must not abort a run the other thirteen would have contributed to; the failure shows up in health tracking instead.

Schema migrations run before the schema script, not after. schema.sql indexes columns that were added later, and CREATE INDEX against a column an older table does not have raises — so adding scored_at made every existing database fail at Store(path), not at the query. Every entry point breaks at once, and only for people who already had data. Migrating first is a no-op on a fresh database and makes the script satisfiable on an old one.

A run that fetched nothing still leaves a row. ingest_runs records the empty run, because "every feed was down" and "the cron never fired" are different problems and a skipped row makes them look identical.

Every clock is an argument. Normalisation, the quality audit and the event lookback all take the run time explicitly rather than reading the wall clock. Replaying a frozen corpus and running live then follow the same code path, and the corpus does not decay into failure with the calendar.

Layout

backend/
  hindsight/
    models.py          canonical schema: RawArticle, Event, corroboration
    config.py          provider limits, documented and centralised
    snapshot.py        the frozen corpus, loaded with its capture clock
    sources/
      base.py          pacing, retries, health tracking
      rss.py           RSS adapter + verified feed registry
      gdelt.py         GDELT DOC 2.0 (no-retry by design)
    pipeline/
      normalize.py     cleaning and filtering, measured
      dedup.py         clustering: batch and incremental, blended similarity
      ingest.py        one run, persisted -- fetch to stored events
    db/
      schema.sql       articles, events, runs, predictions, source health
      store.py         idempotent writes, windowed reads
    scoring/           impact scoring, self-consistency, trends
    eval/              market outcomes, prediction history, the report
    web/
      render.py        the pre-rendered page (and the no-JavaScript fallback)
      dataset.py       the payload the browser app runs on, capped and interned
      client.py        the embedded app: views, router, charts, watchlist
    api/
      service.py       read layer: payload from disk, archive from the store
      server.py        FastAPI routes, read-only by design
  scripts/
    verify_sources.py       live probe of every registered feed
    measure_corroboration.py  replay the corpus as a schedule; batch vs stored
    measure_similarity.py     candidate similarity metrics, measured
  fixtures/
    latest.json               the frozen 652-article regression corpus
    dedup_labels.json         28 headline-only pairs (same-publisher templates)
    dedup_pairs_xpub.json     72 cross-publisher pairs, summaries recoverable
  tests/               287 tests, network-free
frontend/              Next.js 15 + TypeScript + Tailwind + Recharts
  app/                 one route per view, plus /targets/[name]
  components/charts/   six charts, each with a minimum before it will draw
  lib/api.ts           live API when configured, snapshot otherwise

The platform

Three pieces, each of which can fail without taking the others down:

backend/hindsight/     the pipeline — fetch, clean, cluster, score, seal, grade
backend/hindsight/api/ FastAPI, read-only, over what the pipeline produced
frontend/              Next.js 15 + TypeScript + Tailwind + Recharts

Running it

# once, to produce the data everything else reads
cd backend && uv run python -m scripts.build_site

# terminal 1 — the API
uv run python -m scripts.serve                    # :8000, docs at /docs

# terminal 2 — the frontend
cd frontend && npm install
NEXT_PUBLIC_API_URL=http://127.0.0.1:8000 npm run dev   # :3000

Without NEXT_PUBLIC_API_URL the frontend reads a build-time snapshot instead (npm run snapshot copies it in), which is what a static export ships. The same functions serve both modes, so no component knows which it is in.

The API

Read-only on purpose. Nothing here fetches news, scores an event or writes a prediction — those belong to the scheduled jobs, where a failure is retried rather than served to a browser as a 500. A test asserts the app exposes no POST, PUT, PATCH or DELETE route at all.

endpoint	returns
GET /api/summary	mood, pipeline stats, today's trends, the headline record
GET /api/stories	scored stories; ?q= searches reasoning too, not just headlines
GET /api/news	the whole archive, paginated and filterable
GET /api/targets · /api/targets/{name}	every target; one target with its history
GET /api/record	accuracy with its baseline and interval, plus every call
GET /api/sources · /api/timeseries	feed health; stories per day

Two sources behind it, for a reason. Scored stories come from a file: scoring costs model tokens, so the analysis cannot be recomputed per request — build_site writes site/app-data.json and the service reloads it when its mtime changes, so a scheduled rebuild is picked up without a restart. The archive comes from the database, because an API can paginate where the embedded page has to cap.

A missing payload is a 503, not a 500, and the body names the command that fixes it. Nothing is broken — the pipeline has not run — and a 500 sends somebody hunting for a bug that does not exist.

CORS origins come from the environment (HINDSIGHT_CORS_ORIGINS), never a wildcard default. This one is worth stating because getting it wrong is a silent failure: the API answers 200, the browser discards the response, and every page renders empty with a completely clean server log. It cost a debugging cycle here before it was a test.

The frontend

Six routes, real URLs, working back button. Interactive charts via Recharts — tooltips, a clickable movers chart that navigates to the target, and axes that were fought with (see below). / focuses search, T toggles the theme.

Twenty target pages are prerendered at build time from the snapshot, so /targets/Gold is a real page in a static export rather than a client-side route that 404s on refresh.

Charts refuse to draw rather than mislead. Every one has a minimum:

• VolumeChart needs three days before it draws a trend line. On two it says so and prints the counts — a line between two points is not a trend. It also uses linear interpolation, not a spline: a smooth curve through three daily counts implies values on days nobody measured.
• SliceChart renders any group under ten graded calls as a count with an explanation, never as a bar. 4 out of 5 is 80% and means nothing, and drawing it beside a real rate invites exactly the comparison the sample cannot carry.
• IntervalPlot is hand-drawn SVG rather than Recharts, because it is one interval on one axis and a library would add a dependency to draw four shapes.

Three axis bugs were real and worth naming: Recharts picked fractional ticks over a small range and rounded them for display, printing "300, 300, 100, 100"; it chose asymmetric ticks on a diverging axis (−5, −2, +1, +5) that made zero look off-centre; and long publisher domains were clipped mid-word to ctimes.indiatimes.com, which names no source anybody can check. All three are fixed with explicit ticks and an ellipsis formatter.

total and corpus_total are different numbers. total counts matches after filtering; corpus_total describes the archive. Using the first to describe the second makes the page announce "12 stories the pipeline has gathered" the moment somebody searches. There is a test.

A supply-chain fix, found before it shipped

npm install flagged the pinned Next.js (15.1.6) as carrying CVE-2025-66478, and the tree pulled in four postcss advisories including a high-severity path traversal. Upgrading to the patched 15.5.24 backport cleared the first; an overrides entry forcing postcss@^8.5.23 cleared the rest, and one more cleared a ReDoS in ESLint's config parser. The tree is now at zero advisories, and npm audit --audit-level=moderate runs in CI so the next one fails a build instead of shipping.

What still ships without any of this

backend/site/index.html is unchanged: one self-contained file, no server, no build step, every number in the markup. It is what the pipeline publishes on its own, and it stays the thing that works when everything else is down.

The application

cd backend
uv run python -m scripts.build_site --top 10 --open

Writes site/index.html — one file, no framework, no build step, no external asset of any kind, and no server at view time. It is a real application, not a report: six views, client-side routing, search, a watchlist and nine charts, all running on a dataset embedded in the page.

view	what it is for
Today	Mood, your watchlist, today's movers, the verdict, the top stories
Stories	Every story the model scored, with reasoning and caveats; filter by kind and direction
All news	Everything collected — searchable, filterable by outlet, sortable by corroboration
Targets	All 24 targets; each has its own page with history and a hit rate when one is earned
Record	Accuracy with its baseline and interval, four breakdowns, and what has been called
Sources	Which feeds answered and which went dark

Keyboard: / focuses search, T toggles the theme, 1–6 switch views. Every view is a real URL (#news, #target/Gold), so it is linkable and the back button works.

What the data would and would not support

The feature list came from reading the data, not from a wishlist.

Accuracy analytics are deliberately thin, because the data is. 21 graded outcomes across four days. Per-target hit rates, calibration curves and accuracy-over-time would all have rendered noise as insight — a chart is a claim, and drawing one on four results is a false one. So dataset.py refuses to compute a rate below 10 graded calls for a target, and the client renders the count with an explanation instead of a bar. When the sample grows the rates appear on their own.

The Record view is still full, because what has been called is knowable long before whether it was right: most-called targets, the distribution of conviction, the confidence it claims. Those are labelled as counts, not results.

The archive was the biggest thing missing. The store held 1,290 events and the page showed ten. Scoring costs tokens, so only the best-covered stories ever reach the model — but everything collected is now browsable and searchable, which is most of what a reader actually wants to do with it.

Page size is a design constraint

The store never prunes; events accumulate at roughly a hundred a run. Embedding all of them would grow the download every day until it was a multi-megabyte page that nobody noticed getting slower.

So the archive is capped at the most recent 900 events, publisher names are interned into a lookup table rather than repeated 900 times, and default values are dropped rather than serialised. The page is ~540 KB, ~215 KB gzipped, and bounded — it stays that size in a year. The UI says "the most recent 900 of 1,290" rather than implying it has everything, and tests pin the cap, the newest-first ordering and the honesty of that count.

It still works with JavaScript off

The pre-rendered page is inside <div id="ssr"> and the app removes it on boot. If the script never runs, that block is the page — the full plain-English report, with today's stories, the record, the sources and the disclaimer. So the fallback has to be a complete page rather than a spinner, and a test asserts it contains the sealed-call count, the disclaimer and "How this works".

Markup from a news feed is inert by construction

Every string on this page came from a third-party feed, and now it also travels inside a <script> block. Two defences, both tested:

• The client builds nodes with an el() helper that sets textContent. There are exactly two innerHTML assignments in 1,300 lines — the helper's opt-in branch and the theme icon — and a test asserts the opt-in branch is only ever handed string literals, never anything built from payload data.
• The embedded JSON escapes </, so a headline containing </script> cannot close the element early and drop the rest of the payload into the document as markup. A test feeds exactly that headline through and parses the result back.

The charts refuse to draw rather than mislead

chartVolume needs three days before it will draw a trend line; on two it shows the daily counts and says why, because a line between two points is not a trend. chartSlices renders any group below ten graded calls as a count, never as a bar. This is the same rule the accuracy panel has always followed, applied everywhere a percentage could appear.

There is no charting library. Every chart here is a bar, a line or an interval, and the page carries no external asset.

Where the accuracy number stands

Measured, not asserted:

RETROSPECTIVE HARNESS CHECK   (quarantined — not a skill claim)
  n              21
  accuracy       52.4%    95% CI 32% – 72%
  baseline       52.4%
  edge           +0.0%
  verdict        TOO EARLY — not enough calls to say anything

That is the expected result, and the system reporting it rather than dressing it up is the point. Published research puts news-based next-day direction at roughly 52–55%, and at n=21 the interval spans 32–72% — consistent with a coin. 127 calls are recorded; 106 are still sealed and waiting for a session to close.

The by-slice breakdowns are already doing their job: strong (|3+|) calls score 50% against weak (|1-2|) at 53%, which is early evidence that the magnitude the model assigns carries no information and only the sign means anything. If that holds at a real sample size, the scale should be dropped from the UI rather than displayed as though it were meaningful.

Accuracy audit — six defects found by reading every line

A line-by-line pass over the scoring and grading paths, ranked by how much each one was corrupting the number.

1. The model was never told what it was being graded on. The prompt asked "what does this news move" with no timeframe; the scorecard graded the next trading session. The model was free to reason over quarters while being marked against tomorrow. Nothing in the code was wrong — the question and the measurement were simply about different things. The prompt now states the horizon explicitly, including that news already priced in before the next open should score lower.

2. Two targets resolved to one instrument. India and Equities were both graded against the Nifty, so one event could produce two calls that are the same bet. They rise and fall together, yet the statistics treated them as independent evidence — which does not merely inflate n, it narrows the confidence interval, the one number the verdict turns on. Targets sharing an instrument now collapse to one call, and the model is instructed never to say "Equities" without naming the market, because "equities went up" is not a checkable claim.

3. Re-scoring an event minted a second prediction. prediction_id hashed the timestamp, so the same event scored on a later run became a new "independent" call about the same thing. Identity is now (event, target, mode): the first call is the call. A report-time filter also collapses historical duplicates — applied at read time rather than by editing the log, because the log is append-only evidence and must not be rewritten to make statistics tidier.

4. Scores formatted as floats were silently discarded. int("3.5") raises, the exception was swallowed, and the impact vanished — indistinguishable from the model never mentioning it. Parsing is now tolerant of 3, 3.0, "+3".

5. Self-consistency sampling. Each event is now scored three times and only what the samples agree on is recorded. A target raised in one run out of three, or flipped in sign, is noise the model happened to emit rather than a finding. The by-product matters more than the noise reduction: agreement across samples is a measured confidence signal, and it can be tested head-to-head against the model's self-reported high/medium/low. The report now splits accuracy both ways. Whichever separates accuracy better is the one worth showing users — and if neither does, both should go.

6. A schema addition could not read its own history. Adding a NOT NULL column broke importing every log line written before it existed. Missing fields now fall back to declared defaults.

What this does not fix

None of it moves the ceiling. News-based next-day direction sits at roughly 52–55% in published work, and no prompt or parser changes that. What these fixes do is make the number trustworthy: aligned with what was actually asked, counted from independent trials, and not quietly inflated by duplicates.

Whether the changes helped is not yet knowable — 21 graded calls cannot detect a few points of movement. The measurement is set up so that it will be knowable, which is the part worth having.