An on call root cause agent that separates cause from symptom, and knows when to say it does not know.
The on call engineer at a company running a dozen or so microservices. Alone, at 3am, with an incident that just fired two hundred alerts at once and a customer facing outage burning revenue every minute it stays unresolved.
The loudest alert is almost never the broken service. Retries amplify load on the way up the call chain, so a service three hops downstream of the real cause ends up screaming the loudest, while the actual origin might only be slow, never crossing an error rate threshold at all. The telemetry needed to tell the difference, tens of thousands of log lines, metrics, and traces across a twenty minute window, is genuinely beyond what a person can read before the next decision has to be made. The result, teams fix the loudest symptom, the incident recurs, and mean time to resolution is dominated by finding out what to fix, not fixing it.
Solving this matters because the cost is not just engineering time, a wrong guess at 3am means a second outage, and a system that always produces a confident answer even when the evidence does not support one is actively dangerous in this setting. The hardest and most valuable thing an on call agent can do is refuse to guess when refusing is the honest answer.
Nothing here is a modified starter repository, everything was built during the hackathon. Third party components used as is, according to their license and service terms, are DuckDB, the OpenAI Python SDK, FastAPI, and Next.js. No proprietary or real company data is used anywhere, the entire corpus is a synthetic, seeded simulation described below, no credentials or private information are committed.
A deterministic simulator (simulator/) generates 10 incidents over a
synthetic 12 service system. Each incident has a planted fault with
known ground truth, held out from every tool an agent can call, and
used only by the scorer. Fault effects propagate through the call graph
using each service's real timeout and retry policy, so the tell tale
signature of a retry storm, error rate climbing as it moves upstream
while the origin itself stays quiet, falls out of the mechanics rather
than being hand authored.
Three systems are evaluated against the same 10 incidents.
- B1, a deploy correlation heuristic, no LLM, blames whichever alerting service has the most recent deploy, config change, or commit before the first alert. A real pattern teams use today.
- B0, a single LLM prompt, no tools, given the alert list and a tail of error logs.
- Cascade, the advanced solution (
agent/), triage narrows the blast radius from alerts alone, hypothesis investigates with tools and proposes falsifiable claims, falsifier actually runs each falsification query against the real telemetry, then two deterministic checks that involve no model call at all, a precedence checker that rejects any claimed cause that happens after the symptoms it supposedly caused, and an evidence validator that resolves every citation back against the real fixtures. Nothing is remediated automatically, every proposed fix sits behind an explicit human approval gate.
See docs/repro_guide.md for the full walkthrough. Short version.
make setup
make fixtures
make eval-replay
eval-replay reproduces the headline numbers from a committed response
cache, no API key required, no network calls, no cost.
Simulator, tool layer, and eval harness are complete. Both baselines are
fully measured over all 10 incidents, B1 (no LLM) at top1 accuracy 0.60,
B0 (naive single prompt) at top1 accuracy 0.30, see
results/baseline_b1.json and results/b0.json.
Cascade is architecturally complete and hand verified correct on 3 of
10 incidents under gpt-5.6-terra, one from each of the categories that
matter most, a standard case, the deep cascade case (finds a service
that never alerts), and the abstention case (correctly declines to name
a single cause), see results/cascade.json. The full 10 incident number
is not included, on purpose, rather than filled in with something
unverified. OpenAI billing ran out mid sweep, a follow up attempt on
Groq's free tier surfaced a real capability gap on the deep cascade case
for the smaller free model before hitting its own daily quota, and a
serious bug in the code changes made to get Groq working was caught
before it reached a scored run, full account in CHANGELOG.md along with
every other fix along the way.
The FastAPI backend and Next.js frontend are both built and running,
Incident Wall, a live Cascade View with an animated service topology,
streaming triage, hypothesis board, and human approval gate, and a
Scoreboard reading straight from results/*.json. Verified working
against real backend data with headless browser screenshots, not just
unit tested.
See CHANGELOG.md.