Real-time failure detection for production AI agents - Slack alert within 15 seconds.
If Dunetrace helps you, consider giving it a ⭐ on top right, it helps others find the project.
AI agents fail silently:
- ✓ API returns 200 ✓ Logs are clean
- ✗ Agent called the same tool 12 times, burned $10, and gave the user a wrong answer
Langfuse and LangSmith answer "what happened?" — after you already know it broke. Dunetrace answers "is something breaking right now?" and fires an alert in 15 seconds.
| Dunetrace | Langfuse / LangSmith | |
|---|---|---|
| When it fires | Within 15s of run completion | You query it after you notice a problem |
| What it watches | Structural failure patterns | Raw trace data |
| Alert channel | Slack / webhook / Dashboard | Dashboard only |
| Fix path | Auto-apply a policy, or one-click push to a connected prompt store | Manual |
1. Start the backend
git clone https://github.com/dunetrace/dunetrace
cd dunetrace && cp .env.example .env
docker compose -f docker-compose.ghcr.yml up -d
pip install -r requirements.txt2. Install the SDK
pip install dunetrace # Python
npm install dunetrace # Node.js / TypeScript3. Instrument your agent
Python
from dunetrace import Dunetrace
dt = Dunetrace()
@dt.tool
def web_search(query: str) -> list: ...
@dt.trace
def my_agent(question: str) -> str:
return web_search(question)[0]TypeScript / Node.js
import { Dunetrace } from "dunetrace";
import OpenAI from "openai";
const dt = new Dunetrace();
const openai = dt.wrapOpenAI(new OpenAI());
await dt.run("my-agent", { model: "gpt-4o" }, async (run) => {
await openai.chat.completions.create({ model: "gpt-4o", messages });
run.finalAnswer();
});Try the built-in failure scenarios
cd packages/sdk-py
python examples/basic_agent.py # No LLM calls
SCENARIO=tool_loop python examples/langchain_agent.py # TOOL_LOOP via LangChain
SCENARIO=failures python examples/decorator_agent.py # TOOL_LOOP, RETRY_STORM, RAG_EMPTY_RETRIEVAL
SCENARIO=tool_loop python examples/langfuse_agent.py # TOOL_LOOP + Langfuse trace correlationOpen the dashboard: http://localhost:3000
17 detectors run on every completed run — no configuration, no LLM.
| Signal | What it catches |
|---|---|
TOOL_LOOP |
Same tool called repeatedly with identical args |
TOOL_THRASHING |
Oscillating between two tools, unable to commit |
RETRY_STORM |
Tool failing, agent retrying it repeatedly |
CASCADING_TOOL_FAILURE |
Multiple different tools failing in sequence |
CONTEXT_BLOAT |
Prompt tokens growing unsustainably across LLM calls |
LLM_TRUNCATION_LOOP |
Model output truncated repeatedly |
GOAL_ABANDONMENT |
Agent stopped using tools before finishing |
REASONING_STALL |
Too many LLM calls per tool call — agent deliberating in circles |
TOOL_AVOIDANCE |
Agent answered without using any tools |
RAG_EMPTY_RETRIEVAL |
Retrieval returned nothing, agent answered anyway |
EMPTY_LLM_RESPONSE |
Model returned an empty response |
FIRST_STEP_FAILURE |
Failed on the first step — config or setup issue |
SLOW_STEP |
Single step latency well above threshold |
STEP_COUNT_INFLATION |
Far more steps than the agent's baseline |
SESSION_LATENCY |
Wall-clock run time anomalously long vs per-agent baseline |
COST_SPIKE |
Total token consumption unusually high vs per-agent baseline |
PROMPT_INJECTION_SIGNAL |
Input matched adversarial injection patterns |
Each alert includes: what fired, why it matters, a concrete fix, and a rate context line (first occurrence / recurring / systemic).
Custom detectors — write a detector in plain English. Dunetrace translates it to a structured condition set, runs it in shadow mode against real traffic, and lets you review the fire rate before any alert fires. In the dashboard: Config → Custom detectors → Add detector.
Live at http://localhost:3000. Auto-refreshes every 15s.
Slack and generic webhook (PagerDuty, Linear, custom).
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
SLACK_MIN_SEVERITY=LOW # LOW | MEDIUM | HIGH | CRITICALA weekly digest (Monday 9am UTC) summarises top failure types and systemic patterns. Enable with DIGEST_ENABLED=true.
Root-cause analysis is native — no third-party tracer required. Click Explain + on any alert and Dunetrace analyzes the run's own stored events and returns a specific cause and fix. Every fix is one of two kinds:
- Policy fixes (tool loops, retry storms, runaway step counts) → Dunetrace applies a runtime guardrail directly, no code change needed
- Prompt / code fixes → a diff you copy in, or push in one click to a connected prompt store (Langfuse today)
Fix effectiveness is tracked automatically.
connect Langfuse: docs/integrate-langfuse.md
Runtime guardrails that fire mid-run — before a failure propagates.
dt.add_policy(
name="cap tool calls",
condition={"trigger": "tool_call_count", "operator": "gt", "value": 5},
action={"type": "stop"},
)
dt.add_policy(
name="cost cap",
condition={"trigger": "cost_usd", "operator": "gt", "value": 0.50},
action={"type": "switch_model", "params": {"model": "gpt-4o-mini"}},
)Policies can also be created in the dashboard and fetched automatically by the SDK (60s TTL).
Query agent signals directly from Claude Code, Cursor, or Codex — without leaving your editor.
pip install dunetrace-mcp10 tools — ask your editor things like "what failed in the last 24 hours?"
| Tool | What you can ask |
|---|---|
list_agents |
"Which agents are monitored and how healthy are they?" |
get_agent_signals |
"What failures did my agent have today?" |
get_agent_health |
"Show me the health score breakdown for my agent." |
get_signal_detail |
"Show me signal #42 with full evidence and fix code." |
get_agent_patterns |
"Is this failure systemic or a one-off?" |
get_run_detail |
"Walk me through run abc123 step by step." |
get_agent_runs |
"List recent runs for my agent with their status." |
search_signals |
"Show me all CRITICAL signals in the last 24 hours." |
summarize_agent |
"Give me a one-shot diagnosis of my agent." |
get_agent_token_stats |
"How much is my agent wasting on failed runs?" |
Claude Code: registered automatically in ~/.claude.json after pip install dunetrace-mcp. Restart Claude Code to load.
Cursor: add .cursor/mcp.json to your project root:
{
"mcpServers": {
"dunetrace": {
"command": "dunetrace-mcp",
"env": {
"DUNETRACE_API_URL": "http://localhost:8002",
"DUNETRACE_API_KEY": "dt_dev_test"
}
}
}
}Prompts, tool arguments, and model outputs are sent to the backend over TLS and stored there — content-aware detectors (prompt injection in retrieved documents, tool argument fabrication, silent failures) need to see what the agent actually said and did. Trust is handled the way any SaaS handles it: encryption in transit and at rest, SOC2, no training on customer data. If you need an air-gapped deployment, self-host — same code, same detectors, your infrastructure.
Agent Code
└─► Dunetrace SDK (raw content → ingest events)
└─► Ingest API (POST /v1/ingest → Postgres)
├─► Detector (poll → 17 detectors → signals)
├─► Alerts (poll → explain → Slack / webhook)
└─► Customer API (runs, signals, explanations → dashboard)
LangChain, CrewAI, AutoGen, Haystack, LlamaIndex, TypeScript, and more
Fork, branch, change, make test, PR. For larger changes (new detectors, architecture changes), open an issue first.
Requires Python 3.11+, Node.js 18+, Docker + Docker Compose.