| title | IncidentOps Environment Server | |||
|---|---|---|---|---|
| emoji | 🎮 | |||
| colorFrom | yellow | |||
| colorTo | indigo | |||
| sdk | docker | |||
| app_port | 7860 | |||
| tags |
|
IncidentOps is an OpenEnv-native environment designed to evaluate whether an agent can operate like a real on-call Site Reliability Engineer (SRE).
It simulates production incidents across microservices and tests the agent’s ability to observe, diagnose, fix, and resolve issues under realistic constraints.
Unlike toy environments, IncidentOps evaluates:
- Diagnosis before action
- Signal extraction from noisy logs
- Correct root cause identification
- Minimal, safe remediation
- End-to-end incident resolution
The agent must follow a real-world workflow:
Observe → Diagnose → Fix → Verify → Resolve
The system simulates a production SaaS stack with:
- Services:
auth,api,payments,search,notifications,db - Persistent state (SQLite-backed)
- Multi-step incident evolution
- Deterministic scenarios with hidden failure causes
Agents interact using ToolCallAction:
get_alerts()search_logs(service, query)get_metrics(service)inspect_config(service)get_recent_deploys(service)rollback_deploy(service)patch_config(service, key, value)restart_service(service)run_healthcheck(service)resolve_incident(incident_id, root_cause, resolution)
[START] task=single-service-down env=incidentops_env model=mistral
[STEP] step=1 action=get_alerts({}) reward=0.03 done=false error=null
[STEP] step=2 action=search_logs({"service":"auth","query":"401"}) reward=0.11 done=false error=null
[STEP] step=3 action=inspect_config({"service":"auth"}) reward=0.09 done=false error=null
[STEP] step=4 action=patch_config({"service":"auth","key":"jwt_secret_version","value":"v2"}) reward=0.20 done=false error=null
[STEP] step=5 action=resolve_incident({...}) reward=0.90 done=true error=null
[END] success=true steps=5 rewards=0.03,0.11,0.09,0.20,0.90
IncidentOps includes diverse real-world failure modes:
| Task | Type |
|---|---|
single-service-down |
Config failure |
bad-deploy-recovery |
Regression after deploy |
cascading-failure |
System-level bottleneck |
config-drift-with-false-logs |
Silent misconfiguration |
false-alarm |
Self-recovered incident |
resource-leak |
Memory leak via feature flag |
- Correct resolution
- Correct root cause
- Efficient steps
- Proper diagnosis before action
| Behavior | Reward |
|---|---|
| Diagnostic step (logs/metrics) | +0.05 |
| Correct targeted fix | +0.28 |
| Blind action (restart/patch early) | -0.30 |
| Successful resolution (terminal) | up to +0.95 |
uvicorn server.app:app --host 0.0.0.0 --port 7860export HF_TOKEN="hf_xxx"
export MODEL_NAME="mistralai/Mistral-7B-Instruct-v0.2"
export ENV_SERVER_URL="http://localhost:7860"
python3 inference.py| Endpoint | Description |
|---|---|
/reset |
Start new episode |
/step |
Execute action |
/state |
Current state |
/health |
Server health |
server/
app.py → OpenEnv server
tools.py → tool implementations
rubrics.py → reward logic
scenarios/ → task definitions
inference.py → agent runner
- Deterministic evaluation (no randomness in environment state)
- Non-trivial scoring (multi-level reward gradients)
- Realistic failure patterns
- Strict action-reward causality
- Not a toy benchmark — mirrors real incident response
- Rewards reasoning, not brute force
- Penalizes incorrect mental models
- Easy to evaluate, hard to solve optimally
IncidentOps measures whether an agent can:
think like an engineer, not just act like a tool
- OpenEnv Specification (RFC 004)
- SRE Incident Response Practices
- Meta Hackathon Evaluation Guidelines