AgentOps Lab

Durable execution, policy gating, and objective evaluation for tool-using AI agents.

The reference demo is a narrow e-commerce refund agent backed by mock commerce services. The system owns the runtime loop, policy gateway, execution receipts, replay, and deployment controls needed to test agent changes without handing correctness to an orchestration framework.

Current Status

• Phase 0: domain models, mock commerce world, tool registry, and config loader.
• Phase 1: durable runtime with deterministic idempotency keys, receipts, retries, dead lettering, and crash-resume tests.
• Phase 2: deterministic policy gateway with schema, authorization, risk, budget, and durable approval pause checks.
• Phase 3: thin model loop that proposes structured tool calls while the runtime and gateway own execution.
• Phase 4: 50-scenario regression dataset, binary end-state oracle, weighted trajectory rubric, and evaluation CLI.
• Phase 5: structured logs, OpenTelemetry traces, Prometheus runtime metrics, FastAPI control plane, and React dashboard.
• Phase 6: SQLite/Postgres storage, SQLite/Redis queue paths, separate workers, dependency fault injection, and in-process/subprocess crash proofs with duplicate-effect metrics.
• Phase 7 (in progress): integrity-checked replay bundles verified in clean processes, fail-closed side-effect isolation, first-divergence analysis, paired version comparison, API, and dashboard; replay CLI/lineage and advanced workflows remain.
• Phase 8: durable deployment registry, evidence gates, side-effect-free shadow runs, deterministic canary rollback, freeze/kill switches, two-person overrides, multi-window canary health, and automatic promotion of failed canary/shadow runs into the regression corpus.

For the detailed tracker, see implementation.md.

Setup

py -3 -m pip install -e ".[dev]"
py -3 -m pytest tests/ -q

Current backend verification on 2026-07-19: 176 passed, 2 skipped optional external-backend tests. The last dashboard verification reported 5 passed and a successful production build.

Run the deterministic crash proof locally:

agentops chaos crash-proof --iterations 100
agentops chaos fault-matrix

Run a separate process that claims and resumes durable runtime jobs:

agentops worker --db agentops.sqlite3 --max-jobs 10

Omit --max-jobs for a continuously polling worker. Failed handlers retry up to --max-attempts and then move to the durable dead-letter table.

The crash-proof command isolates each run, kills execution after the refund effect but before the runtime receipt is written, resumes from the durable checkpoint, and reports the measured fault-recovery rate and duplicate refund count. The fault matrix separately exercises latency, HTTP 500, malformed JSON, missing-field, stale-data, and unavailable-dependency failures. Transient cases must recover; persistent variants are tested to dead-letter without duplicating side effects. The SQLite job store also locks the claim transaction before selecting work, so concurrent store instances cannot claim the same queued job; expired in-flight claims become eligible again after their visibility timeout.

Control API

Start the FastAPI control plane from the repository root:

py -3 -m uvicorn agentops.api:create_app --factory --reload

By default, it stores control-plane records in agentops.db and loads scenarios from scenarios/refund-regression/. Set AGENTOPS_DB_PATH to choose a different SQLite file. Interactive API documentation is available at http://127.0.0.1:8000/docs.

In a second terminal, start the React dashboard:

cd dashboard
npm install
npm run dev

Open http://127.0.0.1:5173. Vite proxies dashboard /api/* requests to the FastAPI server at http://127.0.0.1:8000. The dashboard supports run/trace inspection and durable approval decisions with required operator reasons. Persisted evaluation reports render as oracle-first scorecards with per-scenario failure details. Registered agent versions and configured scenarios can be selected to launch new durable runs. Replay comparisons and deployment environment pointers/promotion history are also visible.

The main workflow is:

1. POST /versions to register an agent configuration and GET /versions to list versions.
2. GET /scenarios to list the configured scenario catalog.
3. POST /runs to run a named scenario.
4. GET /runs/{run_id}/trace to inspect its persisted trajectory.
5. GET /approvals and POST /approvals/{approval_id} to operate durable pauses.
6. POST /evaluations to produce a persisted oracle-first scorecard.
7. POST /replay/compare to compare paired baseline and candidate runs without executing tools.
8. POST /deployments/versions to register a deployable version.
9. POST /deployments/promotions to promote only when supplied evidence passes every gate.
10. GET /deployments/{name}/status and /history to inspect environment pointers and promotion audit records.
11. GET /metrics for Prometheus scraping.

Replay and Deployment Controls

Replay bundles contain the persisted run, steps, receipts, tool schemas, and config/model/prompt/policy fingerprints behind an integrity hash. A bundle can be loaded and replayed in a spawned Python interpreter without access to its source store, producing the same structured report while keeping side effects mocked. Replay fails closed for unknown tools or changed side-effect classifications. Paired comparison reports sample counts and regressions through the control API and dashboard.

Deployment control has a separate CLI entry point:

agentops-deploy --help
agentops-deploy status --db agentops-deploy.db --name refund-agent --environment staging
agentops-deploy history --db agentops-deploy.db --name refund-agent

Promotions use explicit evidence for dataset/version identity, sample size, freshness, oracle pass rate, policy violations, p95 latency, cost, and regression count. Shadow execution never calls tool handlers, and canary routing/rollback remains an explicitly simulated control-plane feature.

Mock vs OpenAI Provider

The default config at configs/refund-agent.yaml uses the deterministic mock provider:

model:
  provider: mock
  model: mock-refund-scripted

This is the preferred path for tests and evals because it is repeatable and exercises the runtime, gateway, and commerce tools without network access.

An OpenAI-backed provider exists behind provider: openai. It requires the optional llm extra, the openai package, and OPENAI_API_KEY in the environment:

py -3 -m pip install -e ".[dev,llm]"

The model may propose tool calls, but it never executes tools directly. Every action still flows through the deterministic gateway and durable runtime.