Evaluation Driven Development with a Financial Agent

Pydantic AI agent that uses an OpenAI GPT model and a SearchAPI MCP web-search server. The project also includes local observability with Prometheus, Loki, Tempo, and Grafana, plus live and stored-trace eval scripts.

Architecture

The project is organized around a versioned behavior contract. prompt_eval.json keeps the agent identity, prompt, prompt version, agent model, and eval policy together. The same contract drives the application, live evals, and trace evals over stored traces.

flowchart TB
    %% =========================
    %% TOP-LEVEL CONTRACT
    %% =========================
    Contract["📝 <b>prompt_eval.json</b><br/><span style='font-size:16px'>agent_id • prompt • model • eval policy</span>"]

    %% =========================
    %% BLOCK 1 — RUNTIME
    %% =========================
    subgraph Runtime["1. Agent Runtime"]
        direction LR

        Main["⚙️ <b>main.py</b>"]
        Agent["🤖 <b>Pydantic AI Agent</b>"]
        Model["🧠 <b>OpenAI GPT Model</b>"]
        MCP["🔌 <b>SearchAPI MCP Server</b>"]

        Main --> Agent
        Agent --> Model
        Agent -->|"X-MCP-Token"| MCP
    end

    %% =========================
    %% BLOCK 2 — OBSERVABILITY
    %% =========================
    subgraph Telemetry["2. Local Observability"]
        direction LR

        OTel["📡 <b>OpenTelemetry</b>"]
        Metrics["📈 <b>Prometheus</b>"]
        Traces["🔍 <b>Tempo</b>"]
        Logs["📜 <b>Loki</b>"]
        ObjectStore["🪣 <b>SeaweedFS</b><br/>S3-compatible"]
        Grafana["📊 <b>Grafana Dashboard</b>"]

        OTel --> Traces
        OTel --> Logs
        Traces --> ObjectStore

        Metrics --> Grafana
        Traces --> Grafana
        Logs --> Grafana
    end

    %% =========================
    %% BLOCK 3 — EVALUATION
    %% =========================
    subgraph Evaluation["3. Evaluation Layer"]
        direction LR

        Live["🔄 <b>Live Repeated Evals</b>"]
        TraceEval["✅ <b>Deterministic<br/>Trace Regression</b>"]
        Judge["⚖️ <b>LLM-as-a-Judge<br/>Regression</b>"]
        Report["📄 <b>Evaluation Reports</b><br/>Terminal • Markdown • JSON"]

        Live --> TraceEval
        Live --> Judge
        TraceEval --> Report
        Judge --> Report
    end


    %% =========================
    %% CROSS-LAYER FLOWS
    %% =========================
    Contract ==>|"Defines runtime + eval policy"| Main

    Agent ==>|"Traces · Logs"| OTel
    Main ==>|"Usage · Counters"| Metrics

    Traces ==>|"Execution traces"| TraceEval
    Traces ==>|"Execution traces"| Judge

    Contract -.->|"Regression policy"| TraceEval
    Contract -.->|"Evaluation rubric"| Judge

    Contract -.->|"Configures"| Agent


    %% =========================
    %% STYLING
    %% =========================

    %% Contract
    classDef contract fill:#FFF3CD,stroke:#D6A800,stroke-width:3px,color:#3A3000,font-size:18px;

    %% Runtime — Blue
    classDef runtime fill:#E8F1FF,stroke:#2563EB,stroke-width:2.5px,color:#102A56,font-size:17px;

    %% Observability — Purple
    classDef telemetry fill:#F1EAFE,stroke:#7C3AED,stroke-width:2.5px,color:#351267,font-size:17px;

    %% Evaluation — Green
    classDef evaluation fill:#E8F8EE,stroke:#16A34A,stroke-width:2.5px,color:#123D23,font-size:17px;

    class Contract contract;
    class Main,Agent,Model,MCP runtime;
    class OTel,Metrics,Traces,ObjectStore,Logs,Grafana telemetry;
    class Live,TraceEval,Judge,Report evaluation;

    %% Larger connector labels + cleaner arrows
    linkStyle default stroke-width:2.5px,font-size:18px;

    %% Subgraph backgrounds
    style Runtime fill:#F5F9FF,stroke:#2563EB,stroke-width:3px
    style Telemetry fill:#FAF7FF,stroke:#7C3AED,stroke-width:3px
    style Evaluation fill:#F5FCF7,stroke:#16A34A,stroke-width:3px

Loading

The runtime and evaluation paths have separate responsibilities:

• observability records what happened: model requests, tool calls, tokens, logs, and spans;
• evaluation decides whether the behavior met the versioned contract;
• the shell exit status and Markdown report make the result usable as a CI quality gate.

Framework and technology stack

Layer	Framework or technology	Responsibility
Language and environment	Python 3.10+, uv, python-dotenv	Dependency management and loading local configuration from .env
Agent framework	Pydantic AI	Agent execution, tool orchestration, usage limits, run metadata, and instrumentation
Model provider	OpenAI, selected per prompt_eval.json case	Generates tool calls and the final financial-market response
Tool protocol	Model Context Protocol through MCPToolset	Connects the agent to external tools using a standard protocol
Search tool	SearchAPI MCP server	Retrieves current web and financial-market information; authenticated with X-MCP-Token
Behavior contract	prompt_eval.json and EvalCase	Versions the agent prompt and model together with nested eval checks, request/tool limits, and judge rubric
Telemetry standard	OpenTelemetry	Produces and exports Pydantic AI traces and application logs over OTLP/HTTP
Application metrics	Prometheus Python client	Exposes cumulative model-request, tool-call, and token counters on /metrics
Metrics database	Prometheus	Scrapes and stores time-series metrics locally
Trace backend	Grafana Tempo	Receives, compacts, queries, and applies retention to OpenTelemetry traces
Trace object storage	SeaweedFS	Provides the persistent S3-compatible bucket where Tempo stores trace blocks
Log database	Grafana Loki	Indexes logs with Loki TSDB and stores log chunks on its local filesystem backend
Visualization	Grafana	Provides the provisioned dashboard and Explore UI for metrics, traces, and logs
Eval framework	Pydantic Evals	Runs repeated live cases, deterministic evaluators, custom MinToolCalls, and LLMJudge
Eval artifacts	Rich terminal, Markdown, and JSON reports	Shows pass/fail, scores, reasons, prompt versions, and trace identifiers for local, CI, or automated review
Infrastructure	Docker Compose	Runs Prometheus, Tempo, SeaweedFS, Loki, and Grafana as the local observability stack

Local data persistence

The observability stack does not use MySQL or another relational database. Each service uses its own storage engine backed by a persistent Docker volume:

Data	Storage engine	Container path	Docker volume	Retention
Application logs	Loki TSDB index with filesystem chunk storage	/loki	financial_agent2_loki-data	No explicit retention period is configured
Metrics	Prometheus time-series database	/prometheus	financial_agent2_prometheus-data	Prometheus default retention applies
Trace blocks	SeaweedFS S3-compatible object storage	/data	financial_agent2_seaweedfs-data	24 hours, enforced by Tempo's block_retention
Tempo write-ahead log	Tempo local WAL	/var/tempo/wal	financial_agent2_tempo-data	Temporary ingestion buffer; trace blocks are flushed to SeaweedFS
Grafana configuration and state	Grafana embedded database and local data directory	/var/lib/grafana	financial_agent2_grafana-data	Retained until the volume is removed

For logs specifically, infra/loki-config.yml configures:

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks

schema_config:
  configs:
    - store: tsdb
      object_store: filesystem

tsdb here means Loki's time-series index format; it is not MySQL. The loki-data:/loki mount in Docker Compose keeps the log index and chunks across container restarts. Removing that Docker volume removes the locally stored logs.

Tempo uses SeaweedFS through its S3-compatible API at seaweedfs:8333. SeaweedFS pre-creates the tempo bucket on startup and persists its objects in financial_agent2_seaweedfs-data. Removing that volume removes the stored trace blocks. The older local blocks under the Tempo volume are not migrated automatically and are no longer queried after switching to the S3 backend.

There are three evaluation modes:

1. Live repeated evals call the current agent, OpenAI, and SearchAPI MCP server several times to expose stochastic failures.
2. Deterministic trace regression evaluates already-stored Tempo traces without making new model or search calls.
3. LLM-as-judge trace regression evaluates stored answers against the matching versioned rubric and spends only judge-model tokens.

Requirements

• Python 3.10+
• uv
• Docker and Docker Compose
• A .env file with the required API keys and endpoints

Required .env values:

OPENAI_API_KEY=...
SEARCHAPI_MCP_URL=...
SEARCHAPI_TOKEN=...
PROMETHEUS_METRICS_PORT=9109
PROMETHEUS_HOLD_OPEN=true
OTEL_SERVICE_NAME=financial-agent2

The local SeaweedFS service has development credentials with safe defaults for localhost-only access. Override them in .env if needed:

SEAWEEDFS_ACCESS_KEY=tempo
SEAWEEDFS_SECRET_KEY=change-me

Local Observability Stack

Start the local telemetry stack:

docker compose -f infra/docker-compose.yml up -d

Services:

Prometheus: http://localhost:9090  metrics TSDB
Loki:       http://localhost:3100  logs
Tempo:      http://localhost:3200  traces/spans
SeaweedFS:  http://localhost:8333  S3-compatible API
Seaweed UI: http://localhost:23646 object-storage administration
Grafana:    http://localhost:3000  UI

Grafana login:

admin / admin

Provisioned dashboard:

Financial Agent / Financial Agent Observability

Run The Agent

Run the agent:

uv run python main.py

Run a specific versioned agent contract by id:

uv run python main.py --agent-id financial_market_headlines

main.py loads its prompt from:

prompt_eval.json

The default case is:

financial_market_headlines v1

The app:

• loads secrets from .env
• uses the case's agent_model from prompt_eval.json
• calls the SearchAPI MCP server with X-MCP-Token
• exposes Prometheus metrics on PROMETHEUS_METRICS_PORT
• exports traces to Tempo
• exports logs to Loki

Because Prometheus scrapes a running HTTP endpoint, PROMETHEUS_HOLD_OPEN=true keeps the process alive after the agent finishes so Prometheus can scrape /metrics.

Metrics

The app exposes these Prometheus metrics:

financial_agent_llm_requests_total
financial_agent_mcp_tool_calls_total
financial_agent_input_tokens_total
financial_agent_output_tokens_total

For cumulative values in Grafana, use range queries:

round(sum(increase(financial_agent_llm_requests_total[$__range])))
round(sum(increase(financial_agent_mcp_tool_calls_total[$__range])))

Do not use the raw counter as a cross-run total. The Python process owns the counter, so it resets when main.py restarts. Prometheus stores the historical samples, and increase(...) computes the cumulative value over a selected time range.

Unit Evals

Run live Pydantic evals against the agent:

uv run python tests/unittest_eval_agent.py

This makes a fresh model/tool call for each case in prompt_eval.json and evaluates the result with the case's configured checks:

{
  "agent_id": "financial_market_headlines",
  "version": 1,
  "agent_model": "openai:gpt-4",
  "prompt": "What are today's major financial market headlines?",
  "eval": {
    "contains": ["market"],
    "max_model_requests": 3,
    "min_tool_calls": 1,
    "max_tool_calls": 5,
    "judge_rubric": "..."
  }
}

By default, each case runs three times so intermittent failures to call the search tool are visible. Every run must make at least min_tool_calls and no more than max_tool_calls:

uv run python tests/unittest_eval_agent.py
uv run python tests/unittest_eval_agent.py --repeat 5

Repeated runs execute sequentially to avoid bursting model and MCP requests. Use this when you want to test current behavior and are fine with spending OpenAI/SearchAPI quota for every repetition.

The command exits with status 1 when any repeated run fails an assertion, so it can be used directly as a CI quality gate.

MinToolCalls is a small project evaluator because the installed Pydantic Evals release provides MaxToolCalls, but not a corresponding minimum evaluator.

Regression Trace Evals

Run evals against traces already stored in Tempo:

uv run python tests/regression_eval_traces.py

This does not call OpenAI or SearchAPI. It queries Tempo and evaluates stored spans.

Checks come from the matching case in prompt_eval.json:

contains
max_model_requests
min_tool_calls
max_tool_calls

The trace evaluator uses:

• gen_ai.operation.name == "chat" for model request count
• gen_ai.operation.name == "execute_tool" for tool call count
• root span final_result for the contains(...) check
• trace metadata agent_id and prompt_version to look up the matching contract

Evaluate a specific trace:

uv run python tests/regression_eval_traces.py --trace-id d3a378dd2a363988d38f723578968e48

Randomly sample 10% of recent traces:

uv run python tests/regression_eval_traces.py --sample-percent 10 --limit 1000 --seed 42

Search only the last hour:

uv run python tests/regression_eval_traces.py --sample-percent 10 --lookback-minutes 60

Sampling options:

--limit              maximum traces to fetch from Tempo before sampling
--sample-percent     percent of fetched traces to evaluate
--seed               deterministic random seed for reproducible samples
--lookback-minutes   Tempo search window, default 1440

With small trace sets, the evaluator always checks at least one trace.

LLM Judge Trace Regression

Run an LLM-as-judge regression over stored Tempo traces:

uv run python tests/regression_llm_judge_traces.py

This does not rerun the financial agent. It reads stored trace content from Tempo, resolves the matching contract from agent_id/prompt_version, extracts the final answer, and asks a judge model to grade the answer against that contract's eval.judge_rubric.

By default this judges up to 5 recent traces:

--sample-percent 100
--max-traces 5

--max-traces is a cap, not a target. If Tempo only has one matching trace, the runner can only judge one trace.

Default judge model:

openai:gpt-4o-mini

Override it with:

uv run python tests/regression_llm_judge_traces.py --judge-model openai:gpt-4o

Randomly judge 10% of recent traces:

uv run python tests/regression_llm_judge_traces.py --sample-percent 10 --limit 1000 --max-traces 5 --seed 42

Only consider traces from the last 60 minutes:

uv run python tests/regression_llm_judge_traces.py --lookback-minutes 60

--lookback-minutes defaults to 1440 (24 hours) and applies when traces are discovered through Tempo search. It is ignored when --trace-id selects one explicit trace.

Print stored final answers before judging:

uv run python tests/regression_llm_judge_traces.py --show-outputs

Write the Markdown and JSON reports somewhere else:

uv run python tests/regression_llm_judge_traces.py \
  --report-path /tmp/llm-judge-report.md \
  --json-report-path /tmp/llm-judge-report.json

If --json-report-path is omitted, the JSON report uses the Markdown report's basename with a .json suffix. The default report paths are:

tests/reports/regression_llm_judge_traces.md
tests/reports/regression_llm_judge_traces.json

The JSON report is structured for CI and downstream analysis. It includes trace-selection parameters, agent id, prompt version, agent model, judge model, rubric, stored final answer, evaluator scores, assertions, and judge reasons.

The LLM judge emits both:

• judge_pass: pass/fail assertion against the case rubric
• judge_score: numeric judge score with the judge's reason

The report inputs include:

• trace_id
• agent_id
• prompt_version
• agent_model
• judge_model
• prompt

The runner also prints trace-selection counts:

fetched   traces returned by Tempo search
sampled   traces kept after --sample-percent
judging   traces sent to the judge after --max-traces

If a trace fails, the report includes the input, stored output, evaluator reason, and failure table. A very fast failure usually means the evaluator failed locally before a judge-model call completed, for example because the stored trace has no final_result or the judge model/API configuration failed.

This eval spends judge-model tokens, but it does not call the SearchAPI MCP server.

Useful Commands

Check container status:

docker compose -f infra/docker-compose.yml ps

Restart Grafana after dashboard changes:

docker compose -f infra/docker-compose.yml restart grafana

Check Grafana health:

curl http://localhost:3000/api/health

List provisioned Grafana datasources:

curl -u admin:admin http://localhost:3000/api/datasources

Reset only Grafana local state if datasource provisioning gets stuck:

docker compose -f infra/docker-compose.yml stop grafana
docker compose -f infra/docker-compose.yml rm -f grafana
docker volume rm financial_agent2_grafana-data
docker compose -f infra/docker-compose.yml up -d

This preserves Prometheus, Loki, Tempo WAL, and SeaweedFS trace-storage volumes.