ARES RED TEAM: Pi Session Evaluation

Research basis, source attribution and implementation mapping.

Inspect a saved Pi coding-agent session against an explicit tool policy. Each finding points to the entry that produced it. The same session and policy produce the same report.

Run

Python 3.11 or newer. The evaluator uses the standard library.

python ares.py examples/session.jsonl examples/policy.json
python -m pip install pytest
python -m pytest

Exit code 0 means the selected branch passed these checks, 1 means findings were recorded and 2 means invalid input. The example is a fixture, not a recorded model experiment.

Use with Pi

Pi saves sessions as JSONL. Copy a session file into a private local directory and pass its path to the evaluator. Use --leaf ENTRY_ID when a session has multiple branches. Streaming events from --mode json are a different format and are not accepted.

The evaluator checks tool allowlists, unmatched or missing tool results, tool-name mismatches, tool errors and literal canary disclosure in assistant text. Refusal wording never cancels a disclosure. Shell execution entries are surfaced for review. Canaries are identified by index in reports rather than printed.

Session and policy hashes identify the inputs used. They do not prove that a log is authentic or complete. Keep logs outside the repository because they can contain private code, messages and tool output.

Batch evaluation and regression review

Create a manifest next to your private session files:

[{"id":"task-01","session":"task-01.jsonl","leaf":"final-entry-id"}]

python batch.py cases.json policy.json --output report.json --sarif report.sarif
python batch.py cases.json policy.json --output next.json --baseline report.json

The batch evaluator produces per-session findings, input hashes, invalid-session counts and totals by rule. SARIF output can be consumed by code-scanning tools. Finding comparisons identify new observations, observations no longer present, added cases, removed cases and invalid cases. A changed policy is rejected for baseline comparison because it changes what a finding means.

Manifests cannot read outside their own directory tree. Invalid sessions remain visible as failures rather than disappearing from the denominator. The batch exit code is 1 if any session is invalid or has findings.

Policy configuration

{
  "allowed_tools": ["read"],
  "canaries": ["EXAMPLE-ONLY-CANARY"],
  "max_tool_calls": {"read": 3},
  "required_arguments": {"read": ["path"]}
}

Tool limits apply to the selected branch. Required arguments check field presence, not semantic safety. Literal canaries are checked in assistant messages and serialized outgoing tool arguments, with only the canary index included in reports.

The evaluator also checks call/result pairing, duplicate call identifiers, missing results and tool-name mismatches. Selecting a branch avoids counting abandoned calls as though they ran on the final path. An explicit leaf is required when there is more than one branch.

Operational workflow

1. Capture Pi version 3 session files locally.
2. Define the task's permitted tools, limits and synthetic canaries.
3. Select the exact branch to evaluate.
4. Run the batch and inspect the original entries for each finding.
5. Compare against a previous report under the same policy.
6. Keep private session logs separate from public benchmark fixtures.

The tests exercise branch isolation, malformed trees, tool pairing, argument checks, limits, secret-free reports, manifest path containment and baseline-policy mismatches.

Measurement boundary

This is a replay evaluator, not a live permission boundary. It does not execute tool calls or rerun the model. Passing an allowlist check does not establish that a tool's arguments were safe, and absence of literal disclosure does not rule out encoded leakage. Determinism applies to the report, not to model generation.

Research and design

Pi's session format defines the input structure. Branch-aware evaluation avoids mixing outcomes from abandoned and active paths. AgentDojo motivates evaluating tool-using agents on explicit tasks and security outcomes rather than persuasive-looking answers.

Built by Vishnu Kosuri. This repository implements the session-evaluation component of ARES RED TEAM.