browser-eval-realism

Realism and evaluation-awareness measurements for browser-agent and LLM-agent benchmarks.

Status: pre-alpha. API is not stable. v0 ships trajectory-tier judges first; environment-tier judges land in v0.2. See OPEN_SOURCE_PLAN.md for milestones.

What this is

Two complementary measurement tiers:

• Trajectory tier — given an agent's interaction transcript, do the model's own outputs reveal it noticed it was being evaluated? Adapts methods from Needham et al. (2025) and Abdelnabi & Salem (2025).
• Environment tier — given a website artifact (screenshot / HTML / accessibility tree), does the site itself look like a synthetic eval rather than the real product?

Use either tier independently, or both together for a fuller realism picture.

Install

pip install browser-eval-realism

# Optional extras
pip install "browser-eval-realism[agentlab]"   # AgentLab adapter
pip install "browser-eval-realism[inspect]"    # Inspect .eval log adapter
pip install "browser-eval-realism[calibrate]"  # reproduce ROC validation

30-second example

import asyncio
from browser_eval_realism import audit_trajectory, adapters

traj = adapters.openai_responses(messages=[
    {"role": "system", "content": "You are a coding agent..."},
    {"role": "user", "content": "Refactor this function..."},
    {"role": "assistant", "content": "I notice this looks like a benchmark task..."},
])

result = asyncio.run(
    audit_trajectory(traj, judges=["verbalized_awareness", "purpose_continue_5q"])
)
print(result.verbalized.category)        # "Capability Evaluation"
print(result.verbalized.evidence[:1])    # ["I notice this looks like a benchmark task..."]
print(result.p_eval)                     # 0.87
print(result.label_at_threshold(0.85))   # True (looks like an eval)

CLI

warp-audit ~/agentlab_results/some_study --format agentlab --out realism.jsonl
warp-audit env ./my_site_pages --format screenshot --judges p_eval_site,realism_critique
warp-audit calibrate --validation-set needham2025

Calibration

The trajectory-tier operating threshold (default τ=0.85) is calibrated against the agentic subset of Needham et al. (2025). Reproduce the ROC with:

warp-audit calibrate \
    /path/to/agentic_needham_dataset_v2.json \
    --judge purpose_continue_5q \
    --model anthropic/claude-sonnet-4.5 \
    --out-dir calibration_out/
# → calibration_summary.json + per_sample.jsonl + roc_purpose_continue_5q.png

--limit N restricts to the first N samples for a cheap dry run.

Citing

If you use this package, please cite the WARP paper (link TBD) along with the underlying judge methods (Needham et al. 2025; Abdelnabi & Salem 2025).

Status

v0 progress (week 1 milestone — feature-complete):

• Repo scaffold
• Trajectory judges: verbalized_awareness, purpose_continue_5q
• Adapters: AgentLab (from_exp_dir / iter_study_dir), OpenAI Responses, raw messages, Needham (calibration)
• CLI: warp-audit {list-judges,traj,study,calibrate}
• Synthetic AgentLab step_*.pkl.gz fixture + on-disk integration tests
• Calibration on Needham agentic subset (warp-audit calibrate) — pipeline works end-to-end on the real tiny-needham JSON, produces ROC PNG + per-sample JSONL
• Minimal runnable example (examples/audit_one_trajectory.py)
• BrowserGym adapter (AgentLab adapter covers most BG cases; standalone next)
• Run actual calibration against full Needham agentic dataset (costs API)
• Test PyPI smoke release

v0.1 (queued):

• Trajectory judges: probability_third_person, binary_third_person
• Inspect .eval adapter

v0.2 (queued):

• Environment judges: p_eval_site, comparative, realism_critique, open_ended_truesight
• 276-site realism dataset (sister HF dataset)

v0.3:

• PyPI release

License

Apache-2.0.