VCC-experiments

Official experiment code for "View-oriented Conversation Compiler for Agent Trace Analysis" (Paper)

This repo reproduces the academic experiments in the paper. For daily use of VCC, see VCC.

This repo now supports a context learning platform on AppWorld. It trains a memory (MEMORY.md) by running tasks with a student agent, analyzing each execution with a teacher agent, and iteratively refining rules across epochs. The learned memory is then evaluated on test splits.

Install

Requirements: Python 3.10+, Claude Code.

AppWorld is automatically installed and set up on first run (pinned to commit a072b7a). No manual setup needed.

This project runs Claude Code as a subprocess with claude -p --bare. The --bare flag disables OAuth/keychain and requires explicit API credentials. You must set up one of:

• Anthropic API: set ANTHROPIC_API_KEY environment variable
• Vertex AI: set CLAUDE_CODE_USE_VERTEX=1, ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION environment variables

You can also put these in an env.json file in this directory (gitignored, won't be committed):

{
  "CLAUDE_CODE_USE_VERTEX": "1",
  "ANTHROPIC_VERTEX_PROJECT_ID": "your-project-id",
  "CLOUD_ML_REGION": "us-east5"
}

Important:

• Do not run experiments on a Claude Pro/Max subscription — it will get your account banned or burn through your quota instantly.
• Do not remove the --bare flag from the code — it prevents Claude Code's auto-memory and CLAUDE.md from contaminating experiment data.
• These scripts are thread-safe but not process-safe. Use --workers to parallelize, but do not run multiple script instances simultaneously — this will corrupt AppWorld's internal state.

Running all experiments costs roughly $6,500+ USD. You can reproduce the key Sonnet result for under $200 (see below).

Quick Sanity Check

The debug split has only 4 tasks. Use these to verify your setup works.

python train.py --split debug --workers 2 --epochs 1 --model haiku --effort low --teacher-model haiku --teacher-effort low

python evaluate.py --split debug --workers 4 --model haiku --effort low

If both complete without errors, your environment is ready.

How It Works

Train (train.py): epoch-based context learning. Each epoch runs every task once with a student agent, then a teacher agent analyzes the execution trace and edits MEMORY.md. Multiple parallel workers sync via LLM-based memory merging after each batch.

Evaluate (evaluate.py): single-pass execution. Runs tasks with a fixed MEMORY.md (from training or default), measures pass rate.

All outputs go to artifacts/:

artifacts/
├── train/{session}/
│   ├── config.json              # config + progress (for resume)
│   ├── MEMORY.md                # learned rules
│   └── {epoch}/
│       ├── t000_{task_id}/      # per-task: run.jsonl, run.txt, analyse.jsonl, ...
│       └── merge_{batch}/       # memory merge artifacts
└── eval/{session}/
    ├── config.json
    ├── MEMORY.md                # memory used
    ├── results.json             # aggregated pass rate
    └── t000_{task_id}/          # per-task: run.jsonl, run.txt, result.json

Train

python train.py [options]

Arg	Default	Description
--task_id ID [ID ...]	—	Specific task(s). Mutually exclusive with --split
--split SPLIT [...]	—	Load tasks from split(s): train, dev, test_normal, test_challenge, debug
--difficulty N	all	Filter by difficulty (1/2/3). Only with --split
--limit N	all	Take first N tasks. Only with --split
--workers N	1	Parallel sessions
--epochs N	2	Training epochs
--method M	vccp	Teacher trace format: vccp, vcc, vccpf, naive
--model M	CC default	Student model: opus, sonnet, haiku
--effort E	CC default	Student effort: low, medium, high
--teacher-model M	same as --model	Teacher model
--teacher-effort E	same as --effort	Teacher effort
--memory PATH	built-in seed	Initial MEMORY.md file
--session NAME	timestamp	Output dir name (artifacts/train/{name}/). Errors if exists
--shuffle MODE	rnd	Task order: rnd, no, or integer seed
--resume NAME	—	Resume or inherit from existing run. last for most recent

Evaluate

python evaluate.py [options]

Arg	Default	Description
--task_id ID [ID ...]	—	Specific task(s). Mutually exclusive with --split
--split SPLIT [...]	—	Load tasks from split(s)
--difficulty N	all	Filter by difficulty. Only with --split
--limit N	all	Take first N tasks. Only with --split
--workers N	1	Parallel sessions
--model M	CC default	Model: opus, sonnet, haiku
--effort E	CC default	Effort: low, medium, high
--memory PATH	built-in seed	MEMORY.md to use
--session NAME	timestamp	Output dir name (artifacts/eval/{name}/). Errors if exists
--resume NAME	—	Resume or inherit from existing run. last for most recent
--manual	false	Print command and wait for manual execution (single task only)

Resume: --resume NAME with same mode resumes (skips completed tasks). Cross-mode (e.g. --resume a train session from evaluate) inherits config + memory. Add --session NEW to clone instead of resuming in-place.

Run Sonnet Experiments (Recommended)

The most cost-effective way to reproduce the key results. Commands are grouped by cost so you can stop at any point.

Each step is independently resumable. If a run crashes, rerun the same command with --resume {session} to continue from where it left off.

Step 1: Train VCC (~$60)

python train.py --split train --workers 8 --model sonnet --teacher-model sonnet --method vccp --session son_son_vccp

Step 2: Evaluate on test_normal (~$100)

python evaluate.py --resume son_son_vccp --split test_normal --workers 8 --session son_son_vccp_tn

You can stop here for under $200 — this reproduces the core VCC result on test_normal.

Step 3: Train naive for comparison (~$130)

python train.py --split train --workers 8 --model sonnet --teacher-model sonnet --method naive --session son_son_naive

Step 4: Evaluate naive on test_normal (~$100)

python evaluate.py --resume son_son_naive --split test_normal --workers 8 --session son_son_naive_tn

Step 5: Evaluate on test_challenge (~$500)

python evaluate.py --resume son_son_vccp --split test_challenge --workers 8 --session son_son_vccp_tc
python evaluate.py --resume son_son_naive --split test_challenge --workers 8 --session son_son_naive_tc

Step 6: Baselines (~$100 for test_normal, ~$500 for test_challenge)

python evaluate.py --split test_normal --workers 8 --model sonnet --session baseline_son_tn
python evaluate.py --split test_challenge --workers 8 --model sonnet --session baseline_son_tc

Full Sonnet reproduction (all 6 steps) costs roughly $1,000.

Run All Experiments (~$6,500+ USD)

Full reproduction across three model tiers. Each command is independent and resumable. Includes the Sonnet commands above.

# ── Opus + Opus ──
python train.py --split train --workers 8 --model opus --teacher-model opus --method vccp --session opus_opus_vccp
python train.py --split train --workers 8 --model opus --teacher-model opus --method naive --session opus_opus_naive
python evaluate.py --resume opus_opus_vccp --split test_normal --workers 8 --session opus_opus_vccp_tn
python evaluate.py --resume opus_opus_vccp --split test_challenge --workers 8 --session opus_opus_vccp_tc
python evaluate.py --resume opus_opus_naive --split test_normal --workers 8 --session opus_opus_naive_tn
python evaluate.py --resume opus_opus_naive --split test_challenge --workers 8 --session opus_opus_naive_tc

# ── Sonnet + Sonnet ──
python train.py --split train --workers 8 --model sonnet --teacher-model sonnet --method vccp --session son_son_vccp
python train.py --split train --workers 8 --model sonnet --teacher-model sonnet --method naive --session son_son_naive
python evaluate.py --resume son_son_vccp --split test_normal --workers 8 --session son_son_vccp_tn
python evaluate.py --resume son_son_vccp --split test_challenge --workers 8 --session son_son_vccp_tc
python evaluate.py --resume son_son_naive --split test_normal --workers 8 --session son_son_naive_tn
python evaluate.py --resume son_son_naive --split test_challenge --workers 8 --session son_son_naive_tc

# ── Haiku + Sonnet ──
python train.py --split train --workers 8 --model haiku --teacher-model sonnet --method vccp --session haiku_son_vccp
python train.py --split train --workers 8 --model haiku --teacher-model sonnet --method naive --session haiku_son_naive
python evaluate.py --resume haiku_son_vccp --split test_normal --workers 8 --session haiku_son_vccp_tn
python evaluate.py --resume haiku_son_vccp --split test_challenge --workers 8 --session haiku_son_vccp_tc
python evaluate.py --resume haiku_son_naive --split test_normal --workers 8 --session haiku_son_naive_tn
python evaluate.py --resume haiku_son_naive --split test_challenge --workers 8 --session haiku_son_naive_tc

# ── Baselines (no training) ──
python evaluate.py --split test_normal --workers 8 --model opus --session baseline_opus_tn
python evaluate.py --split test_challenge --workers 8 --model opus --session baseline_opus_tc
python evaluate.py --split test_normal --workers 8 --model sonnet --session baseline_son_tn
python evaluate.py --split test_challenge --workers 8 --model sonnet --session baseline_son_tc
python evaluate.py --split test_normal --workers 8 --model haiku --session baseline_haiku_tn
python evaluate.py --split test_challenge --workers 8 --model haiku --session baseline_haiku_tc

Note: Haiku uses Sonnet as teacher because Haiku alone cannot effectively reflect on its own traces.

Results

All numbers are AppWorld official metrics (%). T tok = total reflector tokens during training, T/a = reflector tokens per analysis. Mem = trained MEMORY.md size in characters.

Paper-to-code mapping: VCC = vccp (VCC-preprocessed trace format), JSON = naive (raw JSONL).

Opus generator + Opus reflector

Method	test_normal task_goal				test_normal scenario_goal				test_challenge task_goal				test_challenge scenario_goal				T tok	T/a	Mem
	all	d1	d2	d3	all	d1	d2	d3	all	d1	d2	d3	all	d1	d2	d3
Baseline	79.2	77.2	75.0	84.1	67.9	68.4	56.2	76.2	75.8	69.4	76.7	77.4	64.0	54.2	66.0	66.1	—	—	—
JSON	92.9	96.5	93.8	88.9	87.5	89.5	87.5	85.7	84.7	91.7	84.7	82.0	70.5	83.3	74.0	63.1	22.1M	124K	12.6K
VCC	94.0	98.2	95.8	88.9	91.1	94.7	93.8	85.7	86.3	93.1	84.7	85.1	74.1	87.5	70.0	72.3	7.6M	43K	12.0K

Sonnet generator + Sonnet reflector

Method	test_normal task_goal				test_normal scenario_goal				test_challenge task_goal				test_challenge scenario_goal				T tok	T/a	Mem
	all	d1	d2	d3	all	d1	d2	d3	all	d1	d2	d3	all	d1	d2	d3
Baseline	73.2	77.2	58.3	81.0	67.9	73.7	50.0	76.2	61.4	61.1	59.3	63.1	50.4	45.8	50.0	52.3	—	—	—
JSON	90.5	98.2	91.7	82.5	82.1	94.7	81.2	71.4	71.0	86.1	66.7	68.7	60.4	75.0	54.0	60.0	20.5M	115K	23.4K
VCC	92.9	96.5	95.8	87.3	89.3	94.7	93.8	81.0	74.3	88.9	68.7	73.3	63.3	79.2	56.0	63.1	8.8M	49K	15.5K

Haiku generator + Sonnet reflector

Method	test_normal task_goal				test_normal scenario_goal				test_challenge task_goal				test_challenge scenario_goal				T tok	T/a	Mem
	all	d1	d2	d3	all	d1	d2	d3	all	d1	d2	d3	all	d1	d2	d3
Baseline	39.9	49.1	27.1	41.3	25.0	36.8	12.5	23.8	33.8	31.9	34.0	34.4	17.3	16.7	22.0	13.9	—	—	—
JSON	71.4	86.0	79.2	52.4	62.5	79.0	75.0	38.1	53.5	69.4	45.3	53.8	36.0	45.8	30.0	36.9	23.3M	131K	43.3K
VCC	75.6	87.7	75.0	65.1	62.5	68.4	56.2	61.9	54.7	81.9	47.3	50.3	39.6	66.7	38.0	30.8	12.6M	71K	37.9K

Cite

@article{zhang2026vcc,
  title={View-oriented Conversation Compiler for Agent Trace Analysis},
  author={Lvmin Zhang and Maneesh Agrawala},
  year={2026},
  url={https://github.com/lllyasviel/VCC}
}