TermiGen: High-Fidelity Environments for Terminal Agents

Official implementation and datasets for TermiGen, a framework for training robust terminal agents through verified environments and error-correction trajectories.

📄 Paper: TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents
🤖 Model: TermiGen-32B
🧪 Benchmark: TerminalBench

---

📰 News

• [2026/02/23] TermiGen has integrated with the harbor framework! Simply install harbor and run uvx harbor run -d [email protected]!
• [2026/02/10] Grateful to be selected as the #2 🤗 Huggingface Daily Papers!

🎯 What's Included

This repository provides:

1. 3,500+ Verified Docker Environments - Executable tasks across 11 categories. The TerminalBench 1.0 format is provided as a ZIP archive (because some tasks involve GitHub repositories containing .git metadata), while the Harbor 2.0 format is provided directly as the environments_harbor/ directory.
2. BashAgent - Minimal ReAct-style agent implementation:
   • bash_agent.py - For TerminalBench (tb) framework
   • bash_agent_harbor.py - For Harbor evaluation framework

---

📊 Performance

Our TermiGen-32B achieves:

Benchmark	Pass@1
TerminalBench 1.0	31.3%
TerminalBench 2.0	19.3%
SWE-Bench Verified	21.4%

• +26.8% absolute improvement over base Qwen2.5-Coder-32B on TerminalBench 1.0
• +11.3% absolute improvement over o4-mini with Codex CLI on TerminalBench 1.0

---

🗂️ Environment Categories

Our environments span 11 task categories across 3 tiers:

Tier I: Infrastructure & Core Systems

• 🛠️ Software Build & Compilation: gcc, cmake, rustc, makefile debugging
• ⚙️ System Administration & DevOps: Docker, Kubernetes, systemd, nginx
• 🔐 Security & Reverse Engineering: Ghidra, Wireshark, gdb, Metasploit

Tier II: Data & Algorithm Applications

• 📊 Data Processing & ETL: Spark, Kafka, Parquet, SQL transformations
• 🤖 Machine Learning & MLOps: PyTorch, CUDA, Hugging Face, model debugging
• 🧩 Algorithms & Logic: Graph algorithms, dynamic programming, search

Tier III: Specialized Domains

• 💻 Software Development: React, Django, REST APIs, CI/CD
• 🧪 Scientific Computing: Bioconductor, RDKit, GROMACS, NumPy
• 🎮 Interactive Environments: WebSocket, SSH, Jupyter, REPL
• 🌐 Distributed Computing: MPI, OpenMP, Ray, SLURM
• 🔬 Formal Verification: Coq, Z3, OpenGL, Vulkan

Statistics:

• 420 unique command-line tools
• 16 functional domains
• Average task complexity: 25.5 turns, 8,722 tokens

---

🚀 Quick Start

Step 1: Download Repository

# Clone repository
git clone https://github.com/ucsb-mlsec/terminal-bench-env.git
cd terminal-bench-env

# Extract environments (TerminalBench 1.0 format)
unzip termigen_env.zip -d environments/

# Harbor 2.0 format is already included as environments_harbor/

Step 2: Deploy TermiGen Model

# Install vLLM
pip install vllm

# Deploy model (requires GPU)
vllm serve UCSB-SURFI/TermiGen-32B \
  --port 8000 \
  --tensor-parallel-size 4 \
  --dtype bfloat16

Step 3: Run BashAgent on TerminalBench

# Install dependencies
pip install openai
pip install terminal-bench

# Set environment variables
export MODEL_ENDPOINT="http://localhost:8000/v1"
export MODEL_NAME="UCSB-SURFI/TermiGen-32B"

# Run agent on TerminalBench 1.0 (example task: hello-world)
tb run --dataset terminal-bench-core==0.1.1 --agent-import-path bash_agent:BashAgent --task-id hello-world --log-level debug

Or with Harbor 2.0:

pip install harbor

# Run agent on local tasks
harbor run -p environments_harbor/ \
  --agent-import-path bash_agent_harbor:BashAgent \
  -t hello-world --debug

# Run agent on Terminal-Bench 2.0 (downloads from registry)
harbor run -d [email protected] \
  --agent-import-path bash_agent_harbor:BashAgent \
  --n-concurrent 4 --n-attempts 5 -e e2b

# Run agent on SWE-Bench Verified
harbor run -d swebench-verified \
  --agent-import-path bash_agent_harbor:BashAgent \
  --n-concurrent 4 -e e2b

---

🔧 Environment Details

Using Individual Tasks

After extracting environments (see Step 1):

# TerminalBench 1.0
tb run --agent claude-code --model anthropic/claude-sonnet-4-5-20250929 --dataset-path environments --task-id a_b_testing_models_medium --log-level debug --n-concurrent 1

# Harbor 2.0
harbor run -a claude-code -m anthropic/claude-sonnet-4-5-20250929 -p environments_harbor/ -t a_b_testing_models_medium --debug

Using Full Dataset

# TerminalBench 1.0
tb run --agent claude-code --model anthropic/claude-sonnet-4-5-20250929 --dataset terminal-bench-core==0.1.1 --dataset-path environments --log-level debug --n-concurrent 2

# Harbor 2.0
harbor run -a claude-code -m anthropic/claude-sonnet-4-5-20250929 -p environments_harbor/ -n 2 --debug
# try using smaller n for stability.

Task Structure

Each task is available in two formats:

TerminalBench 1.0 (termigen_env.zip):

task_name/
├── task.yaml              # Task description and metadata
├── Dockerfile             # Environment specification  
├── docker-compose.yaml    # Container orchestration
├── run-tests.sh          # Test execution script
├── tests/                # Unit tests (pytest)
│   └── test_*.py
└── [task files]          # Source code, configs, data, git repos

Harbor 2.0 (environments_harbor/):

task_name/
├── task.toml              # Task metadata
├── instruction.md         # Task description
├── environment/
│   └── Dockerfile         # Environment specification
├── tests/
│   └── test.sh           # Test script with reward logging
└── [task files]          # Source code, configs, data, git repos

---

📁 Repository Structure

terminal-bench-env/
├── README.md                      # This file
├── bash_agent.py                  # BashAgent for TerminalBench (tb) framework
├── bash_agent_harbor.py           # BashAgent for Harbor framework
├── termigen_env.zip               # 3,500+ Docker tasks (TerminalBench 1.0 format)
├── environments/                  # Extracted from termigen_env.zip (after unzip)
└── environments_harbor/           # 3,500+ Docker tasks in Harbor 2.0 format

---

🤖 BashAgent

Two implementations are provided for different evaluation frameworks:

bash_agent.py (TerminalBench / tb framework)

• Interacts with the environment via TmuxSession (send_keys, incremental output)
• Supports asciinema recording and detailed failure mode tracking
• command_duration_sec: Command timeout in seconds (default: 10.0)

bash_agent_harbor.py (Harbor framework)

• Async implementation using Harbor's BaseEnvironment.exec() API
• Compatible with Harbor's e2b, Docker, Daytona, and other environment backends
• timeout_sec: Command timeout in seconds (default: 120)

Shared Configuration

Both agents accept the following via environment variables or constructor args:

• MODEL_ENDPOINT: Model API URL (OpenAI-compatible, default: http://172.17.0.1:8001/v1)
• MODEL_NAME: Model identifier
• max_episodes: Max conversation turns (default: 1000)
• temperature: Sampling temperature (default: 0.6)

Model Compatibility

✅ Fully supported: Qwen2.5-Coder, TermiGen models (support "tool" message role)

⚠️ Requires modification: Models without "tool" role support need the observation message role changed:

# Change from:
{"role": "tool", "content": observation}

# To:
{"role": "user", "content": f"Observation: {observation}"}

---

🔄 Reproducing Benchmark Results

Terminal-Bench 2.0 (19.3%)

# 1. Deploy model
vllm serve UCSB-SURFI/TermiGen-32B --port 8001 --tensor-parallel-size 2

# 2. Run evaluation (89 tasks × 5 attempts)
MODEL_NAME=UCSB-SURFI/TermiGen-32B \
MODEL_ENDPOINT=http://127.0.0.1:8001/v1 \
harbor run -d [email protected] \
  --agent-import-path bash_agent_harbor:BashAgent \
  --job-name tb2_termigen \
  --n-concurrent 4 --n-attempts 5 -e e2b

SWE-Bench Verified (21.4%)

# Same model deployment as above, then:
MODEL_NAME=UCSB-SURFI/TermiGen-32B \
MODEL_ENDPOINT=http://127.0.0.1:8001/v1 \
harbor run -d swebench-verified \
  --agent-import-path bash_agent_harbor:BashAgent \
  --job-name swebench_verified_termigen \
  --n-concurrent 4 --n-attempts 1 -e e2b

Note: These benchmarks use the e2b cloud sandbox (-e e2b). You can also use local Docker (-e docker).

---

📖 Citation

If you use TermiGen in your research, please cite:

@article{zhu2026termigen,
  title={TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents},
  author={Zhu, Kaijie and Nie, Yuzhou and Li, Yijiang and Huang, Yiming and Wu, Jialian and Liu, Jiang and Sun, Ximeng and Yin, Zhenfei and Wang, Lun and Liu, Zicheng and Barsoum, Emad and Wang, William Yang and Guo, Wenbo},
  journal={arXiv preprint arXiv:2602.07274},
  url={https://arxiv.org/abs/2602.07274}, 
  year={2026}
}

---

🤝 Contributing

We welcome contributions! Please feel free to:

• Report bugs or request features via GitHub Issues
• Submit pull requests for bug fixes or improvements for our environments and tasks

---

📧 Contact

• Lead Author: Kaijie Zhu ([email protected])
• Issues: GitHub Issues
• Paper: arXiv

---

🙏 Acknowledgements

• Base Model: Qwen2.5-Coder by Alibaba Cloud
• Benchmark: TerminalBench by Laude Institute
• Compute: AMD MI325X GPUs
• Institutions: UC Santa Barbara, UC San Diego, AMD, University of Oxford, Google