SecureVibeBench

Important

SecureVibeBench is now v0.2! Just try pip install -U securevibebench!

We are updating to new harbor-native evaluation and precompiled images, and we will release them once they are fully prepared. Please report any usage issues you encounter via Issues or my email, and we will investigate and fix as soon as possible. 🙏

✨ News | 🛠️ Quick Start | 🔭 Overview | 📚 Citation | 🙏 Acknowledgments

✨ News

• [2026-10-29] We updated to v0.2, introducing a PyPI package, Harbor-format tasks, and unified model evaluation. The original version is now referred to as v0.1.

• [2026-04-07] 🎉 Our paper has been accepted to ACL 2026.

🛠️ Quick Start

Requirements: Python 3.12+, Linux, and Docker.

Install

We recommend uv for managing Python environments.

uv pip install -U securevibebench

# Or with pip
pip install -U securevibebench

Task containers will be prepared when the real evaluation starts.

For development

git clone https://github.com/iCSawyer/SecureVibeBench.git
cd SecureVibeBench
uv sync  # editable install + development tools

Run evaluation

export OPENAI_API_KEY='<your-api-key>'
export MODEL='openai/<model-name>'

securevibebench eval --model "$MODEL" --task 992
# securevibebench eval --model "$MODEL" # all 105 tasks

Results are saved to runs/<job>/summary.json under the current working directory.

We disable static vulnerability analysis by default. Enable it with --run-sast TRUE and configure Semgrep credentials such as SEMGREP_APP_TOKEN.

Note

For custom endpoints, development setup, and more details, see the Usage Guide. Run securevibebench eval --help for evaluation options.

🔭 Overview

SecureVibeBench is the first SWE-bench-level benchmark for secure vibe coding of agents, consisting of 105 C/C++ coding tasks sourced from real vulnerabilities (OSS-Fuzz/ARVO) covering various projects.

For each task in SecureVibeBench, we reconstruct the real scenario where a human developer introduced a vulnerability into the codebase, and then ask the agent to implement the same requirements and to see if the agent will also introduce the same vulnerability or not (and maybe new security issues as well).

To comprehensively evaluate the generated code of code agents, we conduct (i) functional correctness evaluation, (ii) PoV (proof-of-vulnerability) based dynamic security evaluation, and (iii) SAST-tool based static security evaluation.

Important

Why SecureVibeBench?

• First SWE-bench-level, peer-reviewed benchmark for secure vibe coding.
• Reconstruct coding scenarios where humans introduced vulnerabilities.
• No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.

📚 Citation

If you find our project useful, please consider citing our paper:

@inproceedings{chen2026securevibebench,
  title     = "{S}ecure{V}ibe{B}ench: Benchmarking Secure Vibe Coding of {AI} Agents via Reconstructing Vulnerability-Introducing Scenarios",
  author    = "Chen, Junkai and Huang, Huihui and Lyu, Yunbo and An, Junwen and
               Shi, Jieke and Yang, Chengran and Zhang, Ting and Tian, Haoye and
               Li, Yikun and Li, Zhenhao and Zhou, Xin and Hu, Xing and Lo, David",
  editor    = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
  booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
  year      = "2026", month = jul, pages = "24144--24168",
  address   = "San Diego, California, United States",
  publisher = "Association for Computational Linguistics",
  url       = "https://aclanthology.org/2026.acl-long.1107/",
  ISBN      = "979-8-89176-390-6"
}

🙏 Acknowledgments

Our work builds on the following excellent works, and please consider citing them: OSS-Fuzz, ARVO, habour, and mini-SWE-agent.

@misc{mei2024arvoatlasreproduciblevulnerabilities,
  title         = {ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software},
  author        = {Xiang Mei and Pulkit Singh Singaria and Jordi Del Castillo and Haoran Xi and
                   Abdelouahab and Benchikh and Tiffany Bao and Ruoyu Wang and Yan Shoshitaishvili and
                   Adam Doupé and Hammond Pearce and Brendan Dolan-Gavitt},
  year          = {2024}, eprint = {2408.02153},
  archivePrefix = {arXiv}, primaryClass = {cs.CR},
  url           = {https://arxiv.org/abs/2408.02153}
}

@article{serebryany2017oss,
  title  = {$\{$OSS-Fuzz$\}$-Google's continuous fuzzing service for open source software},
  author = {Serebryany, Kostya},
  year   = {2017}
}

@software{Harbor_Framework,
author = {{Harbor Framework Team}},
title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
year = {2026},
doi = {10.5281/zenodo.20953922},
url = {https://doi.org/10.5281/zenodo.20953922}
}

@inproceedings{yang2024sweagent,
  title={{SWE}-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
  author={John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press},
  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year={2024},
  url={https://arxiv.org/abs/2405.15793}
}