LLM security · AI red teaming · prompt-injection & jailbreak testing · MCP / agent / RAG security · OWASP LLM Top 10
The Armor for AI Security. AgentArmor is an open-source scanner that continuously tests, audits, and red-teams your AI stack — LLM APIs, chatbots, tool-calling agents, MCP servers, and RAG pipelines — for prompt injection, jailbreaks, system-prompt leaks, data exfiltration, and unsafe tool use.
Every scan runs a multi-agent analysis that scores findings on a 0–100 risk model, maps them to the OWASP LLM Top 10, explains in plain English what went wrong and how to fix it, and exports reports for developers and security teams (SARIF, HTML, PDF, CSV).
Latest release: v1.5.0 · Changelog
🔍 Who it's for: AI engineers shipping LLM features, red teamers and pentesters testing GenAI apps, and security teams that need OWASP-mapped, audit-ready reports.
| Capability | Description |
|---|---|
| Endpoint scanning | Test any OpenAI-compatible or auto-detected chat API URL |
| Cloud providers | OpenAI, Anthropic, Gemini, and more via LiteLLM |
| Local models | Offline scanning of .gguf and HuggingFace models |
| Agent / MCP / RAG | Security probes for tool-calling agents, MCP servers, and retrieval pipelines |
| L0 adaptive attacks | 100+ mutation variants per attack goal (jailbreak, prompt leak, exfiltration) |
| Self-play red teaming | Attacker → Target → Judge loop to find vulns static probes miss |
| Agent swarm | Up to 100 specialized agents pursue one goal together, sharing discoveries through a blackboard |
| Benchmarking | Compare models and tools (AgentArmor vs PyRIT, Garak, Promptfoo, Inspect AI) |
| Marketplace | Install community probes and OWASP packs |
| Monitoring | Scheduled rescans with drift detection |
| CI/CD | GitHub Action + agentarmor gate for pipeline security gates |
End users on Windows: download the release installer, double-click, and run scans — no Python, Rust, or Node install needed.
Developers / CLI users: need Python 3.10+. Rust is only for compiling the optional native signature engine or building the Tauri GUI yourself.
- Download
AgentArmor_1.5.0_x64-setup.exeor.msifrom Releases - Run the installer
- Open AgentArmor → choose scan type (API, Local Model, Agent, MCP, RAG, Benchmark)
- Configure target → run scan → review findings → export reports
Portable mode: place a file named PORTABLE next to the executable; data is stored in ./data/.
pip install "agentarmor @ git+https://github.com/YashvantHange/[email protected]"
# Optional: offline local model scanning
pip install "agentarmor[local] @ git+https://github.com/YashvantHange/[email protected]"
# One-time detection model download
agentarmor models downloadNote on
pip install agentarmor: theagentarmorname on PyPI belongs to an unrelated project. Installing it does not get you this tool. Install from this repository until a published package name is settled.
docker build -f docker/Dockerfile -t agentarmor/agentarmor .
docker run -p 8787:8787 agentarmor/agentarmor# OpenAI-compatible chat API (use the POST URL from DevTools, not the HTML page)
agentarmor scan --url http://localhost:8000/v1/chat/completions
# Cloud provider
export OPENAI_API_KEY=sk-...
agentarmor scan --provider openai
# Multi-agent analysis runs on every scan (offline mode was removed in v1.4.0).
# Provide an analysis API key via AGENTARMOR_ANALYSIS_API_KEY or --analysis-api-key.
export AGENTARMOR_ANALYSIS_API_KEY=sk-...
agentarmor scan --url https://api.example.com/v1/chat/completions \
--self-play-enabledRun many specialized agents against one preset goal at once. Each agent gets its own attack-graph node, skill and persona, and they share what they find through a blackboard — so a secret recovered by one agent informs every agent that starts after it.
agentarmor swarm goals # list the preset goals
agentarmor swarm run --url http://localhost:8000/v1/chat/completions --goal extract_system_prompt --agents 24 --concurrency 8Agents run in a worker pool, never all at once: --concurrency caps how many talk to
your target simultaneously (default 8, maximum 16) and the configured
engine.endpoint.rate_limit_rps still applies on top. A swarm sends considerably more
traffic than a single scan, so only run it against systems you own or are authorised
to test.
The report shows what the roster actually spanned, rather than an agent count alone:
Agents 24 | Attack paths 3 | Personas 10 | Strategies 3 | Concurrency 8
Goals are presets — there is no free-text objective. See agentarmor swarm goals, or
the Swarm page in the desktop app for the same flow with live per-agent status and
the shared blackboard.
agentarmor scan --agent crewai --agent-config agent.toml
agentarmor scan --mcp ./filesystem-mcp
agentarmor scan --rag ./corpus --embedder bgepip install "agentarmor[local] @ git+https://github.com/YashvantHange/[email protected]"
agentarmor scan --model llama-3.gguf
agentarmor scan --model ./models/qwen3# Model leaderboard
agentarmor benchmark --providers openai,anthropic,gemini --suite owasp
# Tools comparison
agentarmor benchmark tools --suite owasp-llm01 --targets corpusagentarmor marketplace list
agentarmor marketplace install roleplay-injection
agentarmor monitor add "Daily API" --url https://api.example.com/v1/chat/completions --cron daily
agentarmor monitor run <schedule-id>
agentarmor dataset export -o research.jsonl --anonymizeagentarmor scan --url http://localhost:8000/v1/chat/completions --format html,sarif,pdf -o ./reports/
agentarmor gate --sarif ./reports/findings.sarif --fail-on HIGH,CRITICALagentarmor scan --config AgentArmor.tomlSee AgentArmor.toml for endpoint profiles, L0 attack goals, detection mode, and plugin directories.
The Tauri v2 GUI includes:
| Screen | Purpose |
|---|---|
| Home | Scan type picker + benchmark shortcut |
| Chatbot wizard | Guided API scan with multi-agent analysis |
| Scan progress | Live SSE probe stream |
| Findings | Risk scores, attack chains, evidence graph |
| Reports | Export HTML, SARIF, PDF, CSV, JSON |
| Benchmark | Model leaderboard + tools comparison |
| Marketplace | Install community probes |
| Monitoring | Schedules, drift alerts, dataset export |
| Settings | L0, self-play, detection, API keys |
For GUI development only (not required for end users):
agentarmor serve --port 8787
cd gui && npm install && npm run dev # http://localhost:1420See gui/README.md for Tauri build instructions.
agentarmor serve --port 8787| Endpoint | Description |
|---|---|
GET /health |
Sidecar status |
POST /v1/scans |
Start a scan |
GET /v1/scans/{id} |
Scan status + metadata |
GET /v1/scans/{id}/events |
SSE live progress |
GET /v1/findings |
List findings |
POST /v1/benchmarks |
Run benchmark |
GET /v1/marketplace/rules |
List marketplace packs |
POST /v1/monitoring/schedules |
Create monitor schedule |
POST /v1/datasets/export |
Export research JSONL |
name: AI Security Scan
on: [push, pull_request]
jobs:
agentarmor:
runs-on: ubuntu-latest
steps:
- uses: YashvantHange/AgentArmor/action@v1
with:
url: https://api.example.com/v1/chat/completions
fail-on: HIGH,CRITICALSee action/action.yml.
| Provider | Environment variable |
|---|---|
| OpenAI | OPENAI_API_KEY |
| Anthropic | ANTHROPIC_API_KEY |
| Gemini | GEMINI_API_KEY |
| Mistral | MISTRAL_API_KEY |
| Groq | GROQ_API_KEY |
| Together | TOGETHER_API_KEY |
| OpenRouter | OPENROUTER_API_KEY |
| Azure | AZURE_API_KEY, AZURE_API_BASE |
| Bedrock | AWS_* credentials |
As of v1.4.0 the offline analysis mode was removed — AgentArmor runs multi-agent (cloud) analysis on every scan: capability-aware attack-graph planning, an LLM judge, and confidence scoring. This requires an analysis provider API key. Set AGENTARMOR_ANALYSIS_API_KEY (or a provider key such as OPENAI_API_KEY), or configure it in GUI Settings / AgentArmor.toml. Scans started without a key are rejected with a clear error.
Author probes with the SDK and publish to your local marketplace:
agentarmor marketplace validate probes/custom_probe.py
agentarmor marketplace publish probes/custom_probe.py --id my.probe --name "My Probe"See probes/custom_probe.py and agentarmor.sdk.probe_sdk.
No native Mac .app yet (Windows desktop first). Use PyPI + CLI or dev GUI:
pip install "agentarmor @ git+https://github.com/YashvantHange/[email protected]"
agentarmor models download
agentarmor serve --port 8787See docs/MAC.md.
git clone https://github.com/YashvantHange/AgentArmor.git
cd AgentArmor
pip install -e ".[dev]"
pytestOptional native L1 engine (requires Rust):
cd native/l1_signatures && pip install maturin && maturin develop --releaseThis builds the crate and lets tests/test_l1_rust_parity.py verify the Rust and
Python rule sets agree. It does not change scan behaviour or speed: the runtime
uses the Python engine either way.
The native engine returns no evidence spans, and those carry the matched text, its offsets, the rule, its weight and the OWASP/CWE mapping that reports and findings are built from. While it was wired in, whether a Rust toolchain happened to be present decided whether the detection API returned evidence at all. Measured at the 8,000-character cap applied to every response, it saved roughly 6 ms per probe - not worth reimplementing echo filtering and the full span metadata in a second language and keeping 31 security rules in lockstep across both.
Windows installer build (maintainers):
powershell -File packaging/build-installer.ps1AgentArmor is released under the MIT License.