Summary
Built a security evaluation harness for OpenClaw agents with 270+ attack payloads. Now integrated with AgentTinman's
FailureClassifier for deep analysis.
What it does
- 270+ attack probes across 13 categories
- Synthetic Gateway for isolated testing
- CI integration via SARIF, JUnit, and JSON outputs
- Baseline assertions for regression testing
- Real-time monitoring via Gateway WebSocket
Attack Categories
| Category |
Probes |
Description |
| Prompt Injection |
15 |
Jailbreaks, DAN, instruction override, prompt leaking |
| Tool Exfiltration |
42 |
SSH keys, cloud creds, supply-chain tokens, crypto wallets |
| Context Bleed |
14 |
Cross-session leaks, memory extraction |
| Privilege Escalation |
15 |
Sandbox escape, elevation bypass |
| Supply Chain |
18 |
Malicious skills, dependency attacks |
| Financial |
26 |
Crypto wallets (BTC, ETH, SOL, Base), transactions, exchange APIs |
| Unauthorized Action |
28 |
Actions without consent, implicit execution |
| MCP Attacks |
20 |
MCP tool abuse, server injection, cross-MCP exfil |
| Indirect Injection |
20 |
Injection via files, URLs, documents, configs |
| Evasion Bypass |
30 |
Unicode bypass, URL/base64/hex encoding, shell injection |
| Memory Poisoning |
25 |
Context injection, RAG poisoning, history fabrication |
| Platform Specific |
35 |
Windows (mimikatz, schtasks, PowerShell), macOS (LaunchAgents), Linux (systemd), cloud metadata |
CI Integration
Outputs SARIF (GitHub Code Scanning), JUnit, and JSON formats.
# Example GitHub Actions integration
- run: pip install tinman-openclaw-eval
- run: tinman-eval run --output security-report.sarif --format sarif
- uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: security-report.sarif
Links
- PyPI: https://pypi.org/project/tinman-openclaw-eval/
- GitHub: https://github.com/oliveskin/tinman-openclaw-eval
- Skill Pack: https://clawhub.ai/oliveskin/agent-tinman
- AgentTinman: https://github.com/oliveskin/Agent-Tinman
Install
pip install tinman-openclaw-eval
tinman-eval run
Happy to discuss integration or contribute a PR for CI workflow.
Summary
Built a security evaluation harness for OpenClaw agents with 270+ attack payloads. Now integrated with AgentTinman's
FailureClassifier for deep analysis.
What it does
Attack Categories
CI Integration
Outputs SARIF (GitHub Code Scanning), JUnit, and JSON formats.