Skip to content

Commit 34c17d5

Browse files
committed
docs(readme): position @aictrl/cli as the headless agent harness with benchmark, CI workflow, and dark logo
1 parent 6c661f4 commit 34c17d5

3 files changed

Lines changed: 182 additions & 51 deletions

File tree

‎.github/logo-dark.svg‎

Lines changed: 10 additions & 0 deletions
Loading

‎README.md‎

Lines changed: 162 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,39 @@
11
<p align="center">
2-
<img src=".github/logo.svg" alt="aictrl.dev" width="240" />
2+
<picture>
3+
<source media="(prefers-color-scheme: dark)" srcset=".github/logo-dark.svg" />
4+
<source media="(prefers-color-scheme: light)" srcset=".github/logo.svg" />
5+
<img src=".github/logo.svg" alt="aictrl.dev" width="240" />
6+
</picture>
37
</p>
48

5-
<h3 align="center">Headless execution engine for AI agent skills</h3>
9+
<h3 align="center">The headless agent harness for CI, cron jobs, and container workflows</h3>
610

711
<p align="center">
8-
<a href="https://www.npmjs.com/package/@aictrl/cli"><img src="https://img.shields.io/npm/v/@aictrl/cli" alt="npm" /></a>
9-
<a href="https://github.com/aictrl-dev/cli/blob/main/LICENSE"><img src="https://img.shields.io/github/license/aictrl-dev/cli" alt="license" /></a>
12+
<a href="https://www.npmjs.com/package/@aictrl/cli"><img src="https://img.shields.io/npm/v/@aictrl/cli" alt="npm version" /></a>
13+
<a href="https://www.npmjs.com/package/@aictrl/cli"><img src="https://img.shields.io/npm/dm/@aictrl/cli" alt="npm downloads" /></a>
14+
<a href="https://github.com/aictrl-dev/cli/blob/main/LICENSE"><img src="https://img.shields.io/github/license/aictrl-dev/cli" alt="MIT license" /></a>
15+
<a href="https://aictrl.dev/agent?utm_source=github&utm_medium=readme&utm_campaign=headless-readme"><img src="https://img.shields.io/badge/docs-aictrl.dev%2Fagent-4c6ef5" alt="aictrl.dev/agent" /></a>
16+
<a href="https://aictrl.dev/blog/headless-coding-agent-cli-benchmark?utm_source=github&utm_medium=readme&utm_campaign=headless-readme"><img src="https://img.shields.io/badge/benchmark-6%20CLIs%20compared-15803d" alt="Headless CLI Benchmark" /></a>
1017
</p>
1118

1219
---
1320

14-
Aictrl is a headless, server-side runtime for autonomous AI agent workflows. It is designed for engineers who want to automate complex tasks using agentic models in CI/CD pipelines, cron jobs, or embedded within other applications.
21+
**aictrl CLI** (`@aictrl/cli`) runs an AI coding agent as one non-interactive command in CI/CD pipelines, scheduled jobs, and isolated containers.
22+
23+
Interactive coding agents are built for a person at a terminal: in automation they can hang on permission prompts nobody is watching, exit `0` after a provider error, or bury structured answers in prose. **aictrl CLI has no interactive mode.** Permission prompts are rejected immediately, every run streams a versioned NDJSON event log (`EVENTS.md`), final results are validated against your JSON Schema with corrective turns, and exit codes tell your pipeline whether to proceed, retry, or fail.
24+
25+
## Why a dedicated headless harness?
26+
27+
Engineers running terminal-first coding agents in CI and batch workflows repeatedly hit the same failure modes:
28+
29+
| What breaks in automation | How `aictrl run` handles it |
30+
| :--- | :--- |
31+
| **Silent hangs** (`claude-code-action#865`, `goose#11164`) | **No terminal UI or interactive prompts.** Any unpermitted tool request is rejected immediately; optional stream idle timeouts stop stalled model calls. |
32+
| **False `exit 0` on failure** (`claude-code#79500`, `goose#11909`, `gemini-cli#1088`) | **Distinct exit codes:** `0` accepted result, `1` provider/runtime/timeout error, `2` configuration error, `3` missing or schema-invalid result, `130`/`143` on `SIGINT`/`SIGTERM`. |
33+
| **Unparseable prose or broken JSON** (`codex#19816`, `claude-code#18536`, `gemini-cli#4665`) | **Enforced JSON Schema (`--output-schema`)** with bounded corrective turns and atomic file output (`--output-result`). No prose fallback. |
34+
| **`--dangerously-skip-permissions` in CI** | **Declarative per-tool allow/deny rules.** Tools denied for all patterns are stripped from the model request entirely, saving tokens and shrinking the attack surface. Subagents inherit caller denials. |
35+
| **Cost & token blind spots** (`claude-code#8245`, `codex#13733`) | **608-token system prompt** and a [versioned NDJSON event stream](EVENTS.md) on stdout reporting per-call token usage, cost, tool timing, and permission decisions. |
36+
| **Background auto-updates & telemetry** (`claude-code-action#1290`) | **Zero telemetry, no auto-update, built-in model catalog.** Outbound traffic goes only to the model provider, MCP servers, and web tools you configure. |
1537

1638
## Install
1739

@@ -21,35 +43,113 @@ npm i -g @aictrl/cli
2143

2244
## Quick Start
2345

24-
```bash
25-
# Run a one-off task
26-
aictrl run "analyze the security of this repository"
46+
### 1. Validated JSON output in a terminal or container
2747

28-
# Use a specific model
29-
aictrl run --model anthropic/claude-3-5-sonnet-latest "refactor the auth module"
30-
```
48+
```bash
49+
export ANTHROPIC_API_KEY=<your-key> # or OPENAI_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY, etc.
3150

32-
## Automation & Headless Usage
51+
# Stream NDJSON events and write a schema-validated result atomically
52+
aictrl run --format json \
53+
--model anthropic/claude-sonnet-4-5 \
54+
--output-schema findings.schema.json \
55+
--output-result findings.json \
56+
"Review the changes on this branch and report security or reliability bugs"
3357

34-
Aictrl is "headless first". When run in a non-TTY environment, it automatically switches to a mode optimized for automation.
58+
# Pipe input from stdin
59+
git diff origin/main...HEAD | aictrl run --model openai/gpt-5 "Summarize breaking API changes"
60+
```
3561

36-
### Stdin Piping
62+
### 2. Drop-in GitHub Actions PR review (split-secret pattern)
63+
64+
Save as `.github/workflows/aictrl-review.yml`. No step holds both the model API key and the GitHub token, and the agent runs with read-only tools (`read`, `glob`, `grep` only—no `bash`, no `write`, no repo config):
65+
66+
```yaml
67+
name: aictrl review
68+
69+
on:
70+
pull_request:
71+
types: [opened, synchronize, ready_for_review]
72+
73+
permissions:
74+
contents: read
75+
pull-requests: write
76+
77+
concurrency:
78+
group: aictrl-review-${{ github.event.pull_request.number }}
79+
cancel-in-progress: true
80+
81+
jobs:
82+
review:
83+
# Fork pull requests get no secrets; never switch this to pull_request_target.
84+
if: github.event.pull_request.head.repo.full_name == github.repository && !github.event.pull_request.draft
85+
runs-on: ubuntu-latest
86+
timeout-minutes: 15
87+
env:
88+
PR: ${{ github.event.pull_request.number }}
89+
steps:
90+
- uses: actions/checkout@v6
91+
with:
92+
persist-credentials: false
93+
94+
# 1. Fetch the diff with the GitHub token. The model key is not in this step.
95+
- run: gh pr diff "$PR" > pr.diff
96+
env:
97+
GH_TOKEN: ${{ github.token }}
98+
99+
- run: npm install -g @aictrl/[email protected]
100+
101+
# 2. Review with the model key only: no GitHub token, no shell, read-only tools,
102+
# and no config or plugins from the pull request's .aictrl directory.
103+
- name: Review
104+
env:
105+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
106+
MODEL: ${{ vars.AICTRL_MODEL }} # e.g. anthropic/claude-sonnet-4-5
107+
AICTRL_DISABLE_PROJECT_CONFIG: "true"
108+
AICTRL_MODEL_STREAM_IDLE_TIMEOUT_MS: "300000"
109+
AICTRL_CONFIG_CONTENT: >-
110+
{"permission": {"*": "deny", "read": "allow", "glob": "allow", "grep": "allow"}}
111+
run: |
112+
aictrl run --format json --model "$MODEL" \
113+
"Review the change in pr.diff. Read files in this repository for context.
114+
Report bugs, security and reliability problems with file and line; skip style.
115+
Reply with the review only, in Markdown." > aictrl-events.jsonl
116+
jq -rs '[.[] | select(.type == "text") | .part.text] | last // empty' aictrl-events.jsonl > review.md
117+
test -s review.md
118+
119+
# 3. Post with the GitHub token. The model key is not in this step.
120+
- run: gh pr comment "$PR" --body-file review.md --edit-last --create-if-none
121+
env:
122+
GH_TOKEN: ${{ github.token }}
123+
124+
- uses: actions/upload-artifact@v4
125+
if: always()
126+
with:
127+
name: aictrl-events
128+
path: aictrl-events.jsonl
129+
retention-days: 7
130+
```
37131
38-
You can pipe content directly into `aictrl`. This is useful for processing logs, code, or command output.
132+
## Headless CLI Benchmark (6 Harnesses Compared)
39133
40-
```bash
41-
cat logs.txt | aictrl run "summarize these errors"
42-
```
134+
We benchmarked six coding-agent CLIs (`aictrl 0.5.4`, `pi 1.3.0`, `Goose 1.55.0`, `Codex 0.163.0`, `OpenCode 2.1.0`, and `Hermes v0.22.2`) in fresh 2-CPU Docker containers against a deterministic 50 ms local stub model to isolate harness overhead from remote LLM latency. Full runner, raw JSON captures, and methodology are open in [`aictrl-dev/skill-md-research`](https://github.com/aictrl-dev/skill-md-research/tree/main/experiments/headless-cli-benchmark) and analyzed in the [benchmark post](https://aictrl.dev/blog/headless-coding-agent-cli-benchmark?utm_source=github&utm_medium=readme&utm_campaign=headless-readme).
43135

44-
### JSON Output
136+
| Metric (lower is better, except coverage) | **aictrl 0.5.4** | pi 1.3.0 | Goose 1.55.0 | Codex 0.163.0 | OpenCode 2.1.0 | Hermes 0.22.2 |
137+
| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
138+
| **First response, median** (ms) | 579 | 522 | **392** (1st) | 471 (2nd) | 926 | 5,521 |
139+
| **50-read session time, median** (s) | **4.27** (2nd) | **3.98** (1st) | 7.22 | 5.57 | 6.03 | 12.76 |
140+
| **Harness time per turn, median** (ms) | **12** (2nd) | **9** (1st) | 72 | 45 | 32 | 64 |
141+
| **Peak cgroup memory, median** (MiB) | **207** (2nd) | **154** (1st) | 247 | 132\* | 424 | 344 |
142+
| **Container image increment** (MiB) | **113** (2nd) | **108** (1st) | 292 | 410 | 818 | 388 |
143+
| **System prompt** (`o200k_base` tokens) | 608 | 601 (2nd) | **563** (1st) | 4,047 | 1,857 | 1,053 |
144+
| **First request total** (tokens, default tools) | 3,442 (9 tools) | **1,276** (4 tools) | 1,577 (6 tools) | 8,178 (9 tools) | 4,903 (12 tools) | 3,391 (8 tools) |
145+
| **Short task, 4 calls** (tokens, default tools) | 15,209 (9,805 w/ 4 tools) | **6,074** (1st) | 7,281 (2nd) | 34,078 | 21,489 | 15,562 |
146+
| **NDJSON event stream coverage** (25 facts) | **100%** | **100%** | 68% | 40% | 80% | 72% |
45147

46-
For programmatic consumption, use `--format json` to get raw events.
148+
*\*Codex's shell reader truncates each 800-line file to its first and last ~80 lines, holding ~1/5th of the file in context. `pi`, `Goose`, and `Hermes` ran without a subagent tool in the tested profile; restricting `aictrl` to `pi`'s 4 tools via permission config drops its first request from 3,442 to 2,091 tokens.*
47149

48-
```bash
49-
aictrl run --format json "review this PR" | jq '.type'
50-
```
150+
## Core Contracts for Automation
51151

52-
### Validated Final Results
152+
### Validated Final Results (`--output-schema`)
53153

54154
`aictrl run --output-schema <file>` validates the final JSON result.
55155
Use `--output-schema-retries <n>` for additional corrective turns (0–10, default 2)
@@ -66,21 +166,24 @@ provider failures and stream timeouts use **1**, signals use **130**/**143**.
66166
See the [full contract](packages/cli/README.md#validated-final-results)
67167
and [JSON event reference](EVENTS.md).
68168

69-
### Non-Interactive Execution
169+
### Versioned NDJSON Event Stream (`--format json`)
70170

71-
In headless mode, Aictrl automatically rejects all interactive permission requests, ensuring your pipelines never hang.
171+
Passing `--format json` writes one JSON event per line to stdout (`schemaVersion: "1"`), covering:
72172
73-
### Skill Scope
173+
- **Lifecycle:** `invocation_start`, `session_start`, `retry`, `session_error`, `invocation_error`, `invocation_complete`
174+
- **Model & cost accounting:** `message_complete` and `subagent_message_complete` with input/output/reasoning/cache tokens, cost, and context-window utilization
175+
- **Tools & permissions:** `tool_catalog`, `permission_granted`, `permission_rejected`, `tool_use` (with start/end timestamps), `skill_discovered`, `skill_loaded`
176+
- **Subagents & results:** `subagent_start`, `subagent_complete`, `structured_output`, `structured_output_rejected`
74177

75-
`aictrl run` does not list skills from `~/.claude`, `~/.agents` or `~/.opencode`;
76-
list them explicitly in config `skills.paths` (e.g. `["~/.claude/skills"]`) to use
77-
them. Project skills, the global aictrl
78-
config directories (`~/.config/aictrl`, `~/.aictrl`) and config
79-
`skills.paths`/`skills.urls` are still listed. Pass `--skill <name>` (repeatable)
80-
or set config `skills.allow` to list and load only the named skills.
81-
See the [full contract](packages/cli/README.md#skill-scope).
178+
```bash
179+
aictrl run --format json "review this PR" | jq -c '{type, timestamp, tool: .part.tool}'
180+
```
181+
182+
See [EVENTS.md](EVENTS.md) for the complete schema specification.
82183

83-
### Restricting Dangerous Commands
184+
### Per-Tool Permissions & Restricting Dangerous Commands
185+
186+
In headless mode, `aictrl` automatically rejects any action that would prompt a human. You can scope permissions per tool or command pattern in `aictrl.json` or inline via `AICTRL_CONFIG_CONTENT`. Any tool denied for all patterns (e.g. `"webfetch": "deny"`, `"task": "deny"`) is omitted from the model request, saving tool-definition tokens on every turn.
84187

85188
Tool descriptions describe tools; they are not a security boundary. Block
86189
dangerous commands with `permission.bash` rules, for example denying force
@@ -89,6 +192,16 @@ which also blocks harmless `-c` uses). The CLI ships no such rules by default. S
89192
[configuration reference](packages/cli/README.md#restricting-dangerous-commands)
90193
for an example.
91194

195+
### Skill Scope (`--skill`)
196+
197+
`aictrl run` does not list skills from `~/.claude`, `~/.agents` or `~/.opencode`;
198+
list them explicitly in config `skills.paths` (e.g. `["~/.claude/skills"]`) to use
199+
them. Project skills, the global aictrl
200+
config directories (`~/.config/aictrl`, `~/.aictrl`) and config
201+
`skills.paths`/`skills.urls` are still listed. Pass `--skill <name>` (repeatable)
202+
or set config `skills.allow` to list and load only the named skills.
203+
See the [full contract](packages/cli/README.md#skill-scope).
204+
92205
### Model Stream Idle Timeout
93206

94207
Model stream idle timeouts are disabled by default. Set
@@ -100,21 +213,20 @@ resets after every stream event, so responses that keep making progress are unaf
100213
Tool execution (local or provider-executed) uses a ceiling twelve times the
101214
configured model timeout, capped at 2147483647 ms (one hour for a five-minute timeout).
102215

103-
## MCP & Custom Tools
216+
### Models, MCP & Custom Tools
217+
218+
`aictrl` ships a built-in provider catalog supporting **Anthropic**, **OpenAI**, **Google**, **Amazon Bedrock**, **Google Vertex**, **Azure**, **OpenRouter**, and any OpenAI-compatible local or proxy endpoint (`--model provider/model`).
104219

105-
Aictrl supports the [Model Context Protocol (MCP)](https://modelcontextprotocol.io).
220+
It also supports the [Model Context Protocol (MCP)](https://modelcontextprotocol.io) and custom TypeScript tools in `.aictrl/tool/`:
106221

107222
```bash
108-
# Add an MCP server
223+
# Register an MCP server
109224
aictrl mcp add my-tool --url http://localhost:8080
110-
111-
# Add custom TypeScript tools
112-
# Just drop them in .aictrl/tool/
113225
```
114226

115227
## Programmatic SDK
116228

117-
Embed Aictrl directly into your TypeScript applications.
229+
Embed `aictrl` into TypeScript services via [`@aictrl/sdk`](packages/sdk):
118230

119231
```typescript
120232
import { createAictrlClient } from "@aictrl/sdk"
@@ -124,14 +236,17 @@ const client = createAictrlClient({
124236
})
125237
126238
const session = await client.session.create({
127-
title: "My Automation Task",
239+
title: "Automated PR Triage",
128240
})
129241
```
130242

131-
## Attribution
243+
## Documentation & Links
132244

133-
Aictrl is a fork of the [OpenCode](https://opencode.ai) project and is licensed under the MIT License.
245+
- **[Headless Agent Harness Overview (`aictrl.dev/agent`)](https://aictrl.dev/agent?utm_source=github&utm_medium=readme&utm_campaign=headless-readme)** — architecture principles, interactive telemetry explorer, and CI patterns
246+
- **[Headless Coding-Agent CLI Benchmark](https://aictrl.dev/blog/headless-coding-agent-cli-benchmark?utm_source=github&utm_medium=readme&utm_campaign=headless-readme)** — 6 CLIs measured on startup, turn overhead, memory, image footprint, and tokens
247+
- **[NDJSON Event Stream Reference (`EVENTS.md`)](EVENTS.md)** — full specification of all versioned stdout events
248+
- **[CLI Package & Configuration Reference (`packages/cli/README.md`)](packages/cli/README.md)** — detailed `--output-schema`, `--skill`, and permission contracts
134249

135-
---
250+
## License
136251

137-
[aictrl.dev](https://aictrl.dev/?utm_medium=referral&utm_source=github&utm_campaign=cli&utm_content=readme)
252+
MIT License — see [LICENSE](LICENSE).

‎packages/cli/README.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,18 @@
11
# @aictrl/cli
22

3-
Headless execution engine for AI agent skills.
3+
The headless agent harness for CI, cron jobs, and container workflows.
4+
5+
- **[Overview & CI Quickstart (`aictrl.dev/agent`)](https://aictrl.dev/agent?utm_source=npmjs&utm_medium=readme&utm_campaign=headless-readme)**
6+
- **[Headless CLI Benchmark (6 CLIs compared)](https://aictrl.dev/blog/headless-coding-agent-cli-benchmark?utm_source=npmjs&utm_medium=readme&utm_campaign=headless-readme)**
7+
- **[NDJSON Event Reference (`EVENTS.md`)](https://github.com/aictrl-dev/cli/blob/main/EVENTS.md)**
48

59
## Features
610

7-
- **Headless First:** Designed for automation, CI/CD, and server-side runtimes.
8-
- **MCP Protocol:** First-class support for [Model Context Protocol](https://modelcontextprotocol.io).
9-
- **Custom Tools:** Easy extensibility via TypeScript files.
11+
- **Headless Only:** Built for non-interactive runs in CI/CD, scheduled jobs, and containers—no terminal UI, no hanging permission prompts, and distinct exit codes (`0` accepted, `1` provider/runtime/timeout error, `2` config error, `3` missing/invalid schema result).
12+
- **Schema-Validated Results:** Pass `--output-schema` and `--output-result` to enforce a JSON Schema with corrective turns and atomic file output.
13+
- **Versioned NDJSON Event Stream:** `--format json` streams every lifecycle step, tool call, permission decision, and per-call token/cost count to stdout.
14+
- **Declarative Permissions:** Per-tool allow/deny rules (`permission` config); tools denied everywhere are stripped from the model request to save tokens.
15+
- **Skills & MCP:** Progressive skill loading (`--skill`) and first-class [Model Context Protocol](https://modelcontextprotocol.io) support.
1016

1117
## Automation Usage
1218

0 commit comments

Comments
 (0)