Skip to content

Commit dac1821

Browse files
authored
Merge ef492d7 into 46f98d5
2 parents 46f98d5 + ef492d7 commit dac1821

21 files changed

Lines changed: 2466 additions & 11 deletions

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ on:
44
push:
55
branches: [main]
66
pull_request:
7-
branches: [main]
87

98
permissions:
109
contents: read
@@ -40,5 +39,29 @@ jobs:
4039
sudo apt-get install --no-install-recommends -y ripgrep
4140
rg --version
4241
42+
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
43+
with:
44+
python-version: "3.12"
45+
46+
# Setup may download pinned wheels/ranks. Capture and accounting are offline.
47+
- name: Provision reference tokenizer
48+
run: python3 -I -B script/token-budget/setup.py --directory "$RUNNER_TEMP/token-budget-environment"
49+
50+
- name: Enforce actual SDK prefix token budgets
51+
run: '"$RUNNER_TEMP/token-budget-environment/venv/bin/python3" -I -B script/token-budget/run.py --output "$RUNNER_TEMP/token-budget-evidence"'
52+
53+
- name: Verify token-budget fault controls
54+
run: '"$RUNNER_TEMP/token-budget-environment/venv/bin/python3" -I -B -m unittest discover -s script/token-budget -p "test_*.py" -v'
55+
env:
56+
TOKEN_BUDGET_TEST_CAMPAIGN: ${{ runner.temp }}/token-budget-evidence
57+
58+
- name: Retain token-budget captures
59+
if: always()
60+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
61+
with:
62+
name: token-budget-evidence
63+
path: ${{ runner.temp }}/token-budget-evidence
64+
if-no-files-found: warn
65+
4366
- name: Run Tests
4467
run: bun turbo test --concurrency=100%

‎RELEASING.md‎

Lines changed: 32 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,29 @@
1010
bun test test/cli/run-mcp-discovery.test.ts test/mcp/discovery-recovery.test.ts test/cli/publish-workflow.test.ts test/cli/run-provider-finish.test.ts test/cli/run-signal-cancellation.test.ts test/cli/classify-session-error.test.ts test/session/idle.test.ts test/session/processor-idle.test.ts
1111
```
1212

13+
Before publication, run the separate current-versus-previous native release
14+
comparison. Prepare and review explicit candidate and previous published binary
15+
artifacts, their SHA256/version/provenance pins and one existing immutable
16+
fixture image using [the manifest contract](script/token-budget/release-fixtures/README.md).
17+
Provision the pinned Python counter in a fresh directory (requires Python3.12
18+
and network access during setup only), then capture and count offline:
19+
20+
```bash
21+
python3.12 -I -B script/token-budget/setup.py --directory /absolute/new-counter-env
22+
bun run --no-install --no-env-file script/token-budget/release.ts /absolute/release-inputs.json /absolute/new-release-evidence
23+
/absolute/new-counter-env/venv/bin/python3 -I -B script/token-budget/release-count.py /absolute/new-release-evidence/campaign.json
24+
```
25+
26+
Run these commands from the repository root. Require valid current-default and
27+
previous-default task runs, raw requests/responses and actual Read-result
28+
evidence, verified versions/binary hashes, successful cleanup and
29+
`release-comparison.json`. Review first-input and session input/output/total
30+
deltas before publication and record any accepted regression with rationale.
31+
Minimal arms are measured only when each actual binary advertises `--tools`;
32+
an unsupported previous arm must be labeled UNSUPPORTED, never synthesized
33+
through permission settings. This scripted reference-token gate is separate
34+
from source CI token budgets and from provider billing or model-quality claims.
35+
1336
Wait for CI build, workspace typecheck and tests to pass and address reviews.
1437

1538
3. Merge the approved PR and record the resulting `main` commit SHA. Confirm
@@ -43,15 +66,15 @@
4366
publish workflow publishes each as a platform package. It then lists them in
4467
the wrapper's `optionalDependencies`:
4568

46-
| Package | `os` | `cpu` | `libc` |
47-
| --- | --- | --- | --- |
48-
| `@aictrl/cli-linux-x64-baseline` | linux | x64 | glibc |
49-
| `@aictrl/cli-linux-x64-baseline-musl` | linux | x64 | musl |
50-
| `@aictrl/cli-linux-arm64` | linux | arm64 | glibc |
51-
| `@aictrl/cli-linux-arm64-musl` | linux | arm64 | musl |
52-
| `@aictrl/cli-darwin-x64-baseline` | darwin | x64 | |
53-
| `@aictrl/cli-darwin-arm64` | darwin | arm64 | |
54-
| `@aictrl/cli-windows-x64-baseline` | win32 | x64 | |
69+
| Package | `os` | `cpu` | `libc` |
70+
| ------------------------------------- | ------ | ----- | ------ |
71+
| `@aictrl/cli-linux-x64-baseline` | linux | x64 | glibc |
72+
| `@aictrl/cli-linux-x64-baseline-musl` | linux | x64 | musl |
73+
| `@aictrl/cli-linux-arm64` | linux | arm64 | glibc |
74+
| `@aictrl/cli-linux-arm64-musl` | linux | arm64 | musl |
75+
| `@aictrl/cli-darwin-x64-baseline` | darwin | x64 | |
76+
| `@aictrl/cli-darwin-arm64` | darwin | arm64 | |
77+
| `@aictrl/cli-windows-x64-baseline` | win32 | x64 | |
5578

5679
npm installs only the package that matches the host. Every x64 package is the
5780
baseline (no AVX2) build: Bun ships the same binary for its baseline and AVX2

‎packages/cli/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ Aictrl is built to be used in non-interactive environments.
104104
### Stdin Piping
105105

106106
```bash
107-
cat diff.txt | aictrl run "review this code change"
107+
cat diff.txt | aictrl run "review this code change" -
108108
```
109109

110110
### JSON Events

‎script/token-budget/README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Fixed SDK prefix token guard (Linux)
2+
3+
This guard starts the public CLI twice per configuration with fresh HOME/XDG state and receives its actual OpenAI-compatible SDK request at a loopback endpoint. It verifies the complete nine-tool default catalog and the explicit `--tools read,bash,edit,write` four-tool catalog. It does not import tests, reconstruct production schemas, invoke a paid model, or replace the existing 15,000-character Qwen guard.
4+
5+
Prerequisites: Linux x86_64, Python 3.12 with `venv`/`pip`, Bun from `package.json`, real `rg` and Git on PATH, and the repository's installed frozen Bun dependencies and built SDK. Missing prerequisites fail explicitly.
6+
7+
```sh
8+
# Online setup only: creates a NEW owned directory, pinned official PyPI wheels,
9+
# and checksum-verified OpenAI o200k rank data. It never installs JS packages.
10+
python3.12 -I -B script/token-budget/setup.py --directory /tmp/my-token-environment
11+
12+
# Offline actual CLI capture and accounting; output must be a NEW directory.
13+
/tmp/my-token-environment/venv/bin/python3 -I -B script/token-budget/run.py \
14+
--output /tmp/my-token-evidence
15+
```
16+
17+
The guard uses an exclusive nonblocking lock in the owner-only `/tmp/aictrl-token-budget` directory. Concurrent invocations fail before changing the fixed workspace. Each cold child runs in the literal `/tmp/aictrl-token-budget/workspace`, newly initialized as a Git repository, with UTC and Date fixed before production imports to 21 October 2026. This controls all three raw cwd occurrences, including the two in Bash descriptions/schema. No text normalization is applied. Existing unowned, shared, or symlink fixture directories are rejected. The directory/lock remain for safe reuse; raw captures remain in the requested evidence directory.
18+
19+
The before-import guard observes `globalThis.fetch` calls in the CLI process and rejects origins other than the loopback fixture. It rejects `fetch.preconnect` and direct `Bun.spawn` calls whose executable is `bun` and whose next argument is exactly `add` or `install`. This ledger is partial: other network APIs, `Bun.spawnSync`, Node child-process APIs, shell-launched programs, and package-manager aliases are outside those controls. A clean ledger establishes no rejected operation on the instrumented paths; it does not establish complete network or installation accounting. Child environments are explicitly allowlisted, without provider credentials or inherited project configuration. These trusted fixture controls are not an OS network sandbox.
20+
21+
The real CLI has a mandatory 15-second deadline; each outer capture has 25 seconds. A fresh forked Linux subreaper owns only that invocation's capture and adopted descendants, including descendants that create a new session. After the capture parent exits, already-dead children are reaped without being classified as live survivors. Live adopted children are terminated and reaped, and the capture is rejected even if its parent exited successfully. Cleanup repeatedly handles newly adopted children and fails if it cannot verify completion within its cleanup deadline. The caller's other children are outside this ownership scope. The result records the supervisor and descendant cleanup evidence; the legacy `remaining_group_killed` field now indicates any observed live adopted descendant, not just the original process group.
22+
23+
Evidence output must be a new directory under an existing parent. Its root and per-run directories are owner-only (`0700`), so raw fixture requests and allowlisted environment records are not exposed by an ambient permissive umask. Existing output directories are never overwritten. The fixed workspace directory uses the same ownership/private-directory validation in production and lock controls.
24+
25+
`count.py` verifies the pinned tokenizer version, exact rank SHA256, raw environment, catalog, single request and the recorded process/partial-fetch-ledger evidence. `reference.py` denies socket/DNS use during accounting. The enforced metric is `o200k_base` encoding of compact sorted Unicode JSON containing actual SDK system-role messages, serialized tools, and instructions when present. The fixture user text is excluded. The full first model input is separately reported using `instructions/messages/input/tools/response_format/text`; transport controls are excluded. System/tool diagnostic counts use separate tokenizations and are not additive BPE contributions.
26+
27+
`budgets.json` contains two fixed explicit ceilings. Raising either is a reviewed source edit requiring raw evidence and an explanation. `--observe` records an evaluation without enforcing ceilings and is never used by CI. `--source` selects an isolated source copy for QA source-growth controls; CI always uses the repository entry. The default two cold repetitions must produce identical canonical prefix bytes for each arm.
28+
29+
CI provisions the Python runtime, lock and rank data before capture; capture/accounting do not download. Raw requests, canonical prefixes, network/exit/deadline logs and the complete reference report are retained as `token-budget-evidence`, including when enforcement fails. These counts are reference overhead measurements, not provider billing, cache savings, quality or performance estimates. Release comparisons have a separate current-versus-previous protocol; these two selector arms do not substitute for that requirement.
30+
31+
Initial portable observation: default **3,441** prefix tokens and selected **2,001**, at the controlled path above (15,453 and 8,973 canonical characters). Both raw request bodies and canonical prefixes were byte-identical across two cold captures per arm. Initial ceilings equal those observed counts with **zero headroom**: deterministic prompt/tool growth should require an explicit reviewed explanation rather than consume a hidden allowance. Full first model input was separately 3,457/2,017. The longer earlier evaluation cwd produced different counts and was not used as a ceiling.

‎script/token-budget/budgets.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"default": 3441,
3+
"four": 2001
4+
}

‎script/token-budget/capture.ts‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import path from "node:path"
2+
import fs from "node:fs/promises"
3+
import { guard } from "./guard"
4+
const out = process.env.TOKEN_BUDGET_OUTPUT
5+
const work = process.env.TOKEN_BUDGET_WORKSPACE
6+
const source = process.env.TOKEN_BUDGET_SOURCE
7+
if (!out || !work || !source) throw new Error("Explicit output/workspace/source required")
8+
const binpath = [path.dirname(process.execPath), "/usr/local/bin", "/usr/bin", "/bin"].join(path.delimiter)
9+
const rg = Bun.which("rg", { PATH: binpath })
10+
if (!rg) throw new Error("Real ripgrep prerequisite missing")
11+
const requests: string[] = []
12+
const server = Bun.serve({
13+
hostname: "127.0.0.1",
14+
port: 0,
15+
idleTimeout: 0,
16+
async fetch(request) {
17+
if (new URL(request.url).pathname !== "/v1/chat/completions") throw new Error("Unexpected endpoint " + request.url)
18+
if (request.method !== "POST") throw new Error("Expected POST")
19+
requests.push(await request.text())
20+
const chunks = [
21+
{
22+
id: "eval",
23+
object: "chat.completion.chunk",
24+
choices: [{ index: 0, delta: { role: "assistant", content: "Fixture complete" } }],
25+
},
26+
{ id: "eval", object: "chat.completion.chunk", choices: [{ index: 0, delta: {}, finish_reason: "stop" }] },
27+
]
28+
return new Response(
29+
chunks
30+
.map((chunk) => "data: " + JSON.stringify(chunk))
31+
.concat("data: [DONE]")
32+
.join("\n\n") + "\n\n",
33+
{ headers: { "content-type": "text/event-stream" } },
34+
)
35+
},
36+
})
37+
await fs.mkdir(out, { recursive: true })
38+
await Bun.write(path.join(out, "guard.ts"), guard)
39+
await Bun.write(path.join(out, "network.jsonl"), "")
40+
const config = {
41+
enabled_providers: ["fixture"],
42+
provider: {
43+
fixture: {
44+
npm: "@ai-sdk/openai-compatible",
45+
options: { apiKey: "fixture", baseURL: server.url.origin + "/v1" },
46+
models: { fixture: { name: "fixture", tool_call: true, limit: { context: 200000, output: 1000 } } },
47+
},
48+
},
49+
agent: { title: { disable: true } },
50+
permission: { "*": "allow" },
51+
}
52+
const command = [
53+
process.execPath,
54+
"run",
55+
"--no-install",
56+
"--no-env-file",
57+
"--preload",
58+
path.join(import.meta.dir, "clock.js"),
59+
"--preload",
60+
path.join(out, "guard.ts"),
61+
"--conditions=browser",
62+
source,
63+
"run",
64+
"--model",
65+
"fixture/fixture",
66+
"--agent",
67+
"build",
68+
"--format",
69+
"json",
70+
...(process.env.TOKEN_BUDGET_ARM === "four" ? ["--tools", "read,bash,edit,write"] : []),
71+
"Say Fixture complete without tools.",
72+
]
73+
const env = {
74+
PATH: binpath,
75+
SHELL: "/bin/bash",
76+
TZ: "UTC",
77+
HOME: path.join(out, "home"),
78+
AICTRL_TEST_HOME: path.join(out, "home"),
79+
AICTRL_TEST_MANAGED_CONFIG_DIR: path.join(out, "managed"),
80+
XDG_DATA_HOME: path.join(out, "data"),
81+
XDG_CONFIG_HOME: path.join(out, "config"),
82+
XDG_STATE_HOME: path.join(out, "state"),
83+
XDG_CACHE_HOME: path.join(out, "cache"),
84+
BUN_RUNTIME_TRANSPILER_CACHE_PATH: path.join(out, "bun-cache"),
85+
GIT_CONFIG_NOSYSTEM: "1",
86+
GIT_CONFIG_GLOBAL: "/dev/null",
87+
AICTRL_DISABLE_PROJECT_CONFIG: "true",
88+
AICTRL_DISABLE_DEFAULT_PLUGINS: "true",
89+
AICTRL_DISABLE_AUTOCOMPACT: "true",
90+
AICTRL_MODEL_STREAM_IDLE_TIMEOUT_MS: "3000",
91+
AICTRL_CONFIG_CONTENT: JSON.stringify(config),
92+
SELECTOR_FIXTURE_ORIGIN: server.url.origin,
93+
SELECTOR_FIXTURE_GUARD_LOG: path.join(out, "network.jsonl"),
94+
}
95+
const child = Bun.spawn(command, { cwd: work, env, stdin: "ignore", stdout: "pipe", stderr: "pipe" })
96+
let deadline = false
97+
const timer = setTimeout(() => {
98+
deadline = true
99+
child.kill("SIGKILL")
100+
}, 15000)
101+
try {
102+
const [stdout, stderr, exit] = await Promise.all([
103+
new Response(child.stdout).text(),
104+
new Response(child.stderr).text(),
105+
child.exited,
106+
])
107+
clearTimeout(timer)
108+
await Bun.write(path.join(out, "stdout.jsonl"), stdout)
109+
await Bun.write(path.join(out, "stderr.log"), stderr)
110+
for (const [index, body] of requests.entries()) await Bun.write(path.join(out, `request-${index}.json`), body)
111+
const network = (await Bun.file(path.join(out, "network.jsonl")).text())
112+
.split("\n")
113+
.filter(Boolean)
114+
.map((line) => JSON.parse(line))
115+
const result = { exit, deadline, command, environment: env, rg, requests: requests.length, network }
116+
await Bun.write(path.join(out, "result.json"), JSON.stringify(result, null, 2) + "\n")
117+
if (deadline || exit !== 0 || requests.length !== 1 || network.some((row) => !row.allowed))
118+
throw new Error(
119+
"Capture failed: " + JSON.stringify({ exit, deadline, requests: requests.length, network }) + stderr.slice(-2000),
120+
)
121+
const events = stdout
122+
.split("\n")
123+
.filter(Boolean)
124+
.map((line) => JSON.parse(line))
125+
if (
126+
events
127+
.filter((event) => event.type === "text")
128+
.map((event) => event.part?.text)
129+
.join("") !== "Fixture complete"
130+
)
131+
throw new Error("Exact assistant response missing")
132+
if (
133+
events.filter((event) => event.type === "invocation_complete" && event.status === "completed").length !== 1 ||
134+
events.some((event) => event.type === "error" || event.type === "tool_use")
135+
)
136+
throw new Error("Unexpected invocation lifecycle")
137+
} catch (error) {
138+
console.error(String(error))
139+
process.exitCode = 1
140+
} finally {
141+
clearTimeout(timer)
142+
child.kill("SIGKILL")
143+
server.stop(true)
144+
}

‎script/token-budget/clock.js‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
const epoch = 1792584000000 // 2026-10-21T12:00:00.000Z
2+
const NativeDate = Date
3+
globalThis.Date = new Proxy(NativeDate, {
4+
construct(target, args, next) {
5+
return Reflect.construct(target, args.length ? args : [epoch], next)
6+
},
7+
apply() {
8+
return new NativeDate(epoch).toString()
9+
},
10+
get(target, key, receiver) {
11+
return key === "now" ? () => epoch : Reflect.get(target, key, receiver)
12+
},
13+
})

0 commit comments

Comments
 (0)