Skip to content

Commit ddda435

Browse files
committed
test: enforce headless token budgets and release comparisons
1 parent 9088fa9 commit ddda435

19 files changed

Lines changed: 1981 additions & 9 deletions

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,5 +37,29 @@ jobs:
3737
sudo apt-get install --no-install-recommends -y ripgrep
3838
rg --version
3939
40+
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
41+
with:
42+
python-version: "3.12"
43+
44+
# Setup may download pinned wheels/ranks. Capture and accounting are offline.
45+
- name: Provision reference tokenizer
46+
run: python3 -I -B script/token-budget/setup.py --directory "$RUNNER_TEMP/token-budget-environment"
47+
48+
- name: Enforce actual SDK prefix token budgets
49+
run: '"$RUNNER_TEMP/token-budget-environment/venv/bin/python3" -I -B script/token-budget/run.py --output "$RUNNER_TEMP/token-budget-evidence"'
50+
51+
- name: Verify token-budget fault controls
52+
run: '"$RUNNER_TEMP/token-budget-environment/venv/bin/python3" -I -B -m unittest discover -s script/token-budget -p "test_*.py" -v'
53+
env:
54+
TOKEN_BUDGET_TEST_CAMPAIGN: ${{ runner.temp }}/token-budget-evidence
55+
56+
- name: Retain token-budget captures
57+
if: always()
58+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
59+
with:
60+
name: token-budget-evidence
61+
path: ${{ runner.temp }}/token-budget-evidence
62+
if-no-files-found: warn
63+
4064
- name: Run Tests
4165
run: bun turbo test --concurrency=100%

‎RELEASING.md‎

Lines changed: 32 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,29 @@
1010
bun test test/cli/run-mcp-discovery.test.ts test/mcp/discovery-recovery.test.ts test/cli/publish-workflow.test.ts test/cli/run-provider-finish.test.ts test/cli/run-signal-cancellation.test.ts test/cli/classify-session-error.test.ts test/session/idle.test.ts test/session/processor-idle.test.ts
1111
```
1212

13+
Before publication, run the separate current-versus-previous native release
14+
comparison. Prepare and review explicit candidate and previous published binary
15+
artifacts, their SHA256/version/provenance pins and one existing immutable
16+
fixture image using [the manifest contract](script/token-budget/release-fixtures/README.md).
17+
Provision the pinned Python counter in a fresh directory (requires Python3.12
18+
and network access during setup only), then capture and count offline:
19+
20+
```bash
21+
python3.12 script/token-budget/setup.py --directory /absolute/new-counter-env
22+
bun run --no-install --no-env-file script/token-budget/release.ts /absolute/release-inputs.json /absolute/new-release-evidence
23+
/absolute/new-counter-env/venv/bin/python3 -I -B script/token-budget/release-count.py /absolute/new-release-evidence/campaign.json
24+
```
25+
26+
Run these commands from the repository root. Require valid current-default and
27+
previous-default task runs, raw requests/responses and actual Read-result
28+
evidence, verified versions/binary hashes, successful cleanup and
29+
`release-comparison.json`. Review first-input and session input/output/total
30+
deltas before publication and record any accepted regression with rationale.
31+
Minimal arms are measured only when each actual binary advertises `--tools`;
32+
an unsupported previous arm must be labeled UNSUPPORTED, never synthesized
33+
through permission settings. This scripted reference-token gate is separate
34+
from source CI token budgets and from provider billing or model-quality claims.
35+
1336
Wait for CI build, workspace typecheck and tests to pass and address reviews.
1437

1538
3. Merge the approved PR and record the resulting `main` commit SHA. Confirm
@@ -43,15 +66,15 @@
4366
publish workflow publishes each as a platform package. It then lists them in
4467
the wrapper's `optionalDependencies`:
4568

46-
| Package | `os` | `cpu` | `libc` |
47-
| --- | --- | --- | --- |
48-
| `@aictrl/cli-linux-x64-baseline` | linux | x64 | glibc |
49-
| `@aictrl/cli-linux-x64-baseline-musl` | linux | x64 | musl |
50-
| `@aictrl/cli-linux-arm64` | linux | arm64 | glibc |
51-
| `@aictrl/cli-linux-arm64-musl` | linux | arm64 | musl |
52-
| `@aictrl/cli-darwin-x64-baseline` | darwin | x64 | |
53-
| `@aictrl/cli-darwin-arm64` | darwin | arm64 | |
54-
| `@aictrl/cli-windows-x64-baseline` | win32 | x64 | |
69+
| Package | `os` | `cpu` | `libc` |
70+
| ------------------------------------- | ------ | ----- | ------ |
71+
| `@aictrl/cli-linux-x64-baseline` | linux | x64 | glibc |
72+
| `@aictrl/cli-linux-x64-baseline-musl` | linux | x64 | musl |
73+
| `@aictrl/cli-linux-arm64` | linux | arm64 | glibc |
74+
| `@aictrl/cli-linux-arm64-musl` | linux | arm64 | musl |
75+
| `@aictrl/cli-darwin-x64-baseline` | darwin | x64 | |
76+
| `@aictrl/cli-darwin-arm64` | darwin | arm64 | |
77+
| `@aictrl/cli-windows-x64-baseline` | win32 | x64 | |
5578

5679
npm installs only the package that matches the host. Every x64 package is the
5780
baseline (no AVX2) build: Bun ships the same binary for its baseline and AVX2

‎script/token-budget/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Fixed SDK prefix token guard (Linux)
2+
3+
This guard starts the public CLI twice per configuration with fresh HOME/XDG state and receives its actual OpenAI-compatible SDK request at a loopback endpoint. It verifies the complete nine-tool default catalog and the explicit `--tools read,bash,edit,write` four-tool catalog. It does not import tests, reconstruct production schemas, invoke a paid model, or replace the existing 15,000-character Qwen guard.
4+
5+
Prerequisites: Linux x86_64, Python 3.12 with `venv`/`pip`, Bun from `package.json`, real `rg` and Git on PATH, and the repository's installed frozen Bun dependencies and built SDK. Missing prerequisites fail explicitly.
6+
7+
```sh
8+
# Online setup only: creates a NEW owned directory, pinned official PyPI wheels,
9+
# and checksum-verified OpenAI o200k rank data. It never installs JS packages.
10+
python3.12 -I -B script/token-budget/setup.py --directory /tmp/my-token-environment
11+
12+
# Offline actual CLI capture and accounting; output must be a NEW directory.
13+
/tmp/my-token-environment/venv/bin/python3 -I -B script/token-budget/run.py \
14+
--output /tmp/my-token-evidence
15+
```
16+
17+
The guard uses an exclusive nonblocking lock in the owner-only `/tmp/aictrl-token-budget` directory. Concurrent invocations fail before changing the fixed workspace. Each cold child runs in the literal `/tmp/aictrl-token-budget/workspace`, newly initialized as a Git repository, with UTC and Date fixed before production imports to 21 October 2026. This controls all three raw cwd occurrences, including the two in Bash descriptions/schema. No text normalization is applied. Existing unowned, shared, or symlink fixture directories are rejected. The directory/lock remain for safe reuse; raw captures remain in the requested evidence directory.
18+
19+
Only loopback fixture HTTP is allowed by the before-import fetch/preconnect/install guard reused from the selector fixture. Child environments are explicitly allowlisted, without provider credentials or inherited project configuration. The real CLI has a mandatory 15-second deadline; the outer owned process group has 25 seconds and rejects surviving descendants even if its parent exits successfully. These fixture controls are not an OS network sandbox.
20+
21+
`count.py` verifies the pinned tokenizer version, exact rank SHA256, raw environment, catalog, single request and clean process/network evidence. `reference.py` denies socket/DNS use during accounting. The enforced metric is `o200k_base` encoding of compact sorted Unicode JSON containing actual SDK system-role messages, serialized tools, and instructions when present. The fixture user text is excluded. The full first model input is separately reported using `instructions/messages/input/tools/response_format/text`; transport controls are excluded. System/tool diagnostic counts use separate tokenizations and are not additive BPE contributions.
22+
23+
`budgets.json` contains two fixed explicit ceilings. Raising either is a reviewed source edit requiring raw evidence and an explanation. `--observe` records an evaluation without enforcing ceilings and is never used by CI. `--source` selects an isolated source copy for QA source-growth controls; CI always uses the repository entry. The default two cold repetitions must produce identical canonical prefix bytes for each arm.
24+
25+
CI provisions the Python runtime, lock and rank data before capture; capture/accounting do not download. Raw requests, canonical prefixes, network/exit/deadline logs and the complete reference report are retained as `token-budget-evidence`, including when enforcement fails. These counts are reference overhead measurements, not provider billing, cache savings, quality or performance estimates. Release comparisons have a separate current-versus-previous protocol; these two selector arms do not substitute for that requirement.
26+
27+
Initial portable observation: default **3,441** prefix tokens and selected **2,001**, at the controlled path above (15,453 and 8,973 canonical characters). Both raw request bodies and canonical prefixes were byte-identical across two cold captures per arm. Initial ceilings equal those observed counts with **zero headroom**: deterministic prompt/tool growth should require an explicit reviewed explanation rather than consume a hidden allowance. Full first model input was separately 3,457/2,017. The longer earlier evaluation cwd produced different counts and was not used as a ceiling.

‎script/token-budget/budgets.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"default": 3441,
3+
"four": 2001
4+
}

‎script/token-budget/capture.ts‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import path from "node:path"
2+
import fs from "node:fs/promises"
3+
import { guard } from "./guard"
4+
const out = process.env.TOKEN_BUDGET_OUTPUT
5+
const work = process.env.TOKEN_BUDGET_WORKSPACE
6+
const source = process.env.TOKEN_BUDGET_SOURCE
7+
if (!out || !work || !source) throw new Error("Explicit output/workspace/source required")
8+
const binpath = [path.dirname(process.execPath), "/usr/local/bin", "/usr/bin", "/bin"].join(path.delimiter)
9+
const rg = Bun.which("rg", { PATH: binpath })
10+
if (!rg) throw new Error("Real ripgrep prerequisite missing")
11+
const requests: string[] = []
12+
const server = Bun.serve({
13+
hostname: "127.0.0.1",
14+
port: 0,
15+
idleTimeout: 0,
16+
async fetch(request) {
17+
if (new URL(request.url).pathname !== "/v1/chat/completions") throw new Error("Unexpected endpoint " + request.url)
18+
if (request.method !== "POST") throw new Error("Expected POST")
19+
requests.push(await request.text())
20+
const chunks = [
21+
{
22+
id: "eval",
23+
object: "chat.completion.chunk",
24+
choices: [{ index: 0, delta: { role: "assistant", content: "Fixture complete" } }],
25+
},
26+
{ id: "eval", object: "chat.completion.chunk", choices: [{ index: 0, delta: {}, finish_reason: "stop" }] },
27+
]
28+
return new Response(
29+
chunks
30+
.map((chunk) => "data: " + JSON.stringify(chunk))
31+
.concat("data: [DONE]")
32+
.join("\n\n") + "\n\n",
33+
{ headers: { "content-type": "text/event-stream" } },
34+
)
35+
},
36+
})
37+
await fs.mkdir(out, { recursive: true })
38+
await Bun.write(path.join(out, "guard.ts"), guard)
39+
await Bun.write(path.join(out, "network.jsonl"), "")
40+
const config = {
41+
enabled_providers: ["fixture"],
42+
provider: {
43+
fixture: {
44+
npm: "@ai-sdk/openai-compatible",
45+
options: { apiKey: "fixture", baseURL: server.url.origin + "/v1" },
46+
models: { fixture: { name: "fixture", tool_call: true, limit: { context: 200000, output: 1000 } } },
47+
},
48+
},
49+
agent: { title: { disable: true } },
50+
permission: { "*": "allow" },
51+
}
52+
const command = [
53+
process.execPath,
54+
"run",
55+
"--no-install",
56+
"--no-env-file",
57+
"--preload",
58+
path.join(import.meta.dir, "clock.js"),
59+
"--preload",
60+
path.join(out, "guard.ts"),
61+
"--conditions=browser",
62+
source,
63+
"run",
64+
"--model",
65+
"fixture/fixture",
66+
"--agent",
67+
"build",
68+
"--format",
69+
"json",
70+
...(process.env.TOKEN_BUDGET_ARM === "four" ? ["--tools", "read,bash,edit,write"] : []),
71+
"Say Fixture complete without tools.",
72+
]
73+
const env = {
74+
PATH: binpath,
75+
SHELL: "/bin/bash",
76+
TZ: "UTC",
77+
HOME: path.join(out, "home"),
78+
AICTRL_TEST_HOME: path.join(out, "home"),
79+
AICTRL_TEST_MANAGED_CONFIG_DIR: path.join(out, "managed"),
80+
XDG_DATA_HOME: path.join(out, "data"),
81+
XDG_CONFIG_HOME: path.join(out, "config"),
82+
XDG_STATE_HOME: path.join(out, "state"),
83+
XDG_CACHE_HOME: path.join(out, "cache"),
84+
BUN_RUNTIME_TRANSPILER_CACHE_PATH: path.join(out, "bun-cache"),
85+
GIT_CONFIG_NOSYSTEM: "1",
86+
GIT_CONFIG_GLOBAL: "/dev/null",
87+
AICTRL_DISABLE_PROJECT_CONFIG: "true",
88+
AICTRL_DISABLE_DEFAULT_PLUGINS: "true",
89+
AICTRL_DISABLE_AUTOCOMPACT: "true",
90+
AICTRL_MODEL_STREAM_IDLE_TIMEOUT_MS: "3000",
91+
AICTRL_CONFIG_CONTENT: JSON.stringify(config),
92+
SELECTOR_FIXTURE_ORIGIN: server.url.origin,
93+
SELECTOR_FIXTURE_GUARD_LOG: path.join(out, "network.jsonl"),
94+
}
95+
const child = Bun.spawn(command, { cwd: work, env, stdin: "ignore", stdout: "pipe", stderr: "pipe" })
96+
let deadline = false
97+
const timer = setTimeout(() => {
98+
deadline = true
99+
child.kill("SIGKILL")
100+
}, 15000)
101+
try {
102+
const [stdout, stderr, exit] = await Promise.all([
103+
new Response(child.stdout).text(),
104+
new Response(child.stderr).text(),
105+
child.exited,
106+
])
107+
clearTimeout(timer)
108+
await Bun.write(path.join(out, "stdout.jsonl"), stdout)
109+
await Bun.write(path.join(out, "stderr.log"), stderr)
110+
for (const [index, body] of requests.entries()) await Bun.write(path.join(out, `request-${index}.json`), body)
111+
const network = (await Bun.file(path.join(out, "network.jsonl")).text())
112+
.split("\n")
113+
.filter(Boolean)
114+
.map((line) => JSON.parse(line))
115+
const result = { exit, deadline, command, environment: env, rg, requests: requests.length, network }
116+
await Bun.write(path.join(out, "result.json"), JSON.stringify(result, null, 2) + "\n")
117+
if (deadline || exit !== 0 || requests.length !== 1 || network.some((row) => !row.allowed))
118+
throw new Error(
119+
"Capture failed: " + JSON.stringify({ exit, deadline, requests: requests.length, network }) + stderr.slice(-2000),
120+
)
121+
const events = stdout
122+
.split("\n")
123+
.filter(Boolean)
124+
.map((line) => JSON.parse(line))
125+
if (
126+
events
127+
.filter((event) => event.type === "text")
128+
.map((event) => event.part?.text)
129+
.join("") !== "Fixture complete"
130+
)
131+
throw new Error("Exact assistant response missing")
132+
if (
133+
events.filter((event) => event.type === "invocation_complete" && event.status === "completed").length !== 1 ||
134+
events.some((event) => event.type === "error" || event.type === "tool_use")
135+
)
136+
throw new Error("Unexpected invocation lifecycle")
137+
} catch (error) {
138+
console.error(String(error))
139+
process.exitCode = 1
140+
} finally {
141+
clearTimeout(timer)
142+
child.kill("SIGKILL")
143+
server.stop(true)
144+
}

‎script/token-budget/clock.js‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
const epoch = 1792584000000 // 2026-10-21T12:00:00.000Z
2+
const NativeDate = Date
3+
globalThis.Date = new Proxy(NativeDate, {
4+
construct(target, args, next) {
5+
return Reflect.construct(target, args.length ? args : [epoch], next)
6+
},
7+
apply() {
8+
return new NativeDate(epoch).toString()
9+
},
10+
get(target, key, receiver) {
11+
return key === "now" ? () => epoch : Reflect.get(target, key, receiver)
12+
},
13+
})

0 commit comments

Comments
 (0)