Skip to content

Commit 0fc9676

Browse files
byapparovclaude
andcommitted
feat(run): drop todowrite from the default headless tool set (#155)
todowrite's definition is ~9.4k chars (~2.4k tokens) and goes out on every step of a headless run, yet production runs call it in ~4% of executions. - The run command marks the process with AICTRL_HEADLESS_RUN at runtime, so the full and lite builds behave the same (unlike the AICTRL_HEADLESS build define, #152). - ToolRegistry registers todowrite unless the process is a headless run; AICTRL_ENABLE_TODO_TOOL=1 opts back in, mirroring AICTRL_ENABLE_QUESTION_TOOL. ACP and other sessions are unchanged; todoread stays unregistered. - Add a headless request-prefix budget test (system prompt + tool definitions, no skills) with a per-piece breakdown on failure: 35,188 chars after this change, budget 35,500 (was 44,636). Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 246dfa6 commit 0fc9676

8 files changed

Lines changed: 292 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,10 @@ and [JSON event reference](EVENTS.md).
7070

7171
In headless mode, Aictrl automatically rejects all interactive permission requests (like `question` or `plan_enter`), ensuring your pipelines never hang.
7272

73+
`aictrl run` leaves out the `todowrite` tool by default: its definition adds about 2.4k tokens
74+
to every request, and a one-shot run has nobody following a todo list. Set
75+
`AICTRL_ENABLE_TODO_TOOL=1` to bring it back.
76+
7377
### CI/CD Integration
7478

7579
Set `AICTRL_HEADLESS=true` in your environment to force headless behavior even in pseudo-TTYs.

‎packages/cli/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,10 @@ Provider failures and stream timeouts retain **1**; SIGINT/SIGTERM retain
9797

9898
In headless mode, Aictrl automatically rejects permissions that would otherwise prompt a user.
9999

100+
`aictrl run` leaves out the `todowrite` tool by default: its definition adds about 2.4k tokens
101+
to every request, and a one-shot run has nobody following a todo list. Set
102+
`AICTRL_ENABLE_TODO_TOOL=1` to bring it back.
103+
100104
## GitHub Agent
101105

102106
Install the GitHub agent into any repository:

‎packages/cli/src/cli/cmd/run.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,7 @@ export const RunCommand = cmd({
392392
})
393393
},
394394
handler: async (args) => {
395+
process.env.AICTRL_HEADLESS_RUN = "true"
395396
const invocation = createRunInvocation(args.format === "json")
396397

397398
async function fail(message: string, code: string, exit = 1): Promise<never> {

‎packages/cli/src/flag/flag.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ export namespace Flag {
2222
export const AICTRL_FAKE_VCS = process.env["AICTRL_FAKE_VCS"]
2323
export declare const AICTRL_CLIENT: string
2424
export const AICTRL_ENABLE_QUESTION_TOOL = truthy("AICTRL_ENABLE_QUESTION_TOOL")
25+
export declare const AICTRL_ENABLE_TODO_TOOL: boolean
26+
export declare const AICTRL_HEADLESS_RUN: boolean
2527
export declare const AICTRL_MODEL_STREAM_IDLE_TIMEOUT_MS: number
2628

2729
// Experimental
@@ -93,3 +95,24 @@ Object.defineProperty(Flag, "AICTRL_CLIENT", {
9395
enumerable: true,
9496
configurable: false,
9597
})
98+
99+
// Dynamic getter for AICTRL_ENABLE_TODO_TOOL
100+
// Evaluated at access time, like AICTRL_HEADLESS_RUN, which it overrides
101+
Object.defineProperty(Flag, "AICTRL_ENABLE_TODO_TOOL", {
102+
get() {
103+
return truthy("AICTRL_ENABLE_TODO_TOOL")
104+
},
105+
enumerable: true,
106+
configurable: false,
107+
})
108+
109+
// Dynamic getter for AICTRL_HEADLESS_RUN
110+
// Set by the `run` command at runtime, whichever entry point or build started it,
111+
// so it must be evaluated at access time
112+
Object.defineProperty(Flag, "AICTRL_HEADLESS_RUN", {
113+
get() {
114+
return truthy("AICTRL_HEADLESS_RUN")
115+
},
116+
enumerable: true,
117+
configurable: false,
118+
})

‎packages/cli/src/tool/registry.ts‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,10 +111,21 @@ export namespace ToolRegistry {
111111
custom.push(tool)
112112
}
113113

114+
/**
115+
* todowrite keeps a progress list for someone watching the session. A headless
116+
* `aictrl run` has no such reader, and the tool definition costs ~2.4k tokens on
117+
* every step, so a headless run only gets it with AICTRL_ENABLE_TODO_TOOL (#155).
118+
*/
119+
function todoEnabled() {
120+
if (Flag.AICTRL_ENABLE_TODO_TOOL) return true
121+
return !Flag.AICTRL_HEADLESS_RUN
122+
}
123+
114124
async function all(): Promise<Tool.Info[]> {
115125
const custom = await state().then((x) => x.custom)
116126
const config = await Config.get()
117127
const question = ["app", "cli", "desktop"].includes(Flag.AICTRL_CLIENT) || Flag.AICTRL_ENABLE_QUESTION_TOOL
128+
const todo = todoEnabled()
118129

119130
return [
120131
InvalidTool,
@@ -127,7 +138,7 @@ export namespace ToolRegistry {
127138
WriteTool,
128139
TaskTool,
129140
WebFetchTool,
130-
TodoWriteTool,
141+
...(todo ? [TodoWriteTool] : []),
131142
// TodoReadTool,
132143
WebSearchTool,
133144
CodeSearchTool,
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
import { describe, expect, test } from "bun:test"
2+
import path from "path"
3+
import { tmpdir } from "../fixture/fixture"
4+
5+
const entries = {
6+
full: path.resolve(import.meta.dir, "../../src/index.ts"),
7+
headless: path.resolve(import.meta.dir, "../../src/headless.ts"),
8+
}
9+
10+
/** Runs `aictrl run` against a fixture Gemini server and returns the tool names of the first request. */
11+
async function requestedTools(entry: string, env: Record<string, string> = {}) {
12+
await using tmp = await tmpdir()
13+
const requests: any[] = []
14+
const server = Bun.serve({
15+
hostname: "127.0.0.1",
16+
port: 0,
17+
async fetch(request) {
18+
requests.push(await request.json())
19+
const chunk = {
20+
candidates: [{ index: 0, content: { role: "model", parts: [{ text: "Done." }] }, finishReason: "STOP" }],
21+
usageMetadata: { promptTokenCount: 7, candidatesTokenCount: 3, totalTokenCount: 10 },
22+
}
23+
return new Response(`data: ${JSON.stringify(chunk)}\n\n`, {
24+
headers: { "content-type": "text/event-stream" },
25+
})
26+
},
27+
})
28+
await Bun.write(
29+
path.join(tmp.path, "aictrl.json"),
30+
JSON.stringify({
31+
provider: {
32+
fixture: {
33+
npm: "@ai-sdk/google",
34+
options: { apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}` },
35+
models: { "gemini-fixture": { name: "fixture", limit: { context: 100000, output: 1000 } } },
36+
},
37+
},
38+
agent: { title: { disable: true } },
39+
}),
40+
)
41+
const environment: Record<string, string | undefined> = {
42+
...process.env,
43+
AICTRL_DISABLE_DEFAULT_PLUGINS: "true",
44+
AICTRL_DISABLE_MODELS_FETCH: "true",
45+
AICTRL_DISABLE_AUTOCOMPACT: "true",
46+
...env,
47+
}
48+
// The parent test process may itself run inside a headless run; only the spawned command decides.
49+
delete environment.AICTRL_HEADLESS_RUN
50+
if (!("AICTRL_ENABLE_TODO_TOOL" in env)) delete environment.AICTRL_ENABLE_TODO_TOOL
51+
const proc = Bun.spawn(
52+
["bun", "run", "--conditions=browser", entry, "run", "--model", "fixture/gemini-fixture", "Check this input."],
53+
{ cwd: tmp.path, env: environment, stdout: "pipe", stderr: "pipe" },
54+
)
55+
const timeout = setTimeout(() => proc.kill("SIGKILL"), 15000)
56+
try {
57+
const [stdout, stderr, exit] = await Promise.all([
58+
new Response(proc.stdout).text(),
59+
new Response(proc.stderr).text(),
60+
proc.exited,
61+
])
62+
expect(exit, stderr + stdout).toBe(0)
63+
expect(requests.length, stderr + stdout).toBeGreaterThan(0)
64+
const declarations: { name: string }[] = requests[0].tools?.flatMap((tool: any) => tool.functionDeclarations ?? [])
65+
return declarations.map((declaration) => declaration.name)
66+
} finally {
67+
clearTimeout(timeout)
68+
proc.kill("SIGKILL")
69+
server.stop(true)
70+
}
71+
}
72+
73+
describe("headless run default tools (#155)", () => {
74+
test.each(Object.entries(entries))(
75+
"a %s-build run sends no todowrite or todoread tool",
76+
async (_, entry) => {
77+
const tools = await requestedTools(entry)
78+
expect(tools).toContain("bash")
79+
expect(tools).not.toContain("todowrite")
80+
expect(tools).not.toContain("todoread")
81+
},
82+
20000,
83+
)
84+
85+
test(
86+
"AICTRL_ENABLE_TODO_TOOL=1 brings todowrite back",
87+
async () => {
88+
expect(await requestedTools(entries.full, { AICTRL_ENABLE_TODO_TOOL: "1" })).toContain("todowrite")
89+
},
90+
20000,
91+
)
92+
})
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
2+
import z from "zod"
3+
import { tmpdir } from "../fixture/fixture"
4+
import { Instance } from "../../src/project/instance"
5+
import { Agent } from "../../src/agent/agent"
6+
import { Provider } from "../../src/provider/provider"
7+
import { PermissionNext } from "../../src/permission/next"
8+
import { Skill } from "../../src/skill"
9+
import { SystemPrompt } from "../../src/session/system"
10+
import { InstructionPrompt } from "../../src/session/instruction"
11+
import { ToolRegistry } from "../../src/tool/registry"
12+
13+
/**
14+
* Characters in the fixed prefix (system prompt plus tool definitions) that a
15+
* headless `aictrl run` of the build agent sends on every step, before any
16+
* conversation content. Set just above the current size: lower it whenever a
17+
* change shrinks the prefix, and never raise it without a reason (#160).
18+
*/
19+
const HEADLESS_PREFIX_BUDGET = 35_500
20+
21+
const HEADLESS_ENV = { AICTRL_HEADLESS: "true", AICTRL_HEADLESS_RUN: "true" }
22+
23+
type Piece = { name: string; chars: number }
24+
25+
/** System prompt pieces, assembled the way SessionPrompt and LLM.stream assemble them. */
26+
async function systemPieces(model: Provider.Model): Promise<Piece[]> {
27+
const instructions = await InstructionPrompt.system()
28+
return [
29+
{ name: "system: provider prompt", chars: SystemPrompt.provider(model).join("\n").length },
30+
{ name: "system: environment", chars: (await SystemPrompt.environment(model)).join("\n").length },
31+
{ name: "system: instructions", chars: instructions.join("\n").length },
32+
]
33+
}
34+
35+
/** Tool definitions the model receives: the registry list minus what LLM.stream drops. */
36+
async function toolPieces(model: Provider.Model, agent: Agent.Info): Promise<Piece[]> {
37+
const tools = await ToolRegistry.tools({ providerID: model.providerID, modelID: model.api.id }, agent)
38+
const disabled = PermissionNext.disabled(
39+
tools.map((tool) => tool.id),
40+
agent.permission,
41+
)
42+
return tools
43+
.filter((tool) => tool.id !== "invalid" && !disabled.has(tool.id))
44+
.map((tool) => ({
45+
name: `tool: ${tool.id}`,
46+
chars: tool.id.length + tool.description.length + JSON.stringify(z.toJSONSchema(tool.parameters)).length,
47+
}))
48+
}
49+
50+
function breakdown(pieces: Piece[]) {
51+
return pieces
52+
.toSorted((a, b) => b.chars - a.chars)
53+
.map((piece) => ` ${piece.name.padEnd(28)} ${String(piece.chars).padStart(6)} chars`)
54+
.join("\n")
55+
}
56+
57+
describe("headless request prefix budget (#160)", () => {
58+
const previous: Record<string, string | undefined> = {}
59+
60+
beforeAll(() => {
61+
for (const [key, value] of Object.entries(HEADLESS_ENV)) {
62+
previous[key] = process.env[key]
63+
process.env[key] = value
64+
}
65+
})
66+
67+
afterAll(() => {
68+
for (const [key, value] of Object.entries(previous)) {
69+
if (value === undefined) delete process.env[key]
70+
else process.env[key] = value
71+
}
72+
})
73+
74+
test("system prompt plus tool definitions stay within the budget", async () => {
75+
await using tmp = await tmpdir({
76+
git: true,
77+
config: {
78+
enabled_providers: ["alibaba"],
79+
provider: { alibaba: { options: { apiKey: "test-key" } } },
80+
},
81+
})
82+
83+
await Instance.provide({
84+
directory: tmp.path,
85+
fn: async () => {
86+
// The skill tool lists every discovered skill; the fixture must not find any.
87+
expect(await Skill.all()).toEqual([])
88+
89+
const model = await Provider.getModel("alibaba", "qwen-plus")
90+
const agent = await Agent.get("build")
91+
const pieces = [...(await systemPieces(model)), ...(await toolPieces(model, agent))]
92+
const total = pieces.reduce((sum, piece) => sum + piece.chars, 0)
93+
94+
const report = [
95+
`Headless request prefix is ${total} chars (~${Math.round(total / 4)} tokens); budget is ${HEADLESS_PREFIX_BUDGET}.`,
96+
breakdown(pieces),
97+
].join("\n")
98+
if (total > HEADLESS_PREFIX_BUDGET) throw new Error(report)
99+
console.log(report)
100+
},
101+
})
102+
})
103+
})

‎packages/cli/test/tool/registry.test.ts‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
1-
import { describe, expect, test } from "bun:test"
1+
import { afterEach, describe, expect, test } from "bun:test"
22
import path from "path"
33
import fs from "fs/promises"
44
import os from "os"
55
import { tmpdir } from "../fixture/fixture"
66
import { Instance } from "../../src/project/instance"
77
import { ToolRegistry } from "../../src/tool/registry"
88
import { Config } from "../../src/config/config"
9+
import { Agent } from "../../src/agent/agent"
910

1011
describe("tool.registry", () => {
1112
test("loads tools from .aictrl/tool (singular)", async () => {
@@ -179,3 +180,54 @@ describe("tool.registry", () => {
179180
}
180181
})
181182
})
183+
184+
describe("tool.registry todo tools (#155)", () => {
185+
const keys = ["AICTRL_HEADLESS_RUN", "AICTRL_ENABLE_TODO_TOOL"] as const
186+
const previous = Object.fromEntries(keys.map((key) => [key, process.env[key]]))
187+
188+
afterEach(() => {
189+
for (const key of keys) {
190+
if (previous[key] === undefined) delete process.env[key]
191+
else process.env[key] = previous[key]
192+
}
193+
})
194+
195+
function setEnv(env: Partial<Record<(typeof keys)[number], string>>) {
196+
for (const key of keys) {
197+
if (env[key] === undefined) delete process.env[key]
198+
else process.env[key] = env[key]
199+
}
200+
}
201+
202+
async function toolIDs() {
203+
await using tmp = await tmpdir()
204+
return Instance.provide({
205+
directory: tmp.path,
206+
fn: async () => {
207+
const agent = await Agent.get("build")
208+
const tools = await ToolRegistry.tools({ providerID: "anthropic", modelID: "claude-sonnet-4" }, agent)
209+
return tools.map((tool) => tool.id)
210+
},
211+
})
212+
}
213+
214+
test("a headless run gets neither todowrite nor todoread", async () => {
215+
setEnv({ AICTRL_HEADLESS_RUN: "true" })
216+
const ids = await toolIDs()
217+
expect(ids).toContain("bash")
218+
expect(ids).not.toContain("todowrite")
219+
expect(ids).not.toContain("todoread")
220+
})
221+
222+
test("AICTRL_ENABLE_TODO_TOOL brings todowrite back to a headless run", async () => {
223+
setEnv({ AICTRL_HEADLESS_RUN: "true", AICTRL_ENABLE_TODO_TOOL: "1" })
224+
const ids = await toolIDs()
225+
expect(ids).toContain("todowrite")
226+
expect(ids).not.toContain("todoread")
227+
})
228+
229+
test("sessions outside a headless run keep todowrite", async () => {
230+
setEnv({})
231+
expect(await toolIDs()).toContain("todowrite")
232+
})
233+
})

0 commit comments

Comments
 (0)