Skip to content

Commit 769af84

Browse files
anandgupta42claude
andcommitted
feat: pre-execution critic gate for side-effecting tools
Flag-gated (`ALTIMATE_CRITIC_GATE`, default off) gate that runs before a side-effecting tool (bash/write/edit/sql_execute/dbt_run/patch) executes. On a hard verdict the call is skipped and the reason is fed back so the model can fix-and-retry, instead of executing a bad action. - `tool/critic.ts` — pure, testable gate + pluggable `Verifier` interface. - `ALLOW_ALL` — ungated (opt-out / tests). - `basicSafetyVerifier` — the wired default: a conservative, dependency-free heuristic that blocks catastrophic, unambiguous host-destructive bash (`rm -rf /` incl. system-path/glob/compound/fully-qualified/brace variants, fork bombs, `mkfs`/`dd` on a raw device, recursive chmod of `/`). Best-effort safety NET, not a security boundary — documented as such; defense-in-depth stays with the OS sandbox, the permission system, and a richer verifier a product may inject. - `gate()` is fail-open on verifier error AND on a verifier timeout, so a broken or hung verifier never breaks/hangs the agent. - `session/prompt.ts` — wired into the native-tool execute wrapper just before `item.execute`, marker-wrapped, emits `tool.execute.after` on a block so observability sees it. No-op when off. (MCP tools use a separate wrapper and are intentionally not gated; documented on `DEFAULT_GATED`.) Tests (100): unit, adversarial bypass probes (caught-vs-documented-limits), and e2e driving the REAL BashTool through the gate with real filesystem effects — no mocked tool calls. Hardening from multi-model review (Gemini / GLM-5 / MiniMax): - FIX false positive: `rm -rf *`, `rm -rf .`, `rm -rf ./` no longer blocked (routine workspace cleanup; no workdir context to judge them). - FIX false positive: `rm` mentioned inside another command's argument (`git commit -m "...rm -rf /..."`) no longer blocked — `rm` is only treated as the command when in command position (with a transparent-prefix allowlist so `sudo`/`bash -c` still match). - FIX false negatives: glob wipes of system dirs (`rm -rf /var/*`), `/.`/`/..`, fully-qualified `/bin/rm`, `${HOME}` brace expansion, and long-form/glob recursive chmod (`chmod --recursive 777 /`, `chmod -R 777 /*`). - Add input length cap + verifier timeout; `attachments: []` on blocked result. - Adversarial testing also found+fixed two earlier holes: compound commands where the fatal `rm` wasn't last, and a separator glued to the target. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 79c9ddd commit 769af84

5 files changed

Lines changed: 725 additions & 0 deletions

File tree

‎packages/opencode/src/session/prompt.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ import { NamedError } from "@opencode-ai/util/error"
4242
import { fn } from "@/util/fn"
4343
import { SessionProcessor } from "./processor"
4444
import { TaskTool } from "@/tool/task"
45+
// altimate_change start — critic gate
46+
import { Critic } from "@/tool/critic"
47+
// altimate_change end
4548
import { Tool } from "@/tool/tool"
4649
import { PermissionNext } from "@/permission/next"
4750
import { SessionStatus } from "./status"
@@ -1501,6 +1504,28 @@ export namespace SessionPrompt {
15011504
args,
15021505
},
15031506
)
1507+
// altimate_change start — critic gate (pre-execution safety check)
1508+
// Flag-gated (ALTIMATE_CRITIC_GATE), default off. On a hard verdict we
1509+
// skip execution and feed the reason back so the model can retry, while
1510+
// still emitting tool.execute.after so observability sees the blocked call.
1511+
if (Critic.enabled()) {
1512+
const verdict = await Critic.gate(item.id, args as Record<string, any>, Critic.basicSafetyVerifier)
1513+
if (!verdict.allow) {
1514+
const blocked = {
1515+
title: `Blocked: ${item.id}`,
1516+
metadata: { critic: { blocked: true, reason: verdict.feedback } } as any,
1517+
output: verdict.feedback ?? "Blocked by altimate verifier.",
1518+
attachments: [] as never[],
1519+
}
1520+
await Plugin.trigger(
1521+
"tool.execute.after",
1522+
{ tool: item.id, sessionID: ctx.sessionID, callID: ctx.callID, args },
1523+
blocked,
1524+
)
1525+
return blocked
1526+
}
1527+
}
1528+
// altimate_change end
15041529
const result = await item.execute(args, ctx)
15051530
const output = {
15061531
...result,
Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
/**
2+
* Pre-execution critic gate.
3+
*
4+
* Before a SIDE-EFFECTING tool runs (bash, write, edit, sql_execute, dbt_*), a
5+
* verifier checks the proposed args; on hard failure the call is denied and the
6+
* reason is fed back so the model can retry — instead of executing a bad action.
7+
*
8+
* The judgment plugs in via the `Verifier` interface. Two impls ship here:
9+
* - `ALLOW_ALL` — ungated (open); for tests / opt-out.
10+
* - `basicSafetyVerifier` — the wired default: a conservative, dependency-free
11+
* heuristic that blocks catastrophic, unambiguous
12+
* host-destructive bash (e.g. `rm -rf /`, fork bombs,
13+
* `mkfs`/`dd` on a raw device).
14+
*
15+
* This is a best-effort safety NET, NOT a security boundary or sandbox: a
16+
* determined caller can obfuscate around literal patterns (command substitution,
17+
* base64-decode-pipe-sh, etc.). Defense in depth lives elsewhere (OS sandbox, the
18+
* permission system, and a richer pluggable verifier the caller may inject).
19+
*
20+
* Pure + testable. Wired into session/prompt.ts, just before `item.execute(args, ctx)`.
21+
*/
22+
23+
export namespace Critic {
24+
/**
25+
* Side-effecting tools worth gating. Reads (glob/grep/read) are never gated.
26+
* NOTE: the gate is wired into the native ToolRegistry execute wrapper only;
27+
* MCP-provided tools run through a separate wrapper and are NOT gated. The
28+
* shipped `basicSafetyVerifier` is bash-only (a native tool), so this is a
29+
* no-op gap today — but a product injecting a verifier for `sql_execute`/
30+
* `dbt_run` must confirm those are native, not MCP, tools.
31+
*/
32+
export const DEFAULT_GATED = ["bash", "write", "edit", "sql_execute", "dbt_run", "patch"]
33+
34+
export interface Verdict {
35+
ok: boolean
36+
reason?: string
37+
}
38+
39+
/** The judgment interface. Default impl allows all (open). Product plugs a richer verifier. */
40+
export interface Verifier {
41+
verify(toolName: string, args: Record<string, any>): Verdict | Promise<Verdict>
42+
}
43+
44+
export const ALLOW_ALL: Verifier = { verify: () => ({ ok: true }) }
45+
46+
export function enabled(): boolean {
47+
return process.env["ALTIMATE_CRITIC_GATE"] === "1"
48+
}
49+
50+
export function isGated(toolName: string, gated: string[] = DEFAULT_GATED): boolean {
51+
return gated.includes(toolName)
52+
}
53+
54+
// ── Heuristic bash safety ──────────────────────────────────────────────────
55+
//
56+
// Targets that, combined with a recursive+force `rm`, mean catastrophic loss.
57+
// Deliberately ABSOLUTE only — `.`, `./` and bare `*` are NOT here: clearing
58+
// the working/build directory (`rm -rf *`, `rm -rf .`) is routine and safe in a
59+
// sandboxed workspace, and we have no `workdir` context to judge them.
60+
const RM_FATAL_TARGETS = new Set(["/", "/*", "/.", "/..", "~", "~/", "~/*", "$home", "$home/", "$home/*"])
61+
// Top-level system paths whose recursive deletion bricks the machine.
62+
const RM_FATAL_TOPLEVEL = [
63+
"/etc",
64+
"/usr",
65+
"/var",
66+
"/bin",
67+
"/sbin",
68+
"/boot",
69+
"/lib",
70+
"/lib64",
71+
"/sys",
72+
"/dev",
73+
"/proc",
74+
"/root",
75+
"/home",
76+
"/opt",
77+
]
78+
79+
// Transparent command prefixes: words that may precede the real command without
80+
// changing which command runs. Lets `sudo rm -rf /` and `bash -c "rm -rf /"`
81+
// still be seen as an `rm` in command position, while `git commit -m "rm -rf /"`
82+
// (rm buried in an argument) is not.
83+
const TRANSPARENT_PREFIX = new Set([
84+
"sudo",
85+
"doas",
86+
"nohup",
87+
"time",
88+
"env",
89+
"exec",
90+
"command",
91+
"builtin",
92+
"ionice",
93+
"nice",
94+
"setsid",
95+
"stdbuf",
96+
"then",
97+
"do",
98+
"else",
99+
"bash",
100+
"sh",
101+
"zsh",
102+
"dash",
103+
"ksh",
104+
])
105+
106+
function isFatalRmTarget(tok: string): boolean {
107+
// Normalize bash brace expansion: ${home} -> $home, ${home}/ -> $home/
108+
const norm = tok.replace(/^\$\{([a-z_]+)\}(\/.*)?$/, (_m, v, rest) => "$" + v + (rest ?? ""))
109+
if (RM_FATAL_TARGETS.has(tok) || RM_FATAL_TARGETS.has(norm)) return true
110+
// Catch a system path itself, its trailing-slash form, OR a glob wipe of its
111+
// contents (`/var/*`) — but NOT a scoped subpath like `/home/user/project`.
112+
return RM_FATAL_TOPLEVEL.some((p) => tok === p || tok === p + "/" || tok === p + "/*")
113+
}
114+
115+
/**
116+
* Is the token at index `i` in COMMAND position (the start of a pipeline
117+
* segment), versus an argument to some other command? Walks left skipping
118+
* flags and transparent prefixes; a separator or the start means command
119+
* position, any other bareword means it's an argument.
120+
*/
121+
function isCommandPosition(tokens: string[], i: number, sep: Set<string>): boolean {
122+
for (let j = i - 1; j >= 0; j--) {
123+
const t = tokens[j]
124+
if (sep.has(t)) return true
125+
if (t.startsWith("-")) continue // a flag (e.g. `sudo -E`, `bash -c`)
126+
if (TRANSPARENT_PREFIX.has(t)) continue
127+
return false // a real preceding word -> rm is an argument, not the command
128+
}
129+
return true // reached the start through only flags/prefixes
130+
}
131+
132+
/**
133+
* Detect a catastrophic, unambiguous host-destructive bash command. Returns a
134+
* human reason when dangerous, else null. Conservative on purpose — it only
135+
* fires on the few patterns that are almost never legitimate. Quotes are
136+
* stripped so `bash -c "rm -rf /"` is still seen; this can over-match a command
137+
* that merely mentions such a string (acceptable: the gate is opt-in/off by default).
138+
*/
139+
export function detectDangerousBash(raw: string): string | null {
140+
if (!raw || typeof raw !== "string") return null
141+
if (raw.length > 1_000_000) return null // cap — don't scan pathologically huge input
142+
// Normalize: drop quotes, collapse whitespace, lowercase for keyword/path matching.
143+
const norm = raw.replace(/['"`]/g, "").replace(/\s+/g, " ").trim().toLowerCase()
144+
if (!norm) return null
145+
146+
// Fork bomb: :(){ :|:& };:
147+
if (norm.replace(/\s+/g, "").includes(":|:&") || /:\s*\(\s*\)\s*\{/.test(norm)) {
148+
return "fork bomb"
149+
}
150+
// Explicit intent to remove the root filesystem.
151+
if (norm.includes("--no-preserve-root")) {
152+
return "rm with --no-preserve-root targets the root filesystem"
153+
}
154+
// mkfs on a block device.
155+
if (/\bmkfs(\.\w+)?\b/.test(norm) && /\/dev\//.test(norm)) {
156+
return "mkfs on a block device"
157+
}
158+
// dd writing to a raw disk.
159+
if (/\bdd\b/.test(norm) && /\bof=\/dev\/(sd|nvme|disk|hd|vd|mmcblk)/.test(norm)) {
160+
return "dd writing to a raw block device"
161+
}
162+
// Redirect over a raw disk.
163+
if (/>\s*\/dev\/(sd|nvme|disk|hd|vd|mmcblk)/.test(norm)) {
164+
return "redirect over a raw block device"
165+
}
166+
// Recursive chmod/chown of the root filesystem (short `-R`/`-rf` or long
167+
// `--recursive`, against bare `/` or a `/*` glob wipe).
168+
if (/\bch(mod|own)\b/.test(norm) && /(^|\s)(-[a-z]*r|--recursive)\b/.test(norm) && /\s\/\*?(\s|$)/.test(norm)) {
169+
return "recursive permission/ownership change on /"
170+
}
171+
// Recursive + force rm of a fatal target. Split shell separators into their
172+
// own tokens so they don't glue to a target (`/;`) and so EVERY `rm` in a
173+
// compound command is inspected — not just the last one.
174+
const SEP = new Set([";", "|", "&", ">", "<"])
175+
const tokens = norm
176+
.replace(/([;|&><])/g, " $1 ")
177+
.replace(/\s+/g, " ")
178+
.trim()
179+
.split(" ")
180+
for (let i = 0; i < tokens.length; i++) {
181+
// Match `rm` or a fully-qualified `/bin/rm`, but only in command position
182+
// (not when `rm -rf /` merely appears inside another command's argument).
183+
if (tokens[i] !== "rm" && !tokens[i].endsWith("/rm")) continue
184+
if (!isCommandPosition(tokens, i, SEP)) continue
185+
let recursive = false
186+
let force = false
187+
const targets: string[] = []
188+
for (let j = i + 1; j < tokens.length; j++) {
189+
const t = tokens[j]
190+
if (SEP.has(t)) break // end of this rm invocation
191+
if (t.startsWith("-")) {
192+
if (t === "--recursive" || (/^-[a-z]+$/.test(t) && t.includes("r"))) recursive = true
193+
if (t === "--force" || (/^-[a-z]+$/.test(t) && t.includes("f"))) force = true
194+
} else {
195+
targets.push(t)
196+
}
197+
}
198+
if (recursive && force) {
199+
for (const t of targets) {
200+
if (isFatalRmTarget(t)) return `recursive force-delete of "${t}"`
201+
}
202+
}
203+
}
204+
return null
205+
}
206+
207+
/**
208+
* The wired default verifier. Blocks only catastrophic bash; everything else
209+
* (other gated tools, non-bash) is allowed. A product may inject a richer one.
210+
*/
211+
export const basicSafetyVerifier: Verifier = {
212+
verify(toolName, args) {
213+
if (toolName !== "bash") return { ok: true }
214+
const reason = detectDangerousBash(String(args?.["command"] ?? ""))
215+
return reason ? { ok: false, reason } : { ok: true }
216+
},
217+
}
218+
219+
export interface GateResult {
220+
allow: boolean
221+
/** when blocked, the message to feed back to the model in place of execution. */
222+
feedback?: string
223+
}
224+
225+
/**
226+
* Decide whether a proposed tool call may execute. Non-gated tools always pass.
227+
* Gated tools are checked by the verifier; a not-ok verdict blocks with feedback.
228+
* NEVER throws — a critic failure must not break the agent (fail-open on error).
229+
*/
230+
/** Max time to wait on a verifier before failing open. A hung verifier must
231+
* never hang the agent. */
232+
export const VERIFIER_TIMEOUT_MS = 5000
233+
234+
export async function gate(
235+
toolName: string,
236+
args: Record<string, any>,
237+
verifier: Verifier = ALLOW_ALL,
238+
gated: string[] = DEFAULT_GATED,
239+
timeoutMs: number = VERIFIER_TIMEOUT_MS,
240+
): Promise<GateResult> {
241+
if (!enabled() || !isGated(toolName, gated)) return { allow: true }
242+
try {
243+
// async IIFE so a synchronous throw in verify() rejects the promise (and is
244+
// caught below) rather than escaping the Promise.race.
245+
const verifyPromise = (async () => verifier.verify(toolName, args))()
246+
const timeout = new Promise<Verdict>((resolve) => {
247+
const t = setTimeout(() => resolve({ ok: true, reason: "__timeout__" }), timeoutMs)
248+
// don't keep the event loop alive for this guard timer
249+
;(t as any)?.unref?.()
250+
})
251+
const v = await Promise.race([verifyPromise, timeout])
252+
if (v.ok) return { allow: true }
253+
return {
254+
allow: false,
255+
feedback: `Blocked by altimate verifier before execution: ${v.reason ?? "failed validation"}. Fix and retry.`,
256+
}
257+
} catch {
258+
// Fail-open: observability/governance must never break core functionality.
259+
return { allow: true }
260+
}
261+
}
262+
}
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
/**
2+
* Adversarial probes for the critic gate's default safety heuristic.
3+
*
4+
* Two purposes:
5+
* 1. Confirm the heuristic catches the obfuscations it CLAIMS to catch
6+
* (whitespace, quote-wrapping, flag reordering, sudo/compound prefixes).
7+
* 2. Honestly DOCUMENT the obfuscations it does NOT catch. The default verifier
8+
* is a best-effort safety net, not a security boundary — a determined caller
9+
* can always evade literal pattern matching. These "known bypass" tests are
10+
* guardrails against over-claiming: if one ever starts being caught, great,
11+
* but we never depend on it. Real isolation is the OS sandbox + permission
12+
* system + a richer pluggable verifier.
13+
*/
14+
import { describe, expect, test } from "bun:test"
15+
import { Critic } from "../../src/tool/critic"
16+
17+
describe("critic adversarial — obfuscations that ARE caught", () => {
18+
const caught = [
19+
"rm -rf /", // arbitrary whitespace
20+
"\trm -rf /\n", // surrounding control whitespace
21+
"RM -RF /", // case (heuristic lowercases; harmless over-match)
22+
'bash -c "rm -rf /"', // shell wrapper, quotes stripped
23+
"sh -c 'rm -rf /'", // single-quote wrapper
24+
'r""m -rf /', // quote splicing inside the command name
25+
"sudo rm -rf /", // privilege prefix
26+
"nohup rm -rf / &", // backgrounded
27+
"true; rm -rf /", // statement separator
28+
"ls && rm -rf ~", // conjunction, home target
29+
"rm -rf /var", // system path
30+
"/bin/rm -rf /", // fully-qualified rm path
31+
"/usr/bin/rm -rf /", // fully-qualified rm in /usr/bin
32+
"sudo -E rm -rf /", // privilege prefix with a flag
33+
"rm -rf ${HOME}", // brace-expanded $HOME
34+
]
35+
for (const cmd of caught) {
36+
test(`caught: ${JSON.stringify(cmd)}`, () => {
37+
expect(Critic.detectDangerousBash(cmd)).not.toBeNull()
38+
})
39+
}
40+
})
41+
42+
describe("critic adversarial — KNOWN bypasses (documented, not depended upon)", () => {
43+
// These evade literal matching. We assert current behavior so the limitation is
44+
// explicit and reviewed, NOT because allowing them is desirable.
45+
const knownBypasses = [
46+
"$(echo rm) -rf /", // command substitution splits the rm token
47+
"`echo rm` -rf /", // backtick substitution — `echo` is not a transparent prefix, so rm isn't in command position
48+
"echo cm0gLXJmIC8K | base64 -d | sh", // base64-encoded `rm -rf /`
49+
"find / -delete", // destructive without rm
50+
"rm -rf $TARGET", // target hidden behind a variable
51+
"alias x='rm -rf /'; x", // aliased
52+
"xargs rm -rf < /tmp/list", // target via stdin (no literal fatal target token)
53+
]
54+
for (const cmd of knownBypasses) {
55+
test(`known bypass (returns null today): ${JSON.stringify(cmd)}`, () => {
56+
expect(Critic.detectDangerousBash(cmd)).toBeNull()
57+
})
58+
}
59+
})
60+
61+
describe("critic adversarial — gate never throws, regardless of input", () => {
62+
test("malformed/pathological args do not crash the gate (fail-open)", async () => {
63+
process.env["ALTIMATE_CRITIC_GATE"] = "1"
64+
const inputs: any[] = [
65+
undefined,
66+
null,
67+
{},
68+
{ command: 123 },
69+
{ command: { nested: true } },
70+
{ command: "x".repeat(100_000) },
71+
{ command: Array(1000).fill("rm -rf /").join(" && ") },
72+
]
73+
for (const args of inputs) {
74+
const g = await Critic.gate("bash", args ?? {}, Critic.basicSafetyVerifier)
75+
expect(typeof g.allow).toBe("boolean")
76+
}
77+
delete process.env["ALTIMATE_CRITIC_GATE"]
78+
})
79+
80+
test("a verifier that returns a malformed verdict is tolerated", async () => {
81+
process.env["ALTIMATE_CRITIC_GATE"] = "1"
82+
const weird: Critic.Verifier = { verify: () => ({}) as any }
83+
// ok is undefined -> falsy -> treated as a block (safe direction), never throws.
84+
const g = await Critic.gate("bash", { command: "ls" }, weird)
85+
expect(typeof g.allow).toBe("boolean")
86+
delete process.env["ALTIMATE_CRITIC_GATE"]
87+
})
88+
89+
test("ReDoS guard: a huge whitespace-heavy command returns quickly", () => {
90+
const big = "rm" + " ".repeat(200_000) + "-rf /"
91+
const start = performance.now()
92+
const r = Critic.detectDangerousBash(big)
93+
const ms = performance.now() - start
94+
expect(r).not.toBeNull() // collapsed whitespace still matches
95+
expect(ms).toBeLessThan(250) // no catastrophic backtracking
96+
})
97+
})

0 commit comments

Comments
 (0)