Skip to content

Commit d25189d

Browse files
only21milclaude
andcommitted
fix(server): keep the shared Codex app-server scope alive after a tool OOM kill
Since #17662 every agent scope starts with OOMPolicy=stop. The Codex app-server is shared by every thread on a provider instance, and the commands its tools run live in the same scope. When the kernel OOM-killed one tool process, systemd stopped the whole scope, so the app-server died and every Codex thread on that instance failed. AgentScope.wrap now takes `shared`, and the Codex app-server spawn passes it. On systemd 253 or newer, shared scopes start with OOMPolicy=continue, so systemd logs the kill and the app-server keeps running. Claude and terminal scopes keep OOMPolicy=stop. The probe and the plain-scope fallback for older systemd are unchanged. With OOMPolicy=continue systemd never records Result=oom-kill, so such a scope can never explain one thread's failure. It is no longer tracked for the thread that opened it. Otherwise every later failed turn on that thread would poll the scope for about 2 seconds, waiting for a stop that never comes. Launching it still ends the thread's previous answer, and nothing reads its result, so it gets --collect like thread-less scopes. Older user managers cannot set OOMPolicy on a scope. There a shared scope starts as before and stays tracked for its thread without --collect, because systemd can still stop it on an OOM kill and record Result=oom-kill (systemd 252 does). Not changed: - The kernel still picks the victim. If it kills the app-server itself, every thread on the instance still loses it. - A systemd-oomd kill still ends the whole scope. - On systemd 253 or newer, a Codex turn that fails because its app-server was itself killed for memory is no longer labeled "Killed: out of memory.", since systemd records no OOM result under OOMPolicy=continue. Before, only the thread that opened the session got that label. - Recovering a session whose app-server died is #17046 and #18139. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent bed9239 commit d25189d

4 files changed

Lines changed: 121 additions & 14 deletions

File tree

‎apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1369,6 +1369,8 @@ export const makeCodexAppServerSpawnCommand = Effect.fn(
13691369
name: "codex",
13701370
threadId: input.threadId,
13711371
env: input.env,
1372+
// Every thread on a provider instance uses this app-server.
1373+
shared: true,
13721374
});
13731375
const spawnCommand = yield* resolveSpawnCommand(launch.command, launch.args, {
13741376
...(input.env === undefined ? {} : { env: input.env }),

‎apps/server/src/process/agentScope.test.ts‎

Lines changed: 96 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,15 @@ import * as NodeOS from "node:os";
44
import * as NodePath from "node:path";
55

66
import * as NodeServices from "@effect/platform-node/NodeServices";
7-
import { describe, expect, it } from "@effect/vitest";
7+
import { assert, describe, expect, it } from "@effect/vitest";
8+
import { AgentScope, type AgentScopeShape } from "@t3tools/shared/AgentScope";
89
import * as HostProcess from "@t3tools/shared/HostProcess";
910
import * as Effect from "effect/Effect";
10-
import { ChildProcessSpawner } from "effect/process";
11+
import * as Fiber from "effect/Fiber";
12+
import * as FileSystem from "effect/FileSystem";
13+
import { ChildProcess, ChildProcessSpawner } from "effect/process";
1114

15+
import { makeCodexAppServerSpawnCommand } from "../orchestration-v2/Adapters/CodexAdapterV2.ts";
1216
import * as ProcessRunner from "../processRunner.ts";
1317
import { agentSliceMemoryLimits, classifyScope, make, type ScopeState } from "./agentScope.ts";
1418

@@ -54,24 +58,33 @@ describe("agentSliceMemoryLimits", () => {
5458
});
5559
});
5660

57-
// Runs the real service against a fake user manager. `units` holds the
58-
// ActiveState and Result that `systemctl show` reports for each scope.
61+
interface FakeUnit {
62+
activeState: string;
63+
result: string;
64+
/** `oom_kill` in the scope's memory.events. */
65+
oomKills?: number;
66+
}
67+
68+
// Runs the real service against a fake user manager. `units` holds what
69+
// `systemctl show` and the cgroup files report for each scope. A manager
70+
// without `oomPolicy` (before systemd 253) rejects OOMPolicy on scopes.
5971
const withFakeSystemd = <A, E>(
6072
body: (input: {
6173
readonly scope: Effect.Success<typeof make>;
62-
readonly units: Map<string, { activeState: string; result: string }>;
74+
readonly units: Map<string, FakeUnit>;
6375
}) => Effect.Effect<A, E>,
76+
{ oomPolicy = true }: { readonly oomPolicy?: boolean } = {},
6477
) =>
6578
Effect.gen(function* () {
6679
const bin = NodeFS.mkdtempSync(NodePath.join(NodeOS.tmpdir(), "t3-agent-scope-"));
6780
for (const name of ["systemd-run", "systemctl", "agent"]) {
6881
NodeFS.writeFileSync(NodePath.join(bin, name), "", { mode: 0o755 });
6982
}
70-
const units = new Map<string, { activeState: string; result: string }>();
71-
const output = (stdout: string): ProcessRunner.ProcessRunOutput => ({
83+
const units = new Map<string, FakeUnit>();
84+
const output = (stdout: string, code = 0): ProcessRunner.ProcessRunOutput => ({
7285
stdout,
7386
stderr: "",
74-
code: ChildProcessSpawner.ExitCode(0),
87+
code: ChildProcessSpawner.ExitCode(code),
7588
timedOut: false,
7689
stdoutTruncated: false,
7790
stderrTruncated: false,
@@ -81,19 +94,34 @@ const withFakeSystemd = <A, E>(
8194
const runner = ProcessRunner.ProcessRunner.of({
8295
run: ({ args }) =>
8396
Effect.sync(() => {
97+
if (!oomPolicy && args.some((arg) => arg.startsWith("--property=OOMPolicy="))) {
98+
return output("", 1);
99+
}
84100
const [, verb, unit = ""] = args;
85101
if (verb === "reset-failed") units.delete(unit);
86102
if (verb !== "show") return output("");
87103
const state = units.get(unit);
88104
return output(
89105
state === undefined
90106
? "LoadState=not-found\nActiveState=inactive\nResult=success\n"
91-
: `LoadState=loaded\nActiveState=${state.activeState}\nResult=${state.result}\n`,
107+
: `LoadState=loaded\nActiveState=${state.activeState}\nResult=${state.result}\nControlGroup=/test.slice/${unit}\n`,
92108
);
93109
}),
94110
});
111+
const fileSystem = yield* FileSystem.FileSystem;
95112
const scope = yield* make.pipe(
96113
Effect.provideService(ProcessRunner.ProcessRunner, runner),
114+
Effect.provideService(FileSystem.FileSystem, {
115+
...fileSystem,
116+
readFileString: (file, encoding) =>
117+
file.startsWith("/sys/fs/cgroup/")
118+
? Effect.succeed(
119+
file.endsWith("/memory.events")
120+
? `oom_kill ${units.get(NodePath.basename(NodePath.dirname(file)))?.oomKills ?? 0}\n`
121+
: "",
122+
)
123+
: fileSystem.readFileString(file, encoding),
124+
}),
97125
Effect.provideService(HostProcess.Platform, "linux"),
98126
Effect.provideService(HostProcess.Environment, { PATH: bin, XDG_RUNTIME_DIR: "/run/user/1" }),
99127
);
@@ -105,6 +133,17 @@ const withFakeSystemd = <A, E>(
105133
const unitOf = (launch: { readonly args: ReadonlyArray<string> }) =>
106134
launch.args.find((arg) => arg.startsWith("--unit="))?.slice("--unit=".length) ?? "";
107135

136+
// Starts the Codex app-server the way its adapter does.
137+
const launchCodex = (scope: AgentScopeShape, threadId: string) =>
138+
makeCodexAppServerSpawnCommand({ command: "agent", args: ["app-server"], threadId }).pipe(
139+
Effect.provideService(AgentScope, scope),
140+
Effect.provideService(HostProcess.Platform, "linux"),
141+
Effect.map((command) => {
142+
assert(ChildProcess.isStandardCommand(command));
143+
return command;
144+
}),
145+
);
146+
108147
describe("AgentScope service", () => {
109148
it.effect("labels a thread from its newest scope until its next session clears it", () =>
110149
withFakeSystemd(({ scope, units }) =>
@@ -129,6 +168,54 @@ describe("AgentScope service", () => {
129168
),
130169
);
131170

171+
it.effect("keeps the shared Codex app-server running after a tool's OOM kill", () =>
172+
withFakeSystemd(({ scope }) =>
173+
Effect.gen(function* () {
174+
const codex = yield* launchCodex(scope, "a");
175+
const agent = yield* scope.wrap({ command: "agent", args: [], name: "t", threadId: "b" });
176+
// OOMPolicy=stop would end the app-server of every Codex thread.
177+
expect(codex.args).toContain("--property=OOMPolicy=continue");
178+
expect(codex.args).not.toContain("--property=OOMPolicy=stop");
179+
expect(agent.args).toContain("--property=OOMPolicy=stop");
180+
}),
181+
),
182+
);
183+
184+
it.effect("does not wait on a shared scope that outlived an OOM kill", () =>
185+
withFakeSystemd(({ scope, units }) =>
186+
Effect.gen(function* () {
187+
const agent = yield* scope.wrap({ command: "agent", args: [], name: "t", threadId: "a" });
188+
units.set(unitOf(agent), { activeState: "failed", result: "oom-kill" });
189+
expect(yield* scope.oomKilled("a")).toBe(true);
190+
191+
// The thread moves to the shared app-server, and the kernel kills one
192+
// of its tools. The scope keeps running and never records an OOM kill.
193+
const codex = yield* launchCodex(scope, "a");
194+
units.set(unitOf(codex), { activeState: "active", result: "success", oomKills: 1 });
195+
const answer = yield* scope
196+
.oomKilled("a")
197+
.pipe(Effect.forkChild({ startImmediately: true }));
198+
expect(answer.pollUnsafe(), "answers without waiting for a stop").toBeDefined();
199+
expect(yield* Fiber.join(answer)).toBe(false);
200+
}),
201+
),
202+
);
203+
204+
it.effect("labels the shared Codex app-server's OOM kill on systemd without OOMPolicy", () =>
205+
withFakeSystemd(
206+
({ scope, units }) =>
207+
Effect.gen(function* () {
208+
const codex = yield* launchCodex(scope, "a");
209+
// Older managers still stop a scope on an OOM kill and record it.
210+
units.set(unitOf(codex), { activeState: "failed", result: "oom-kill" });
211+
expect(yield* scope.oomKilled("a")).toBe(true);
212+
expect(codex.args.filter((arg) => arg.includes("OOMPolicy"))).toEqual([]);
213+
expect(codex.args).not.toContain("--collect");
214+
}),
215+
{ oomPolicy: false },
216+
),
217+
);
218+
132219
it.effect("keeps a running scope while hundreds of other threads launch", () =>
133220
withFakeSystemd(({ scope, units }) =>
134221
Effect.gen(function* () {

‎apps/server/src/process/agentScope.ts‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,9 @@ export const make = Effect.gen(function* () {
169169
systemctl: systemctl.value,
170170
runtimeDir,
171171
scopeOptions,
172+
// A shared process keeps serving its other threads when the kernel
173+
// kills one of its tools, so its scope keeps running instead.
174+
sharedScopeOptions: scopeOptions.length > 0 ? ["--property=OOMPolicy=continue"] : [],
172175
};
173176
}),
174177
);
@@ -291,22 +294,29 @@ export const make = Effect.gen(function* () {
291294
});
292295

293296
return {
294-
wrap: Effect.fn("AgentScope.wrap")(function* ({ command, args, name, threadId, env }) {
297+
wrap: Effect.fn("AgentScope.wrap")(function* ({ command, args, name, threadId, env, shared }) {
295298
const resolved = yield* resolve(command, env ?? hostEnvironment);
296299
// Leave unresolved commands alone so the spawn reports the missing binary.
297300
if (resolved._tag === "None") return { command, args };
298301
const scope = yield* systemd;
299302
if (scope === undefined) return agentScopeCommand({ command: resolved.value, args });
300303
const unit = `t3code-${name}-${NodeCrypto.randomUUID().slice(0, 8)}.scope`;
301-
if (threadId !== undefined) yield* track(scope.systemctl, threadId, unit);
304+
// With OOMPolicy=continue systemd never records `Result=oom-kill`, so a
305+
// shared scope answers for no thread. Launching one still ends the
306+
// thread's previous answer. Without OOMPolicy systemd may still stop
307+
// the scope and record it, so the thread tracks it like any other.
308+
const continuing = shared === true && scope.sharedScopeOptions.length > 0;
309+
const owned = threadId !== undefined && !continuing;
310+
if (threadId !== undefined) yield* track(scope.systemctl, threadId, owned ? unit : undefined);
311+
const options = continuing ? scope.sharedScopeOptions : scope.scopeOptions;
302312
const systemdScope: AgentSystemdScope = {
303313
systemdRun: scope.systemdRun,
304314
runtimeDir: scope.runtimeDir,
305315
slice: AGENT_SLICE,
306316
unit,
307-
// Without a thread nothing reads the result, so let systemd unload
308-
// the unit even when it fails.
309-
options: threadId === undefined ? [...scope.scopeOptions, "--collect"] : scope.scopeOptions,
317+
// Without an owning thread nothing reads the result, so let systemd
318+
// unload the unit even when it fails.
319+
options: owned ? options : [...options, "--collect"],
310320
};
311321
return agentScopeCommand({ command: resolved.value, args, scope: systemdScope });
312322
}),

‎packages/shared/src/AgentScope.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,14 @@ export interface AgentScopeShape {
2525
readonly name: string;
2626
readonly threadId?: string | undefined;
2727
readonly env?: NodeJS.ProcessEnv | undefined;
28+
/**
29+
* Several threads use this process, as with the Codex app-server. On
30+
* systemd 253 or newer its scope runs with OOMPolicy=continue: it keeps
31+
* running when the kernel kills one of its processes for memory, and
32+
* `oomKilled` answers false for it at once. Older systemd treats it like
33+
* any other scope.
34+
*/
35+
readonly shared?: boolean | undefined;
2836
}) => Effect.Effect<AgentScopeCommand>;
2937
/**
3038
* Whether the latest agent scope of this thread was killed because the

0 commit comments

Comments
 (0)