Skip to content

Commit 25b1873

Browse files
committed
🤖 Settle a wedged battery command instead of hanging forever
deno bundle and the esbuild service it spawns can deadlock under concurrent load: esbuild_client's two bounded protocol channels fill on both sides under CPU starvation, its select loop and handler task each park in a send the other can never drain, and both processes idle at 0% CPU indefinitely. CI's composability job hit this on both of its runs, hanging silently until cancelled an hour in. Three ceilings turn that into bounded, reported failure: - scripts/lib/retry.ts: retryStuck halts an attempt that outlives its ceiling — tearing down the bundler pair — and runs a fresh one. bundleClient allows three 120s attempts; only a stuck attempt retries, a failing one still fails on the spot. - scripts/lib/verify.ts: every battery command races a 20-minute ceiling; a command settled by it reports (timed out) with everything it had written, and the rest of the battery still finishes. - ci.yml: composability gets timeout-minutes: 60 as the job backstop. A halted build cannot orphan its esbuild service (the service exits on pipe EOF), and a live wedge held exactly one service — the stuck bundle's own child — so the halt-mid-bundle tests are unrelated and stay as they are.
1 parent 092f465 commit 25b1873

6 files changed

Lines changed: 244 additions & 17 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,10 @@ jobs:
209209
# whole chain end to end rather than each link separately.
210210
composability:
211211
runs-on: ubuntu-latest
212+
# A wedged `deno bundle` ↔ esbuild pair inside the battery would otherwise
213+
# hang this job to the six-hour default with nothing printed. The battery
214+
# itself settles a stuck command at 20 minutes; this is the backstop.
215+
timeout-minutes: 60
212216
steps:
213217
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
214218

‎scripts/build-web-client.ts‎

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ import { resolve } from "node:path";
6969
import { pathToFileURL } from "node:url";
7070

7171
import { assertSideEffectFree, sideEffectFreeManifests } from "./lib/side-effect-free.ts";
72+
import { retryStuck } from "./lib/retry.ts";
7273
import { byteLength, generatedModule } from "./lib/web-client-module.ts";
7374

7475
const repoRoot = new URL("../", import.meta.url);
@@ -116,22 +117,37 @@ function* run(command: string, args: string[]): Operation<void> {
116117
yield* exec(command, { arguments: args, cwd: new URL(repoRoot).pathname }).expect();
117118
}
118119

120+
/**
121+
* A wedged bundler pair sits at 0% CPU forever (see `lib/retry.ts`), so the
122+
* ceiling only has to clear the slowest healthy bundle — ~1s alone, well
123+
* under a minute even inside the concurrent battery. Halting the attempt
124+
* kills the pair: the esbuild service exits when its parent's pipe closes,
125+
* which the halt-mid-bundle test observes.
126+
*/
127+
const BUNDLE_STUCK_MILLISECONDS = 120_000;
128+
const BUNDLE_ATTEMPTS = 3;
129+
119130
function* bundleClient(scratch?: string): Operation<string> {
120131
return yield* scoped(function* () {
121132
const output = yield* until(Deno.makeTempFile({ dir: scratch, suffix: ".js" }));
122133
yield* ensure(() => rm(output));
123-
yield* run(Deno.execPath(), [
124-
"bundle",
125-
"--platform=browser",
126-
"--minify",
127-
"--packages=bundle",
128-
"--node-modules-dir=manual",
129-
"--no-remote",
130-
"--frozen",
131-
"--output",
132-
output,
133-
CLIENT_ENTRY,
134-
]);
134+
yield* retryStuck(
135+
"deno bundle",
136+
() =>
137+
run(Deno.execPath(), [
138+
"bundle",
139+
"--platform=browser",
140+
"--minify",
141+
"--packages=bundle",
142+
"--node-modules-dir=manual",
143+
"--no-remote",
144+
"--frozen",
145+
"--output",
146+
output,
147+
CLIENT_ENTRY,
148+
]),
149+
{ milliseconds: BUNDLE_STUCK_MILLISECONDS, attempts: BUNDLE_ATTEMPTS },
150+
);
135151
return yield* readTextFile(output);
136152
});
137153
}

‎scripts/lib/retry.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
import { Err, Ok, race, sleep } from "effection";
2+
import type { Operation, Result } from "effection";
3+
4+
/** How long one attempt may run, and how many attempts there are. */
5+
export interface RetryStuckOptions {
6+
milliseconds: number;
7+
attempts: number;
8+
}
9+
10+
function* settled<T>(operation: () => Operation<T>): Operation<Result<T>> {
11+
return Ok(yield* operation());
12+
}
13+
14+
function* expired<T>(subject: string, milliseconds: number): Operation<Result<T>> {
15+
yield* sleep(milliseconds);
16+
return Err(new Error(`${subject} was still running after ${milliseconds}ms`));
17+
}
18+
19+
/**
20+
* Run `operation`; if it is still running when the ceiling passes, halt it —
21+
* tearing down everything it started — and run a fresh one, up to `attempts`
22+
* times in all.
23+
*
24+
* This retries only an operation that never settles. One that fails has
25+
* decided, and its error propagates on the spot.
26+
*
27+
* It exists for `deno bundle`: under concurrent load the bundler and the
28+
* esbuild service it spawns can lose their protocol handshake and wait on
29+
* each other forever, both at 0% CPU (denoland/esbuild_client's bounded
30+
* protocol channels can mutually fill). No observed wedge ever resolved
31+
* itself, so an attempt that outlives a generous ceiling is stuck, not slow
32+
* — and a fresh attempt is overwhelmingly likely to succeed.
33+
*/
34+
export function* retryStuck<T>(
35+
subject: string,
36+
operation: () => Operation<T>,
37+
{ milliseconds, attempts }: RetryStuckOptions,
38+
): Operation<T> {
39+
for (let attempt = 1; attempt < attempts; attempt++) {
40+
const result = yield* race([settled(operation), expired<T>(subject, milliseconds)]);
41+
if (result.ok) {
42+
return result.value;
43+
}
44+
}
45+
const last = yield* race([settled(operation), expired<T>(subject, milliseconds)]);
46+
if (last.ok) {
47+
return last.value;
48+
}
49+
throw new Error(
50+
`${subject} did not finish within ${milliseconds}ms in any of ${attempts} attempts`,
51+
);
52+
}

‎scripts/lib/verify.ts‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,13 @@
2626
* before anything is decided: one runtime failing must not cancel another, and
2727
* must not skip the fingerprint that runs afterwards — a failing battery is
2828
* exactly when a dirtied tree would otherwise go unnoticed.
29+
*
30+
* A command that outlives `COMMAND_TIMEOUT_MILLISECONDS` is settled by the
31+
* battery itself — killed, counted failed, reported with whatever it wrote —
32+
* because a battery with no ceiling hangs silently when a command wedges.
2933
*/
3034

31-
import { allSettled } from "effection";
35+
import { allSettled, race, sleep } from "effection";
3236
import type { Operation, Result } from "effection";
3337

3438
import { parseNameStatus, parsePorcelain, touches } from "./changed-paths.ts";
@@ -73,8 +77,21 @@ export const BATTERY: readonly CommandSpec[] = [
7377
export interface Settled {
7478
code: number;
7579
milliseconds: number;
80+
/** Settled by the ceiling rather than by its own exit. */
81+
timedOut?: boolean;
7682
}
7783

84+
/**
85+
* How long one command may run before the battery gives up on it.
86+
*
87+
* `deno bundle` and its esbuild service can deadlock under concurrent load —
88+
* both idle at 0% CPU, never recovering — and a battery with no ceiling then
89+
* hangs to whatever kills it from outside, with nothing printed. The ceiling
90+
* clears every healthy command by a wide margin: the slowest observed leg of
91+
* the battery is minutes, not tens of minutes.
92+
*/
93+
export const COMMAND_TIMEOUT_MILLISECONDS = 20 * 60 * 1000;
94+
7895
export interface VerifyHost {
7996
/** Start `command`, spool its output, and settle with its status. */
8097
run(command: CommandSpec): Operation<Settled>;
@@ -93,6 +110,8 @@ export interface VerifyHost {
93110
export interface VerifyOptions {
94111
/** `off` is how `verify:clean` runs the battery inside a fresh clone. */
95112
site: "auto" | "off";
113+
/** Milliseconds one command may run; `COMMAND_TIMEOUT_MILLISECONDS` when absent. */
114+
timeout?: number;
96115
}
97116

98117
interface Applicability {
@@ -144,6 +163,16 @@ function seconds(milliseconds: number): string {
144163
return `${Math.round(milliseconds / 100) / 10}s`;
145164
}
146165

166+
/**
167+
* Losing the race halts `host.run`, whose scope teardown signals the
168+
* command's process group and closes its spool — so a command settled this
169+
* way still reports with everything it had written.
170+
*/
171+
function* expiry(milliseconds: number): Operation<Settled> {
172+
yield* sleep(milliseconds);
173+
return { code: 1, milliseconds, timedOut: true };
174+
}
175+
147176
/**
148177
* Run the applicable battery and report it.
149178
*
@@ -153,9 +182,12 @@ function seconds(milliseconds: number): string {
153182
export function* verify(host: VerifyHost, options: VerifyOptions): Operation<number> {
154183
const before = yield* host.fingerprint();
155184
const { commands, reason } = yield* applicable(host, options);
185+
const timeout = options.timeout ?? COMMAND_TIMEOUT_MILLISECONDS;
156186

157187
host.log(`verifying ${commands.length} commands concurrently (${reason})`);
158-
const settled = yield* allSettled(commands.map((command) => host.run(command)));
188+
const settled = yield* allSettled(
189+
commands.map((command) => race([host.run(command), expiry(timeout)])),
190+
);
159191
const after = yield* host.fingerprint();
160192

161193
const failures: CommandSpec[] = [];
@@ -167,9 +199,8 @@ export function* verify(host: VerifyHost, options: VerifyOptions): Operation<num
167199
continue;
168200
}
169201
failures.push(command);
170-
host.log(
171-
` FAILED ${command.id.padEnd(10)} ${seconds(result.milliseconds)} (exit ${result.code})`,
172-
);
202+
const why = result.timedOut ? "timed out" : `exit ${result.code}`;
203+
host.log(` FAILED ${command.id.padEnd(10)} ${seconds(result.milliseconds)} (${why})`);
173204
}
174205

175206
const [first, ...rest] = failures;

‎scripts/tests/retry.test.ts‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
import { describe, it } from "@executablemd/test-support/bdd";
2+
import { expect } from "@executablemd/test-support/expect";
3+
import { ensure, sleep, suspend } from "effection";
4+
5+
import { retryStuck } from "../lib/retry.ts";
6+
7+
describe("retryStuck", () => {
8+
it("returns a settled operation's value from the first attempt", function* () {
9+
let attempts = 0;
10+
11+
const value = yield* retryStuck(
12+
"quick",
13+
function* () {
14+
attempts += 1;
15+
return "done";
16+
},
17+
{ milliseconds: 1000, attempts: 3 },
18+
);
19+
20+
expect(value).toBe("done");
21+
expect(attempts).toBe(1);
22+
});
23+
24+
it("halts a stuck attempt, tears it down, and starts over", function* () {
25+
let attempts = 0;
26+
const torndown: number[] = [];
27+
28+
const value = yield* retryStuck(
29+
"stuck once",
30+
function* () {
31+
attempts += 1;
32+
const attempt = attempts;
33+
yield* ensure(() => {
34+
torndown.push(attempt);
35+
});
36+
if (attempt === 1) {
37+
yield* suspend();
38+
}
39+
return "recovered";
40+
},
41+
{ milliseconds: 20, attempts: 3 },
42+
);
43+
44+
expect(value).toBe("recovered");
45+
expect(attempts).toBe(2);
46+
expect(torndown).toEqual([1, 2]);
47+
});
48+
49+
it("gives up after the last attempt and names the ceiling", function* () {
50+
let attempts = 0;
51+
let failure: unknown;
52+
53+
try {
54+
yield* retryStuck(
55+
"always stuck",
56+
function* () {
57+
attempts += 1;
58+
yield* suspend();
59+
return "unreachable";
60+
},
61+
{ milliseconds: 10, attempts: 2 },
62+
);
63+
} catch (error) {
64+
failure = error;
65+
}
66+
67+
expect(attempts).toBe(2);
68+
expect(String(failure)).toContain("always stuck");
69+
expect(String(failure)).toContain("2 attempts");
70+
});
71+
72+
it("propagates a failure without retrying: failing is settling", function* () {
73+
let attempts = 0;
74+
let failure: unknown;
75+
76+
try {
77+
yield* retryStuck(
78+
"failing",
79+
function* (): Generator<never, void> {
80+
attempts += 1;
81+
throw new Error("the attempt itself failed");
82+
},
83+
{ milliseconds: 1000, attempts: 3 },
84+
);
85+
} catch (error) {
86+
failure = error;
87+
}
88+
89+
expect(attempts).toBe(1);
90+
expect(String(failure)).toContain("the attempt itself failed");
91+
});
92+
93+
it("lets a slow-but-alive attempt finish under the ceiling", function* () {
94+
const value = yield* retryStuck(
95+
"slow",
96+
function* () {
97+
yield* sleep(10);
98+
return 42;
99+
},
100+
{ milliseconds: 5000, attempts: 2 },
101+
);
102+
103+
expect(value).toBe(42);
104+
});
105+
});

‎scripts/tests/verify-coordinator.test.ts‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,25 @@ describe("verify", () => {
187187
expect(results(lines)).toContain("FAILED test:node");
188188
});
189189

190+
it("settles a command that outlives the ceiling and reports it timed out", function* () {
191+
const { host, lines } = recorder({ delays: { test: 60_000 } });
192+
expect(yield* verify(host, { site: "off", timeout: 25 })).toEqual(1);
193+
expect(results(lines)).toContain("FAILED test");
194+
expect(lines.join("\n")).toContain("(timed out)");
195+
});
196+
197+
it("prints what a timed-out command had already written", function* () {
198+
const { host, emitted } = recorder({ delays: { test: 60_000 } });
199+
yield* verify(host, { site: "off", timeout: 25 });
200+
expect(emitted).toEqual(["<output of test>"]);
201+
});
202+
203+
it("lets the rest of the battery finish when one command times out", function* () {
204+
const { host, lines } = recorder({ delays: { test: 60_000, "test:bun": 10 } });
205+
yield* verify(host, { site: "off", timeout: 40 });
206+
expect(results(lines)).toContain("ok test:bun");
207+
});
208+
190209
/** The fingerprint runs after everything settles, so a failing run still proves it. */
191210
it("fails when the tracked tree moved, even though every command passed", function* () {
192211
const moved: TrackedState = new Map([["a.ts", { ...FILE, executable: true }]]);

0 commit comments

Comments
 (0)