Skip to content

Commit d8e136e

Browse files
authored
⏱️ Give the runtime suites deadlines that clear them by a margin (#538) (#539)
On `main` at 5542c92 the composability battery killed the complete Deno suite at its thirty-minute ceiling, and the suite's own last line was `ok | 677 passed (4667 steps) | 0 failed (29m30s)`. That is #482 again one ceiling higher: a deadline firing on a healthy command, costing the signal it exists to give and supplying a false one. The thirty minutes #485 gave the suite tracked the duration observed then. The suite has grown since — on a runner of its own it went 1102s → 1150s → 1314s → 1327s over the last four pushes to `main`, and the battery stretches it by about a third (1427s → 1534s → 1697s → killed at 1800s). A ceiling that tracks a growing suite closely fires as soon as the suite grows into it, so the Deno suite now carries forty-five minutes: wide of the observed duration rather than just past it. The Node and Bun suites run under the same load, and Bun's is stretched to more than twice its standalone time: in the same run it finished healthy at 1,031s against the shared 1,200s ceiling, with Node's at 856s. Both now carry thirty minutes, so the next growth does not open the next `ci-main-red` issue for the same reason. Everything else keeps the twenty-minute default. `commandTimeout()` is still the one place a deadline is answered, the opening line still names every command whose deadline differs, and an explicit coordinator timeout still overrides them all. Co-authored-by: Taras Mankovski <[email protected]>
1 parent 5542c92 commit d8e136e

2 files changed

Lines changed: 57 additions & 24 deletions

File tree

‎scripts/lib/verify.ts‎

Lines changed: 37 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,11 @@
3434
* visible instead of being absorbed by an attempt count.
3535
*
3636
* The deadline is per command, because the commands are not the same size. Most
37-
* clear `COMMAND_TIMEOUT_MILLISECONDS` by a wide margin; the complete Deno suite
38-
* does not, and under the concurrent load of the whole battery it finished with
39-
* zero failures and was killed anyway (#482). A ceiling that fires on a healthy
40-
* command reports a defect that is not there and hides the one that is, so that
41-
* command carries its own.
37+
* clear `COMMAND_TIMEOUT_MILLISECONDS` by a wide margin; the three complete
38+
* runtime suites do not, and under the concurrent load of the whole battery the
39+
* Deno suite finished with zero failures and was killed anyway (#482). A ceiling
40+
* that fires on a healthy command reports a defect that is not there and hides
41+
* the one that is, so each suite carries its own.
4242
*
4343
* Every deadline is announced when the battery starts and named again in the
4444
* line that reports a command it settled, so it is never something a reader has
@@ -86,13 +86,25 @@ export const COMMAND_TIMEOUT_MILLISECONDS = 20 * 60 * 1000;
8686
* How long the complete Deno suite may run.
8787
*
8888
* It is the longest leg of the battery by a wide margin, and it runs beside ten
89-
* others competing for the same cores. At the shared ceiling it was killed on
90-
* `main` twice in a row having reported zero failed tests (#482) — a deadline
91-
* firing on a healthy command, which costs a real signal and supplies a false
92-
* one. This clears the observed duration rather than tracking it closely;
93-
* a ceiling exists to catch a wedge, not to police a slow afternoon.
89+
* others competing for the same cores, which stretches it by about a third over
90+
* what it takes on a runner of its own — and it grows with every test added. A
91+
* ceiling exists to catch a wedge, not to police a slow afternoon: one that
92+
* tracks the observed duration closely fires on a healthy suite as soon as the
93+
* suite grows into it, which costs a real signal and supplies a false one
94+
* (#482). This one clears the observed duration by a wide margin rather than by
95+
* a little.
9496
*/
95-
export const TEST_TIMEOUT_MILLISECONDS = 30 * 60 * 1000;
97+
export const TEST_TIMEOUT_MILLISECONDS = 45 * 60 * 1000;
98+
99+
/**
100+
* How long the complete Node and Bun suites may run.
101+
*
102+
* Each is the same suite under another runtime, and the same concurrent load
103+
* stretches Bun's to more than twice what it takes on a runner of its own,
104+
* within sight of the shared ceiling. The margin that keeps the Deno suite's
105+
* ceiling from firing on a healthy run is the margin these need too.
106+
*/
107+
export const RUNTIME_SUITE_TIMEOUT_MILLISECONDS = 30 * 60 * 1000;
96108

97109
/** The battery, in the order every report uses. */
98110
export const BATTERY: readonly CommandSpec[] = [
@@ -106,8 +118,18 @@ export const BATTERY: readonly CommandSpec[] = [
106118
program: "pnpm",
107119
args: ["exec", "tsc", "--project", "tsconfig.node.json", "--noEmit"],
108120
},
109-
{ id: "test:node", program: "pnpm", args: ["test:node"] },
110-
{ id: "test:bun", program: "bun", args: ["run", "test:bun"] },
121+
{
122+
id: "test:node",
123+
program: "pnpm",
124+
args: ["test:node"],
125+
timeout: RUNTIME_SUITE_TIMEOUT_MILLISECONDS,
126+
},
127+
{
128+
id: "test:bun",
129+
program: "bun",
130+
args: ["run", "test:bun"],
131+
timeout: RUNTIME_SUITE_TIMEOUT_MILLISECONDS,
132+
},
111133
{
112134
id: "docs",
113135
program: "deno",
@@ -232,8 +254,8 @@ function shared(deadlines: readonly number[]): number {
232254
/**
233255
* What the battery announces about its deadlines.
234256
*
235-
* "20m deadline each" while one command has thirty would be a plain untruth in
236-
* the first line of every report, so the commands that differ are named.
257+
* "20m deadline each" while one command has forty-five would be a plain untruth
258+
* in the first line of every report, so the commands that differ are named.
237259
*/
238260
function announced(commands: readonly CommandSpec[], deadlines: readonly number[]): string {
239261
const common = shared(deadlines);

‎scripts/tests/verify-coordinator.test.ts‎

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -280,24 +280,33 @@ describe("line", () => {
280280
/**
281281
* The battery's commands are not the same size, and #482 is what happens when
282282
* one ceiling pretends they are: the complete Deno suite reported zero failed
283-
* tests and was killed anyway, twice in a row on `main`.
283+
* tests and was killed anyway, twice in a row on `main` — and a ceiling that
284+
* tracks a growing suite closely does the same thing a little later.
284285
*
285-
* The race that fires cannot be exercised here — twenty and thirty minutes are
286-
* not durations a test waits out — so what these hold is the single function
286+
* The race that fires cannot be exercised here — twenty to forty-five minutes
287+
* are not durations a test waits out — so what these hold is the single function
287288
* both the race and the report read. A second source of truth is exactly the
288289
* defect: a report naming a deadline that is not the one that settles the
289290
* command is worse than no report.
290291
*/
291292
describe("commandTimeout", () => {
292293
const command = (id: string): CommandSpec => BATTERY.find((entry) => entry.id === id)!;
293294

294-
it("gives the complete Deno suite thirty minutes", function* () {
295-
expect(commandTimeout(command("test"), NO_SITE)).toEqual(30 * 60 * 1000);
296-
expect(command("test").timeout).toEqual(30 * 60 * 1000);
295+
it("gives the complete Deno suite forty-five minutes", function* () {
296+
expect(commandTimeout(command("test"), NO_SITE)).toEqual(45 * 60 * 1000);
297+
expect(command("test").timeout).toEqual(45 * 60 * 1000);
298+
});
299+
300+
it("gives the complete Node and Bun suites thirty minutes", function* () {
301+
for (const id of ["test:node", "test:bun"]) {
302+
expect(commandTimeout(command(id), NO_SITE)).toEqual(30 * 60 * 1000);
303+
expect(command(id).timeout).toEqual(30 * 60 * 1000);
304+
}
297305
});
298306

299307
it("leaves every other command at twenty minutes", function* () {
300-
const others = BATTERY.filter((entry) => entry.id !== "test");
308+
const suites = new Set(["test", "test:node", "test:bun"]);
309+
const others = BATTERY.filter((entry) => !suites.has(entry.id));
301310
expect(others.length).toBeGreaterThan(5);
302311
for (const entry of others) {
303312
expect(commandTimeout(entry, NO_SITE)).toEqual(COMMAND_TIMEOUT_MILLISECONDS);
@@ -317,7 +326,7 @@ describe("deadline reporting", () => {
317326
it("names the commands whose deadline differs instead of claiming one", function* () {
318327
const { host, lines } = recorder({});
319328
yield* verify(host, NO_SITE);
320-
expect(lines[0]).toContain("20m deadline each except test 30m");
329+
expect(lines[0]).toContain("20m deadline each except test 45m, test:node 30m, test:bun 30m");
321330
});
322331

323332
it("claims one deadline only when there is one", function* () {
@@ -339,7 +348,9 @@ describe("deadline reporting", () => {
339348
for (const entry of BATTERY.filter((command) => !command.site)) {
340349
const deadline = commandTimeout(entry, NO_SITE);
341350
const stated =
342-
deadline === COMMAND_TIMEOUT_MILLISECONDS ? "20m deadline each" : `${entry.id} 30m`;
351+
deadline === COMMAND_TIMEOUT_MILLISECONDS
352+
? "20m deadline each"
353+
: `${entry.id} ${deadline / 60_000}m`;
343354
expect(announcement).toContain(stated);
344355
}
345356
});

0 commit comments

Comments
 (0)