Skip to content

run-slot gauges lock test flakes: eight looks with no reading run top 6 times #956

Closed

Description

@nrjdalal

Problem

tests/scripts/ops/run-slot-gauges-lock.test.ts failed once on canary at d216f23 and passed on a rerun of the same commit, so it is a flake. It turned Audit, Lint, and Build red on canary and held a ship for one rerun.

The failing assertion:

(fail) the gauges refresh under one lock > eight looks with no reading yet run top once [1631.38ms]

49 |   test("eight looks with no reading yet run top once", async () => {
50 |     const m = machine({ topDelay: 1 })
51 |     await slotsTogether(m, 8)
52 |     expect(tops(m)).toBe(1)

Expected: 1
Received: 6

A lead, not yet checked

gauges() in scripts/ops/run-slot.sh stamps the reading it writes with now, the time the look began, not the time the reading landed. On a loaded runner, a refresh that takes 10 s or more writes a reading that is already stale, so the next look takes the freed lock and runs top again. The six tops would then be refreshes one after another, not at once. This is a reading of the code; a run that shows it is part of the fix.

Done when

  • The test's cause is shown by a run, for example by slowing the fake top or the fake runner.
  • The fix makes the test hold under that slowdown, or the test stops depending on the runner's speed.
  • The test passes 20 runs in a row under bun run test:file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions