Skip to content

usageDeadlineWaitsForAnInFlightRateLimitUpdateDecision misses the usage-limit event when both fixture sessions win the mkdir role lock #1016

Closed

Description

@martinfrancois

Symptom

CodexAppServerClientTest.usageDeadlineWaitsForAnInFlightRateLimitUpdateDecision failed in the CI test job three times on 2026-10-24, on PRs that do not touch CodexAppServerClient or its test:

Each time the accepted=false case waited the full five seconds and failed:

CodexAppServerClientTest.usageDeadlineWaitsForAnInFlightRateLimitUpdateDecision:1201 [the usage-limit event is published within 5 seconds]
Expecting value to be true but was false

All three runs came after aebc42e moved the test job to Ubuntu 26.04. None of the 60 most recent failed CI runs before that commit (back to 2026-10-04) failed this test.

Cause

The fixture starts two app-server processes from one script, and each picks its role with if mkdir <dir>; then session=rate-limit; else session=usage; fi. That only works if exactly one of two concurrent mkdir calls succeeds.

Ubuntu 26.04 ships uutils coreutils 0.10.0 as mkdir. Its non-recursive mkdir treats a failed mkdir(2) as success when the path is a directory by then, so when both processes race, both can succeed. Both sessions then send account/rateLimits/updated, nobody sends the usage-limit failure, and the latch at line 1201 times out. uutils fixed this in uutils/coreutils#13986 (issue uutils/coreutils#13970), released in 0.11.0.

Two concurrent mkdir calls on one new path, repeated 1,000 times:

Image mkdir Both succeeded
ubuntu:26.04 uutils coreutils 0.10.0 299
ubuntu:24.04 GNU coreutils 9.4 0

Reproduction

Running only this test 30 times in eclipse-temurin:25-jdk-resolute (Ubuntu 26.04) with eight nice -n 19 yes loads failed 4 of 30 runs, always at line 1201, in both parameter cases. The same loop on a host with GNU coreutils passed 30 of 30.

atomicallySharesConcurrentSparseRateLimitsWithUsageDeadline in the same class picks roles the same way for two concurrent sessions, so it can fail the same way. sharesSparseRateLimitsAndMetadataAcrossSequentialAppServerSessions uses the same mkdir lock, but its sessions run one after the other.

Fix

Pick each fixture session's role from something the test controls instead of a lock, for example the workspace directory the session runs in. Then the test also knows which future is the usage-limit run and no longer has to find out from the published event.

Compatibility

No user-facing contract change. This is test-only work.

Activity

  1. added
    bugSomething isn't working
    priority mediumMedium-priority issue: setup validation, diagnostics accuracy, or meaningful UX correctness
    flakyConfirmed nondeterministic test that requires root-cause repair and recurrence tracking
    size SExpected change: 50–199 lines (additions plus deletions, including tests and docs).
    on Oct 24, 2026
  2. added 2 commits that reference this issue on Oct 24, 2026
    cf56e6a
    5a7d070
  3. added a commit that references this issue on Oct 29, 2026
    dc56a88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingflakyConfirmed nondeterministic test that requires root-cause repair and recurrence trackingpriority mediumMedium-priority issue: setup validation, diagnostics accuracy, or meaningful UX correctnesssize SExpected change: 50–199 lines (additions plus deletions, including tests and docs).

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions