Skip to content

fix(ci): flaky unit tests — prompt-during-run and hook timeout #121

Closed

Description

@terisuke

Summary

Two unit tests fail intermittently in CI (GitHub Actions Linux runner), causing unit (linux) job to report failure. These are pre-existing flakes unrelated to any recent PR changes.

Failing Tests

1. prompt submitted during an active run is included in the next LLM input

  • Location: Likely in test/session/ or test/prompt/
  • Failure mode: Timing-dependent — test expects a prompt submitted during an active LLM run to be queued and included in the next input. Race condition between prompt submission and run completion.
  • Frequency: Fails consistently in CI, may pass locally due to faster I/O
  • Root cause hypothesis: Non-deterministic timing in async prompt submission pipeline. Test lacks explicit synchronization barrier.
  • Suggested fix: Add explicit waitFor/flush before assertion, or mock the timing to remove non-determinism.

2. hook.execute > runHook > timeout returns pass

  • Location: Likely in test/hook/
  • Failure mode: Takes 5000.98ms — just barely exceeds the 5000ms timeout threshold
  • Frequency: Fails in CI where runners are slower, passes locally
  • Root cause hypothesis: The timeout boundary is too tight for CI environments. The test expects the hook to resolve within exactly 5000ms but CI jitter pushes it over.
  • Suggested fix: Either increase the timeout threshold by a small margin (e.g., 5100ms) or use a more generous timeout in CI environments via process.env.CI detection.

Impact

  • These failures are non-blocking for PRs that don't touch session/hook code
  • unit results (linux) secondary job also reports failure as a consequence
  • Developers may waste time investigating false positives

Evidence

CI logs from PR #117-#120 (all show identical failures):

(fail) prompt submitted during an active run is included in the next LLM input [283.00ms]
(fail) hook.execute > runHook > timeout returns pass [5000.98ms]

Suggested Actions

  • Investigate and fix the timing issue in prompt-during-run test
  • Adjust hook timeout test to account for CI jitter
  • Consider adding @flaky tag or retry annotation as interim measure
  • Check if these are known upstream issues (anomalyco/opencode)

Ref

Observed across PRs #117, #118, #119, #120 (2026-04-06)

Activity

  1. terisuke commented on Apr 6, 2026

    @terisuke
    Author

    PR #127 investigation results

    Hook timeout test — FIXED

    • Root cause: sleep 10 + 200ms kill timeout overhead exceeded test harness limit
    • Fix: increased test timeout from 5s → 15s
    • CI result: now passes consistently

    Prompt-during-run test — PRE-EXISTING RACE CONDITION

    • Root cause: NOT a timeout issue. Test fails at ~300-400ms (well within 10s limit)
    • The test expects prompt.prompt() to properly queue while a held LLM call is active
    • Effect.sleep(50) barriers and increased polling intervals did not resolve it
    • This is a genuine race condition in the upstream session prompt pipeline
    • The test's llm.wait(1) + immediate second prompt submission creates a window where the session state may not be consistent
    • CI shows fail_on_failure: false — this is informational, not a PR blocker

    Recommendation

    • The hook timeout fix is complete and should be merged
    • The prompt-during-run flake needs a deeper fix in packages/opencode/src/session/prompt.ts (synchronization barrier between concurrent prompt submissions)
    • This is upstream test code — consider filing upstream or accepting as known flake
  2. terisuke commented on Apr 6, 2026

    @terisuke
    Author

    Root cause identified: NOT a flaky test. CI runner downgrade from blacksmith-4vcpu to ubuntu-latest (2 shared vCPU) causes timing-sensitive tests to fail. Superseded by #129 which tracks the proper fix.

    Hook timeout fix included in PR #127 (5s → 15s). Prompt-during-run race needs runner restoration (Option A in #129) or deeper synchronization fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions