Skip to content

Cut CI test time by 68% while preserving the full test matrix - #913

Merged
sds merged 6 commits into
mainfrom
perf/parallel-ci-tests
Nov 5, 2026
Merged

sds merged 6 commits into
mainfrom
perf/parallel-ci-tests

Conversation

@sds

@sds sds commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

The complete Tests workflow drops from 15m 42s to 5m 04s (68% faster) on GitHub Actions. Every one of the 1,679 examples still runs on each of the existing 16 Ruby/OS combinations: 26,864 verified example executions, with no examples or matrix entries removed.

Changes:

  • Run examples in isolated processes: eight workers on Linux and four on Windows. Distribute individual examples so large spec files cannot monopolize a worker. Check the executed example IDs against discovery to catch missing or duplicate examples, propagate failures, and merge all worker coverage before Coveralls reporting.
  • Set temporary repositories' local defaults with one config write instead of five Git subprocesses per repository.
  • Resolve Gemfile fixtures from installed gems, check setup command success, and isolate generated executables and commit-message files for concurrent tests.
  • Exclude lint dependencies from test installs, remove the obsolete ffi dependency (ChildProcess 5 uses Ruby's process API), skip unused Windows build tools, and invoke Git directly rather than through its Windows launcher.

GitHub Actions verification on afbe8ed:

Measurement Before After
Complete Tests workflow, including queue/setup/coverage 15m 42s 5m 04s
Slowest Linux test step 1m 34s 40s
Slowest Windows test step 12m 32s 3m 44s

Baseline run · Verified complete run · Passing lint

All 16 test jobs and coverage finalization pass. Logs confirm 1,679/1,679 examples in every job. Local checks also verify that the runner exits unsuccessfully for both failing tests and omitted examples.

Sub-minute limitation: that target was not achieved on the existing hosted runners. Even the fastest complete Windows suite takes 1m 53s; the slowest takes 3m 44s, before setup and coverage reporting. These tests exercise real Git repositories, hooks, and Ruby subprocesses. Increasing worker count caused severe contention on older Windows Rubies; four workers produced the verified passing result. Reaching a reliable sub-minute result would require additional/faster prewarmed Windows runner capacity and further sharding, or reducing the platform/version coverage. This change preserves that coverage.

@sds sds changed the title Reduce CI test time with process parallelism and faster fixtures Cut CI test time by 68% while preserving the full test matrix Nov 5, 2026
@sds
sds marked this pull request as ready for review November 5, 2026 07:40
@sds
sds merged commit 3b8832a into main Nov 5, 2026
35 checks passed
@sds
sds deleted the perf/parallel-ci-tests branch November 5, 2026 07:40
sds added a commit to guillaume-d/overcommit that referenced this pull request Nov 10, 2026
* origin/main: (108 commits)
  Remember all previously approved configurations and hooks (sds#665)
  Keep deleted and renamed paths absent while stashing changes (sds#928)
  Document the case-sensitive KtLint hook name (sds#927)
  Update CI actions and pin Ubuntu to clear annotations (sds#926)
  Preserve standard input bytes when forwarding hooks (sds#925)
  Preserve Git-quoted filenames when running all hooks (sds#924)
  Fail when Git cannot enumerate tracked files (sds#923)
  Add note about disabling default hooks (sds#922)
  Close files inspected by LineEndings (sds#921)
  Use Script Host before PowerShell for Windows parent query fallback (sds#920)
  Cache Windows dependency resolution with daily refreshes (sds#919)
  Avoid PowerShell startup for Windows parent command queries (sds#918)
  Avoid repeated CLI startup when preparing integration fixtures (sds#917)
  Allocate more CI runners to Windows with asymmetric sharding (sds#916)
  Shard CI tests across runners and add one required CI check (sds#915)
  Remove EOL Ruby versions from the CI matrix (sds#914)
  Cut CI test time by 68% while preserving the full test matrix (sds#913)
  Cut version 0.75.0 (sds#912)
  Re-enable the full Windows test matrix and fix platform failures (sds#911)
  Read .overcommit.yml as UTF-8 in the hook template (sds#910)
  Update Overcommit logo (sds#907)
  Cut version 0.74.0 (sds#906)
  Keep each worktree's signatures separate (sds#903)
  Recognize amend aliases on macOS (sds#904)
  Exclude files deleted from the working tree from all_files (sds#905)
  Restore the Ruby 2.6-3.1 test checks (sds#902)
  Delete subprocess output tempfiles once they are read (sds#901)
  Stop wrapping commands in cmd.exe on Windows (sds#898)
  Allow URL-only lines to exceed TextWidth body limit (sds#900)
  Cut version 0.73.0 (sds#896)
  Add automated release flow (sds#895)
  Instruct oxfmt not to fail if no files are matched (sds#894)
  Add filetypes to oxc tools (sds#893)
  Add pre commit hook for herb-lint (sds#892)
  Cut version 0.72.0 (sds#891)
  Remove maintainability badge (sds#890)
  Require patched REXML version (sds#889)
  Fix operation state restoration in linked worktrees (sds#887)
  Fail Flay hook when flay crashes instead of silently passing (sds#885)
  Fix Solargraph regexp capture warning (sds#886)
  Cut version 0.71.0 (sds#883)
  Prepare for the new solargraph typecheck format (sds#873)
  Make hook context input reads thread-safe (sds#882)
  Cut version 0.70.0 (sds#880)
  Add oxc pre-commit hooks (sds#879)
  Add Ruby 3.4 and 4.0 to CI matrix (sds#878)
  Cut version 0.69.0 (sds#877)
  Use Etc.nprocessors and handle sandboxed environments (sds#876)
  Cut version 0.68.0 (sds#872)
  Add solargraph pre-commit hook for typechecking (sds#871)
  Update RuboCop in Gemfile to silence a deprecation warning in build (sds#870)
  Cut version 0.67.1
  adds 'set' requirements to diff hook context. (sds#866)
  Add note about unsupported YAML features in `gemfile:` line in default.yml (sds#865)
  Cut version 0.67.0 (sds#864)
  Restore ability to specify "gemfile: false" in config (sds#863)
  Cut version 0.66.0 (sds#861)
  Add `-diff` cli option for running precommit hooks against diffs (sds#860)
  Cut version 0.65.0
  Load bundled gems on expected version (sds#859)
  Update changelog
  Bump rexml to >= 3.3.9 to resolve GHSA-2rxp-v6pw-ch6m (sds#857)
  Expand tildes(~) for hooksPath (sds#853)
  Fix spec compatibility with recent git versions (sds#854)
  Update changelog
  Cut version 0.64.0 (sds#852)
  Add changelog_uri to gemspec (sds#851)
  Support stylelint 16+, which emits messages on stderr (sds#848)
  Add Ruby 3.3 to CI matrix (sds#849)
  Update README with latest requirements (sds#845)
  Fix README badges (sds#844)
  Cut version 0.63.0 (sds#843)
  Add RSpec Hook to PreCommit, Add Applicable files if set (sds#842)
  Fix RuboCop warnings (sds#841)
  [FEATURE] New pre-commit hook: Sorbet (Issue sds#825) (sds#826)
  Green CI PoC (sds#833)
  Revert "Get suite running smoothly again (sds#824)" (sds#840)
  Get suite running smoothly again (sds#824)
  Cut version 0.62.0 (sds#821)
  Allow childprocess 5 (sds#820)
  Update branch for CI from `master` to `main` (sds#819)
  Update RuboCop (sds#818)
  Adding Ruby 3.2 to test matrix (sds#816)
  Cut version 0.61.0 (sds#817)
  fix: local overcommit file merges with existing .overcommit.yml file (sds#815)
  Allow ReplaceBranch to use skip_if (sds#809)
  Cut version 0.60.0 (sds#807)
  Add non ASCII encoding schema support to RailsSchemaUpToDate hook (sds#786)
  Rescue Psych::DisallowedClass in YamlSyntax (sds#805)
  Add optional additional local configuration (sds#799)
  Add MixTest hook (sds#802)
  Add MixFormat hook (sds#801)
  support trailing space in replace_branch's 'replacement_text' config (sds#804)
  Allow overriding Gemfile.lock (sds#796)
  Cut version 0.59.1 (sds#785)
  Revert addition of `--disable-pending-cops` flag to RuboCop (sds#783)
  Revert `Fix encoding of process output under windows` (sds#784)
  Cut version 0.59.0 (sds#780)
  Partial log flushing to make long hooks intuitive (sds#777)
  Fix pronto hooks swallowing generic errors (sds#779)
  ...

# Conflicts:
#	CHANGELOG.md
#	README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant