Skip to content

Split rspec-package-tests into parallel generator/unit shards#3134

Merged
justin808 merged 3 commits intomainfrom
jg/3131-split-rspec-shards
Apr 13, 2026
Merged

Split rspec-package-tests into parallel generator/unit shards#3134
justin808 merged 3 commits intomainfrom
jg/3131-split-rspec-shards

Conversation

@justin808
Copy link
Copy Markdown
Member

@justin808 justin808 commented Apr 13, 2026

Summary

  • Splits the rspec-package-tests CI job into two parallel matrix shards: generators (429 examples with heavy filesystem I/O) and unit (584 examples covering everything else)
  • Applies to both PR (latest-only) and main (latest + minimum) matrix configurations
  • Artifact names updated to include shard name to avoid collisions

Expected CI time reduction: ~24 min → ~12 min.

Closes #3131

Test plan

  • Verify both generators and unit shard jobs appear in the CI matrix
  • Confirm generator shard runs only spec/react_on_rails/generators
  • Confirm unit shard runs spec/react_on_rails excluding generators
  • Check artifact names are unique per shard
  • Verify full matrix on main includes 4 jobs (2 Ruby versions × 2 shards)

🤖 Generated with Claude Code


Note

Medium Risk
Medium risk because it changes CI execution by splitting the RSpec suite into shard-specific commands and adjusts test initialization to suppress Minitest/Test::Unit autorun behavior, which could affect what tests run and how failures surface.

Overview
Splits rspec-package-tests into two parallel matrix shards (generators vs unit) for both PR and full (main/full-ci) runs, and runs shard-specific RSpec commands (generators-only vs everything excluding generators).

Updates uploaded artifact names to include the shard to avoid collisions, and tweaks both spec_helper.rb files to require "minitest" and disable Minitest/Test::Unit autorun hooks so RSpec CLI args don’t get parsed by Minitest during CI/test boot.

Reviewed by Cursor Bugbot for commit 7b8a9c8. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Chores
    • Updated CI to run tests in parallel shards (generators and unit), improving feedback speed and artifact reporting.
  • Bug Fixes
    • Stabilized test boot process to prevent interfering test runners during spec runs, improving test reliability.

The RSpec package test job ran all ~1,013 examples in a single process
(~24 min). Split it into two parallel matrix shards:
- generators: spec/react_on_rails/generators (429 examples, heavy I/O)
- unit: everything else under spec/react_on_rails (584 examples)

This applies to both PR (latest-only) and main (latest + minimum)
matrix configurations, cutting CI time roughly in half.

Closes #3131

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 13, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 99caec31-3365-45e3-af03-f4c4d64087f3

📥 Commits

Reviewing files that changed from the base of the PR and between 29b9b3e and 7b8a9c8.

📒 Files selected for processing (2)
  • react_on_rails/spec/react_on_rails/spec_helper.rb
  • react_on_rails_pro/spec/react_on_rails_pro/spec_helper.rb

Walkthrough

The CI test matrix gains a shard dimension to split RSpec runs into generators and unit shards; rspec invocation and artifact/test-result names are driven by the shard. Test helpers set Minitest’s at-exit flag to avoid its runner interfering with RSpec runs.

Changes

Cohort / File(s) Summary
CI Test Matrix Sharding
.github/workflows/gem-tests.yml
Adds shard matrix entries (generators, unit); expands matrix for full and PR runs; conditional rspec command to run spec/react_on_rails/generators or run spec/react_on_rails with --exclude-pattern "**/generators/**"; artifact/test-result names include shard.
RSpec Test Bootstrapping
react_on_rails/spec/react_on_rails/spec_helper.rb, react_on_rails_pro/spec/react_on_rails_pro/spec_helper.rb
Adds require "minitest" and sets Minitest.class_variable_set(:@@installed_at_exit, true) when defined to prevent Minitest’s at-exit runner from processing RSpec-only CLI args; comment wording adjusted.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 I hopped through CI with a twitchy nose,
I split the tests where the wild code grows,
Generators dig, units leap and run,
Together they finish the job much more fun. 🥕✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: splitting RSpec tests into parallel generator and unit shards for CI optimization.
Linked Issues check ✅ Passed The PR successfully implements all coding requirements from issue #3131: adds shard matrix dimension to workflows, splits tests between generators and unit shards with correct spec paths/exclude patterns, updates artifact names, and applies changes to both PR and full-CI workflows.
Out of Scope Changes check ✅ Passed All changes are directly related to the sharding objectives: CI workflow restructuring, spec_helper.rb Minitest configuration to prevent CLI argument conflicts, and artifact naming updates—no unrelated modifications present.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jg/3131-split-rspec-shards

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Apr 13, 2026

Greptile Summary

This PR splits the rspec-package-tests CI job into two parallel matrix shards (generators and unit) by adding a setup-gem-tests-matrix job, and uses an environment variable for SHARD to follow GitHub Actions security-hardening best practices. The implementation looks correct: the --exclude-pattern "**/generators/**" RSpec flag properly partitions the specs, artifact names are uniquified with the shard suffix, and the conditional logic for PR vs. main/full-ci matrix builds is sound.

Confidence Score: 5/5

Safe to merge — changes are isolated to CI configuration with no runtime code impact.

The only finding is a P2 style/consistency note about inline expressions in the matrix-setup step, which poses no real security risk given the values are trusted GitHub context outputs. All logic (shard routing, exclusion pattern, artifact naming, matrix dimensions) looks correct.

No files require special attention.

Important Files Changed

Filename Overview
.github/workflows/gem-tests.yml Splits rspec-package-tests into generators/unit shards via a new setup-gem-tests-matrix job; adds SHARD env-var pattern for security hardening; artifact names updated to include shard suffix.

Sequence Diagram

sequenceDiagram
    participant GH as GitHub Actions
    participant DC as detect-changes
    participant SM as setup-gem-tests-matrix
    participant G as rspec-package-tests (shard: generators)
    participant U as rspec-package-tests (shard: unit)

    GH->>DC: trigger (push / PR / dispatch)
    DC-->>SM: docs_only, run_ruby_tests, has_full_ci_label

    alt main branch OR full-ci label
        SM-->>G: ruby 3.4 latest + ruby 3.2 minimum
        SM-->>U: ruby 3.4 latest + ruby 3.2 minimum
    else PR (latest only)
        SM-->>G: ruby 3.4 latest
        SM-->>U: ruby 3.4 latest
    end

    par Generators shard
        G->>G: bundle exec rspec spec/react_on_rails/generators
        G-->>GH: upload artifact (…-generators)
    and Unit shard
        U->>U: bundle exec rspec spec/react_on_rails --exclude-pattern **/generators/**
        U-->>GH: upload artifact (…-unit)
    end
Loading

Reviews (2): Last reviewed commit: "fix: use env var for matrix.shard to fol..." | Re-trigger Greptile

Comment thread .github/workflows/gem-tests.yml
Comment thread .github/workflows/gem-tests.yml Outdated
Comment thread .github/workflows/gem-tests.yml
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 13, 2026

Review: Split rspec-package-tests into parallel shards

Overall this is a clean, well-scoped CI-only change. The shard boundary (generators vs. everything else) is a sensible split — generators are the heaviest filesystem consumers — and the artifact naming update correctly prevents collisions.

What looks good

  • fail-fast: false is already set, so a flaky generator spec won't kill the unit shard mid-run.
  • Gem cache key deliberately excludes the shard (both shards share the same gems), which is correct and efficient.
  • The --exclude-pattern "**/generators/**" glob is correct for RSpec; it will match spec/react_on_rails/generators/ and any nested paths.
  • The 2×2 main matrix (2 Ruby versions × 2 shards) and 1×2 PR matrix are structured correctly.

Two items to address

  1. Script injection pattern (inline comment on line 156) — ${{ matrix.shard }} is interpolated directly into a shell if condition. No real risk here since the values are workflow-controlled, but GitHub's hardening guide recommends routing through an env var instead. Easy fix.

  2. Long JSON on lines 91/94 (inline comment) — minor readability nit; not blocking.

One implicit behaviour to be aware of

If matrix.shard ever takes an unexpected value (e.g., a typo), the else branch silently runs the unit suite instead of failing loudly. You could add an explicit guard:

elif [ "$SHARD" = "unit" ]; then
  bundle exec rspec spec/react_on_rails --exclude-pattern "**/generators/**"
else
  echo "Unknown shard: $SHARD" && exit 1
fi

Not required for this PR, but worth considering if/when more shards are added.

…ardening

Route ${{ matrix.shard }} through an env var instead of direct shell
interpolation, per GitHub's security hardening guide. Also switch from
POSIX [ ] to bash [[ ]] to match surrounding workflow style.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@justin808
Copy link
Copy Markdown
Member Author

Review comments addressed

I (Claude Code) triaged all 6 review comments and addressed them as follows:

Fixed (1):

  • Script injection hardening (@claude[bot]) — Moved ${{ matrix.shard }} to an env: block and switched [ ][[ ]] for bash consistency. Fixed in 29b9b3e.

Acknowledged with rationale (2):

  • dev/pack_generator_spec.rb in unit shard (@greptile-apps[bot]) — Spec is lightweight; runtime split is balanced.
  • Long matrix JSON (@claude[bot]) — Only 2–4 entries today; will revisit if matrix grows.

Skipped (3):

  • CodeRabbit rate limit warning — bot status post
  • Greptile summary + "Git commit step" observation — bot summary, harmless no-op
  • Claude review summary — duplicates the inline comments above

All 3 review threads have been resolved.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 13, 2026

Good optimization overall. Isolating heavy filesystem-I/O generator specs into their own shard is the right approach. No blockers, but a few notes:

Exclude pattern edge case: The pattern --exclude-pattern '**/generators/**' correctly partitions the current layout (all generator specs are under spec/react_on_rails/generators/). Worth noting: a future top-level file like generators_integration_spec.rb would NOT be excluded since the pattern is directory-based. This is fine for now but worth keeping in mind as the suite grows.

Bash comparison style (minor): [[ "$SHARD" = "generators" ]] uses single = — valid in [[ ]] but == is the idiomatic form for string comparison in bash.

Long single-line JSON (readability): The matrix JSON on lines 91 and 94 are very long one-liners. Consider a multi-line variable or heredoc approach before echoing to $GITHUB_OUTPUT to make the matrix easier to audit and extend.

Git Stuff step comment is misleading: The step runs for both shards when dependency-level == minimum and the comment says 'stop generators from complaining about uncommitted code' — but the commit is actually needed to clean up files modified by the convert script before any tests run, not just generators. Behavior is correct; just the comment that needs updating.

# Full matrix: test both latest and minimum supported versions
echo 'matrix={"include":[{"ruby-version":"3.4","dependency-level":"latest"},{"ruby-version":"3.2","dependency-level":"minimum"}]}' >> $GITHUB_OUTPUT
# Full matrix: test both latest and minimum supported versions × 2 shards
echo 'matrix={"include":[{"ruby-version":"3.4","dependency-level":"latest","shard":"generators"},{"ruby-version":"3.4","dependency-level":"latest","shard":"unit"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"generators"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"unit"}]}' >> $GITHUB_OUTPUT
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These long JSON one-liners are hard to read and extend. Consider a multi-line approach for maintainability:

Suggested change
echo 'matrix={"include":[{"ruby-version":"3.4","dependency-level":"latest","shard":"generators"},{"ruby-version":"3.4","dependency-level":"latest","shard":"unit"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"generators"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"unit"}]}' >> $GITHUB_OUTPUT
matrix=$(cat <<'EOF'
{"include":[
{"ruby-version":"3.4","dependency-level":"latest","shard":"generators"},
{"ruby-version":"3.4","dependency-level":"latest","shard":"unit"},
{"ruby-version":"3.2","dependency-level":"minimum","shard":"generators"},
{"ruby-version":"3.2","dependency-level":"minimum","shard":"unit"}
]}
EOF
)
echo "matrix=$matrix" >> $GITHUB_OUTPUT

This makes adding or removing matrix entries much easier to review in diffs.

SHARD: ${{ matrix.shard }}
run: |
cd react_on_rails
if [[ "$SHARD" = "generators" ]]; then
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nit: prefer == over = for string comparison in [[ ]] — both are valid in bash but == is the conventional form and more clearly signals "comparison" vs "assignment".

Suggested change
if [[ "$SHARD" = "generators" ]]; then
if [[ "$SHARD" == "generators" ]]; then

# Prevent Minitest's at_exit runner from parsing RSpec-only CLI args after
# rails/test_help loads ActiveSupport's autorun hooks.
# rubocop:disable Style/ClassVars
Minitest.class_variable_set(:@@installed_at_exit, true) if Minitest.class_variable_defined?(:@@installed_at_exit)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patches a private Minitest implementation detail (@@installed_at_exit). It works today, but it could silently break if Minitest renames or removes that class variable in a future version — with no error, just the autorun hook re-appearing.

A slightly more resilient alternative is to prevent the hook from firing without touching internals:

Suggested change
Minitest.class_variable_set(:@@installed_at_exit, true) if Minitest.class_variable_defined?(:@@installed_at_exit)
Minitest.class_variable_set(:@@installed_at_exit, true) if Minitest.class_variable_defined?(:@@installed_at_exit)

The current guard (class_variable_defined?) already mitigates the worst-case (no-op on unknown Minitest versions), so this is low risk. Just worth noting in a follow-up that the real fix would be avoiding require "rails/test_help" entirely if it's not needed, since that's the source of the autorun registration.

# Full matrix: test both latest and minimum supported versions
echo 'matrix={"include":[{"ruby-version":"3.4","dependency-level":"latest"},{"ruby-version":"3.2","dependency-level":"minimum"}]}' >> $GITHUB_OUTPUT
# Full matrix: test both latest and minimum supported versions × 2 shards
echo 'matrix={"include":[{"ruby-version":"3.4","dependency-level":"latest","shard":"generators"},{"ruby-version":"3.4","dependency-level":"latest","shard":"unit"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"generators"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"unit"}]}' >> $GITHUB_OUTPUT
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These long one-liner JSON strings are hard to read and diff. GitHub Actions supports multi-line echo with >>, so you can split this cleanly:

Suggested change
echo 'matrix={"include":[{"ruby-version":"3.4","dependency-level":"latest","shard":"generators"},{"ruby-version":"3.4","dependency-level":"latest","shard":"unit"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"generators"},{"ruby-version":"3.2","dependency-level":"minimum","shard":"unit"}]}' >> $GITHUB_OUTPUT
echo 'matrix={"include":[' \
'{"ruby-version":"3.4","dependency-level":"latest","shard":"generators"},' \
'{"ruby-version":"3.4","dependency-level":"latest","shard":"unit"},' \
'{"ruby-version":"3.2","dependency-level":"minimum","shard":"generators"},' \
'{"ruby-version":"3.2","dependency-level":"minimum","shard":"unit"}' \
']}' >> $GITHUB_OUTPUT

Not a correctness issue — just a maintainability nit for when the matrix inevitably grows again.

if [[ "$SHARD" = "generators" ]]; then
bundle exec rspec spec/react_on_rails/generators
else
bundle exec rspec spec/react_on_rails --exclude-pattern "**/generators/**"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth confirming the --exclude-pattern glob behaves as expected with RSpec. The pattern **/generators/** excludes files inside a generators/ subdirectory, but spec files directly named generators_spec.rb at the top level (if any were ever added) would still be picked up by the unit shard. Currently the directory structure only has spec/react_on_rails/generators/*.rb, so this is fine — just something to keep in mind if flat spec files named after generators are added later.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 13, 2026

Review: Split rspec-package-tests into parallel generator/unit shards

Overall: Clean, well-motivated change. The shard split correctly targets the I/O-heavy generator specs and the artifact naming is updated consistently. A few observations below.


Correctness

  • Shard coverage is complete. The generators shard runs spec/react_on_rails/generators directly; the unit shard uses --exclude-pattern '**/generators/**' which correctly excludes everything under that directory. No tests are double-counted or dropped.
  • fail-fast: false is already set, so one failing shard does not cancel the other — good.

Minitest suppression (both spec_helpers)

The new Minitest.class_variable_set(:@@installed_at_exit, true) block is needed because rails/test_help registers a Minitest at_exit hook, and when RSpec exits with --exclude-pattern in ARGV, Minitest's hook tries to parse those args and errors. The existing Test::Unit.run = true guard was already handling the older Test::Unit layer; this extends it to Minitest 5+. The guard (class_variable_defined? check) makes it safe against future Minitest versions that remove the variable.

One question: react_on_rails_pro/spec_helper.rb is updated here, but gem-tests.yml only runs the open-source gem (cd react_on_rails). The pro spec_helper change is not exercised by this workflow. Is this a defensive fix bundled in, or will a separate pro CI PR follow?

Artifact path (pre-existing)

The log/test.log artifact path was there before this PR. Worth noting: cd react_on_rails only changes the shell working directory, not where actions/upload-artifact looks — it resolves paths relative to $GITHUB_WORKSPACE. If the log is written to react_on_rails/log/test.log, the artifact path should be react_on_rails/log/test.log. Pre-existing issue, not introduced here, but worth checking.


No blocking issues. The three inline comments are nits/notes — the implementation is sound and achieves the stated ~12 min reduction goal.

@justin808 justin808 merged commit 73f9085 into main Apr 13, 2026
44 checks passed
@justin808 justin808 deleted the jg/3131-split-rspec-shards branch April 13, 2026 07:09
justin808 added a commit that referenced this pull request Apr 18, 2026
…ages

* origin/main:
  Fix initial page startup race for late-loading client bundles (#3151)
  chore: apply prettier formatting to tracked docs files (#3153)
  docs: comprehensive RSC API documentation and registration consolidation (#3140)
  Split rspec-package-tests into parallel generator/unit shards (#3134)
  fix: add concurrency groups to long-running CI workflows (#3133)
  refactor: add RenderRequest, JsCodeBuilder, and RenderingStrategy abstractions (#3094)
  fix: address deferred review items from PR #2849 (#3093)
  Add complimentary OSS license policy for React on Rails Pro (#3123)
  fix: centralize CI docs-only detection and add CLI flag validation (#3091)
  refactor: replace stub-throw + Object.assign with capability-based composition (#3096)
  Enhance address-review with parallel fixes, self-review, and Greptile verification (#3121)
  fix: Doctor no longer fails custom projects for missing bin/dev (#3117)
  fix: cap webpack <5.106.0 to prevent ExecJS SSR breakage (#3095)
  Add Rspack + RSC compatibility tests and documentation (#1828) (#3120)
  Add error scenarios hub and test pages (#2497)
  docs: document polyfill requirements for web-targeted server bundles (#3092)
  docs: RSC integration pitfalls from tutorial app (#3087)
  docs: fix render function/helper API documentation (#3088)
  Doctor: accept TS/TSX server bundle suffixes (#3111)
  feat: add CI guard requiring sidebar updates when adding docs (#3089)
justin808 added a commit that referenced this pull request Apr 18, 2026
## Summary
- Splits the `rspec-package-tests` CI job into two parallel matrix
shards: **generators** (429 examples with heavy filesystem I/O) and
**unit** (584 examples covering everything else)
- Applies to both PR (latest-only) and main (latest + minimum) matrix
configurations
- Artifact names updated to include shard name to avoid collisions

Expected CI time reduction: ~24 min → ~12 min.

Closes #3131

## Test plan
- [ ] Verify both `generators` and `unit` shard jobs appear in the CI
matrix
- [ ] Confirm generator shard runs only `spec/react_on_rails/generators`
- [ ] Confirm unit shard runs `spec/react_on_rails` excluding generators
- [ ] Check artifact names are unique per shard
- [ ] Verify full matrix on main includes 4 jobs (2 Ruby versions × 2
shards)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Medium risk because it changes CI execution by splitting the RSpec
suite into shard-specific commands and adjusts test initialization to
suppress Minitest/Test::Unit autorun behavior, which could affect what
tests run and how failures surface.
> 
> **Overview**
> Splits `rspec-package-tests` into two parallel matrix shards
(`generators` vs `unit`) for both PR and full (main/`full-ci`) runs, and
runs shard-specific RSpec commands (generators-only vs everything
excluding generators).
> 
> Updates uploaded artifact names to include the shard to avoid
collisions, and tweaks both `spec_helper.rb` files to `require
"minitest"` and disable Minitest/Test::Unit autorun hooks so RSpec CLI
args don’t get parsed by Minitest during CI/test boot.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
7b8a9c8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated CI to run tests in parallel shards (generators and unit),
improving feedback speed and artifact reporting.
* **Bug Fixes**
* Stabilized test boot process to prevent interfering test runners
during spec runs, improving test reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Split RSpec package tests into parallel shards for faster CI

1 participant