Add manual dev environment testing checklist for coding agents#3074
Add manual dev environment testing checklist for coding agents#3074
Conversation
Automated tests can pass while bin/dev is completely broken (as seen in shakacode/react-webpack-rails-tutorial#723). This adds a structured checklist that agents must follow when changes affect how the app starts, builds, or serves. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
WalkthroughAdded a new manual dev-environment testing checklist and updated repository instructions to require manual verification for PRs that affect application startup, build, or serving behavior. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4db9e56455
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Greptile SummaryAdds Confidence Score: 5/5Safe to merge — documentation-only change with no runtime impact. All three files are documentation/config only. Cross-references resolve correctly, the prose is clear and actionable, and the single finding (Phase 1 omitted from two table rows) is a P2 clarity nit that doesn't block merge. No files require special attention.
|
| Filename | Overview |
|---|---|
| .claude/docs/manual-dev-environment-testing.md | New doc defining 5 checklist phases for manual dev-env verification; cross-references (pr-testing-guide.md, testing-build-scripts.md) both exist; minor table ambiguity where Phase 1 is omitted for rows that implicitly depend on it. |
| AGENTS.md | Single line added under Testing section linking to the new manual-dev-environment-testing.md guide; path and prose are correct. |
| CLAUDE.md | One entry appended to the Claude-Specific Workflow doc list pointing to the new guide; no issues. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[PR changes detected] --> B{Touches startup/build/serve?}
B -- No --> C[Manual testing optional]
B -- Yes --> D[Consult phase matrix]
D --> E[Phase 1: Dev Server Startup\nBLOCKING]
E -- Fails --> F[🛑 Stop — fix before proceeding]
E -- Passes --> G{Required phases\nper change type}
G --> H[Phase 2: Page Smoke Test]
G --> I[Phase 3: SSR Verification]
G --> J[Phase 4: Interactive Functionality]
G --> K[Phase 5: Process Health]
H & I & J & K --> L[Report results in PR description]
L --> M{Can't run bin/dev?}
M -- Yes --> N[⚠️ Declare DEV ENVIRONMENT UNTESTED]
M -- No --> O[✅ Submit PR]
Reviews (1): Last reviewed commit: "Add manual dev environment testing check..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
.claude/docs/manual-dev-environment-testing.md (1)
72-72: Use hyphenated compound adjective for readability.
full page reloads→full-page reloads.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.claude/docs/manual-dev-environment-testing.md at line 72, Update the checklist line that currently reads "Client-side navigation works without full page reloads (for SPA routes)" to use the hyphenated compound adjective: change "full page reloads" to "full-page reloads" so the line becomes "Client-side navigation works without full-page reloads (for SPA routes)"; locate and edit that exact checklist item in the document to apply the fix.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/docs/manual-dev-environment-testing.md:
- Around line 31-35: Replace the "yarn install" command in the shell snippet
with "pnpm install" so the JS dependency step uses pnpm only; specifically
update the code block that currently contains "yarn install # or pnpm install,
per project" to instead read "pnpm install" (remove the yarn alternative) so the
manual-dev-environment-testing instructions enforce pnpm for JavaScript
installs.
---
Nitpick comments:
In @.claude/docs/manual-dev-environment-testing.md:
- Line 72: Update the checklist line that currently reads "Client-side
navigation works without full page reloads (for SPA routes)" to use the
hyphenated compound adjective: change "full page reloads" to "full-page reloads"
so the line becomes "Client-side navigation works without full-page reloads (for
SPA routes)"; locate and edit that exact checklist item in the document to apply
the fix.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b9f04e74-4245-48a2-86b3-b725bdaccf4d
📒 Files selected for processing (3)
.claude/docs/manual-dev-environment-testing.mdAGENTS.mdCLAUDE.md
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Review: Add manual dev environment testing checklistWell-motivated PR — the incident in shakacode/react-webpack-rails-tutorial#723 is exactly the kind of failure that structured checklists prevent. The structure (phases, trigger matrix, failure modes, reporting template) is solid and follows the pattern of existing docs in IssuesMinor corrections (see inline comments):
No blocking concernsThe AGENTS.md and CLAUDE.md changes are clean — the new bullet slots in naturally and the link path is consistent with other references in those files. The |
|
Skipped-item rationale for summary-only comments:
|
|
Review cleanup summary for PR #3074. Scan window: full PR history (no previous summary cutoff; no Mattered
Skipped
Inline feedback threads for handled items were replied to and resolved; skipped-summary rationale was posted at Future full-PR scans should start after this summary comment unless |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7bda6896da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ```bash | ||
| # 1. Install dependencies | ||
| bundle install | ||
| pnpm install # or your project's required JS package manager |
There was a problem hiding this comment.
Fresh evidence after the earlier yarn-specific wording was removed: this line still states pnpm install “or your project's required JS package manager,” which tells agents that non-pnpm managers are acceptable. In this repository, AGENTS.md requires pnpm for all JS operations and forbids npm/yarn, so following this checklist during mandatory dev-environment verification can still produce unsupported dependency state (including stray lockfiles) and inconsistent local results.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
.claude/docs/manual-dev-environment-testing.md (1)
34-34:⚠️ Potential issue | 🟠 MajorEnforce pnpm-only wording for agent instructions
Line 34 still allows alternate JS package managers (
or your project's required JS package manager), which conflicts with repo policy for agent-facing workflows. Please make this command strictly pnpm-only in this document.Suggested edit
-pnpm install # or your project's required JS package manager +pnpm installBased on learnings: "Follow
AGENTS.mdas the canonical policy for commands, tests, lint workflow, formatting, style requirements, git/PR safety boundaries, and project directory boundaries." and "Usepnpmfor all JavaScript/Node operations; never usenpmoryarn.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.claude/docs/manual-dev-environment-testing.md at line 34, Remove the alternative package manager text and make the instruction strictly pnpm-only by replacing the existing line containing "pnpm install # or your project's required JS package manager" with a single command "pnpm install" so the agent-facing doc enforces the repo policy; update the line that contains the pnpm install command to exactly "pnpm install" (no trailing commentary).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/docs/manual-dev-environment-testing.md:
- Line 72: Update the checklist item text "[ ] Client-side navigation works
without full page reloads (for SPA routes)" by hyphenating the compound
adjective to read "full-page reloads" so the line becomes "[ ] Client-side
navigation works without full-page reloads (for SPA routes)"; locate and edit
that exact string in the .claude/docs/manual-dev-environment-testing.md file.
---
Duplicate comments:
In @.claude/docs/manual-dev-environment-testing.md:
- Line 34: Remove the alternative package manager text and make the instruction
strictly pnpm-only by replacing the existing line containing "pnpm install # or
your project's required JS package manager" with a single command "pnpm install"
so the agent-facing doc enforces the repo policy; update the line that contains
the pnpm install command to exactly "pnpm install" (no trailing commentary).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e031c27f-6926-4b96-b2c8-099a6e8e1e6b
📒 Files selected for processing (1)
.claude/docs/manual-dev-environment-testing.md
| ### Phase 4: Interactive Functionality | ||
|
|
||
| - [ ] Forms submit and produce expected results | ||
| - [ ] Client-side navigation works without full page reloads (for SPA routes) |
There was a problem hiding this comment.
Hyphenate compound adjective for clarity
On Line 72, prefer “full-page reloads” (instead of “full page reloads”) for cleaner technical writing.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~72-~72: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ... ] Client-side navigation works without full page reloads (for SPA routes) - [ ] Interact...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.claude/docs/manual-dev-environment-testing.md at line 72, Update the
checklist item text "[ ] Client-side navigation works without full page reloads
(for SPA routes)" by hyphenating the compound adjective to read "full-page
reloads" so the line becomes "[ ] Client-side navigation works without full-page
reloads (for SPA routes)"; locate and edit that exact string in the
.claude/docs/manual-dev-environment-testing.md file.
### Summary Adds a structured checklist that coding agents must follow to verify the dev environment works before submitting PRs. This was motivated by shakacode/react-webpack-rails-tutorial#723 where all 38 rspec tests passed but `bin/dev` wouldn't even start — the agent never tested the actual development experience. The new doc at `.claude/docs/manual-dev-environment-testing.md` defines 5 phases (dev server startup, page smoke test, SSR verification, interactive functionality, process health), a matrix mapping change types to required phases, common failure modes, and a reporting template. AGENTS.md and CLAUDE.md are updated to reference it. ### Pull Request checklist - [x] ~Add/update test to cover these changes~ - [x] Update documentation - [x] ~Update CHANGELOG file~ ### Other Information Kept abstract and repo-agnostic — no hardcoded paths or routes from the tutorial app. The checklist applies to any Rails + React on Rails app with a `Procfile.dev`-based dev setup. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Low risk: documentation-only changes that add contributor guidance and do not affect runtime code paths. > > **Overview** > Adds a new `.claude/docs/manual-dev-environment-testing.md` guide requiring **manual `bin/dev` verification** (startup, page smoke, SSR checks, interactivity, and process health) for PRs that change startup/build/serve behavior, plus a reporting template for PR descriptions. > > Updates `AGENTS.md` and `CLAUDE.md` to reference this checklist so agents consistently run dev-environment smoke tests in addition to automated suites. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7bda689. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a comprehensive manual development-environment testing guide that requires PRs affecting app startup, build, or serving to include manual verification. Includes phased checklists (startup, per-page smoke tests, SSR, interactivity, process health), change-type-to-required-phase mapping, common failure modes, and reporting templates. * Updated agent and operational docs to reference the new testing guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
### Summary Adds a structured checklist that coding agents must follow to verify the dev environment works before submitting PRs. This was motivated by shakacode/react-webpack-rails-tutorial#723 where all 38 rspec tests passed but `bin/dev` wouldn't even start — the agent never tested the actual development experience. The new doc at `.claude/docs/manual-dev-environment-testing.md` defines 5 phases (dev server startup, page smoke test, SSR verification, interactive functionality, process health), a matrix mapping change types to required phases, common failure modes, and a reporting template. AGENTS.md and CLAUDE.md are updated to reference it. ### Pull Request checklist - [x] ~Add/update test to cover these changes~ - [x] Update documentation - [x] ~Update CHANGELOG file~ ### Other Information Kept abstract and repo-agnostic — no hardcoded paths or routes from the tutorial app. The checklist applies to any Rails + React on Rails app with a `Procfile.dev`-based dev setup. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Low risk: documentation-only changes that add contributor guidance and do not affect runtime code paths. > > **Overview** > Adds a new `.claude/docs/manual-dev-environment-testing.md` guide requiring **manual `bin/dev` verification** (startup, page smoke, SSR checks, interactivity, and process health) for PRs that change startup/build/serve behavior, plus a reporting template for PR descriptions. > > Updates `AGENTS.md` and `CLAUDE.md` to reference this checklist so agents consistently run dev-environment smoke tests in addition to automated suites. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7bda689. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a comprehensive manual development-environment testing guide that requires PRs affecting app startup, build, or serving to include manual verification. Includes phased checklists (startup, per-page smoke tests, SSR, interactivity, process health), change-type-to-required-phase mapping, common failure modes, and reporting templates. * Updated agent and operational docs to reference the new testing guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
### Summary Adds a structured checklist that coding agents must follow to verify the dev environment works before submitting PRs. This was motivated by shakacode/react-webpack-rails-tutorial#723 where all 38 rspec tests passed but `bin/dev` wouldn't even start — the agent never tested the actual development experience. The new doc at `.claude/docs/manual-dev-environment-testing.md` defines 5 phases (dev server startup, page smoke test, SSR verification, interactive functionality, process health), a matrix mapping change types to required phases, common failure modes, and a reporting template. AGENTS.md and CLAUDE.md are updated to reference it. ### Pull Request checklist - [x] ~Add/update test to cover these changes~ - [x] Update documentation - [x] ~Update CHANGELOG file~ ### Other Information Kept abstract and repo-agnostic — no hardcoded paths or routes from the tutorial app. The checklist applies to any Rails + React on Rails app with a `Procfile.dev`-based dev setup. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Low risk: documentation-only changes that add contributor guidance and do not affect runtime code paths. > > **Overview** > Adds a new `.claude/docs/manual-dev-environment-testing.md` guide requiring **manual `bin/dev` verification** (startup, page smoke, SSR checks, interactivity, and process health) for PRs that change startup/build/serve behavior, plus a reporting template for PR descriptions. > > Updates `AGENTS.md` and `CLAUDE.md` to reference this checklist so agents consistently run dev-environment smoke tests in addition to automated suites. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 7bda689. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a comprehensive manual development-environment testing guide that requires PRs affecting app startup, build, or serving to include manual verification. Includes phased checklists (startup, per-page smoke tests, SSR, interactivity, process health), change-type-to-required-phase mapping, common failure modes, and reporting templates. * Updated agent and operational docs to reference the new testing guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
…ages * origin/main: (44 commits) Consolidate CSP nonce sanitization into shared module (#2828) Add comprehensive --rsc-pro generator tests (#3098) fix: cross-env validation and docs for renderer password (#3090) Improve package metadata and Pro upgrade CTAs (#3112) docs: standardize warning syntax to GFM alert format (#3115) docs: improve react-intl documentation for React Server Components (#3085) Fix generator CI SSR regression on main (#3110) Refocus GitHub README on docs navigation (#3113) Add manual dev environment testing checklist for coding agents (#3074) Bump version to 16.6.0 Update CHANGELOG.md for 16.6.0 (#3078) fix: node-renderer diagnostic improvements (#3086) fix: pin third-party npm deps in generator to prevent peer dep conflicts (#3083) chore(deps): bump lodash from 4.17.23 to 4.18.1 in the npm-security group across 1 directory (#2920) fix: refactor formatExceptionMessage to accept generic request context (#2877) Bump version to 16.6.0.rc.1 Update CHANGELOG.md for 16.6.0.rc.1 (#3079) Update CHANGELOG.md unreleased section (#3077) Fix Content-Length mismatch and null renderingRequest errors in node renderer (#3069) Improve memory debugging docs with simpler heap snapshot approach (#3072) ... # Conflicts: # docs/pro/home-pro.md # docs/pro/react-on-rails-pro.md # docs/sidebars.ts
1 participant
Summary
Adds a structured checklist that coding agents must follow to verify the dev environment works before submitting PRs. This was motivated by shakacode/react-webpack-rails-tutorial#723 where all 38 rspec tests passed but
bin/devwouldn't even start — the agent never tested the actual development experience.The new doc at
.claude/docs/manual-dev-environment-testing.mddefines 5 phases (dev server startup, page smoke test, SSR verification, interactive functionality, process health), a matrix mapping change types to required phases, common failure modes, and a reporting template. AGENTS.md and CLAUDE.md are updated to reference it.Pull Request checklist
Add/update test to cover these changesUpdate CHANGELOG fileOther Information
Kept abstract and repo-agnostic — no hardcoded paths or routes from the tutorial app. The checklist applies to any Rails + React on Rails app with a
Procfile.dev-based dev setup.Note
Low Risk
Low risk: documentation-only changes that add contributor guidance and do not affect runtime code paths.
Overview
Adds a new
.claude/docs/manual-dev-environment-testing.mdguide requiring manualbin/devverification (startup, page smoke, SSR checks, interactivity, and process health) for PRs that change startup/build/serve behavior, plus a reporting template for PR descriptions.Updates
AGENTS.mdandCLAUDE.mdto reference this checklist so agents consistently run dev-environment smoke tests in addition to automated suites.Reviewed by Cursor Bugbot for commit 7bda689. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit