feat(agents): add preflight system and doctor command#130
Conversation
Implement a comprehensive preflight verification system that helps users diagnose agent configuration issues before running tasks. Changes: - Add AgentPreflightResult interface to agent types - Implement preflight() method in BaseAgentPlugin that runs a minimal test prompt and verifies agent responsiveness - Add getPreflightSuggestion() overrides in Claude, OpenCode, and Droid plugins with agent-specific troubleshooting guidance - Create doctor command (ralph-tui doctor) for standalone diagnostics with --json output for CI/scripting use - Integrate preflight verification into setup wizard after saving config - Integrate preflight verification into create-prd before chat mode - Add --verify flag to run command for explicit preflight check Test coverage: - doctor.ts: 80.12% - wizard.ts: 79.68% - base.ts preflight method: covered (lines 630-701) Closes #56
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (2)
WalkthroughAdds a new CLI Changes
Sequence Diagram(s)sequenceDiagram
participant User as User/CLI
participant CLI as CLI Dispatcher
participant Registry as Agent Registry
participant Agent as Agent Plugin
participant External as External Service
User->>CLI: Run 'doctor' command
CLI->>Registry: load config & register agents
Registry-->>CLI: registry ready
CLI->>Registry: resolve agent plugin
Registry-->>CLI: plugin factory/instance
CLI->>Agent: detect()
Agent->>External: check availability (binary/API)
External-->>Agent: response (available/unavailable)
Agent-->>CLI: detection result (path/version/status)
alt detection success
CLI->>Agent: preflight({timeout:30000})
Agent->>External: send minimal test request
External-->>Agent: test response
Agent-->>CLI: preflight result (success/duration)
else detection/preflight failure
Agent-->>CLI: error, suggestion, duration
end
CLI->>CLI: format human or JSON output
CLI-->>User: print report and exit (0 or 1)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The mock was returning true for ALL plugins, which caused the validateConfig test to fail when wizard tests ran first (mock pollution). Now properly checks if the plugin is one of the known test plugins.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #130 +/- ##
==========================================
- Coverage 46.73% 46.08% -0.66%
==========================================
Files 58 59 +1
Lines 12968 13337 +369
==========================================
+ Hits 6061 6146 +85
- Misses 6907 7191 +284
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@src/commands/doctor.test.ts`:
- Around line 286-346: The tests for doctor result structure are flaky because
they run in the repo cwd; update the suite to run in an isolated temp directory
by saving the original cwd, creating a temp dir (e.g., fs.mkdtempSync), calling
process.chdir(tempDir) in beforeEach (or at the start of this describe block)
and restoring process.chdir(originalCwd) in afterEach, and remove the temp dir
after tests; make this change around the describe('doctor result structure'...)
block and ensure executeDoctorCommand still executes but now in the isolated cwd
so detection/preflight assertions (executeDoctorCommand, DoctorResult parsing)
become deterministic.
In `@src/plugins/agents/base.test.ts`:
- Around line 243-268: The tests use expect(...).resolves but don’t await the
promise, so update the assertions to await the resolves to ensure proper async
assertion handling: add await before each
expect(agent.isReady()).resolves.toBe(...) in the 'initialize' and 'dispose'
tests (references: agent.initialize, agent.dispose, agent.isReady) so the test
runner waits for the promise resolution and reports failures correctly.
🧹 Nitpick comments (4)
src/plugins/agents/base.ts (1)
622-701: Surface stderr on preflight failures for clearer diagnostics.
When the process exits non‑zero without anerrorevent,result.erroris empty, so users see a generic failure. Falling back toresult.stderrgives more actionable feedback.♻️ Suggested adjustment
- if (result.status === 'failed') { - return { - success: false, - error: result.error ?? 'Agent execution failed', - suggestion: this.getPreflightSuggestion(), - durationMs, - }; - } + if (result.status === 'failed') { + const failureDetail = result.error || result.stderr || 'Agent execution failed'; + return { + success: false, + error: failureDetail, + suggestion: this.getPreflightSuggestion(), + durationMs, + }; + }src/setup/wizard.ts (1)
384-424: Consider disposing the agent instance after preflight check.The
agentInstanceobtained from the registry is used for the preflight check but never disposed. While this may not cause immediate issues since the wizard exits shortly after, it's good practice to clean up resources.Additionally, consider wrapping the
getInstancecall in a try-catch block to handle potential initialisation failures gracefully, similar to how the doctor command handles this.♻️ Suggested improvement
// Get a fresh agent instance with the configured options const agentRegistry = getAgentRegistry(); - const agentInstance = await agentRegistry.getInstance({ - name: selectedAgent, - plugin: selectedAgent, - options: agentOptions, - }); + let agentInstance; + try { + agentInstance = await agentRegistry.getInstance({ + name: selectedAgent, + plugin: selectedAgent, + options: agentOptions, + }); + } catch (error) { + printError(`Failed to initialise agent: ${error instanceof Error ? error.message : String(error)}`); + printInfo('Configuration saved, but the agent could not be initialised.'); + printInfo('Run "ralph-tui doctor" to diagnose issues.'); + console.log(); + // Continue to show tracker-specific instructions + } - // Run preflight check - const preflightResult = await agentInstance.preflight({ timeout: 30000 }); + if (agentInstance) { + // Run preflight check + const preflightResult = await agentInstance.preflight({ timeout: 30000 }); + // ... existing success/failure handling ... + + // Clean up + await agentInstance.dispose(); + }src/commands/run.tsx (1)
1362-1395: Consider disposing the agent instance after preflight check.Similar to the wizard implementation, the
agentInstanceobtained for the preflight check is not disposed after use. Whilst the process may exit on failure, on success the agent instance remains allocated until the next agent is instantiated for the actual run.♻️ Suggested improvement
const agentRegistry = getAgentRegistry(); const agentInstance = await agentRegistry.getInstance(config.agent); const preflightResult = await agentInstance.preflight({ timeout: 30000 }); if (preflightResult.success) { console.log('✓ Agent is ready'); if (preflightResult.durationMs) { console.log(` Response time: ${preflightResult.durationMs}ms`); } console.log(''); + // Note: Don't dispose here as the registry may reuse the instance } else { console.error(''); console.error('❌ Agent preflight check failed'); // ... error handling ... + await agentInstance.dispose(); process.exit(1); }src/commands/doctor.ts (1)
38-128: Consider disposing the agent instance after diagnostics.The
agentinstance obtained from the registry is never disposed after the diagnostics complete. This could leave resources allocated, particularly if the doctor command is used programmatically.♻️ Suggested improvement
// Run preflight log('\n Step 2: Preflight (testing if agent can respond)...'); log(' Running test prompt...'); const preflight = await agent.preflight({ timeout: 30000 }); + // Clean up agent resources + await agent.dispose(); + if (!preflight.success) { return { healthy: false, agent: { name: agent.meta.name, plugin: agent.meta.id }, detection, preflight, message: preflight.error ?? 'Agent failed preflight check', }; }Note: You may also want to ensure disposal happens in the early return paths (lines 90-96) and wrap in try/finally for robustness.
- doctor.test.ts: Run 'doctor result structure' tests in isolated temp directory to prevent flaky behavior from repo cwd state - base.test.ts: Add await to expect().resolves assertions for proper async assertion handling
…stem feat(agents): add preflight system and doctor command
Summary
Implement a comprehensive preflight verification system that helps users diagnose agent configuration issues before running tasks.
AgentPreflightResultinterface andpreflight()method to agent plugin systemralph-tui doctorcommand for standalone diagnostics with--jsonoutput for CI/scripting--verifyflag)Test plan
bun run typecheckpassesbun run buildsucceedsManual testing
Closes #56
Summary by CodeRabbit
New Features
Tests
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.