Skip to content

Comments

feat(agents): add preflight system and doctor command#130

Merged
subsy merged 3 commits intomainfrom
feature/issue-56-preflight-system
Jan 17, 2026
Merged

feat(agents): add preflight system and doctor command#130
subsy merged 3 commits intomainfrom
feature/issue-56-preflight-system

Conversation

@subsy
Copy link
Owner

@subsy subsy commented Jan 17, 2026

Summary

Implement a comprehensive preflight verification system that helps users diagnose agent configuration issues before running tasks.

  • Add AgentPreflightResult interface and preflight() method to agent plugin system
  • Create ralph-tui doctor command for standalone diagnostics with --json output for CI/scripting
  • Integrate preflight verification into setup wizard, create-prd command, and run command (--verify flag)
  • Add agent-specific troubleshooting suggestions for Claude, OpenCode, and Droid plugins

Test plan

  • All 1189 tests pass
  • doctor.ts coverage: 80.12%
  • wizard.ts coverage: 79.68%
  • base.ts preflight method (lines 630-701): covered
  • bun run typecheck passes
  • bun run build succeeds

Manual testing

# Test doctor command
ralph-tui doctor
ralph-tui doctor --json
ralph-tui doctor --agent opencode

# Test preflight in setup wizard
ralph-tui setup --force

# Test preflight in create-prd
ralph-tui create-prd --chat

# Test --verify flag
ralph-tui run --verify

Closes #56

Summary by CodeRabbit

  • New Features

    • Added a doctor command to diagnose agent configuration and health, with human and JSON output.
    • Introduced a --verify flag to run preflight agent checks before executing run commands.
    • Agent configuration now runs an immediate preflight verification after setup.
    • Agents provide targeted preflight guidance/suggestions when checks fail.
  • Tests

    • Added comprehensive tests covering doctor, preflight flows and agent lifecycle scenarios.
  • Documentation

    • CLI help updated with doctor examples and --verify usage.

✏️ Tip: You can customize this high-level summary in your review settings.

Implement a comprehensive preflight verification system that helps users
diagnose agent configuration issues before running tasks.

Changes:
- Add AgentPreflightResult interface to agent types
- Implement preflight() method in BaseAgentPlugin that runs a minimal
  test prompt and verifies agent responsiveness
- Add getPreflightSuggestion() overrides in Claude, OpenCode, and Droid
  plugins with agent-specific troubleshooting guidance
- Create doctor command (ralph-tui doctor) for standalone diagnostics
  with --json output for CI/scripting use
- Integrate preflight verification into setup wizard after saving config
- Integrate preflight verification into create-prd before chat mode
- Add --verify flag to run command for explicit preflight check

Test coverage:
- doctor.ts: 80.12%
- wizard.ts: 79.68%
- base.ts preflight method: covered (lines 630-701)

Closes #56
@vercel
Copy link

vercel bot commented Jan 17, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Review Updated (UTC)
ralph-tui Ignored Ignored Jan 17, 2026 7:16pm

Request Review

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jan 17, 2026

Warning

Rate limit exceeded

@subsy has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 15 minutes and 14 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between fd709e9 and fd4e518.

📒 Files selected for processing (2)
  • src/commands/doctor.test.ts
  • src/plugins/agents/base.test.ts

Walkthrough

Adds a new CLI doctor command and a preflight health-check system for AI agents, extends agent types with a preflight API, integrates preflight checks into run/create-prd/setup flows, and provides agent-specific preflight guidance and tests.

Changes

Cohort / File(s) Summary
CLI core & command exports
src/cli.tsx, src/commands/index.ts
Expose and wire the new doctor subcommand into the CLI entrypoint and command index exports.
Doctor command & tests
src/commands/doctor.ts, src/commands/doctor.test.ts
New executeDoctorCommand, printDoctorHelp, parsing of --cwd/--json/--agent/--help, diagnostics flow (detect + preflight), human/JSON output, exit codes; comprehensive unit tests.
Agent types & base implementation
src/plugins/agents/types.ts, src/plugins/agents/base.ts, src/plugins/agents/base.test.ts
Add AgentPreflightResult type and preflight(...) to AgentPlugin interface; implement preflight and getPreflightSuggestion() in BaseAgentPlugin; tests for preflight behaviour.
Agent-specific guidance
src/plugins/agents/builtin/claude.ts, src/plugins/agents/builtin/opencode.ts, src/plugins/agents/droid/index.ts
Implement protected override getPreflightSuggestion(): string in built-in plugins (provides agent-specific diagnostic guidance).
Run / create-prd integration
src/commands/run.tsx, src/commands/create-prd.tsx
Add --verify flag / verify?: boolean option; invoke agent preflight when requested (or in create-prd) and handle failures with messages and exit(1).
Setup wizard & tests
src/setup/wizard.ts, src/setup/wizard.test.ts
After saving config, run a 30s preflight on the selected agent and report success or suggestions; tests updated to mock agent registry and preflight.

Sequence Diagram(s)

sequenceDiagram
    participant User as User/CLI
    participant CLI as CLI Dispatcher
    participant Registry as Agent Registry
    participant Agent as Agent Plugin
    participant External as External Service

    User->>CLI: Run 'doctor' command
    CLI->>Registry: load config & register agents
    Registry-->>CLI: registry ready

    CLI->>Registry: resolve agent plugin
    Registry-->>CLI: plugin factory/instance

    CLI->>Agent: detect()
    Agent->>External: check availability (binary/API)
    External-->>Agent: response (available/unavailable)
    Agent-->>CLI: detection result (path/version/status)

    alt detection success
        CLI->>Agent: preflight({timeout:30000})
        Agent->>External: send minimal test request
        External-->>Agent: test response
        Agent-->>CLI: preflight result (success/duration)
    else detection/preflight failure
        Agent-->>CLI: error, suggestion, duration
    end

    CLI->>CLI: format human or JSON output
    CLI-->>User: print report and exit (0 or 1)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Poem

🐰 I hopped in with a tiny byte,

Checked agents morning, noon and night,
Detection chirped and preflight gleamed,
Now diagnostics mend what once just dreamed,
A cheerful hop—your agents are bright!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main changes: a preflight verification system and a doctor command for diagnostics.
Linked Issues check ✅ Passed The PR implements comprehensive preflight verification and diagnostic tooling to address agent connectivity issues, including preflight checks integrated into setup, create-prd, and run flows, plus a doctor command for diagnostics.
Out of Scope Changes check ✅ Passed All changes are scoped to implementing the preflight system and doctor command; test updates and agent plugin modifications directly support these objectives.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

The mock was returning true for ALL plugins, which caused the
validateConfig test to fail when wizard tests ran first (mock pollution).
Now properly checks if the plugin is one of the known test plugins.
@codecov
Copy link

codecov bot commented Jan 17, 2026

Codecov Report

❌ Patch coverage is 70.26144% with 91 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.08%. Comparing base (efc0df0) to head (fd4e518).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/commands/run.tsx 2.63% 37 Missing ⚠️
src/commands/doctor.ts 80.12% 33 Missing ⚠️
src/setup/wizard.ts 53.12% 15 Missing ⚠️
src/plugins/agents/builtin/claude.ts 33.33% 2 Missing ⚠️
src/plugins/agents/builtin/opencode.ts 33.33% 2 Missing ⚠️
src/plugins/agents/droid/index.ts 33.33% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
- Coverage   46.73%   46.08%   -0.66%     
==========================================
  Files          58       59       +1     
  Lines       12968    13337     +369     
==========================================
+ Hits         6061     6146      +85     
- Misses       6907     7191     +284     
Files with missing lines Coverage Δ
src/plugins/agents/base.ts 24.76% <100.00%> (-2.49%) ⬇️
src/plugins/agents/builtin/claude.ts 47.54% <33.33%> (-6.07%) ⬇️
src/plugins/agents/builtin/opencode.ts 34.91% <33.33%> (-7.78%) ⬇️
src/plugins/agents/droid/index.ts 12.45% <33.33%> (-12.55%) ⬇️
src/setup/wizard.ts 79.68% <53.12%> (-2.65%) ⬇️
src/commands/doctor.ts 80.12% <80.12%> (ø)
src/commands/run.tsx 9.75% <2.63%> (-0.21%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@src/commands/doctor.test.ts`:
- Around line 286-346: The tests for doctor result structure are flaky because
they run in the repo cwd; update the suite to run in an isolated temp directory
by saving the original cwd, creating a temp dir (e.g., fs.mkdtempSync), calling
process.chdir(tempDir) in beforeEach (or at the start of this describe block)
and restoring process.chdir(originalCwd) in afterEach, and remove the temp dir
after tests; make this change around the describe('doctor result structure'...)
block and ensure executeDoctorCommand still executes but now in the isolated cwd
so detection/preflight assertions (executeDoctorCommand, DoctorResult parsing)
become deterministic.

In `@src/plugins/agents/base.test.ts`:
- Around line 243-268: The tests use expect(...).resolves but don’t await the
promise, so update the assertions to await the resolves to ensure proper async
assertion handling: add await before each
expect(agent.isReady()).resolves.toBe(...) in the 'initialize' and 'dispose'
tests (references: agent.initialize, agent.dispose, agent.isReady) so the test
runner waits for the promise resolution and reports failures correctly.
🧹 Nitpick comments (4)
src/plugins/agents/base.ts (1)

622-701: Surface stderr on preflight failures for clearer diagnostics.
When the process exits non‑zero without an error event, result.error is empty, so users see a generic failure. Falling back to result.stderr gives more actionable feedback.

♻️ Suggested adjustment
-      if (result.status === 'failed') {
-        return {
-          success: false,
-          error: result.error ?? 'Agent execution failed',
-          suggestion: this.getPreflightSuggestion(),
-          durationMs,
-        };
-      }
+      if (result.status === 'failed') {
+        const failureDetail = result.error || result.stderr || 'Agent execution failed';
+        return {
+          success: false,
+          error: failureDetail,
+          suggestion: this.getPreflightSuggestion(),
+          durationMs,
+        };
+      }
src/setup/wizard.ts (1)

384-424: Consider disposing the agent instance after preflight check.

The agentInstance obtained from the registry is used for the preflight check but never disposed. While this may not cause immediate issues since the wizard exits shortly after, it's good practice to clean up resources.

Additionally, consider wrapping the getInstance call in a try-catch block to handle potential initialisation failures gracefully, similar to how the doctor command handles this.

♻️ Suggested improvement
     // Get a fresh agent instance with the configured options
     const agentRegistry = getAgentRegistry();
-    const agentInstance = await agentRegistry.getInstance({
-      name: selectedAgent,
-      plugin: selectedAgent,
-      options: agentOptions,
-    });
+    let agentInstance;
+    try {
+      agentInstance = await agentRegistry.getInstance({
+        name: selectedAgent,
+        plugin: selectedAgent,
+        options: agentOptions,
+      });
+    } catch (error) {
+      printError(`Failed to initialise agent: ${error instanceof Error ? error.message : String(error)}`);
+      printInfo('Configuration saved, but the agent could not be initialised.');
+      printInfo('Run "ralph-tui doctor" to diagnose issues.');
+      console.log();
+      // Continue to show tracker-specific instructions
+    }

-    // Run preflight check
-    const preflightResult = await agentInstance.preflight({ timeout: 30000 });
+    if (agentInstance) {
+      // Run preflight check
+      const preflightResult = await agentInstance.preflight({ timeout: 30000 });
+      // ... existing success/failure handling ...
+
+      // Clean up
+      await agentInstance.dispose();
+    }
src/commands/run.tsx (1)

1362-1395: Consider disposing the agent instance after preflight check.

Similar to the wizard implementation, the agentInstance obtained for the preflight check is not disposed after use. Whilst the process may exit on failure, on success the agent instance remains allocated until the next agent is instantiated for the actual run.

♻️ Suggested improvement
     const agentRegistry = getAgentRegistry();
     const agentInstance = await agentRegistry.getInstance(config.agent);

     const preflightResult = await agentInstance.preflight({ timeout: 30000 });

     if (preflightResult.success) {
       console.log('✓ Agent is ready');
       if (preflightResult.durationMs) {
         console.log(`  Response time: ${preflightResult.durationMs}ms`);
       }
       console.log('');
+      // Note: Don't dispose here as the registry may reuse the instance
     } else {
       console.error('');
       console.error('❌ Agent preflight check failed');
       // ... error handling ...
+      await agentInstance.dispose();
       process.exit(1);
     }
src/commands/doctor.ts (1)

38-128: Consider disposing the agent instance after diagnostics.

The agent instance obtained from the registry is never disposed after the diagnostics complete. This could leave resources allocated, particularly if the doctor command is used programmatically.

♻️ Suggested improvement
   // Run preflight
   log('\n  Step 2: Preflight (testing if agent can respond)...');
   log('    Running test prompt...');

   const preflight = await agent.preflight({ timeout: 30000 });

+  // Clean up agent resources
+  await agent.dispose();
+
   if (!preflight.success) {
     return {
       healthy: false,
       agent: { name: agent.meta.name, plugin: agent.meta.id },
       detection,
       preflight,
       message: preflight.error ?? 'Agent failed preflight check',
     };
   }

Note: You may also want to ensure disposal happens in the early return paths (lines 90-96) and wrap in try/finally for robustness.

- doctor.test.ts: Run 'doctor result structure' tests in isolated temp
  directory to prevent flaky behavior from repo cwd state
- base.test.ts: Add await to expect().resolves assertions for proper
  async assertion handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

not creating PRD with opencode

1 participant