Fix Hardware Advisor Agent 'No Plan Found' warning for simple queries (#210)#216
Conversation
- Added SIMPLE_TOOLS class attribute to the Agent class to define tools that can execute directly without requiring a multi-step plan. - Updated HardwareAdvisorAgent to utilize SIMPLE_TOOLS for quick hardware information retrieval. - Implemented unit tests to validate SIMPLE_TOOLS behavior, ensuring tools execute without plan requirements and trigger warnings appropriately. - Created integration tests for agents with and without SIMPLE_TOOLS to verify end-to-end functionality. - Added tests for BlenderAgent and HardwareAdvisorAgent to ensure correct tool registration and execution.
There was a problem hiding this comment.
Pull request overview
This PR implements SIMPLE_TOOLS support for the GAIA agent framework to resolve issue #210, eliminating unnecessary "No plan found" warnings for straightforward information retrieval operations. The change enables agents to mark specific tools for direct execution without requiring multi-step plan creation.
Changes:
- Added
SIMPLE_TOOLSclass attribute support in the Hardware Advisor Agent example, marking three information retrieval tools as simple - Created comprehensive test suite (41 tests total) covering unit, integration, and agent-specific scenarios
- Added detailed documentation in both API specification and conceptual guides explaining when and how to use SIMPLE_TOOLS
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/hardware_advisor_agent.py | Adds SIMPLE_TOOLS list with three hardware query tools to eliminate warnings for simple queries |
| tests/unit/test_simple_tools.py | Unit tests for base Agent SIMPLE_TOOLS validation logic (15 tests) |
| tests/integration/test_simple_tools_integration.py | End-to-end integration tests for SIMPLE_TOOLS behavior (11 tests) |
| tests/test_hardware_advisor_agent.py | Hardware Advisor Agent-specific tests (15 tests) |
| tests/test_chat_agent.py | Adds SIMPLE_TOOLS verification tests for existing ChatAgent |
| tests/test_blender_agent.py | Creates SIMPLE_TOOLS verification tests for BlenderAgent |
| docs/spec/agent-base.mdx | Documents SIMPLE_TOOLS class attribute with technical implementation details |
| docs/sdk/core/agent-system.mdx | Adds comprehensive "Simple Tools (Direct Execution)" conceptual guide with examples |
| .claude/settings.local.json | Updates permissions for pytest commands and MCP server access |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Changes: - Delete tests/test_blender_agent.py (BlenderAgent out of scope) - Revert TestChatAgentSimpleTools class from tests/test_chat_agent.py (ChatAgent out of scope) - Fix unused variable warnings in tests/integration/test_simple_tools_integration.py All SIMPLE_TOOLS tests pass (27/27).
|
@claude review this PR |
|
Claude finished @itomek's task —— View job PR #216 Review CompleteStatus: 🟢 Approve with minor suggestions SummaryThis PR correctly implements What's Good
Copilot Review Comments - ClarificationSeveral Copilot comments are based on incorrect assumptions:
Minor SuggestionIn RecommendationMerge when CI passes. The implementation is correct and solves issue #210. |
|
@claude review my comments. Also answer why do we need to differentiate with SIMPLE_TOOLS? This seems like an over-engineered solution. Cant' we solve this with careful prompt optimization and tweaking the warnings? Why not just disable the warning if the agent figures out it's just a single step to execute the tool? |
|
Claude finished @kovtcharov's task —— View job Reviewing Comments and SIMPLE_TOOLS Architecture
Response to @kovtcharov's Question: Is SIMPLE_TOOLS Over-Engineered?Short answer: Yes, this could be solved more elegantly. The Current ProblemWhen an agent calls a tool in step 1 without first creating a plan, the warning appears at warning_msg = f"No plan found in step {step} response. The agent should create a plan for all tasks."Why SIMPLE_TOOLS Is Over-EngineeredYou're right that the LLM can figure this out. The issue is:
Alternative Solutions (Simpler)Option 1: Prompt Optimization - Add to system prompt: Option 2: Disable Warning for Single-Tool Responses - In # If response contains only one tool call, allow it without a plan
if "tool" in parsed and "plan" not in parsed and step == 1:
# Single tool call - no plan needed
returnOption 3: Use @simple_tool # Marks tool for direct execution
def get_hardware_info() -> dict:
...RecommendationI agree with @kovtcharov's assessment:
Review of @kovtcharov's Specific Comments
My RecommendationRequest changes to the PR author:
The simplest fix might be a 3-line change to --- | Branch: |
- Removed SIMPLE_TOOLS attribute from base Agent class and specific agent implementations. - Updated agent documentation to reflect changes in SIMPLE_TOOLS usage. - Introduced atomic tool execution with @tool(atomic=True) decorator for direct execution without planning. - Removed integration and unit tests related to SIMPLE_TOOLS, focusing on atomic tool behavior. - Adjusted Hardware Advisor Agent tests to verify hardware information retrieval without SIMPLE_TOOLS.
|
@claude please review |
|
Claude finished @itomek's task —— View job PR #216 Review Complete
Status: 🟢 ApproveSummaryThis PR has been significantly improved by implementing @kovtcharov's feedback. The Key Changes
What's Good
Minor Observations
RecommendationMerge when CI passes. The refactored implementation addresses @kovtcharov's feedback and follows GAIA's established decorator patterns. --- | Branch: |
Pull request was converted to draft
|
@claude review this please |
|
Claude finished @itomek's task —— View job PR #216 Review
Status: 🟢 ApproveSummaryThis PR has been refactored well based on @kovtcharov's feedback. The Key Changes
What's Good
Minor Observations
Addresses @kovtcharov's Feedback
RecommendationMerge when CI passes. The refactored implementation is clean, follows established patterns, and addresses all review feedback. | Branch: |
|
@claude review the latest updates to this branch. |
|
Claude finished @kovtcharov's task —— View job PR #216 Review - Latest Updates
Status: 🟢 Approve (with one action item)SummaryThe PR has been cleanly refactored from the What's Good
🟡 Action Item: @kovtcharov's Comment
The Recommendation: Add a brief section to ### Atomic Tools (Direct Execution)
For simple, self-contained operations that don't require multi-step planning:
@tool(atomic=True)
def get_system_info() -> dict:
"""Get current system information."""
return {"platform": platform.system(), "memory_gb": ...}
Use `atomic=True` for:
- Status/info retrieval (e.g., `get_scene_info`, `rag_status`)
- Simple queries that return immediately
- Read-only operations with no side effectsFiles Changed Summary
RecommendationMerge when the conceptual guide is updated (or create a follow-up issue to add | Branch: |
Summary
Fixes #210 by implementing
SIMPLE_TOOLSsupport for the Hardware Advisor Agent, allowing quick information retrieval operations to execute directly without requiring multi-step plans.Problem
When running the Hardware Advisor Agent with simple queries like "show me my system specs", users saw an unnecessary warning:
This warning appeared because all tools were treated as complex operations requiring plans, even for straightforward information retrieval.
Solution
Code Changes
examples/hardware_advisor_agent.py): AddedSIMPLE_TOOLS = ["get_hardware_info", "list_available_models", "recommend_models"]to mark quick information retrieval toolsTest Coverage (41 tests passing)
tests/unit/test_simple_tools.py): 15 tests for base Agent SIMPLE_TOOLS validation logictests/integration/test_simple_tools_integration.py): 11 tests for end-to-end agent behavior (no mocking)tests/test_hardware_advisor_agent.py): 15 tests for Hardware Advisor Agent implementationtests/test_chat_agent.py): Added SIMPLE_TOOLS verification for existing ChatAgenttests/test_blender_agent.py): Created SIMPLE_TOOLS verification testsDocumentation
docs/sdk/core/agent-system.mdx): Added comprehensive "Simple Tools (Direct Execution)" section with examples and best practicesdocs/spec/agent-base.mdx): DocumentedSIMPLE_TOOLSclass attribute with technical detailsTest Plan
uv run python -m pytest tests/unit/test_simple_tools.py -xvs✅ 15/15 passeduv run python -m pytest tests/integration/test_simple_tools_integration.py -xvs✅ 11/11 passeduv run python -m pytest tests/test_hardware_advisor_agent.py -xvs✅ 15/15 passeduv run python util/lint.py --all --fix✅ All checks passedVerification
After this change:
🤖 Generated with Claude Code