Skip to content

Fix Hardware Advisor Agent 'No Plan Found' warning for simple queries (#210)#216

Merged
kovtcharov-amd merged 7 commits intomainfrom
210-hardware-advisor-agent-playbook-warning-message
Jan 22, 2026
Merged

Fix Hardware Advisor Agent 'No Plan Found' warning for simple queries (#210)#216
kovtcharov-amd merged 7 commits intomainfrom
210-hardware-advisor-agent-playbook-warning-message

Conversation

@itomek
Copy link
Collaborator

@itomek itomek commented Jan 21, 2026

Summary

Fixes #210 by implementing SIMPLE_TOOLS support for the Hardware Advisor Agent, allowing quick information retrieval operations to execute directly without requiring multi-step plans.

Problem

When running the Hardware Advisor Agent with simple queries like "show me my system specs", users saw an unnecessary warning:

⚠️  No plan found in step 1 response. The agent should create a plan for all tasks.

This warning appeared because all tools were treated as complex operations requiring plans, even for straightforward information retrieval.

Solution

Code Changes

  • Hardware Advisor Agent (examples/hardware_advisor_agent.py): Added SIMPLE_TOOLS = ["get_hardware_info", "list_available_models", "recommend_models"] to mark quick information retrieval tools

Test Coverage (41 tests passing)

  • Unit tests (tests/unit/test_simple_tools.py): 15 tests for base Agent SIMPLE_TOOLS validation logic
  • Integration tests (tests/integration/test_simple_tools_integration.py): 11 tests for end-to-end agent behavior (no mocking)
  • Hardware Advisor tests (tests/test_hardware_advisor_agent.py): 15 tests for Hardware Advisor Agent implementation
  • Chat Agent tests (tests/test_chat_agent.py): Added SIMPLE_TOOLS verification for existing ChatAgent
  • Blender Agent tests (tests/test_blender_agent.py): Created SIMPLE_TOOLS verification tests

Documentation

  • Conceptual Guide (docs/sdk/core/agent-system.mdx): Added comprehensive "Simple Tools (Direct Execution)" section with examples and best practices
  • API Specification (docs/spec/agent-base.mdx): Documented SIMPLE_TOOLS class attribute with technical details

Test Plan

  1. Run unit tests: uv run python -m pytest tests/unit/test_simple_tools.py -xvs ✅ 15/15 passed
  2. Run integration tests: uv run python -m pytest tests/integration/test_simple_tools_integration.py -xvs ✅ 11/11 passed
  3. Run Hardware Advisor tests: uv run python -m pytest tests/test_hardware_advisor_agent.py -xvs ✅ 15/15 passed
  4. Linting: uv run python util/lint.py --all --fix ✅ All checks passed

Verification

After this change:

  • ✅ Simple queries execute directly without warning
  • ✅ Complex operations still trigger plan creation
  • ✅ All existing functionality preserved
  • ✅ Documentation explains when to use SIMPLE_TOOLS

🤖 Generated with Claude Code

- Added SIMPLE_TOOLS class attribute to the Agent class to define tools that can execute directly without requiring a multi-step plan.
- Updated HardwareAdvisorAgent to utilize SIMPLE_TOOLS for quick hardware information retrieval.
- Implemented unit tests to validate SIMPLE_TOOLS behavior, ensuring tools execute without plan requirements and trigger warnings appropriately.
- Created integration tests for agents with and without SIMPLE_TOOLS to verify end-to-end functionality.
- Added tests for BlenderAgent and HardwareAdvisorAgent to ensure correct tool registration and execution.
Copilot AI review requested due to automatic review settings January 21, 2026 15:16
@itomek itomek linked an issue Jan 21, 2026 that may be closed by this pull request
@github-actions github-actions bot added documentation Documentation changes tests Test changes labels Jan 21, 2026
@itomek itomek self-assigned this Jan 21, 2026
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements SIMPLE_TOOLS support for the GAIA agent framework to resolve issue #210, eliminating unnecessary "No plan found" warnings for straightforward information retrieval operations. The change enables agents to mark specific tools for direct execution without requiring multi-step plan creation.

Changes:

  • Added SIMPLE_TOOLS class attribute support in the Hardware Advisor Agent example, marking three information retrieval tools as simple
  • Created comprehensive test suite (41 tests total) covering unit, integration, and agent-specific scenarios
  • Added detailed documentation in both API specification and conceptual guides explaining when and how to use SIMPLE_TOOLS

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
examples/hardware_advisor_agent.py Adds SIMPLE_TOOLS list with three hardware query tools to eliminate warnings for simple queries
tests/unit/test_simple_tools.py Unit tests for base Agent SIMPLE_TOOLS validation logic (15 tests)
tests/integration/test_simple_tools_integration.py End-to-end integration tests for SIMPLE_TOOLS behavior (11 tests)
tests/test_hardware_advisor_agent.py Hardware Advisor Agent-specific tests (15 tests)
tests/test_chat_agent.py Adds SIMPLE_TOOLS verification tests for existing ChatAgent
tests/test_blender_agent.py Creates SIMPLE_TOOLS verification tests for BlenderAgent
docs/spec/agent-base.mdx Documents SIMPLE_TOOLS class attribute with technical implementation details
docs/sdk/core/agent-system.mdx Adds comprehensive "Simple Tools (Direct Execution)" conceptual guide with examples
.claude/settings.local.json Updates permissions for pytest commands and MCP server access

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@itomek itomek marked this pull request as draft January 21, 2026 16:23
Changes:
- Delete tests/test_blender_agent.py (BlenderAgent out of scope)
- Revert TestChatAgentSimpleTools class from tests/test_chat_agent.py (ChatAgent out of scope)
- Fix unused variable warnings in tests/integration/test_simple_tools_integration.py

All SIMPLE_TOOLS tests pass (27/27).
@itomek
Copy link
Collaborator Author

itomek commented Jan 21, 2026

@claude review this PR

@github-actions
Copy link

github-actions bot commented Jan 21, 2026

Claude finished @itomek's task —— View job


PR #216 Review Complete

Status: 🟢 Approve with minor suggestions


Summary

This PR correctly implements SIMPLE_TOOLS support for the Hardware Advisor Agent, eliminating unnecessary "No plan found" warnings for simple queries like "show me my system specs".


What's Good

Aspect Assessment
Implementation Clean addition of SIMPLE_TOOLS at examples/hardware_advisor_agent.py:29
Test coverage 41 tests across unit/integration/agent-specific
Documentation Comprehensive updates to docs/sdk/core/agent-system.mdx and docs/spec/agent-base.mdx
Base Agent validation _validate_plan_required() at src/gaia/agents/base/agent.py:2136-2177 correctly handles simple tools

Copilot Review Comments - Clarification

Several Copilot comments are based on incorrect assumptions:

  1. tests/test_chat_agent.py and tests/test_blender_agent.py - These files are NOT modified by this PR. The existing tests are unchanged.

  2. ChatAgent SIMPLE_TOOLS with write_file/run_shell_command - This is a pre-existing design in src/gaia/agents/chat/agent.py:82-96, not introduced by this PR.

  3. BlenderAgent silent_mode - Not relevant to this PR's test files.


Minor Suggestion

In tests/integration/test_simple_tools_integration.py, some agent variables are created only to register tools. Consider using _agent prefix to indicate intentional non-use (e.g., _agent = QuickInfoAgent()).


Recommendation

Merge when CI passes. The implementation is correct and solves issue #210.


@itomek itomek marked this pull request as ready for review January 21, 2026 16:39
@itomek itomek enabled auto-merge January 21, 2026 16:39
@kovtcharov
Copy link
Collaborator

@claude review my comments. Also answer why do we need to differentiate with SIMPLE_TOOLS? This seems like an over-engineered solution. Cant' we solve this with careful prompt optimization and tweaking the warnings? Why not just disable the warning if the agent figures out it's just a single step to execute the tool?

@github-actions
Copy link

github-actions bot commented Jan 21, 2026

Claude finished @kovtcharov's task —— View job


Reviewing Comments and SIMPLE_TOOLS Architecture

  • Read the PR diff and changed files
  • Review @kovtcharov's comments
  • Analyze whether SIMPLE_TOOLS is necessary vs prompt optimization
  • Provide comprehensive response

Response to @kovtcharov's Question: Is SIMPLE_TOOLS Over-Engineered?

Short answer: Yes, this could be solved more elegantly.

The Current Problem

When an agent calls a tool in step 1 without first creating a plan, the warning appears at src/gaia/agents/base/agent.py:2160:

warning_msg = f"No plan found in step {step} response. The agent should create a plan for all tasks."

Why SIMPLE_TOOLS Is Over-Engineered

You're right that the LLM can figure this out. The issue is:

  1. The system prompt tells the agent to create plans (lines 178-188) but doesn't explain when to skip planning
  2. The warning logic is too strict - it expects a plan for ANY tool call in step 1

Alternative Solutions (Simpler)

Option 1: Prompt Optimization - Add to system prompt:

For simple information retrieval (like status checks or listing data), 
you may call the tool directly without a plan.

Option 2: Disable Warning for Single-Tool Responses - In _validate_plan_required():

# If response contains only one tool call, allow it without a plan
if "tool" in parsed and "plan" not in parsed and step == 1:
    # Single tool call - no plan needed
    return

Option 3: Use @simple_tool Decorator (Your suggestion) - More Pythonic:

@simple_tool  # Marks tool for direct execution
def get_hardware_info() -> dict:
    ...

Recommendation

I agree with @kovtcharov's assessment:

  1. Option 2 (disable warning for single-tool step-1 responses) is the simplest fix
  2. If explicit marking is needed, Option 3 (@simple_tool decorator) is cleaner than a class attribute list
  3. The current PR's SIMPLE_TOOLS = [...] list approach requires manual sync between tool registration and the list

Review of @kovtcharov's Specific Comments

File Comment Assessment
docs/sdk/core/agent-system.mdx:439 Rename QuickInfoAgent() to SysInfoAgent() 🟢 Reasonable
docs/sdk/core/agent-system.mdx:452 "One example is enough" 🟢 Agree - documentation is verbose
docs/sdk/core/agent-system.mdx:517 "I don't think we need this" 🟢 Agree - remove technical details section
examples/hardware_advisor_agent.py:29 Use @simple_tool decorator 🟢 Better pattern
docs/spec/agent-base.mdx:146 Use @simple_tool decorator pattern 🟢 Consistent with existing @tool
docs/sdk/core/agent-system.mdx:385 "Why can't the agent figure this out?" 🟡 See analysis above
docs/sdk/core/agent-system.mdx:407 Move example to examples/ folder 🟢 Reasonable

My Recommendation

Request changes to the PR author:

  1. Consider alternative approach: Modify _validate_plan_required() to allow single-tool calls without a plan (no explicit marking needed)
  2. If explicit marking is required: Implement @simple_tool decorator instead of class attribute list
  3. Reduce documentation verbosity: Address the specific comments about removing redundant examples

The simplest fix might be a 3-line change to _validate_plan_required() rather than 1000+ lines of new code.

--- | Branch: 210-hardware-advisor-agent-playbook-warning-message

- Removed SIMPLE_TOOLS attribute from base Agent class and specific agent implementations.
- Updated agent documentation to reflect changes in SIMPLE_TOOLS usage.
- Introduced atomic tool execution with @tool(atomic=True) decorator for direct execution without planning.
- Removed integration and unit tests related to SIMPLE_TOOLS, focusing on atomic tool behavior.
- Adjusted Hardware Advisor Agent tests to verify hardware information retrieval without SIMPLE_TOOLS.
@github-actions github-actions bot added the agents Agent system changes label Jan 21, 2026
@itomek
Copy link
Collaborator Author

itomek commented Jan 21, 2026

@claude please review

@github-actions
Copy link

github-actions bot commented Jan 21, 2026

Claude finished @itomek's task —— View job


PR #216 Review Complete

  • Read PR diff and changed files
  • Analyze the refactored implementation
  • Review @kovtcharov's feedback and previous discussion
  • Provide comprehensive review

Status: 🟢 Approve


Summary

This PR has been significantly improved by implementing @kovtcharov's feedback. The SIMPLE_TOOLS class attribute approach has been replaced with a cleaner @tool(atomic=True) decorator pattern, which follows established GAIA conventions.


Key Changes

Aspect Before After
Approach SIMPLE_TOOLS = ["tool1", "tool2"] class attribute @tool(atomic=True) decorator
Tool decorator @tool only @tool(atomic=True) for atomic tools
Validation logic _validate_plan_required() with 45 lines Removed (simplified)
Code location Per-agent class attribute lists Inline with tool definition

What's Good

  1. Clean decorator pattern - @tool(atomic=True) at src/gaia/agents/base/tools.py:19-24 follows existing @tool conventions
  2. Removed over-engineering - The 45-line _validate_plan_required() method was removed; replaced with simple get_atomic_tools() at src/gaia/agents/base/agent.py:2126-2137
  3. Consistent across agents - BlenderAgent, ChatAgent, and Hardware Advisor Agent all use the same pattern
  4. Self-documenting - Tool behavior is declared at the tool definition site, not in a separate list
  5. Tests focused - Tests verify actual tool functionality, not just list membership

Minor Observations

  1. get_atomic_tools() method unused - The new method at agent.py:2126-2137 is defined but not called anywhere in the codebase. Consider:

    • Removing it if not needed
    • Or documenting its intended use case
  2. Documentation updates - The docs/spec/*.mdx files were updated to remove SIMPLE_TOOLS references, but no documentation was added explaining the @tool(atomic=True) pattern. Consider adding this to docs/sdk/core/tools.mdx.

  3. Test coverage - The new test file (tests/test_hardware_advisor_agent.py) focuses on tool functionality but doesn't test the atomic=True metadata registration.


Recommendation

Merge when CI passes. The refactored implementation addresses @kovtcharov's feedback and follows GAIA's established decorator patterns.

--- | Branch: 210-hardware-advisor-agent-playbook-warning-message

@itomek itomek marked this pull request as draft January 21, 2026 20:11
auto-merge was automatically disabled January 21, 2026 20:11

Pull request was converted to draft

@itomek
Copy link
Collaborator Author

itomek commented Jan 21, 2026

@claude review this please

@github-actions
Copy link

github-actions bot commented Jan 21, 2026

Claude finished @itomek's task —— View job


PR #216 Review

  • Read PR diff and changed files
  • Analyze the @tool(atomic=True) implementation
  • Review test coverage
  • Check documentation updates
  • Provide comprehensive review

Status: 🟢 Approve


Summary

This PR has been refactored well based on @kovtcharov's feedback. The SIMPLE_TOOLS class attribute list approach has been replaced with a cleaner @tool(atomic=True) decorator pattern, which follows GAIA's established conventions and is more Pythonic.


Key Changes

Aspect Before After
Approach SIMPLE_TOOLS = ["tool1", "tool2"] class attribute @tool(atomic=True) decorator parameter
Tool decorator @tool only @tool(atomic=True) for atomic tools
Validation logic 45-line _validate_plan_required() method Removed entirely
Code location Per-agent class attribute lists Inline with tool definition

What's Good

  1. Clean decorator pattern - @tool(atomic=True) at src/gaia/agents/base/tools.py:19-24 follows the existing @tool convention elegantly

  2. Self-documenting - Tool behavior is declared at the tool definition site, not in a separate list that must be kept in sync

  3. Removed complexity - The 45-line _validate_plan_required() method was removed from agent.py:2132-2177, simplifying the base Agent

  4. Consistent across agents - All agents now use the same pattern:

    • examples/hardware_advisor_agent.py:124,174,217 - Hardware Advisor tools
    • src/gaia/agents/blender/agent.py:169,408 - BlenderAgent tools
    • src/gaia/agents/chat/tools/rag_tools.py - RAG tools (7 tools marked atomic)
    • src/gaia/agents/chat/tools/shell_tools.py:105 - Shell tools
    • src/gaia/agents/tools/file_tools.py - File tools (5 tools marked atomic)
  5. Good test coverage - New test file tests/unit/test_tool_decorator.py with 7 tests covering:

    • Basic registration
    • atomic default to False
    • atomic=True flag storage
    • Mixed atomic values across tools
    • Parameter capture
    • Function callability

Minor Observations

Issue Severity Location Notes
atomic flag unused 🟡 Minor src/gaia/agents/base/tools.py:76 The atomic flag is stored in the registry but never read. This is metadata for future use - acceptable.
No documentation for atomic in conceptual guide 🟢 Low N/A The API spec (docs/spec/tool-decorator.mdx) is updated, but no conceptual guide for when to use atomic=True. Consider adding to docs/sdk/core/tools.mdx in a follow-up.

Addresses @kovtcharov's Feedback

Comment Resolution
"Use @simple_tool decorator pattern" ✅ Implemented as @tool(atomic=True) parameter instead of separate decorator - cleaner, less code duplication
"Why can't the agent figure this out?" ✅ Removed the warning/validation logic entirely. The atomic flag is now just metadata; agents don't force planning
"This is over-engineered" ✅ Simplified significantly - removed _validate_plan_required() method and SIMPLE_TOOLS attribute

Recommendation

Merge when CI passes. The refactored implementation is clean, follows established patterns, and addresses all review feedback.


| Branch: 210-hardware-advisor-agent-playbook-warning-message

@itomek itomek marked this pull request as ready for review January 21, 2026 21:03
@itomek itomek enabled auto-merge January 21, 2026 21:05
@kovtcharov
Copy link
Collaborator

@claude review the latest updates to this branch.

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Claude finished @kovtcharov's task —— View job


PR #216 Review - Latest Updates

  • Read PR diff and changed files
  • Analyze the @tool(atomic=True) implementation
  • Review documentation updates
  • Check @kovtcharov's comment about atomic documentation
  • Provide comprehensive review

Status: 🟢 Approve (with one action item)


Summary

The PR has been cleanly refactored from the SIMPLE_TOOLS class attribute list to the @tool(atomic=True) decorator pattern. This is a much cleaner approach that follows GAIA's established patterns.


What's Good

Aspect Assessment
Implementation Clean @tool(atomic=True) decorator at src/gaia/agents/base/tools.py:19-24
Removed complexity 53 lines removed from agent.py including _validate_plan_required() and SIMPLE_TOOLS attribute
Self-documenting Tool behavior declared inline with tool definition
Test coverage New tests/unit/test_tool_decorator.py with 7 tests covering atomic flag
API spec updated docs/spec/tool-decorator.mdx fully documents the atomic parameter

🟡 Action Item: @kovtcharov's Comment

"Make sure atomic is described in the documentation / userguide the first time it's mentioned."

The atomic parameter is documented in the API spec (docs/spec/tool-decorator.mdx:79,93-95,194-227) but is NOT in the conceptual guide (docs/sdk/core/tools.mdx).

Recommendation: Add a brief section to docs/sdk/core/tools.mdx explaining when to use @tool(atomic=True):

### Atomic Tools (Direct Execution)

For simple, self-contained operations that don't require multi-step planning:

@tool(atomic=True)
def get_system_info() -> dict:
    """Get current system information."""
    return {"platform": platform.system(), "memory_gb": ...}

Use `atomic=True` for:
- Status/info retrieval (e.g., `get_scene_info`, `rag_status`)
- Simple queries that return immediately
- Read-only operations with no side effects

Files Changed Summary

File Change
src/gaia/agents/base/tools.py Added atomic parameter to @tool decorator
src/gaia/agents/base/agent.py Removed SIMPLE_TOOLS and _validate_plan_required() (-53 lines)
examples/hardware_advisor_agent.py Uses @tool(atomic=True) for 3 tools
src/gaia/agents/blender/agent.py Uses @tool(atomic=True) for clear_scene, get_scene_info
src/gaia/agents/chat/tools/*.py Uses @tool(atomic=True) for RAG, shell, and file tools
docs/spec/tool-decorator.mdx Documented atomic parameter with examples
tests/unit/test_tool_decorator.py Added 7 tests for atomic flag

Recommendation

Merge when the conceptual guide is updated (or create a follow-up issue to add atomic docs to docs/sdk/core/tools.mdx).


| Branch: 210-hardware-advisor-agent-playbook-warning-message

@itomek itomek added this pull request to the merge queue Jan 22, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 22, 2026
@kovtcharov-amd kovtcharov-amd added this pull request to the merge queue Jan 22, 2026
Merged via the queue into main with commit 24dde65 Jan 22, 2026
62 checks passed
@kovtcharov-amd kovtcharov-amd deleted the 210-hardware-advisor-agent-playbook-warning-message branch January 22, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent system changes documentation Documentation changes tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hardware Advisor Agent playbook warning message

4 participants