Skip to content

SKChatCompletionAdapter Expects BaseTools, but AssistantAgent provides ToolSchema instead. #6582

@foskaos

Description

@foskaos

What happened?

Describe the bug
When using the Semantic Kernel Bedrock connector and SKChatCompletionAdapter, there seems to be an issue with the tools not being registered by the Assistant Agent. It seems that the SKChatCompletionAdapter is expecting a list of BaseTool, but the agent is providing a list of tool schemas:

Eg: in _sk_chat_completion_adapter.py:

 def _sync_tools_with_kernel(self, kernel: Kernel, tools: Sequence[Tool | ToolSchema]) -> None:
        """Sync tools with kernel by updating the plugin"""
        # Get current tool names in plugin
        current_tool_names = set(self._tools_plugin.functions.keys())

        # Get new tool names
        new_tool_names = {tool.schema["name"] if isinstance(tool, Tool) else tool["name"] for tool in tools}

        # Remove tools that are no longer needed
        for tool_name in current_tool_names - new_tool_names:
            del self._tools_plugin.functions[tool_name]

        # Add or update tools
        for tool in tools:
            if isinstance(tool, BaseTool):
                # Convert Tool to KernelFunction using KernelFunctionFromTool
                kernel_function = KernelFunctionFromTool(tool)  # type: ignore
                self._tools_plugin.functions[tool.schema["name"]] = kernel_function

        kernel.add_plugin(self._tools_plugin)

AND
in _assistant_agent.py:

    async def _call_llm(
        cls,
        model_client: ChatCompletionClient,
        model_client_stream: bool,
        system_messages: List[SystemMessage],
        model_context: ChatCompletionContext,
        workbench: Workbench,
        handoff_tools: List[BaseTool[Any, Any]],
        agent_name: str,
        cancellation_token: CancellationToken,
        output_content_type: type[BaseModel] | None,
    ) -> AsyncGenerator[Union[CreateResult, ModelClientStreamingChunkEvent], None]:
        """
        Perform a model inference and yield either streaming chunk events or the final CreateResult.
        """
        all_messages = await model_context.get_messages()
        llm_messages = cls._get_compatible_context(model_client=model_client, messages=system_messages + all_messages)

        tools = (await workbench.list_tools()) + handoff_tools
...

To Reproduce
I used a version of the sample code from:
https://microsoft.github.io/autogen/stable//reference/python/autogen_ext.models.semantic_kernel.html

import asyncio
import logging

import boto3
from autogen_agentchat import EVENT_LOGGER_NAME, TRACE_LOGGER_NAME
from autogen_agentchat.agents import AssistantAgent
from autogen_core.models import ModelFamily, ModelInfo
from autogen_core.tools import FunctionTool
from autogen_ext.models.semantic_kernel import SKChatCompletionAdapter
from botocore.config import Config
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.bedrock import (
    BedrockChatCompletion,
    BedrockChatPromptExecutionSettings,
)
from semantic_kernel.memory.null_memory import NullMemory

logging.basicConfig(level=logging.DEBUG)

# For trace logging.
trace_logger = logging.getLogger(TRACE_LOGGER_NAME)
trace_logger.addHandler(logging.StreamHandler())
trace_logger.setLevel(logging.DEBUG)

event_logger = logging.getLogger(EVENT_LOGGER_NAME)
event_logger.addHandler(logging.StreamHandler())
event_logger.setLevel(logging.DEBUG)

my_config = Config(
    region_name='us-east-1',
)
client_bedrock = boto3.client('bedrock', config=my_config)

runtime_client = boto3.client('bedrock-runtime', config=my_config)


async def top_song(city:str) -> str:
    """Get the most popular song for a given city"""
    print(f'calling top song tool for {city}')
    return "The top song is called: Hey JumJumXXuzz"


top_song_tool = FunctionTool(top_song, description="Get the most popular song for a provided city")

async def main():
    sk_client = BedrockChatCompletion(
        model_id="anthropic.claude-3-5-sonnet-20240620-v1:0",#"anthropic.claude-3-5-sonnet-20241022-v2:0",#"anthropic.claude-3-sonnet-20240229-v1:0",
        client=client_bedrock,
        runtime_client=runtime_client
    )
    settings = BedrockChatPromptExecutionSettings()
    model_client = SKChatCompletionAdapter(
        sk_client,
        kernel=Kernel(memory=NullMemory()),
        prompt_settings=settings,
        model_info= ModelInfo(
            function_calling= True,
            json_output= True,
            vision= False,
            family= ModelFamily.CLAUDE_3_5_SONNET,
            structured_output= True,
        ),
    )

    agent = AssistantAgent(
        "assistant",
        model_client=model_client,
        system_message="You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.",
        tools=[top_song_tool],
    )

    result = await agent.run(
        task="what is the top song in madric?? "
    )
    result2 = await agent.run()

    print("=== ALL MESSAGES ===")
    for i, msg in enumerate(result.messages):
        print(f"Message {i}: {type(msg).__name__}")
        print(f"Content: {getattr(msg, 'content', 'No content')}")
        if hasattr(msg, "tool_calls") and msg.tool_calls:
            print(f"Tool calls: {msg.tool_calls}")
        print("---")

    for i, msg in enumerate(result2.messages):
        print(f"Message {i}: {type(msg).__name__}")
        print(f"Content: {getattr(msg, 'content', 'No content')}")
        if hasattr(msg, "tool_calls") and msg.tool_calls:
            print(f"Tool calls: {msg.tool_calls}")
        print("---")


if __name__ == "__main__":
    asyncio.run(main())

I enabled debugging mode to see what actual calls are being made and confirmed that _call_llm was not sending any toolconfig object.

To fix this i can change the way the agent gets a list of tools by calling workbench._tools rather than workbench.list_tools()

_assitant_agent.py:936

tools = workbench._tools

this then get the tools into the http request to bedrock and the agent can then use the tool normally
Expected behavior

I would expect _call_llm to have tool config. I guess there needs to be some consideration for how the assistant agent registers tools? Also possible that this is being used incorrectly by me, but worth considering that following the documentation for skchatcompletions adapter doesn't seem to work with assistant agent.

Which packages was the bug in?

Python AgentChat (autogen-agentchat>=0.4.0)

AutoGen library version.

Python 0.5.7

Other library version.

No response

Model used

No response

Model provider

None

Other model provider

No response

Python version

3.13

.NET version

None

Operating system

None

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions