What happened?
Describe the bug
When using the Semantic Kernel Bedrock connector and SKChatCompletionAdapter, there seems to be an issue with the tools not being registered by the Assistant Agent. It seems that the SKChatCompletionAdapter is expecting a list of BaseTool, but the agent is providing a list of tool schemas:
Eg: in _sk_chat_completion_adapter.py:
def _sync_tools_with_kernel(self, kernel: Kernel, tools: Sequence[Tool | ToolSchema]) -> None:
"""Sync tools with kernel by updating the plugin"""
# Get current tool names in plugin
current_tool_names = set(self._tools_plugin.functions.keys())
# Get new tool names
new_tool_names = {tool.schema["name"] if isinstance(tool, Tool) else tool["name"] for tool in tools}
# Remove tools that are no longer needed
for tool_name in current_tool_names - new_tool_names:
del self._tools_plugin.functions[tool_name]
# Add or update tools
for tool in tools:
if isinstance(tool, BaseTool):
# Convert Tool to KernelFunction using KernelFunctionFromTool
kernel_function = KernelFunctionFromTool(tool) # type: ignore
self._tools_plugin.functions[tool.schema["name"]] = kernel_function
kernel.add_plugin(self._tools_plugin)
AND
in _assistant_agent.py:
async def _call_llm(
cls,
model_client: ChatCompletionClient,
model_client_stream: bool,
system_messages: List[SystemMessage],
model_context: ChatCompletionContext,
workbench: Workbench,
handoff_tools: List[BaseTool[Any, Any]],
agent_name: str,
cancellation_token: CancellationToken,
output_content_type: type[BaseModel] | None,
) -> AsyncGenerator[Union[CreateResult, ModelClientStreamingChunkEvent], None]:
"""
Perform a model inference and yield either streaming chunk events or the final CreateResult.
"""
all_messages = await model_context.get_messages()
llm_messages = cls._get_compatible_context(model_client=model_client, messages=system_messages + all_messages)
tools = (await workbench.list_tools()) + handoff_tools
...
To Reproduce
I used a version of the sample code from:
https://microsoft.github.io/autogen/stable//reference/python/autogen_ext.models.semantic_kernel.html
import asyncio
import logging
import boto3
from autogen_agentchat import EVENT_LOGGER_NAME, TRACE_LOGGER_NAME
from autogen_agentchat.agents import AssistantAgent
from autogen_core.models import ModelFamily, ModelInfo
from autogen_core.tools import FunctionTool
from autogen_ext.models.semantic_kernel import SKChatCompletionAdapter
from botocore.config import Config
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.bedrock import (
BedrockChatCompletion,
BedrockChatPromptExecutionSettings,
)
from semantic_kernel.memory.null_memory import NullMemory
logging.basicConfig(level=logging.DEBUG)
# For trace logging.
trace_logger = logging.getLogger(TRACE_LOGGER_NAME)
trace_logger.addHandler(logging.StreamHandler())
trace_logger.setLevel(logging.DEBUG)
event_logger = logging.getLogger(EVENT_LOGGER_NAME)
event_logger.addHandler(logging.StreamHandler())
event_logger.setLevel(logging.DEBUG)
my_config = Config(
region_name='us-east-1',
)
client_bedrock = boto3.client('bedrock', config=my_config)
runtime_client = boto3.client('bedrock-runtime', config=my_config)
async def top_song(city:str) -> str:
"""Get the most popular song for a given city"""
print(f'calling top song tool for {city}')
return "The top song is called: Hey JumJumXXuzz"
top_song_tool = FunctionTool(top_song, description="Get the most popular song for a provided city")
async def main():
sk_client = BedrockChatCompletion(
model_id="anthropic.claude-3-5-sonnet-20240620-v1:0",#"anthropic.claude-3-5-sonnet-20241022-v2:0",#"anthropic.claude-3-sonnet-20240229-v1:0",
client=client_bedrock,
runtime_client=runtime_client
)
settings = BedrockChatPromptExecutionSettings()
model_client = SKChatCompletionAdapter(
sk_client,
kernel=Kernel(memory=NullMemory()),
prompt_settings=settings,
model_info= ModelInfo(
function_calling= True,
json_output= True,
vision= False,
family= ModelFamily.CLAUDE_3_5_SONNET,
structured_output= True,
),
)
agent = AssistantAgent(
"assistant",
model_client=model_client,
system_message="You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.",
tools=[top_song_tool],
)
result = await agent.run(
task="what is the top song in madric?? "
)
result2 = await agent.run()
print("=== ALL MESSAGES ===")
for i, msg in enumerate(result.messages):
print(f"Message {i}: {type(msg).__name__}")
print(f"Content: {getattr(msg, 'content', 'No content')}")
if hasattr(msg, "tool_calls") and msg.tool_calls:
print(f"Tool calls: {msg.tool_calls}")
print("---")
for i, msg in enumerate(result2.messages):
print(f"Message {i}: {type(msg).__name__}")
print(f"Content: {getattr(msg, 'content', 'No content')}")
if hasattr(msg, "tool_calls") and msg.tool_calls:
print(f"Tool calls: {msg.tool_calls}")
print("---")
if __name__ == "__main__":
asyncio.run(main())
I enabled debugging mode to see what actual calls are being made and confirmed that _call_llm was not sending any toolconfig object.
To fix this i can change the way the agent gets a list of tools by calling workbench._tools rather than workbench.list_tools()
_assitant_agent.py:936
this then get the tools into the http request to bedrock and the agent can then use the tool normally
Expected behavior
I would expect _call_llm to have tool config. I guess there needs to be some consideration for how the assistant agent registers tools? Also possible that this is being used incorrectly by me, but worth considering that following the documentation for skchatcompletions adapter doesn't seem to work with assistant agent.
Which packages was the bug in?
Python AgentChat (autogen-agentchat>=0.4.0)
AutoGen library version.
Python 0.5.7
Other library version.
No response
Model used
No response
Model provider
None
Other model provider
No response
Python version
3.13
.NET version
None
Operating system
None
What happened?
Describe the bug
When using the Semantic Kernel Bedrock connector and SKChatCompletionAdapter, there seems to be an issue with the tools not being registered by the Assistant Agent. It seems that the SKChatCompletionAdapter is expecting a list of BaseTool, but the agent is providing a list of tool schemas:
Eg: in _sk_chat_completion_adapter.py:
AND
in _assistant_agent.py:
To Reproduce
I used a version of the sample code from:
https://microsoft.github.io/autogen/stable//reference/python/autogen_ext.models.semantic_kernel.html
I enabled debugging mode to see what actual calls are being made and confirmed that _call_llm was not sending any toolconfig object.
To fix this i can change the way the agent gets a list of tools by calling workbench._tools rather than workbench.list_tools()
_assitant_agent.py:936
this then get the tools into the http request to bedrock and the agent can then use the tool normally
Expected behavior
I would expect _call_llm to have tool config. I guess there needs to be some consideration for how the assistant agent registers tools? Also possible that this is being used incorrectly by me, but worth considering that following the documentation for skchatcompletions adapter doesn't seem to work with assistant agent.
Which packages was the bug in?
Python AgentChat (autogen-agentchat>=0.4.0)
AutoGen library version.
Python 0.5.7
Other library version.
No response
Model used
No response
Model provider
None
Other model provider
No response
Python version
3.13
.NET version
None
Operating system
None