.Net: Add Bedrock Connector by charlizeyeh · Pull Request #7587 · microsoft/semantic-kernel

Charlize Yeh (charlizeyeh) · 2024-07-31T22:30:10Z

Motivation and Context

There has been a strong demand from developers in the community to be able to utilize Bedrock with the Semantic Kernel. With the wide variety of foundation models offered including specialized models tailored for various domains and use cases, Bedrock's AI capabilities can now be leveraged within the Semantic Kernel's unified ecosystem with this connector.

Description

This pull request implements a connector for Amazon Bedrock, allowing developers to leverage Bedrock's foundation models within the Semantic Kernel framework. Key features of this implementation include:

Core Services: Implementation of IChatCompletionService and ITextGenerationService from the Semantic Kernel.
Authentication and Security: Secure client authentication and API communication with the Amazon Bedrock Runtime.
Modular Design: A directory organization and namespace structure similar to other connectors in the Semantic Kernel, including folders for core, services, models, and extensions.
Testing: Inclusion of unit tests and integration tests for available Bedrock models to demonstrate successful communication with Bedrock API, request/response mapping, and output generation.

The implementation leverages the following Amazon Bedrock Runtime actions:

InvokeModelAsync and InvokeModelWithResponseStreamAsync for GetTextContentsAsync and GetStreamingTextContentsAsync, respectively.
ConverseAsync and ConverseStreamAsync for GetChatMessageContentsAsync and GetStreamingChatMessageContentsAsync, respectively.

The following base models from various providers are currently connected and available for use under Bedrock:

AI21 Labs Jamba
AI21 Labs Jurassic-2 series
Amazon Titan
Anthropic Claude
Cohere Command
Meta Llama 3
Mistral AI Mistral

The kernel builder extension methods support two authentication scenarios:

User-provided IAmazonBedrockRuntime object: Developers can provide a pre-configured IAmazonBedrockRuntime object, allowing for customized configurations such as specifying custom credentials, endpoints, or other settings.
Automatic IAmazonBedrockRuntime creation: If developers do not provide an IAmazonBedrockRuntime object, an instance is automatically created and registered in the dependency injection (DI) container using the AWSSDK.Extensions.NETCore.Setup package. In this case, the IAmazonBedrockRuntime object will be configured with default settings, such as credentials obtained from environment variables, AWS credentials files, or instance profile credentials.

Resolve #3094
Resolve #4335

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

…sample program and run

…ttings

…his part yet)

…mmand r)

… r (diff bodies)

… supported by model

…models functional for all actions supported

…ce fixes in connector

Co-authored-by: Roger Barreto <[email protected]>

…e and IBedrockTextGenerationIOService

…mplicity

Charlize Yeh (charlizeyeh) · 2024-08-13T21:50:06Z

Apologies, I think there were extra usings in execution settings classes. dotnet format should be fixed now

Roger Barreto (rogerbarreto) · 2024-08-13T21:54:40Z

I still see some XmlDocs without description for params and return, suggest doing a full overview.

Search for should help:

<returns></returns>
></param>

Overall LGTM

Roger Barreto (rogerbarreto) · 2024-08-13T21:55:45Z

The usings I believe you will only be able to solve, removing the <ImplicitUsings>enable</ImplicitUsings> from the .csproj

…internal IO classes

Roger Barreto (rogerbarreto)

Charlize Yeh (@charlizeyeh) Thank you soo much for all changes made. Tomorrow plan to merge this in the feature branch.

Charlize Yeh (charlizeyeh) · 2024-08-14T22:05:16Z

Roger Barreto (@RogerBarret0)
Thank you for taking the time to review and excited to have this merged! The next PR (which adds text embedding and text to image service will include the prompt execution settings naming update). One question - will this branch be merged with main and if so, are there any steps on our end for that?

Roger Barreto (rogerbarreto) · 2024-08-19T08:14:46Z

Charlize Yeh (@charlizeyeh), yes, all connectors feature branches as of now intend to merge in main at some point.
The merging in main process goes around more series of reviews PR in the future and may arise as the team analyses it. Normally I try to do the most of this checks but another suggestions may come as part of it. From your side I just need to understand what are the missing features you plan to add prior to merging.

One feature that we want to add as part of the connectors is Function Calling/Tool Calling using our SK AutoFunctionInvocation abstractions but this is currently being finalized by SergeyMenshykh.

Manuel Naujoks (halllo) · 2024-08-19T10:52:02Z

Charlize Yeh (@charlizeyeh) thanks for your connector! I am looking forward to using it. Does it already support function calling (with Claude)?

Anushka Srinivasa (as14692) · 2024-09-05T19:55:57Z

Charlize Yeh (@charlizeyeh), yes, all connectors feature branches as of now intend to merge in main at some point. The merging in main process goes around more series of reviews PR in the future and may arise as the team analyses it. Normally I try to do the most of this checks but another suggestions may come as part of it. From your side I just need to understand what are the missing features you plan to add prior to merging.

One feature that we want to add as part of the connectors is Function Calling/Tool Calling using our SK AutoFunctionInvocation abstractions but this is currently being finalized by SergeyMenshykh.

Roger Barreto (@RogerBarret0) Could you please merge this PR into main which has the implementation of IChatCompletionService and ITextGenerationService. We will publish a new PR for the Function Calling/Tool Calling once the abstractions are ready. I will be contributing to the .NET Bedrock connector moving forward.

Beau Gosse (Beau-Gosse-dev) · 2024-09-12T18:13:14Z

Roger Barreto (@rogerbarreto) Any update on the question above? We'd love to get this out.

Roger Barreto (rogerbarreto) · 2024-09-13T10:41:06Z

Beau Gosse (@Beau-Gosse-dev) yes, we have plan on moving this one forward next sprint 35 (Sep-23 / Oct-06).

.Net: Add connector for AWS Bedrock #4335

Roger Barreto (rogerbarreto) · 2024-09-13T10:43:09Z

Do you have models for TextEmbeddings? Would be nice to have some options for the ITextEmbeddingGenerationService interface in the connector.

Beau Gosse (Beau-Gosse-dev) · 2024-09-13T19:12:55Z

Roger Barreto (@rogerbarreto) Thanks for the update. There is a first draft of TextEmbeddings support in our fork, but we've been pulled into other priorities at the moment and wouldn't be able to get it fully ready before your next sprint, so hopefully the first release can go without it and we can support TextEmbeddings in a later release.

Charlize Yeh (charyeh) added 30 commits July 1, 2024 14:54

initial chat completion start charyeh

6362c1e

created diff clients per service

ce24af6

working on generatechatmessageasync, converse req and res bodies needed

394ebed

rough text and chat completion with titan written out, need to write …

d32a026

…sample program and run

sample converse call gets a response

909d59b

titan chat completion connected, titan text gen request obj still buggy

7ebee0b

got titan text gen connected (rough untemplated)

924b585

titan text generation and chat completion connected

9c97764

mistral request body built

66be7e4

mistral chat connected too, this one accounts for prompt execution se…

ff102d3

…ttings

mistral chat iterative works, chat history retained

76a042c

claude chat completion connected, tool calling enabled (didn't test t…

cc5adfe

…his part yet)

ai21 chat completion connected

cdd249c

cohere chat completion connected, tool config still buggy

ab567eb

llama chat completion connected

bfc7d29

BedrockTest sample moved to Demos

d16d533

mistral text gen connected

78035c3

ai21 jamba text gen connected

4617cb2

claude text gen connected

9059a1d

command text gen connected (command r for chat, command for text

93abebf

llama text gen connected

19eb570

stream text gen added for titan, claude, mistral, and command (not co…

0bc7928

…mmand r)

chat streaming connected for mistral and titan

4bd51cc

command r and meta chat completion connected

a0ca077

connected command r text gen. differentiated btwn command and command…

2a31ae2

… r (diff bodies)

cohere command r chat history bug fix

6b26c1d

fixed IO naming conventions

5f4c170

connected text gen/invoke for ai21 jurassic. streaming & converse not…

09e8a9a

… supported by model

Code clean up. Removed/replaced all initial hard-coded settings. All …

6e9da83

…models functional for all actions supported

basic chat completion service unit test added. minor client and servi…

1749a1a

…ce fixes in connector

Charlize Yeh (charyeh) and others added 7 commits August 13, 2024 10:26

fix: remove concatenation with + for user agent

0e39a0f

fix: simplify model util role

f86fe33

Co-authored-by: Roger Barreto <[email protected]>

fix: cannot set authorrole

d29db9a

fix: remove memory stream allocation for get invoke body

465f4ec

fix: split IBedrockModelIOService into IBedrockChatCompletionIOServic…

5735b3a

…e and IBedrockTextGenerationIOService

fix: keep all within AI21JambaResponse as 1 level deep at most for si…

76e1c57

…mplicity

fix: dotnet format removed unused imports

5ade57e

Roger Barreto (rogerbarreto) reviewed Aug 13, 2024

View reviewed changes

Comment thread dotnet/src/Connectors/Connectors.Amazon/Bedrock/Extensions/BedrockKernelBuilderExtensions.cs Resolved

Add ServiceCollection Extensions

027104a

Roger Barreto (rogerbarreto) reviewed Aug 13, 2024

View reviewed changes

Comment thread ...et/src/Connectors/Connectors.Amazon/Bedrock/Extensions/BedrockServiceCollectionExtensions.cs Resolved

Charlize Yeh (charyeh) added 3 commits August 13, 2024 16:17

fix: removed implicit usings, reverted back to public methods inside …

dacf296

…internal IO classes

fix: filled in rest of XML descriptions

d519727

feat: added UT for service collection extensions

c6f7844

Roger Barreto (rogerbarreto) approved these changes Aug 14, 2024

View reviewed changes

Comment thread dotnet/src/Connectors/Connectors.Amazon/Bedrock/Core/BedrockModelUtilities.cs Resolved

Comment thread dotnet/src/Connectors/Connectors.Amazon/Bedrock/Extensions/BedrockKernelBuilderExtensions.cs Resolved

Charlize Yeh (charyeh) added 3 commits August 13, 2024 17:09

fix: minor XML and spacing corrections

c4ebd1b

fix: XML exceptions

cc9a795

fix: links to types in descriptions

6ab5927

Roger Barreto (rogerbarreto) merged commit 95b82dd into microsoft:feature-connectors-amazon Aug 14, 2024

Roger Barreto (RogerBarret0) reviewed Aug 14, 2024

View reviewed changes

Comment thread dotnet/src/Connectors/Connectors.Amazon/Bedrock/Settings/AmazonClaudeExecutionSettings.cs Resolved

Conversation

Charlize Yeh (charlizeyeh) commented Jul 31, 2024 • edited by rogerbarreto Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Description

Contribution Checklist

Uh oh!

Charlize Yeh (charlizeyeh) commented Aug 13, 2024

Uh oh!

Roger Barreto (rogerbarreto) commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Roger Barreto (rogerbarreto) commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roger Barreto (rogerbarreto) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Charlize Yeh (charlizeyeh) commented Aug 14, 2024

Uh oh!

Roger Barreto (rogerbarreto) commented Aug 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Manuel Naujoks (halllo) commented Aug 19, 2024

Uh oh!

Anushka Srinivasa (as14692) commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Beau Gosse (Beau-Gosse-dev) commented Sep 12, 2024

Uh oh!

Roger Barreto (rogerbarreto) commented Sep 13, 2024

Uh oh!

Roger Barreto (rogerbarreto) commented Sep 13, 2024

Uh oh!

Beau Gosse (Beau-Gosse-dev) commented Sep 13, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Charlize Yeh (charlizeyeh) commented Jul 31, 2024 •

edited by rogerbarreto

Loading

Roger Barreto (rogerbarreto) commented Aug 13, 2024 •

edited

Loading

Roger Barreto (rogerbarreto) commented Aug 13, 2024 •

edited

Loading

Roger Barreto (rogerbarreto) left a comment •

edited

Loading

Roger Barreto (rogerbarreto) commented Aug 19, 2024 •

edited

Loading

Anushka Srinivasa (as14692) commented Sep 5, 2024 •

edited

Loading