Skip to content

Commit 6305e3e

Browse files
Python: feat(python): Add embedding abstractions and OpenAI implementation (Phase 1) (#4153)
* feat(python): Add embedding abstractions and OpenAI implementation (Phase 1) This PR contains two parts: 1. **Overall migration plan** for porting vector stores and embeddings from Semantic Kernel to Agent Framework (docs/features/vector-stores-and-embeddings/README.md) covering all 10 phases from core abstractions through connectors and TextSearch. 2. **Phase 1 implementation** — core embedding abstractions and OpenAI/Azure OpenAI embedding clients: Core types (_types.py): - EmbeddingGenerationOptions TypedDict (total=False) - Embedding[EmbeddingT] generic class with model_id, dimensions, created_at - GeneratedEmbeddings[EmbeddingT, EmbeddingOptionsT] list container with options, usage - EmbeddingInputT (default str) and EmbeddingT (default list[float]) TypeVars Protocol + base class (_clients.py): - SupportsGetEmbeddings protocol — Generic[EmbeddingInputT, EmbeddingT, OptionsContraT] - BaseEmbeddingClient ABC — Generic[EmbeddingInputT, EmbeddingT, OptionsCoT] Telemetry (observability.py): - EmbeddingTelemetryLayer with gen_ai.operation.name = "embeddings" OpenAI implementation (openai/_embedding_client.py): - RawOpenAIEmbeddingClient, OpenAIEmbeddingClient, OpenAIEmbeddingOptions - Uses _ensure_client() factory pattern Azure OpenAI implementation (azure/_embedding_client.py): - AzureOpenAIEmbeddingClient following AzureOpenAIChatClient pattern - Supports API key, Entra ID credentials, env var configuration Tests: - 47 unit tests for types, protocol, base class, OpenAI, and Azure clients - 6 integration tests (gated behind RUN_INTEGRATION_TESTS + credentials) Samples: - samples/02-agents/embeddings/openai_embeddings.py - samples/02-agents/embeddings/azure_openai_embeddings.py Co-authored-by: Copilot <[email protected]> * fix: Add AzureOpenAIEmbeddingClient to azure __init__.pyi stub Co-authored-by: Copilot <[email protected]> * ci: Add embedding env vars to Python integration tests Map OPENAI_EMBEDDING_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME from GitHub vars to the integration test environment. Co-authored-by: Copilot <[email protected]> * fix: Handle base64 encoding_format in OpenAI embedding client When encoding_format='base64' is used, the OpenAI API returns base64-encoded floats instead of a JSON array. Decode these automatically to list[float] so the return type stays consistent regardless of encoding format. Also adds a unit test for base64 decoding and fixes minor docstring/import issues. Co-authored-by: Copilot <[email protected]> * fix: Only record INPUT_TOKENS for embedding telemetry Embeddings have no output/completion tokens. Remove OUTPUT_TOKENS recording which was double-counting prompt_tokens via the total_tokens fallback. Co-authored-by: Copilot <[email protected]> * fix: Resolve mypy variance error and lint warning Use contravariant/covariant TypeVars for SupportsGetEmbeddings Protocol. Combine nested if into single statement in telemetry layer. Co-authored-by: Copilot <[email protected]> * fix: Make EmbeddingCoT invariant for mypy compatibility GeneratedEmbeddings is invariant in its type param, so the Protocol TypeVar cannot be covariant. Co-authored-by: Copilot <[email protected]> * fix: Address PR review - empty values guard, service_url for telemetry - Add early return for empty values in get_embeddings to avoid unnecessary API calls - Add service_url() method to RawOpenAIEmbeddingClient for proper telemetry endpoint reporting - Add test for empty values behavior Co-authored-by: Copilot <[email protected]> * Python: Fix OpenAI chat client compatibility with third-party endpoints and OTel 0.4.14 (#4161) * Fix system message content sent as list instead of string Some OpenAI-compatible endpoints (e.g. NVIDIA NIM) reject system messages when content is a list of content parts. This change flattens system and developer message content to a plain string in the Chat Completions client. Fixes #1407 Co-authored-by: Copilot <[email protected]> * Fix compatibility with opentelemetry-semantic-conventions-ai 0.4.14 Version 0.4.14 removed several LLM_* attributes from SpanAttributes (LLM_SYSTEM, LLM_REQUEST_MODEL, LLM_RESPONSE_MODEL, LLM_REQUEST_MAX_TOKENS, LLM_REQUEST_TEMPERATURE, LLM_REQUEST_TOP_P, LLM_TOKEN_TYPE). Move these to the OtelAttr enum with their well-known gen_ai.* string values and update all references in observability.py and tests. Fixes #4160 Co-authored-by: Copilot <[email protected]> * Flatten text-only message content to string for all roles Extend the system/developer fix to all message roles. Text-only content lists are now post-processed into plain strings, while multimodal content (text + images/audio) remains as a list. This fixes compatibility with OpenAI-like endpoints that cannot deserialize list content (e.g. Foundry Local's Neutron backend). Partially fixes #4084 Co-authored-by: Copilot <[email protected]> * Fix streaming text lost when usage data in same chunk Some providers (e.g. Gemini) include both usage data and text content in the same streaming chunk. The early return on chunk.usage caused text and tool call parsing to be skipped entirely. Remove the early return and process usage alongside text/tool calls. Fixes #3434 Co-authored-by: Copilot <[email protected]> * Fix mypy errors in _chat_client.py Rename shadowed variable 'args' in system/developer branch to 'sys_args' and rename loop variable 'content' to 'msg_content' to avoid type conflict. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * reorder imports * fix: Use OtelAttr.REQUEST_MODEL instead of removed SpanAttributes.LLM_REQUEST_MODEL Co-authored-by: Copilot <[email protected]> * docs: Add score_threshold to vector store plan Reference SK .NET PR #13501 for score threshold filtering semantics. Include score_threshold in SearchOptions from Phase 3. Co-authored-by: Copilot <[email protected]> * docs: Add reference to roji's SK .NET MEVD work for SQL connectors Co-authored-by: Copilot <[email protected]> * fix: Clear env vars in construction tests to avoid CI leakage Tests for missing API key / model ID now use monkeypatch.delenv to ensure env vars from the integration test environment don't prevent the expected ValueError from being raised. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>
1 parent 7b24d91 commit 6305e3e

18 files changed

Lines changed: 1890 additions & 3 deletions

File tree

‎.github/workflows/python-merge-tests.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,10 @@ jobs:
6262
OPENAI_API_KEY: ${{ secrets.OPENAI__APIKEY }}
6363
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
6464
ANTHROPIC_CHAT_MODEL_ID: ${{ vars.ANTHROPIC_CHAT_MODEL_ID }}
65+
OPENAI_EMBEDDING_MODEL_ID: ${{ vars.OPENAI__EMBEDDINGMODELID }}
6566
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
6667
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
68+
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__EMBEDDINGDEPLOYMENTNAME }}
6769
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
6870
LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
6971
# For Azure Functions integration tests

‎docs/features/vector-stores-and-embeddings/README.md‎

Lines changed: 389 additions & 0 deletions
Large diffs are not rendered by default.

‎python/packages/core/agent_framework/__init__.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@
2020
from ._agents import Agent, BaseAgent, RawAgent, SupportsAgentRun
2121
from ._clients import (
2222
BaseChatClient,
23+
BaseEmbeddingClient,
2324
SupportsChatGetResponse,
2425
SupportsCodeInterpreterTool,
2526
SupportsFileSearchTool,
27+
SupportsGetEmbeddings,
2628
SupportsImageGenerationTool,
2729
SupportsMCPTool,
2830
SupportsWebSearchTool,
@@ -82,9 +84,14 @@
8284
ChatResponseUpdate,
8385
Content,
8486
ContinuationToken,
87+
Embedding,
88+
EmbeddingGenerationOptions,
89+
EmbeddingInputT,
90+
EmbeddingT,
8591
FinalT,
8692
FinishReason,
8793
FinishReasonLiteral,
94+
GeneratedEmbeddings,
8895
Message,
8996
OuterFinalT,
9097
OuterUpdateT,
@@ -201,6 +208,7 @@
201208
"BaseAgent",
202209
"BaseChatClient",
203210
"BaseContextProvider",
211+
"BaseEmbeddingClient",
204212
"BaseHistoryProvider",
205213
"Case",
206214
"ChatAndFunctionMiddlewareTypes",
@@ -218,6 +226,10 @@
218226
"Edge",
219227
"EdgeCondition",
220228
"EdgeDuplicationError",
229+
"Embedding",
230+
"EmbeddingGenerationOptions",
231+
"EmbeddingInputT",
232+
"EmbeddingT",
221233
"Executor",
222234
"FanInEdgeGroup",
223235
"FanOutEdgeGroup",
@@ -232,6 +244,7 @@
232244
"FunctionMiddleware",
233245
"FunctionMiddlewareTypes",
234246
"FunctionTool",
247+
"GeneratedEmbeddings",
235248
"GraphConnectivityError",
236249
"InMemoryCheckpointStorage",
237250
"InMemoryHistoryProvider",
@@ -261,6 +274,7 @@
261274
"SupportsChatGetResponse",
262275
"SupportsCodeInterpreterTool",
263276
"SupportsFileSearchTool",
277+
"SupportsGetEmbeddings",
264278
"SupportsImageGenerationTool",
265279
"SupportsMCPTool",
266280
"SupportsWebSearchTool",

‎python/packages/core/agent_framework/_clients.py‎

Lines changed: 141 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@
3535
from ._types import (
3636
ChatResponse,
3737
ChatResponseUpdate,
38+
EmbeddingGenerationOptions,
39+
EmbeddingInputT,
40+
EmbeddingT,
41+
GeneratedEmbeddings,
3842
Message,
3943
ResponseStream,
4044
validate_chat_options,
@@ -56,7 +60,6 @@
5660

5761
InputT = TypeVar("InputT", contravariant=True)
5862

59-
EmbeddingT = TypeVar("EmbeddingT")
6063
BaseChatClientT = TypeVar("BaseChatClientT", bound="BaseChatClient")
6164

6265
logger = logging.getLogger("agent_framework")
@@ -660,3 +663,140 @@ def get_file_search_tool(**kwargs: Any) -> Any:
660663

661664

662665
# endregion
666+
667+
668+
# region SupportsGetEmbeddings Protocol
669+
670+
# Contravariant/covariant TypeVars for the Protocol
671+
EmbeddingInputContraT = TypeVar(
672+
"EmbeddingInputContraT",
673+
default="str",
674+
contravariant=True,
675+
)
676+
EmbeddingCoT = TypeVar(
677+
"EmbeddingCoT",
678+
default="list[float]",
679+
)
680+
EmbeddingOptionsContraT = TypeVar(
681+
"EmbeddingOptionsContraT",
682+
bound=TypedDict, # type: ignore[valid-type]
683+
default="EmbeddingGenerationOptions",
684+
contravariant=True,
685+
)
686+
687+
688+
@runtime_checkable
689+
class SupportsGetEmbeddings(Protocol[EmbeddingInputContraT, EmbeddingCoT, EmbeddingOptionsContraT]):
690+
"""Protocol for an embedding client that can generate embeddings.
691+
692+
This protocol enables duck-typing for embedding generation. Any class that
693+
implements ``get_embeddings`` with a compatible signature satisfies this protocol.
694+
695+
Generic over the input type (defaults to ``str``), output embedding type
696+
(defaults to ``list[float]``), and options type.
697+
698+
Examples:
699+
.. code-block:: python
700+
701+
from agent_framework import SupportsGetEmbeddings
702+
703+
704+
async def use_embeddings(client: SupportsGetEmbeddings) -> None:
705+
result = await client.get_embeddings(["Hello, world!"])
706+
for embedding in result:
707+
print(embedding.vector)
708+
"""
709+
710+
additional_properties: dict[str, Any]
711+
712+
def get_embeddings(
713+
self,
714+
values: Sequence[EmbeddingInputContraT],
715+
*,
716+
options: EmbeddingOptionsContraT | None = None,
717+
) -> Awaitable[GeneratedEmbeddings[EmbeddingCoT]]:
718+
"""Generate embeddings for the given values.
719+
720+
Args:
721+
values: The values to generate embeddings for.
722+
options: Optional embedding generation options.
723+
724+
Returns:
725+
Generated embeddings with metadata.
726+
"""
727+
...
728+
729+
730+
# endregion
731+
732+
733+
# region BaseEmbeddingClient
734+
735+
# Covariant for the BaseEmbeddingClient
736+
EmbeddingOptionsCoT = TypeVar(
737+
"EmbeddingOptionsCoT",
738+
bound=TypedDict, # type: ignore[valid-type]
739+
default="EmbeddingGenerationOptions",
740+
covariant=True,
741+
)
742+
743+
744+
class BaseEmbeddingClient(SerializationMixin, ABC, Generic[EmbeddingInputT, EmbeddingT, EmbeddingOptionsCoT]):
745+
"""Abstract base class for embedding clients.
746+
747+
Subclasses implement ``get_embeddings`` to provide the actual
748+
embedding generation logic.
749+
750+
Generic over the input type (defaults to ``str``), output embedding type
751+
(defaults to ``list[float]``), and options type.
752+
753+
Examples:
754+
.. code-block:: python
755+
756+
from agent_framework import BaseEmbeddingClient, Embedding, GeneratedEmbeddings
757+
from collections.abc import Sequence
758+
759+
760+
class CustomEmbeddingClient(BaseEmbeddingClient):
761+
async def get_embeddings(self, values, *, options=None):
762+
return GeneratedEmbeddings([Embedding(vector=[0.1, 0.2, 0.3]) for _ in values])
763+
"""
764+
765+
OTEL_PROVIDER_NAME: ClassVar[str] = "unknown"
766+
DEFAULT_EXCLUDE: ClassVar[set[str]] = {"additional_properties"}
767+
768+
def __init__(
769+
self,
770+
*,
771+
additional_properties: dict[str, Any] | None = None,
772+
**kwargs: Any,
773+
) -> None:
774+
"""Initialize a BaseEmbeddingClient instance.
775+
776+
Args:
777+
additional_properties: Additional properties to pass to the client.
778+
**kwargs: Additional keyword arguments passed to parent classes (for MRO).
779+
"""
780+
self.additional_properties = additional_properties or {}
781+
super().__init__(**kwargs)
782+
783+
@abstractmethod
784+
async def get_embeddings(
785+
self,
786+
values: Sequence[EmbeddingInputT],
787+
*,
788+
options: EmbeddingOptionsCoT | None = None,
789+
) -> GeneratedEmbeddings[EmbeddingT]:
790+
"""Generate embeddings for the given values.
791+
792+
Args:
793+
values: The values to generate embeddings for.
794+
options: Optional embedding generation options.
795+
796+
Returns:
797+
Generated embeddings with metadata.
798+
"""
799+
...
800+
801+
802+
# endregion

‎python/packages/core/agent_framework/_types.py‎

Lines changed: 139 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,18 @@
88
import re
99
import sys
1010
from asyncio import iscoroutine
11-
from collections.abc import AsyncIterable, AsyncIterator, Awaitable, Callable, Mapping, MutableMapping, Sequence
11+
from collections.abc import (
12+
AsyncIterable,
13+
AsyncIterator,
14+
Awaitable,
15+
Callable,
16+
Iterable,
17+
Mapping,
18+
MutableMapping,
19+
Sequence,
20+
)
1221
from copy import deepcopy
22+
from datetime import datetime
1323
from typing import TYPE_CHECKING, Any, ClassVar, Final, Generic, Literal, NewType, cast, overload
1424

1525
from pydantic import BaseModel
@@ -272,7 +282,8 @@ def _serialize_value(value: Any, exclude_none: bool) -> Any:
272282

273283
# region Constants and types
274284
_T = TypeVar("_T")
275-
EmbeddingT = TypeVar("EmbeddingT")
285+
EmbeddingT = TypeVar("EmbeddingT", default="list[float]")
286+
EmbeddingInputT = TypeVar("EmbeddingInputT", default="str")
276287
ChatResponseT = TypeVar("ChatResponseT", bound="ChatResponse")
277288
ToolModeT = TypeVar("ToolModeT", bound="ToolMode")
278289
AgentResponseT = TypeVar("AgentResponseT", bound="AgentResponse")
@@ -3162,3 +3173,129 @@ def merge_chat_options(
31623173
result[key] = value
31633174

31643175
return result
3176+
3177+
3178+
# region Embedding Types
3179+
3180+
3181+
class EmbeddingGenerationOptions(TypedDict, total=False):
3182+
"""Common request settings for embedding generation.
3183+
3184+
All fields are optional (total=False) to allow partial specification.
3185+
Provider-specific TypedDicts extend this with additional options.
3186+
3187+
Examples:
3188+
.. code-block:: python
3189+
3190+
from agent_framework import EmbeddingGenerationOptions
3191+
3192+
options: EmbeddingGenerationOptions = {
3193+
"model_id": "text-embedding-3-small",
3194+
"dimensions": 1536,
3195+
}
3196+
"""
3197+
3198+
model_id: str
3199+
dimensions: int
3200+
3201+
3202+
class Embedding(Generic[EmbeddingT]):
3203+
"""A single embedding vector with metadata.
3204+
3205+
Generic over the embedding vector type, e.g. ``Embedding[list[float]]``,
3206+
``Embedding[list[int]]``, or ``Embedding[bytes]``.
3207+
3208+
Args:
3209+
vector: The embedding vector data.
3210+
model_id: The model used to generate this embedding.
3211+
dimensions: Explicit dimension count (computed from vector length if omitted).
3212+
created_at: Timestamp of when the embedding was generated.
3213+
additional_properties: Additional metadata.
3214+
3215+
Examples:
3216+
.. code-block:: python
3217+
3218+
from agent_framework import Embedding
3219+
3220+
embedding = Embedding(
3221+
vector=[0.1, 0.2, 0.3],
3222+
model_id="text-embedding-3-small",
3223+
)
3224+
assert embedding.dimensions == 3
3225+
"""
3226+
3227+
def __init__(
3228+
self,
3229+
vector: EmbeddingT,
3230+
*,
3231+
model_id: str | None = None,
3232+
dimensions: int | None = None,
3233+
created_at: datetime | None = None,
3234+
additional_properties: dict[str, Any] | None = None,
3235+
) -> None:
3236+
self.vector = vector
3237+
self._dimensions = dimensions
3238+
self.model_id = model_id
3239+
self.created_at = created_at
3240+
self.additional_properties = additional_properties or {}
3241+
3242+
@property
3243+
def dimensions(self) -> int | None:
3244+
"""Return the number of dimensions in the embedding vector.
3245+
3246+
Uses the explicitly provided value if set, otherwise computes from vector length.
3247+
"""
3248+
if self._dimensions is not None:
3249+
return self._dimensions
3250+
if isinstance(self.vector, (list, tuple, bytes)):
3251+
return len(self.vector)
3252+
return None
3253+
3254+
3255+
EmbeddingOptionsT = TypeVar(
3256+
"EmbeddingOptionsT",
3257+
bound=TypedDict, # type: ignore[valid-type]
3258+
default="EmbeddingGenerationOptions",
3259+
)
3260+
3261+
3262+
class GeneratedEmbeddings(list[Embedding[EmbeddingT]], Generic[EmbeddingT, EmbeddingOptionsT]):
3263+
"""A list of generated embeddings with usage metadata.
3264+
3265+
Extends list for direct iteration and indexing.
3266+
Generic over both the embedding vector type and the options type used for generation.
3267+
3268+
Args:
3269+
embeddings: Sequence of Embedding objects.
3270+
options: The options used to generate these embeddings.
3271+
usage: Token usage information (e.g. prompt_tokens, total_tokens).
3272+
additional_properties: Additional metadata.
3273+
3274+
Examples:
3275+
.. code-block:: python
3276+
3277+
from agent_framework import Embedding, GeneratedEmbeddings
3278+
3279+
embeddings = GeneratedEmbeddings(
3280+
[Embedding(vector=[0.1, 0.2]), Embedding(vector=[0.3, 0.4])],
3281+
usage={"prompt_tokens": 10, "total_tokens": 10},
3282+
)
3283+
assert len(embeddings) == 2
3284+
assert embeddings.usage["prompt_tokens"] == 10
3285+
"""
3286+
3287+
def __init__(
3288+
self,
3289+
embeddings: Iterable[Embedding[EmbeddingT]] | None = None,
3290+
*,
3291+
options: EmbeddingOptionsT | None = None,
3292+
usage: dict[str, Any] | None = None,
3293+
additional_properties: dict[str, Any] | None = None,
3294+
) -> None:
3295+
super().__init__(embeddings or [])
3296+
self.options = options
3297+
self.usage = usage
3298+
self.additional_properties = additional_properties or {}
3299+
3300+
3301+
# endregion

0 commit comments

Comments
 (0)