Repository navigation
Commit 6305e3e
Python: feat(python): Add embedding abstractions and OpenAI implementation (Phase 1) (#4153)
* feat(python): Add embedding abstractions and OpenAI implementation (Phase 1)
This PR contains two parts:
1. **Overall migration plan** for porting vector stores and embeddings from
Semantic Kernel to Agent Framework (docs/features/vector-stores-and-embeddings/README.md)
covering all 10 phases from core abstractions through connectors and TextSearch.
2. **Phase 1 implementation** — core embedding abstractions and OpenAI/Azure OpenAI
embedding clients:
Core types (_types.py):
- EmbeddingGenerationOptions TypedDict (total=False)
- Embedding[EmbeddingT] generic class with model_id, dimensions, created_at
- GeneratedEmbeddings[EmbeddingT, EmbeddingOptionsT] list container with options, usage
- EmbeddingInputT (default str) and EmbeddingT (default list[float]) TypeVars
Protocol + base class (_clients.py):
- SupportsGetEmbeddings protocol — Generic[EmbeddingInputT, EmbeddingT, OptionsContraT]
- BaseEmbeddingClient ABC — Generic[EmbeddingInputT, EmbeddingT, OptionsCoT]
Telemetry (observability.py):
- EmbeddingTelemetryLayer with gen_ai.operation.name = "embeddings"
OpenAI implementation (openai/_embedding_client.py):
- RawOpenAIEmbeddingClient, OpenAIEmbeddingClient, OpenAIEmbeddingOptions
- Uses _ensure_client() factory pattern
Azure OpenAI implementation (azure/_embedding_client.py):
- AzureOpenAIEmbeddingClient following AzureOpenAIChatClient pattern
- Supports API key, Entra ID credentials, env var configuration
Tests:
- 47 unit tests for types, protocol, base class, OpenAI, and Azure clients
- 6 integration tests (gated behind RUN_INTEGRATION_TESTS + credentials)
Samples:
- samples/02-agents/embeddings/openai_embeddings.py
- samples/02-agents/embeddings/azure_openai_embeddings.py
Co-authored-by: Copilot <[email protected]>
* fix: Add AzureOpenAIEmbeddingClient to azure __init__.pyi stub
Co-authored-by: Copilot <[email protected]>
* ci: Add embedding env vars to Python integration tests
Map OPENAI_EMBEDDING_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME
from GitHub vars to the integration test environment.
Co-authored-by: Copilot <[email protected]>
* fix: Handle base64 encoding_format in OpenAI embedding client
When encoding_format='base64' is used, the OpenAI API returns base64-encoded
floats instead of a JSON array. Decode these automatically to list[float]
so the return type stays consistent regardless of encoding format.
Also adds a unit test for base64 decoding and fixes minor docstring/import issues.
Co-authored-by: Copilot <[email protected]>
* fix: Only record INPUT_TOKENS for embedding telemetry
Embeddings have no output/completion tokens. Remove OUTPUT_TOKENS recording
which was double-counting prompt_tokens via the total_tokens fallback.
Co-authored-by: Copilot <[email protected]>
* fix: Resolve mypy variance error and lint warning
Use contravariant/covariant TypeVars for SupportsGetEmbeddings Protocol.
Combine nested if into single statement in telemetry layer.
Co-authored-by: Copilot <[email protected]>
* fix: Make EmbeddingCoT invariant for mypy compatibility
GeneratedEmbeddings is invariant in its type param, so the Protocol
TypeVar cannot be covariant.
Co-authored-by: Copilot <[email protected]>
* fix: Address PR review - empty values guard, service_url for telemetry
- Add early return for empty values in get_embeddings to avoid unnecessary API calls
- Add service_url() method to RawOpenAIEmbeddingClient for proper telemetry endpoint reporting
- Add test for empty values behavior
Co-authored-by: Copilot <[email protected]>
* Python: Fix OpenAI chat client compatibility with third-party endpoints and OTel 0.4.14 (#4161)
* Fix system message content sent as list instead of string
Some OpenAI-compatible endpoints (e.g. NVIDIA NIM) reject system messages
when content is a list of content parts. This change flattens system and
developer message content to a plain string in the Chat Completions client.
Fixes #1407
Co-authored-by: Copilot <[email protected]>
* Fix compatibility with opentelemetry-semantic-conventions-ai 0.4.14
Version 0.4.14 removed several LLM_* attributes from SpanAttributes
(LLM_SYSTEM, LLM_REQUEST_MODEL, LLM_RESPONSE_MODEL, LLM_REQUEST_MAX_TOKENS,
LLM_REQUEST_TEMPERATURE, LLM_REQUEST_TOP_P, LLM_TOKEN_TYPE).
Move these to the OtelAttr enum with their well-known gen_ai.* string values
and update all references in observability.py and tests.
Fixes #4160
Co-authored-by: Copilot <[email protected]>
* Flatten text-only message content to string for all roles
Extend the system/developer fix to all message roles. Text-only content
lists are now post-processed into plain strings, while multimodal content
(text + images/audio) remains as a list. This fixes compatibility with
OpenAI-like endpoints that cannot deserialize list content (e.g. Foundry
Local's Neutron backend).
Partially fixes #4084
Co-authored-by: Copilot <[email protected]>
* Fix streaming text lost when usage data in same chunk
Some providers (e.g. Gemini) include both usage data and text content
in the same streaming chunk. The early return on chunk.usage caused
text and tool call parsing to be skipped entirely. Remove the early
return and process usage alongside text/tool calls.
Fixes #3434
Co-authored-by: Copilot <[email protected]>
* Fix mypy errors in _chat_client.py
Rename shadowed variable 'args' in system/developer branch to 'sys_args'
and rename loop variable 'content' to 'msg_content' to avoid type conflict.
Co-authored-by: Copilot <[email protected]>
---------
Co-authored-by: Copilot <[email protected]>
* reorder imports
* fix: Use OtelAttr.REQUEST_MODEL instead of removed SpanAttributes.LLM_REQUEST_MODEL
Co-authored-by: Copilot <[email protected]>
* docs: Add score_threshold to vector store plan
Reference SK .NET PR #13501 for score threshold filtering semantics.
Include score_threshold in SearchOptions from Phase 3.
Co-authored-by: Copilot <[email protected]>
* docs: Add reference to roji's SK .NET MEVD work for SQL connectors
Co-authored-by: Copilot <[email protected]>
* fix: Clear env vars in construction tests to avoid CI leakage
Tests for missing API key / model ID now use monkeypatch.delenv to ensure
env vars from the integration test environment don't prevent the expected
ValueError from being raised.
Co-authored-by: Copilot <[email protected]>
---------
Co-authored-by: Copilot <[email protected]>1 parent 7b24d91 commit 6305e3e
18 files changed
Lines changed: 1890 additions & 3 deletions
File tree
- .github/workflows
- docs/features/vector-stores-and-embeddings
- python
- packages/core
- agent_framework
- azure
- openai
- tests
- core
- openai
- samples/02-agents/embeddings
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
65 | 66 | | |
66 | 67 | | |
| 68 | + | |
67 | 69 | | |
68 | 70 | | |
69 | 71 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
82 | 84 | | |
83 | 85 | | |
84 | 86 | | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
85 | 91 | | |
86 | 92 | | |
87 | 93 | | |
| 94 | + | |
88 | 95 | | |
89 | 96 | | |
90 | 97 | | |
| |||
201 | 208 | | |
202 | 209 | | |
203 | 210 | | |
| 211 | + | |
204 | 212 | | |
205 | 213 | | |
206 | 214 | | |
| |||
218 | 226 | | |
219 | 227 | | |
220 | 228 | | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
221 | 233 | | |
222 | 234 | | |
223 | 235 | | |
| |||
232 | 244 | | |
233 | 245 | | |
234 | 246 | | |
| 247 | + | |
235 | 248 | | |
236 | 249 | | |
237 | 250 | | |
| |||
261 | 274 | | |
262 | 275 | | |
263 | 276 | | |
| 277 | + | |
264 | 278 | | |
265 | 279 | | |
266 | 280 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
| |||
56 | 60 | | |
57 | 61 | | |
58 | 62 | | |
59 | | - | |
60 | 63 | | |
61 | 64 | | |
62 | 65 | | |
| |||
660 | 663 | | |
661 | 664 | | |
662 | 665 | | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
12 | 21 | | |
| 22 | + | |
13 | 23 | | |
14 | 24 | | |
15 | 25 | | |
| |||
272 | 282 | | |
273 | 283 | | |
274 | 284 | | |
275 | | - | |
| 285 | + | |
| 286 | + | |
276 | 287 | | |
277 | 288 | | |
278 | 289 | | |
| |||
3162 | 3173 | | |
3163 | 3174 | | |
3164 | 3175 | | |
| 3176 | + | |
| 3177 | + | |
| 3178 | + | |
| 3179 | + | |
| 3180 | + | |
| 3181 | + | |
| 3182 | + | |
| 3183 | + | |
| 3184 | + | |
| 3185 | + | |
| 3186 | + | |
| 3187 | + | |
| 3188 | + | |
| 3189 | + | |
| 3190 | + | |
| 3191 | + | |
| 3192 | + | |
| 3193 | + | |
| 3194 | + | |
| 3195 | + | |
| 3196 | + | |
| 3197 | + | |
| 3198 | + | |
| 3199 | + | |
| 3200 | + | |
| 3201 | + | |
| 3202 | + | |
| 3203 | + | |
| 3204 | + | |
| 3205 | + | |
| 3206 | + | |
| 3207 | + | |
| 3208 | + | |
| 3209 | + | |
| 3210 | + | |
| 3211 | + | |
| 3212 | + | |
| 3213 | + | |
| 3214 | + | |
| 3215 | + | |
| 3216 | + | |
| 3217 | + | |
| 3218 | + | |
| 3219 | + | |
| 3220 | + | |
| 3221 | + | |
| 3222 | + | |
| 3223 | + | |
| 3224 | + | |
| 3225 | + | |
| 3226 | + | |
| 3227 | + | |
| 3228 | + | |
| 3229 | + | |
| 3230 | + | |
| 3231 | + | |
| 3232 | + | |
| 3233 | + | |
| 3234 | + | |
| 3235 | + | |
| 3236 | + | |
| 3237 | + | |
| 3238 | + | |
| 3239 | + | |
| 3240 | + | |
| 3241 | + | |
| 3242 | + | |
| 3243 | + | |
| 3244 | + | |
| 3245 | + | |
| 3246 | + | |
| 3247 | + | |
| 3248 | + | |
| 3249 | + | |
| 3250 | + | |
| 3251 | + | |
| 3252 | + | |
| 3253 | + | |
| 3254 | + | |
| 3255 | + | |
| 3256 | + | |
| 3257 | + | |
| 3258 | + | |
| 3259 | + | |
| 3260 | + | |
| 3261 | + | |
| 3262 | + | |
| 3263 | + | |
| 3264 | + | |
| 3265 | + | |
| 3266 | + | |
| 3267 | + | |
| 3268 | + | |
| 3269 | + | |
| 3270 | + | |
| 3271 | + | |
| 3272 | + | |
| 3273 | + | |
| 3274 | + | |
| 3275 | + | |
| 3276 | + | |
| 3277 | + | |
| 3278 | + | |
| 3279 | + | |
| 3280 | + | |
| 3281 | + | |
| 3282 | + | |
| 3283 | + | |
| 3284 | + | |
| 3285 | + | |
| 3286 | + | |
| 3287 | + | |
| 3288 | + | |
| 3289 | + | |
| 3290 | + | |
| 3291 | + | |
| 3292 | + | |
| 3293 | + | |
| 3294 | + | |
| 3295 | + | |
| 3296 | + | |
| 3297 | + | |
| 3298 | + | |
| 3299 | + | |
| 3300 | + | |
| 3301 | + | |
0 commit comments