Skip to content

Python: Add Vector Store to PostgreSQL connector - #8951

Merged
Evan Mattson (moonbox3) merged 42 commits into
microsoft:mainfrom
lossyrob:feature/postgres-memory
Oct 7, 2024
Merged

Evan Mattson (moonbox3) merged 42 commits into
microsoft:mainfrom
lossyrob:feature/postgres-memory

Conversation

@lossyrob

@lossyrob Rob Emanuele (lossyrob) commented Sep 23, 2024 •

Copy link
Copy Markdown
Contributor

Motivation and Context

This PR updates the postgres memory connector to the new memory architecture.

Description

It adds a PostgresStore and PostgresCollection to semantickernel.connectors.memory.postgres, along with unit tests, integration tests, inclusion in the new_memory.py sample, and a getting started notebook.

Contribution Checklist

@markwallace-microsoft Mark Wallace (markwallace-microsoft) added python Pull requests for the Python Semantic Kernel memory labels Sep 23, 2024
@markwallace-microsoft

Mark Wallace (markwallace-microsoft) commented Sep 23, 2024 •

Copy link
Copy Markdown
Contributor

Python Test Coverage

Python Test Coverage Report
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py1994776%148, 159, 163, 313–316, 423, 437–480
semantic_kernel/agents/group_chat
   agent_chat.py124298%78, 171
   agent_group_chat.py100298%151, 201
   broadcast_queue.py72199%35
semantic_kernel/agents/open_ai
   assistant_content_generation.py133993%96–97, 281, 291–294, 335, 337
   open_ai_assistant_base.py449898%259, 337–338, 746, 867, 870, 932, 990
semantic_kernel/connectors/ai
   chat_completion_client_base.py116298%382, 392
   completion_usage.py8188%17
semantic_kernel/connectors/ai/anthropic/services
   anthropic_chat_completion.py176597%147, 165, 169, 223, 419
semantic_kernel/connectors/ai/azure_ai_inference/services
   azure_ai_inference_chat_completion.py119794%120, 146–149, 159, 180, 202
   azure_ai_inference_text_embedding.py41198%87
semantic_kernel/connectors/ai/embeddings
   embedding_generator_base.py8188%50
semantic_kernel/connectors/ai/google
   shared_utils.py26196%56
semantic_kernel/connectors/ai/google/google_ai/services
   google_ai_chat_completion.py119497%127, 153, 176, 178
   google_ai_text_completion.py63297%98, 121
   utils.py65395%140, 160–165
semantic_kernel/connectors/ai/google/vertex_ai/services
   utils.py66395%141, 161–166
   vertex_ai_chat_completion.py119497%121, 147, 170, 172
   vertex_ai_text_completion.py62297%95, 116
semantic_kernel/connectors/ai/hugging_face/services
   hf_text_completion.py60395%103, 112, 127
   hf_text_embedding.py32584%79–83
semantic_kernel/connectors/ai/mistral_ai/services
   mistral_ai_chat_completion.py118794%118–121, 307–310
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py60592%95–98, 108, 143
   ollama_text_completion.py55591%87–90, 100, 128
semantic_kernel/connectors/ai/open_ai/prompt_execution_settings
   open_ai_prompt_execution_settings.py94199%112
semantic_kernel/connectors/ai/open_ai/services
   azure_chat_completion.py107595%118, 123, 157, 166, 169
   azure_text_completion.py28293%82, 87
   azure_text_embedding.py30293%84, 89
   open_ai_chat_completion_base.py127596%71, 121, 141, 177, 287
   open_ai_handler.py63395%86, 95–96
   open_ai_text_completion_base.py80298%56, 161
semantic_kernel/connectors/ai/open_ai/settings
   azure_open_ai_settings.py22195%99
semantic_kernel/connectors/memory/azure_ai_search
   azure_ai_search_collection.py87298%150, 152
semantic_kernel/connectors/memory/redis
   redis_collection.py160299%146, 316
   utils.py451176%145–146, 164, 166, 173–188
semantic_kernel/connectors/openapi_plugin
   openapi_manager.py58297%110–111
   openapi_parser.py88298%71, 128
   openapi_runner.py105298%181–182
semantic_kernel/connectors/openapi_plugin/models
   rest_api_operation.py129199%242
semantic_kernel/contents
   function_call_content.py97199%201
   streaming_chat_message_content.py68199%210
   streaming_content_mixin.py39295%37, 64
semantic_kernel/core_plugins/sessions_python_tool
   sessions_python_plugin.py134894%69, 82–91, 99
   sessions_python_settings.py39490%84–87
semantic_kernel/data
   vector_store_record_collection.py2491992%410, 470–474, 482–486, 526–529, 536–539
   vector_store_record_utils.py26292%50, 52
semantic_kernel/functions
   kernel_function_decorator.py98199%102
   kernel_function_from_method.py96199%153
   kernel_function_from_prompt.py154795%165–166, 180, 201, 219, 239, 322
   kernel_function_log_messages.py36683%37–43
   kernel_plugin.py187299%472, 475
semantic_kernel/planners
   plan.py2344581%54, 163–165, 197, 214–227, 264, 269, 277–278, 288–291, 308, 313, 329, 332–337, 355, 360, 363, 365, 372, 386–388, 393–397
semantic_kernel/planners/function_calling_stepwise_planner
   function_calling_stepwise_planner.py116497%145, 189–190, 198
semantic_kernel/planners/sequential_planner
   sequential_planner.py64691%71, 75, 109, 125, 134–135
   sequential_planner_extensions.py50982%31–32, 56, 110–124
   sequential_planner_parser.py771284%66–74, 93, 117–120
semantic_kernel/schema
   kernel_json_schema_builder.py129993%53, 90, 186, 194, 205, 213, 228, 232–233
semantic_kernel/services
   ai_service_client_base.py22195%64
semantic_kernel/template_engine/blocks
   code_block.py77199%119
   named_arg_block.py43198%98
semantic_kernel/utils/authentication
   entra_id_authentication.py15287%26, 38
semantic_kernel/utils/telemetry
   user_agent.py16288%18–19
semantic_kernel/utils/telemetry/model_diagnostics
   decorators.py171498%372–375
TOTAL1143032397% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
2528 4 💤 0 ❌ 0 🔥 1m 13s ⏱️

@lossyrob
Rob Emanuele (lossyrob) force-pushed the feature/postgres-memory branch 4 times, most recently from 34fa01d to 3408fe0 Compare September 25, 2024 13:52
github-merge-queue Bot pushed a commit that referenced this pull request Sep 25, 2024
…ent in environment variable. (#8985)

### Motivation and Context

While working on #8951 I found that interacting with an Azure OpenAI
instance I had an API key for, but my identity didn't have IAM access to
the API for, that I was getting a permission error:

```
semantic_kernel.exceptions.service_exceptions.ServiceResponseException: ("<class 'semantic_kernel.connectors.ai.open_ai.services.azure_text_embedding.AzureTextEmbedding'> service failed to generate embeddings", AuthenticationError("Error code: 401 - {'error': {'code': 'PermissionDenied', 'message': 'Principal does not have access to API/Operation.'}}"))
```

I found that this was because, though my API key was set in the
environment, the codepath only recognized an explicitly passed in API
key when deciding whether to issue an AD token while initializing an
AzureTextEmbedding instance.

This PR fixes that by looking at the setting object explicitly, which
may pull the API key from the environment.


### Contribution Checklist

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
@lossyrob
Rob Emanuele (lossyrob) marked this pull request as ready for review September 27, 2024 16:23
@lossyrob
Rob Emanuele (lossyrob) requested a review from a team as a code owner September 27, 2024 16:23
@lossyrob
Rob Emanuele (lossyrob) force-pushed the feature/postgres-memory branch 6 times, most recently from fad399d to c88bc86 Compare September 28, 2024 02:22
Update the settings to take individual connection config intead of only a connection string.
Also enable setting min and max pool size through settings.
Extract constants from memory store in preperation for vector store implementation.
Changes made to make mock postgres cursor accept and return same values as psycopg
This change moves to utilize psycopg's async connection pool, connection and cursor.
Changes were made dealing with how the connection pool is created.
A VectorStore itself is not a context manager, so managing closing the connection pool
was not possible without async tasks being cancelled incorrectly, even with the __del__ methods
that other VectorStores use.

Now, if a user uses a PostgresStore, they must pass in a connection pool they manage.
If they create a PostgresCollection without a connection pool, they have to use the
collection as a context manager which will close the connection pool on __aexit__.
- Make creation of stores Callable, so if you're not using a particular store it won't throw setting errors for settings that aren't valid
- Allow user to pass in store name from command line, and choose to use Azure OpenAI instead of OpenAI for embeddings.
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Oct 4, 2024
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 4, 2024
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Oct 4, 2024
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 4, 2024
See https://pytest-asyncio.readthedocs.io/en/v0.24.0/reference/markers/
Without this scope, tests cannot connect to pool.
Note that in pytest-asyncio 0.24.0, the
scope parameter is renamed to "loop_scope", and will
need to be updated.
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Oct 7, 2024
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 7, 2024
This adds an __enter__ and __exit__ function to the PostgresMemoryStore
so that it can be used as a context manager.
Also modify the integration tests to utilize the context manager,
so that connections are freed up after the tests are run.
Without this change, the tests will hold onto the connections
and cause the database to run out of connections.
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Oct 7, 2024
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 7, 2024
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Oct 7, 2024
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 7, 2024
With the current version of pytest-asyncio we're using,
there's an issue with using async fixtures cached at
different scopes when they need the same event loop
scope.

See: pytest-dev/pytest-asyncio#871

An API breaking change that fixes this is available in 0.24,
but fixing this with a context manager here to avoid increasing
the blast radius.
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Oct 7, 2024
Merged via the queue into microsoft:main with commit 4cafb95 Oct 7, 2024
@TaoChenOSU Tao Chen (TaoChenOSU) linked an issue Oct 14, 2024 that may be closed by this pull request
github-merge-queue Bot pushed a commit that referenced this pull request Jan 30, 2025
### Motivation and Context

Following up on #8951, this PR adds an implementation of
`VectorSearchBase` to `PostgresCollection`. This implementation provides
vectorized search and does not implement text search or vectorizable
text search.

Unit and integration tests are added, and the
`python/samples/getting_started/third_party/postgres-memory.ipynb`
notebook was expanded to include vector search in the example.

### Contribution Checklist

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
jcruzmot-te pushed a commit to thousandeyes/aia-semantic-kernel that referenced this pull request Sep 15, 2025
### Motivation and Context

Following up on microsoft#8951, this PR adds an implementation of
`VectorSearchBase` to `PostgresCollection`. This implementation provides
vectorized search and does not implement text search or vectorizable
text search.

Unit and integration tests are added, and the
`python/samples/getting_started/third_party/postgres-memory.ipynb`
notebook was expanded to include vector search in the example.

### Contribution Checklist

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

memory python Pull requests for the Python Semantic Kernel

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Python: Update Postgres Connector to new Design

4 participants