Repository navigation
Python: Add Vector Store to PostgreSQL connector - #8951
Merged
Evan Mattson (moonbox3) merged 42 commits intoOct 7, 2024
Merged
Conversation
Contributor
Python Test Coverage ReportPython Unit Test Overview
|
Rob Emanuele (lossyrob)
force-pushed
the
feature/postgres-memory
branch
4 times, most recently
from
September 25, 2024 13:52
34fa01d to
3408fe0
Compare
4 tasks done
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Sep 25, 2024
…ent in environment variable. (#8985) ### Motivation and Context While working on #8951 I found that interacting with an Azure OpenAI instance I had an API key for, but my identity didn't have IAM access to the API for, that I was getting a permission error: ``` semantic_kernel.exceptions.service_exceptions.ServiceResponseException: ("<class 'semantic_kernel.connectors.ai.open_ai.services.azure_text_embedding.AzureTextEmbedding'> service failed to generate embeddings", AuthenticationError("Error code: 401 - {'error': {'code': 'PermissionDenied', 'message': 'Principal does not have access to API/Operation.'}}")) ``` I found that this was because, though my API key was set in the environment, the codepath only recognized an explicitly passed in API key when deciding whether to issue an AD token while initializing an AzureTextEmbedding instance. This PR fixes that by looking at the setting object explicitly, which may pull the API key from the environment. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
Rob Emanuele (lossyrob)
force-pushed
the
feature/postgres-memory
branch
from
September 27, 2024 16:22
3408fe0 to
c26dcd4
Compare
Rob Emanuele (lossyrob)
marked this pull request as ready for review
September 27, 2024 16:23
Rob Emanuele (lossyrob)
force-pushed
the
feature/postgres-memory
branch
6 times, most recently
from
September 28, 2024 02:22
fad399d to
c88bc86
Compare
Update the settings to take individual connection config intead of only a connection string. Also enable setting min and max pool size through settings. Extract constants from memory store in preperation for vector store implementation.
Changes made to make mock postgres cursor accept and return same values as psycopg
This change moves to utilize psycopg's async connection pool, connection and cursor. Changes were made dealing with how the connection pool is created. A VectorStore itself is not a context manager, so managing closing the connection pool was not possible without async tasks being cancelled incorrectly, even with the __del__ methods that other VectorStores use. Now, if a user uses a PostgresStore, they must pass in a connection pool they manage. If they create a PostgresCollection without a connection pool, they have to use the collection as a context manager which will close the connection pool on __aexit__.
- Make creation of stores Callable, so if you're not using a particular store it won't throw setting errors for settings that aren't valid - Allow user to pass in store name from command line, and choose to use Azure OpenAI instead of OpenAI for embeddings.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 4, 2024
Evan Mattson (moonbox3)
approved these changes
Oct 4, 2024
Evan Mattson (moonbox3)
enabled auto-merge
October 4, 2024 14:39
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 4, 2024
See https://pytest-asyncio.readthedocs.io/en/v0.24.0/reference/markers/ Without this scope, tests cannot connect to pool. Note that in pytest-asyncio 0.24.0, the scope parameter is renamed to "loop_scope", and will need to be updated.
Evan Mattson (moonbox3)
approved these changes
Oct 7, 2024
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 7, 2024
This adds an __enter__ and __exit__ function to the PostgresMemoryStore so that it can be used as a context manager. Also modify the integration tests to utilize the context manager, so that connections are freed up after the tests are run. Without this change, the tests will hold onto the connections and cause the database to run out of connections.
Evan Mattson (moonbox3)
approved these changes
Oct 7, 2024
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 7, 2024
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 7, 2024
With the current version of pytest-asyncio we're using, there's an issue with using async fixtures cached at different scopes when they need the same event loop scope. See: pytest-dev/pytest-asyncio#871 An API breaking change that fixes this is available in 0.24, but fixing this with a context manager here to avoid increasing the blast radius.
Evan Mattson (moonbox3)
approved these changes
Oct 7, 2024
Evan Mattson (moonbox3)
enabled auto-merge
October 7, 2024 21:01
4 tasks done
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Jan 30, 2025
### Motivation and Context Following up on #8951, this PR adds an implementation of `VectorSearchBase` to `PostgresCollection`. This implementation provides vectorized search and does not implement text search or vectorizable text search. Unit and integration tests are added, and the `python/samples/getting_started/third_party/postgres-memory.ipynb` notebook was expanded to include vector search in the example. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
jcruzmot-te
pushed a commit
to thousandeyes/aia-semantic-kernel
that referenced
this pull request
Sep 15, 2025
### Motivation and Context Following up on microsoft#8951, this PR adds an implementation of `VectorSearchBase` to `PostgresCollection`. This implementation provides vectorized search and does not implement text search or vectorizable text search. Unit and integration tests are added, and the `python/samples/getting_started/third_party/postgres-memory.ipynb` notebook was expanded to include vector search in the example. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
This PR updates the postgres memory connector to the new memory architecture.
Description
It adds a
PostgresStoreandPostgresCollectiontosemantickernel.connectors.memory.postgres, along with unit tests, integration tests, inclusion in thenew_memory.pysample, and a getting started notebook.Contribution Checklist