Repository navigation
Feature python integration tests - #1134
Merged
Merged
Conversation
### Motivation and Context This PR slims down the number of integration tests running against AOAI and OAI models - reducing time and resources spent running them. Additional justification for slimming down these tests is to reduce the chance of throttling from these endpoints (leads to unstable integration tests). The scenarios previously being covered are already handled using HF models which are much less expensive to test against and do not run the risk of throttling. ### Description - added a top-level test `conftest.py` to handle kernel creation and OpenAI model secret handling. I had originally intended `create_kernel` to do more than just create a kernel, but additional setup wasn't needed. I've left the pytest fixture though since other fixtures can call it, and using it in tests can making importing Kernel from semantic_kernel unncessary. - added a completions `conftest.py` to set up completions tests. For example, setup_hf_text_completion_function allows for testing text2text_generation and text_generation models using the same test file. - Common test code is now in pytest fixtures instead of common methods. - For a number of the completion tests, I have broken out the asserts to individual tests instead of running one giant test. This makes it easier to identify regressions in specific patterns around invoking skills. - Added retry logic to conversationSummarySkill - renamed tests to be more descriptive.
Devis Lucato (dluc)
approved these changes
May 21, 2023
Devis Lucato (dluc)
enabled auto-merge (squash)
May 21, 2023 06:02
Alex Chao (alexchaomander)
self-requested a review
May 21, 2023 16:11
Shawn Callegari (shawncal)
pushed a commit
to shawncal/semantic-kernel
that referenced
this pull request
Jul 6, 2023
### Motivation and Context Python integration tests were failing consistently for 2 reasons. First, all ubuntu system tests were failing due to missing shared object files related to cuda. Second, macOS + python 3.11 tests were failing due to hsnwlib unable to install on native hardware. See microsoft#1066 for description of changes to integration test code to condense test code, make better use of pytest fixtures, and reduce the number of calls to external APIs (OpenAI, AOAI) ### Description - Integration tests were failing on Linux due to a regression in python torch=2.0.1. Downgrading the version to 2.0.0 resolved these failures. - For MacOS + 3.11 specifically, setting the environment variable `HNSWLIB_NO_NATIVE=1` ensures that the hnswlib wheel can be built and installed properly on M1 hardware. - Python integration tests reliably and consistently passing: https://github.com/microsoft/semantic-kernel/actions/runs/5035748596
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Python integration tests were failing consistently for 2 reasons. First, all ubuntu system tests were failing due to missing shared object files related to cuda. Second, macOS + python 3.11 tests were failing due to hsnwlib unable to install on native hardware.
See #1066 for description of changes to integration test code to condense test code, make better use of pytest fixtures, and reduce the number of calls to external APIs (OpenAI, AOAI)
Description
HNSWLIB_NO_NATIVE=1ensures that the hnswlib wheel can be built and installed properly on M1 hardware.Contribution Checklist
dotnet format