Refs: (overarching issue) #19098

• macOS @DonJayamanne
• linux @karthiknadig
• windows @Yoyokrazy
• anyOS @lszomoru
  Complexity: 5

Author: @eleanorjboyd

Create Issue

Thank you for doing this testing! I apologize for how long it is and please let me know if you have any questions!

Prerequisites:

1. Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information:
This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!). Secondly, debugging is not supported yet so there is no need to test debugging on the rewrite which will also be reflected in the TPI steps.

Steps:

Part 0: Setup

1. launch vscode with the environment variable ENABLE_PYTHON_TESTING_REWRITE = '1'
2. Now open the inc_dec_example_repo/inc_dec_example_test_suite testing repo folder in vscode
3. use the create environment command from the command palette, select venv and whatever python version you want >=3.7
4. in the terminal run python -m pip install pytest
5. from the command palette run python: configure tests select unittest and "." (root directory) and test_*.py

Part 1: General

1. Go to the testing explorer, and confirm all the tests are discovered and nested correctly (please see attached image for reference on how it should look or compare it to the stable version of the extension since it will look the same)
2. There are two types of test cases in unittest, regular test cases and subtests test cases (examples of both are in the test repo). Please try the following for both
   • Add a test case, reload
   • Remove a test case, reload
   • Run via the gutter
   • Run via the test explorer (individually as as part of a folder)
   • Run select, non-adjacent tests (cmd+click to select individual tests)
3. Now go to the output tab and select python: output logs, run tests and confirm that the output appears with all the necessary information (how many tests were run, which succeeded, explanation of failure if that occurred)

Part 2: Error Handling

1. Make one of your test files have a syntax error then run test discovery
2. Discover tests, then make a syntax error, then try and run the test (check error message appears)

Part 3: Adding Unittest Args

1. Add -v to your pytest args in your settings.json and run the test folder, check that this arg worked (it works if it stops after a single test failure)

Part 4: Additional Commands

1. Try out different commands from the command palette related to testing to see if they work
2. Try out the different items on the top righthand side of the testing panel such as collapse test cases and clear all results

Part 5: Different Testing Pattern

1. Rerun the command python: configure tests select unittest and "." (root directory) now instead select *test.py
2. reload the window and go to test discovery
3. confirm that the only test files that are found are the ones with the matching pattern, here is a screenshot of how it should look

test explorer reference image (from Part 1)