The test suite is slow. It takes a long time to run them and to get results. Because of this, occasionally the CI builds timeout due to how long it takes for the builds to complete.

There are various issues that affect the speed of the test suite. This issue is an overall tracking issue for them. I'll take a shot at each of those, given that I have the time. The intent is to try to speed up the test suite without compromising the degree of confidence on it.

• Investigate CI improvements
  • Containers for Travis (Travis: Improve build speeds and upgrade PyPy version #4590)
  • Breakup tests across 2 workers (CI: Split PyPy tests to speed up end-to-end running time #5436)
• Look into speeding up the virtualenv fixture (Speedup testsuite - venv creation #4706)
• Require all non-pip invoking tests to be unit tests
• Categorize all non-script tests as unit tests
• Increasing unit test coverage
• Remove redundant tests -- Most of the tests in tests/functional/test_install_cleanup.py aren't testing anything #2640
• Simplify or Break up slow tests -- tests take too long to run #1721
• Speed up pip's startup time (Speedup startup time #4768)
• In-Memory tests where appropriate
• Removing exhaustive file checks in scripttest (WIP: Remove exhaustive file checks #7141)

(I'll add more as they come)