{"id":24945,"date":"2019-12-10T12:15:06","date_gmt":"2019-12-10T10:15:06","guid":{"rendered":"https:\/\/www.webcodegeeks.com\/?p=24945"},"modified":"2020-06-23T15:00:36","modified_gmt":"2020-06-23T12:00:36","slug":"how-to-performance-test-python-code-timeit-cprofile-and-more","status":"publish","type":"post","link":"https:\/\/www.webcodegeeks.com\/python\/how-to-performance-test-python-code-timeit-cprofile-and-more\/","title":{"rendered":"How to Performance Test Python Code: timeit, cProfile, and More"},"content":{"rendered":"\n

A lot of the articles in this series take advantage of a feature of Python which allows us to performance test our code, and I finally wanted to get around to explaining how it works and how to use it.<\/p>\n\n\n\n

In this article, I cover three main techniques: brute force, timeit<\/code>, and cProfile<\/code>. Personally, I performance test my code with timeit<\/code> because I find it easy to understand, but you may find the various profiling tools helpful. At the end, I\u2019ll ask you to demonstrate your newfound skills with a challenge.<\/strong><\/p>\n\n\n\n

Problem Introduction<\/h2>\n\n\n\n

In order to start talking about how to performance test Python code, we need to define it. At a high level, a performance test is anything that verifies the speed, reliability, scalability, and\/or stability of software. For our purposes, we\u2019ll be looking at speed. In particular, we\u2019ll be looking at different ways to compare the speed of two programs using their relative execution times.<\/p>\n\n\n\n

When it comes to performance testing software, the process isn\u2019t always easy or obvious. In particular, there are a lot of pitfalls. For example, performance tests can be influenced by all sorts of factors like background processes (i.e. Spotify, Eclipse, GitHub Desktop, etc.).<\/p>\n\n\n\n

In addition, performance tests aren\u2019t always written in a way that fairly accounts for differences in implementation. For example, I might have two snippets of code that have the same behavior except one requires a library to be imported. When I run my test, I don\u2019t want the import to affect the test outcome. As a result, I should write my tests such that I don\u2019t start timing until after the library is imported.<\/p>\n\n\n\n

On top of all that, it\u2019s important to take into account different types of scenarios when performance testing. For instance, if we have two similar snippets, one might have better performance for larger data sets. It\u2019s important to test a range of data sets for that reason.<\/p>\n\n\n\n

At any rate, the goal of this article is to look at a few different ways we can performance test code in Python. Let\u2019s dig in!<\/p>\n\n\n\n

Solutions<\/h2>\n\n\n\n

As always, I like to share a few ways to accomplish our task. Of course, if you\u2019ve been following along in this series, you know that I prefer to use the timeit<\/code> library to test snippets. Luckily, there are more options if timeit<\/code> isn\u2019t for you.<\/p>\n\n\n\n

Performance Testing by Brute Force<\/h3>\n\n\n\n

If you\u2019ve never done any performance testing before, you probably have a gist of how to get started. Typically, we want to take a timestamp before and after we run our code snippet. Then, we can calculate the difference between those times and use the result in our comparison with other snippets.<\/p>\n\n\n\n

To do this in Python, we can take advantage of the datetime<\/code> library:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n<\/td>\n
\n
\n
import datetime<\/code><\/div>\n
start_time = datetime.datetime.now()<\/code><\/div>\n
# insert code snippet here<\/code><\/div>\n
end_time = datetime.datetime.now()<\/code><\/div>\n
print<\/code> end_time - start_time<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

Of course, this solution leaves a lot to be desired. For example, it only gives us a single data point. Ideally, we\u2019d want to run this a few times to collect an average or at least a lower bound, but this can do in a pinch.<\/p>\n\n\n\n

Performance Testing Using the timeit<\/code> Library<\/h3>\n\n\n\n

If you\u2019d prefer to have all this timestamp garbage abstracted away with the addition of a few perks, check out the timeit<\/code> library. With the timeit<\/code> library, there are basically two main ways to test code: command line or inline. For our purposes, we\u2019ll take a look at the inline version since that\u2019s what I use for all my testing.<\/p>\n\n\n\n

To test code using the timeit<\/code> library, you\u2019ll need to call either the timeit<\/code> function or the repeat<\/code> function. Either one is fine, but the repeat<\/code> function gives a bit more control.<\/p>\n\n\n\n

As an example, we\u2019ll test the following code snippet from an earlier article on list comprehensions<\/a>:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n<\/td>\n
\n
\n
[(a, b) <\/code>for<\/code> a in (1, 3, 5) <\/code>for<\/code> b in (2, 4, 6)]<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

In this snippet, we\u2019re generating a list of pairs from two tuples. To test it, we could use the timeit<\/code> function:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
import timeit<\/code><\/div>\n
timeit.timeit(<\/code>\"[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]\"<\/code>)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

If done correctly, this will run the snippet a million times and return an average execution time as a result. Of course, you\u2019re welcome to change the number of iterations using the number<\/code> keyword argument:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
import timeit<\/code><\/div>\n
timeit.timeit(<\/code>\"[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]\"<\/code>, number=1000)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

Naturally, we can take this test a step further by running it multiple times using the repeat function:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
import timeit<\/code><\/div>\n
timeit.repeat(<\/code>\"[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]\"<\/code>)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

Instead of returning an execution time, this function returns a list of execution times. In this case, the list will contain three separate execution times. Of course, we don\u2019t need all those times. Instead, we can return the smallest execution time, so we can get an idea of the lower bound of the snippet:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
import timeit<\/code><\/div>\n
min(timeit.repeat(<\/code>\"[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]\"<\/code>))<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

If you\u2019ve been around a bit, you\u2019ve probably seen this exact syntax in my performance tests in other articles or videos. Of course, I go the extra mile and increase the number of repetitions, but it\u2019s probably overkill. In any case, this is a great way of performance testing Python snippets.<\/p>\n\n\n\n

Performance Testing Using the cProfile Library<\/h3>\n\n\n\n

Outside of timeit<\/code> and outright brute force, you can always leverage other profiling tools like cProfile<\/a>. Like timeit<\/code>, we can leverage cProfile<\/code> to get runtime statistics from a section of code. Of course, cProfile is quite a bit more detailed. For example, we can run the same list comprehension from above as follows:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n<\/td>\n
\n
\n
import cProfile<\/code><\/div>\n
cProfile.run(<\/code>\"[(a, b) for a in (1, 3, 5) for b in (2, 4, 6)]\"<\/code>)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

As a result, you get a nice report that looks like this:<\/p>\n\n\n\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n
6<\/div>\n
7<\/div>\n
8<\/div>\n
9<\/div>\n<\/td>\n
\n
\n
4 <\/code>function<\/code> calls in 0.000 seconds<\/code><\/div>\n
 <\/div>\n
   <\/code>Ordered by: standard name<\/code><\/div>\n
 <\/div>\n
   <\/code>ncalls  tottime  percall  cumtime  percall filename:lineno(<\/code>function<\/code>)<\/code><\/div>\n
        <\/code>1    0.000    0.000    0.000    0.000 <string>:1(<listcomp>)<\/code><\/div>\n
        <\/code>1    0.000    0.000    0.000    0.000 <string>:1(<module>)<\/code><\/div>\n
        <\/code>1    0.000    0.000    0.000    0.000 {built-in method builtins.<\/code>exec<\/code>}<\/code><\/div>\n
        <\/code>1    0.000    0.000    0.000    0.000 {method <\/code>'disable'<\/code> of <\/code>'_lsprof.Profiler'<\/code> objects}<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n

Here, we get a nice table which includes a lot of helpful information. Specifically, each row indicates a function that was executed, and each column breaks down a different runtime segment. For example, the <listcomp><\/code> function was called once (ncalls<\/code>) and took 0.000 seconds (tottime<\/code>) excluding calls to subfunctions. To understand everything else in this table, check out the following breakdown of all six columns:<\/p>\n\n\n\n