Benchmarking Gotchas

While benchmarking at its core is a simple idea, a surprising number of gotchas can completely derail your experiments. These gotchas are particularly sneaky since timeit will always spit out a number even if your experiment isn’t set up correctly. Everything may seem to be in order, but your results may be totally meaningless. We’ll look at these gotchas throughout the remainder of this chapter.

Gotcha: Using Two Different Sets of Numbers

One of the most important things to keep in mind when benchmarking, or when conducting any scientific experiment for that matter, is to ensure that your experiment is controlled. This means that if you’re comparing two different algorithms, all other factors besides the algorithms themselves should be identical.

Take our append vs. insert experiment, for example. Our experiment is only valid if both snippets of code are creating arrays of the same size. If, however, our append code created 1,000,000 elements while our insert code created only 100 elements, our benchmarking results would be skewed. The fact that our append code ran more slowly wouldn’t prove that append is slower than insert; perhaps it ran more slowly only because it was busy generating so many more elements.

Indeed, when I run the insert code with just 100 elements, I get the supersonic result of 3.69548797607e-05. This is way faster than the results of my append benchmark of one million elements, and if I wasn’t paying attention, I might mistakenly conclude that insert is much faster than append.

Because of this, when benchmarking two competing code snippets, you always want to double-check that both algorithms are working with the same data.

Gotcha: Only Benchmarking One Time

Obviously, a benchmarking experiment would not be controlled if you tested Algorithm A on one computer and Algorithm B on another. Perhaps the only reason why one algorithm runs faster than the other is that it was executed on the more powerful computer!

Similarly, it’s possible that even when you benchmark two algorithms on the same computer, the computer happens to be more powerful when executing one algorithm than when executing the other. This is very common since a computer is always running multiple processes at any given time. The benchmarking code you run is never executed in a vacuum. You may have other applications running, including that Internet browser with 85 tabs open. (You should probably do something about that.)

Based on this, it would be unwise to benchmark Algorithm A and then decide to play a massive multiplayer video game while benchmarking Algorithm B. Perhaps the results of Algorithm B are slower because the game is running in the background.

However, even if you don’t turn on that game, you can’t know for certain that your computer isn’t secretly downloading some security update while you happen to be benchmarking Algorithm B. Basically, your computer is always running all sorts of processes in the background, and there’s not much you can do about it.

One way to help with this is to run your benchmarking experiments multiple times. It’s less likely that the same background process will keep occurring each time you run your experiment.

timeit’s Repeat Method

The timeit module provides a method called repeat that will conveniently run your benchmark multiple times so you don’t have to do it manually. The repeat method is essentially identical to the timeit method, except that repeat also accepts a repeat argument where you set the number of times that your code should execute:

I’ll explain the difference between the repeat argument and the number argument soon, but for now, we’ll focus on repeat.

With this code, our benchmark will run 5 times. This will return an array of results like these:

These are the results of running my insert benchmark on 1,000,000 elements. The results are all similar, but the fastest among these results is 0.2981288433074951.

Now, here’s an important point. In theory, a piece of code should take the same amount of time each time we execute it. So why aren’t the results all exactly the same? Again, this is because the computer’s background processes will always skew the results somewhat.

Because of this, the truest of the results is, in fact, 0.2981288433074951. The only reason why the other results were slightly slower than this is that the computer’s background processes got in the way. So instead of using the average of our results to get the most accurate speed, professional benchmarkers use the fastest result.

However, this assumes that our code has no randomness involved. If our data is randomized each time we run it, the varying speeds may be a result of the fact that the data is different each time we run our benchmark. This will come into play when we benchmark sorting algorithms, as you’ll soon see.

Repeat vs. Number

Both the repeat argument and the number argument allow you to execute your code numerous times in a row. However, there are two key differences between the two arguments.

One key difference is with regard to how the results are displayed. The repeat argument spits out an array of different results, as you saw earlier in this chapter. However, if we keep repeat at 1 and instead change number to 5, we get one result that is the amount of time that it took for all five rounds to execute in total. So, if I get the result of 1.56467604637146, this means I would have to divide that number by 5 to see how fast each individual round took on average.

Whether you use repeat or number, you’ll always want to benchmark Algorithm A and Algorithm B the same number of times to keep your experiment controlled.

In any case, I’ll be using the repeat approach going forward, generally setting the repeat argument to 5.

There’s another key difference between repeat and number, but before we look at that, let me address another gotcha.

Gotcha: Not Making Sure Your Code Works

I once read an online tutorial on benchmarking that used this example code:

It’s basically the same append experiment we used earlier, except that all the code is wrapped inside a function called creative_massive_array.

When I benchmark this code, I get very strange results:

What on Earth? What’s with all the zeroes? And why are all of these numbers so different from each other?

Before I reveal the solution, I want to highlight that this should be your most important takeaway from this chapter: always keep your brain on. If you see surprising results, you shouldn’t blindly accept them. Instead, investigate why you’re getting those results. If something smells fishy, it probably is.

One of the first steps to take when discovering fishy results is to make sure your code does what you think it should do. For this, printing to the console is your friend. Let’s go ahead and add a print(array) command at the end of our test_code:

When I benchmark this revised code, no array gets outputted to the console. I still get wonky benchmark numbers, but I should have also seen a massive array displayed in the console. What’s going on? Wait … facepalm emoji. (Is that a phrase?) The code for generating the array never gets executed! Sure, my test_code defines a function that will generate an array, but this function never gets called anywhere. No wonder this code ran so fast.

As to why we got those strange benchmarking numbers, that has to do with Python internals. If you benchmark any code that hardly does anything, you’ll get similar wonky results.

To avoid this gotcha, it’s worthwhile to always use print or a similar technique to ensure that your code is doing what it should.

To fix this particular problem, we need to call the function within our test_code itself:

After we’ve verified that our code works, we can then eliminate the print statement and benchmark our code properly. In fact, we should indeed make a point of removing the print statement. This is because printing to the console consumes considerable time in its own right, and we only want to benchmark the actual algorithm, not the printing.