What We Talk About When We Talk About Performance

TL;DR – the phrase “90% faster” is interpreted by people in two different ways:

1. 90% less time
2. The speed (ops/sec or km/h) is 90% greater

I strongly prefer #2 but due to the ambiguity this phrase should never be used. The preferred unambiguous phrase should be “1.9 times as fast” (or “ten times as fast”, if that is what you were trying to say). To do otherwise is to risk being misunderstood by a significant portion of your audience.

Onwards…

Describing performance improvements exists at the intersection of mathematics and linguistics. It is quite common to use incorrect math to describe performance improvements, and it is possible to use incorrect, misleading, or just sub-optimal rhetoric to describe your math.

Consider this hypothetical press release:

AirTrain Inc. is proud to announce the new AirTrain-8000. This revolutionary new plane can fly from London to Seattle at an average speed of 7,700 km/h – a huge improvement over the 770 km/h of other jets. This drops the travel time from ten hours to just one, making the AirTrain-8000 90% faster than our competitors.

A press release like this would never be released. The new plane is ten times as fast as previous planes (7,700 km/h divided by 770 km/h), and no marketing team should allow this improvement to be summarized as “90% faster”, which to many people means “almost twice as fast.” And yet, when talking about computers – where ten-times speedups happen fairly often – this mistake is made quite frequently.

This abuse of percentages has made them meaningless for describing optimizations – we need to stop using them. The AirTrain-8000 is ten times as fast, full stop.

In the AirTrain example above the elapsed time for the journey was reduced by 90%. It would be reasonable for the press release to say that. It would be accurate (but uselessly ambiguous) to call this plane “90% better.” But saying that a plane that goes 7,700 km/h instead of 770 km/h is “90% faster” means that many readers will interpret the new plane as being 1.9 times as fast. To my mind “90% faster” means that the speed of the new plane is 1.9 times greater (isn’t that what faster means?) but to some other smart people “90% faster” means “90% less elapsed time.” That interpretation mystifies me, but cannot be ignored.

You sit on a throne of lies

It’s quite easy to find examples of this rhetorical error. A google search for “90% faster” (with the quotes) finds many articles talking about reductions in elapsed time of 90% or more, where the new process is actually at least 900% faster. Here are just a few examples:

• 90% Faster Game Loads? – more than ten times as fast
• Get a business education 90% faster – more than thirteen times as fast
• 90% Faster Processing – fifteen times as fast
• Finds Files 90% Faster! – ten times as fast
• 67% faster than HTTP –  3.07 times as fast

I could go on.

My bold claim

If you optimize a task so that it takes 90% less time than before then that is a ten-times speedup, and it should be described as such. It doesn’t matter whether the task is flying a plane from London to Seattle, loading a game, or finding a file. A 90% reduction in time is a ten-times speedup, period, and the five examples above are all incorrect, or at least poor communication. Why would you want to describe your achievement in a way that makes it sound less significant to many people?

What 90% faster means

Let’s imagine that I ride my scooter at 10 m/s and a friend rides theirs at 19 m/s. Going 19 m/s instead of 10 m/s is, mathematically, 1.9-times the speed. We convert that to a percentage increase by going:

100% * 1.9 – 100% = 90%

Therefore their speed is 90% greater than mine. Another way to say 90% more speed is to say 90% faster.

Aside: the subtracting of 100% is because, by convention, a percentage change always refers to just the change, whereas in “1.9-times speedup” the “1.9” is a ratio of the two speeds. I don’t make the rules, but that is the linguistic/mathematical convention. Some people disagree and think that 1.9-times the speed should be called 190% faster but that is non-standard.

What 90% faster doesn’t mean

If I do some task in 100 s and a friend does it in 10 s then my friend is doing it in 90% less time. And many people seem content to say that in this situation my friend is 90% faster than I. But wait. What if the task that we are doing is riding our scooters 1,000 m to work. In that case I am riding at 10 m/s and my friend (propelled by a jet pack?) is riding at 100 m/s.

It seems to make little sense to use the term “90% faster” to mean “going 19 m/s instead of 10 m/s” and also to mean “going 100 m/s instead of 10 m/s”. It can’t mean both, and this second usage of 90% faster is wrong – in this case my friend is actually 900% faster. Some people call it 1,000% faster (see previous section), which is unfortunate.

Reality check

Let’s look at one concrete example. In this article a gamer describes a technique they have found for getting a game to load in 50 s instead of 510 s. They then describe that as “90% Faster Game Loads?”.

This confusion seems to happen whenever an improvement is a reduction in how long it takes to do a task, so let’s go through the math.

Before: the game loaded in 510 s which means 0.00196 game loads per second. Alternately, you can load this game 7.06 times per hour.

After: with the suggested hack in use the game loads in 50 s, which means you can load the game 0.02 times per second, 72 times per hour.

The faster method works out to be a bit more than ten times the game loads per time period, regardless of what time period is chosen. It is 900% faster, not 90% faster.

The game loading example might “feel” weird because nobody is going to repeatedly load the game for an hour and be pleased that they could load it 72 times instead of 7.06 times. But that doesn’t change the fact that game loading is more than ten times as fast. Skydivers can freefall at about ten times the speed of casual cyclists, but we don’t discount this accomplishment just because they can only do it for a minute or so at a time. They travel about two miles in a minute, which is a speed of 120 miles per hour, and their failure to maintain that speed for an hour does not diminish that accomplishment.

Calculating speedups

If we have an old speed and a new speed then the speedup ratio can be calculated as:

speedup ratio = x = NewSpeed / OldSpeed

We can then say that the new speed is x times as fast as the old speed, ten times as fast in the case of the AirTrain-8000.

If we have old and new elapsed times for a fixed task then translating a reduction in time into an “x times as fast” sentence is just as easy. It’s as simple as

speedup ratio = x = OldTime / NewTime

It’s just as simple as calculating the speedup ratio from speed numbers, although it is old/new instead of new/old. This calculation works for any sort of fixed task that you speed up. You can work out this formula by calculating new and old speed with this formula:

speed = tasks/s = NumTasks / ElapsedTimeInSeconds

and then inserting the two speeds into the NewSpeed / OldSpeed formula until NumTasks cancels out. Simple algebra.

Recommendations

I originally wanted to recommend that people use “90% speedup” when they make something 1.9 times as fast. But my research made me realize that this is a terrible idea – the multiple incorrect usages are too pervasive and you simply can’t count on people understanding what you mean, with the possible exception of small percentages like “20% faster” where the ambiguity is modest.

Therefore, when you reduce how long a task takes – any task – you should calculate the speedup factor as NewSpeed / OldSpeed and use that speedup to describe your achievement. You should also share your before/after numbers so readers can check your math.

When you hear something described as a 90% to 99.9% speedup you should be wary. Unless the before and after data is shared you really have no idea what was achieved. It usually means “I’m good at optimizing but poor at communication.”

You should never describe something as being a “90% improvement” or “90% better” – these phrases are meaningless. Instead, embrace the big and accurate 10-times as fast number. It’s 10 times better!

Putting my money where my mouth is, when I reduced the render times for a particular fractal zoom movie from six months to 18 hours I didn’t call it 99.6% faster, I called it 240 times as fast. I’m glad that I preemptively followed my own advice.

Twitter discussion is here, reddit discussion is here.

Update: it was pointed out by a reader that “two times faster” and “two times as fast” mean different things. “two times faster” refers to the increase in speed and means that the speed ratio is 3:1, while “two times as fast” means that the speed ratio is 2:1. I’ve updated the article for clarity. At the very least “two times faster” is ambiguous and subject to misinterpretation.