What We Talk About When We Talk About Performance

Describing performance improvements exists at the intersection of mathematics and linguistics. It is quite common to use incorrect math to describe performance improvements, and it is possible to use incorrect, misleading, or just sub-optimal rhetoric to describe your math.

Consider this hypothetical press release:

AirTrain Inc. is proud to announce the new AirTrain-8000. This revolutionary new plane can fly from London to Seattle at an average speed of 7,700 km/h – a huge improvement over the 770 km/h of other jets. This drops the travel time from ten hours to just one, making the AirTrain-8000 90% faster than our competitors.

AirTrain-8000 takes 1 hour, other jets take 10 hoursA press release like this would never be released. The new plane is ten times faster than previous planes (7,700 km/h divided by 770 km/h), and no marketing team would allow this improvement to be summarized as “90% faster”, which means “almost twice as fast.” And yet, when talking about computers – where ten-times speedups happen fairly often – this mistake is made quite frequently.

This abuse of percentages has made them meaningless for describing optimizations – we need to stop using them. The AirTrain-8000 is ten times as fast, full stop.

In the AirTrain example above the elapsed time for the journey was reduced by 90%. It would be reasonable for the press release to say that. It would be accurate (but uselessly ambiguous) to call this plane “90% better.” But saying that a plane that goes 7,700 km/h instead of 770 km/h is “90% faster” is just plain wrong, and self defeating.

You sit on a throne of lies

It’s quite easy to find examples of this rhetorical error. A google search for “90% faster” (with the quotes) almost exclusively finds articles talking about reductions in elapsed time of 90% or more, where the new process is actually at least 900% faster. Here are just a few examples:

I could go on.

My bold claim

If you optimize a task so that it takes 90% less time than before then that is a ten-times speedup, and it should be described as such. It doesn’t matter whether the task is flying a plane from London to Seattle, loading a game, or finding a file. A 90% reduction in time is a ten-times speedup, period, and the five examples above are all incorrect.

Graph showing speedup dramatically increasing as reduction in time goes to 95%

What 90% faster means

Let’s imagine that I ride my scooter at 10 m/s and a friend rides theirs at 19 m/s. Going 19 m/s instead of 10 m/s is, mathematically, 1.9-times the speed. We convert that to a percentage increase by going:

100% * 1.9 – 100% = 90%

Therefore their speed is 90% greater than mine. Another way to say 90% more speed is to say 90% faster.

Aside: the subtracting of 100% is because, by convention, a percentage change always refers to just the change, whereas in “1.9-times speedup” the “1.9” is a ratio of the two speeds. I don’t make the rules, but that is the linguistic/mathematical convention. Some people disagree and think that 1.9-times the speed should be called 190% faster but that is non-standard.

What 90% faster doesn’t mean

Unicycle Speed Trap - CopyIf I do some task in 100 s and a friend does it in 10 s then my friend is doing it in 90% less time. And many people seem content to say that in this situation my friend is 90% faster than I. But wait. What if the task that we are doing is riding our scooters 1,000 m to work. In that case I am riding at 10 m/s and my friend (propelled by a jet pack?) is riding at 100 m/s.

It makes no sense to use the term “90% faster” to mean “going 19 m/s instead of 10 m/s” and also to mean “going 100 m/s instead of 10 m/s”. It can’t mean both, and this second usage of 90% faster is wrong – in this case my friend is actually 900% faster. Some people call it 1,000% faster (see previous section), which is unfortunate.

Reality check

Let’s look at one concrete example. In this article a gamer describes a technique they have found for getting a game to load in 50 s instead of 510 s. They then describe that as “90% Faster Game Loads?”.

This confusion seems to happen whenever an improvement is a reduction in how long it takes to do a task, so let’s go through the math.

Before: the game loaded in 510 s which means 0.00196 game loads per second. Alternately, you can load this game 7.06 times per hour.

After: with the suggested hack in use the game loads in 50 s, which means you can load the game 0.02 times per second, 72 times per hour.

The faster method works out to be a bit more than ten times the game loads per time period, regardless of what time period is chosen. It is 900% faster, not 90% faster.

The near red car is 90% faster than the far red carThe game loading example might “feel” weird because nobody is going to repeatedly load the game for an hour and be pleased that they could load it 72 times instead of 7.06 times. But that doesn’t change the fact that game loading is more than ten times as fast. Skydivers can freefall at about ten times the speed of casual cyclists, but we don’t discount this accomplishment just because they can only do it for a minute or so at a time. They travel about two miles in a minute, which is a speed of 120 miles per hour, and their failure to maintain that speed for an hour does not diminish that accomplishment.

Calculating speedups

If we have an old speed and a new speed then the speedup ratio can be calculated as:

speedup ratio = x = NewSpeed / OldSpeed

We can then say that the new speed is x times as fast as the old speed, ten times as fast in the case of the AirTrain-8000.

If we have old and new elapsed times for a fixed task then translating a reduction in time into an “x times as fast” sentence is just as easy. It’s as simple as

speedup ratio = x = OldTime / NewTime

It’s just as simple as calculating the speedup ratio from speed numbers, although it is old/new instead of new/old. This calculation works for any sort of fixed task that you speed up. You can work out this formula by calculating new and old speed with this formula:

speed = tasks/s = NumTasks / ElapsedTimeInSeconds

and then inserting the two speeds into the NewSpeed / OldSpeed formula until NumTasks cancels out. Simple algebra.

Recommendations

I originally wanted to recommend that people use “90% speedup” when they make something 1.9 times as fast. But my research made me realize that this is a terrible idea – the multiple incorrect usages are too pervasive and you simply can’t count on people understanding what you mean, with the possible exception of small percentages like “20% faster” where the ambiguity is modest.

Therefore, when you reduce how long a task takes – any task – you should calculate the speedup factor as NewSpeed / OldSpeed and use that speedup to describe your achievement. You should also share your before/after numbers so readers can check your math.

AirTrain-8000 goes 7,700 km/h, other jets go 770 km/hWhen you hear something described as a 90% to 99.9% speedup you should be wary. Unless the before and after data is shared you really have no idea what was achieved. It usually means “I’m good at optimizing but poor at communication.”

You should never describe something as being a “90% improvement” or “90% better” – these phrases are meaningless. Instead, embrace the big and accurate 10-times as fast number. It’s 10 times better!

Putting my money where my mouth is, when I reduced the render times for a particular fractal zoom movie from six months to 18 hours I didn’t call it 99.6% faster, I called it 240 times as fast. I’m glad that I preemptively followed my own advice.

Twitter discussion is here, reddit discussion is here.

Update: it was pointed out by a reader that “two times faster” and “two times as fast” mean different things. “two times faster” refers to the increase in speed and means that the speed ratio is 3:1, while “two times as fast” means that the speed ratio is 2:1. I’ve updated the article for clarity. At the very least “two times faster” is ambiguous and subject to misinterpretation.

Advertisements

About brucedawson

I'm a programmer, working for Google, focusing on optimization and reliability. Nothing's more fun than making code run 10x faster. Unless it's eliminating large numbers of bugs. I also unicycle. And play (ice) hockey. And sled hockey. And juggle. And worry about whether this blog should have been called randomutf-8.
This entry was posted in Math, Performance, Rants and tagged , , . Bookmark the permalink.

21 Responses to What We Talk About When We Talk About Performance

  1. Aankhen says:

    Hear, hear! It’s not something I get worked up about, but I’m glad someone’s pointed it out.

  2. dithermaster says:

    I’m completely in agreement. Percentages are misunderstood, by both writers and readers. I try to avoid them and just use factors (e.g., “1.5x faster”) and share actual measurements. For speedups, the only place percentages make sense are in the “more than 1x, less than 2x” range, as in “30% faster” in place of “1.3x faster”. Above 2x they get confusing (3x faster is “200% faster”, and parsing things like “1700% faster” shouldn’t be necessary). I think reduction in something is harder to describe. Sometimes percentages make more sense (e.g., “30% fewer crashes” versus “0.7x as many crashes”). I hate it when people use “times” for reductions, as in “3 times less crashes” — three times what? Maybe they mean one third as many crashes, but who knows? A related issue is describing something (for example video encoding) relative to realtime. If something is “3x realtime” does that mean it takes one third the time (e.g, 20 minutes to encode an hours worth of content), or does it mean it takes three times the time (e.g., 3 hours to encode an hours worth of content)? Be specific or you risk having your reader think the opposite.

  3. Great post!

    If you’re reporting a speedup as a ratio, then how about plotting it using a logarithmic scale? https://johnwickerson.wordpress.com/2016/08/06/ratios-on-logarithmic-scales/
    That way, it’s very easy to spot if a speedup or slowdown has occurred.

  4. Guy Davidson says:

    YES YES YES!!! This is such a common mistake, it blights discourse about all things numerical. I’ve heard people who claim to be ECONOMISTS make exactly this error. I have entirely given up the use of percentages to describe change and only use them for describing portions of things, e.g. “that’s 80% of our income”, “that’s 140% of the original investment”. You’re on a road to ambiguity when you say things like “that’s 80% smaller” or “that’s 140% faster”.

    Thanks for writing this. Now I have somewhere to point people to…

    • brucedawson says:

      It makes me sad to have to give up on saying things like “50% faster” because I know what I mean and I am strongly believe that my interpretation is correct, but… language is about communication and if a significant percentage (!!!) of people will misinterpret or be unsure of my meaning then I must phrase things differently. 1.5 times faster it is.

      • Adrian says:

        Surely you mean 0.5 times faster?! When talking about a change, don’t you subtract one (just as you would subtract 100% if you were expressing it as a percentage)? 😉

        Also, your press release didn’t use consistent units: The new plane goes 7700 km/h, but the old ones go 770 km. That is, the old ones have a distance limit rather than a speed limit. 😉

        • brucedawson says:

          Good catches. I’ve fixed them both. The km versus km/h was just sloppy. The “x times faster” versus “x times as fast” issue caught me unawares. Apparently I am guilty of one of the sins of which I have accused others. I think that using “0.5 times faster” to mean “1.5 times as fast” is *wicked* confusing so I have resolved to always use the “x times as fast” phrasing in the future.

          Your feedback has made this blog post up to 50% better, or more!

  5. jrrr says:

    Rants are so much better when they have this much rigor.

  6. Pingback: New top story on Hacker News: What We Talk About When We Talk About Performance – Tech + Hckr News

  7. Pingback: What We Talk About When We Talk About Performance – posted at February 05, 2018 at 12:01PM by ingve – Startup News 2018

  8. Pingback: New top story on Hacker News: What We Talk About When We Talk About Performance – ÇlusterAssets Inc.,

  9. John Payson says:

    How about using the term “order of magnitude”? While phraseology might sometimes make it unclear whether it’s a precise or imprecise measure, it has the advantage that it’s commutative. If X is an order of magnitude faster than Y, Y is an order of magnitude faster than X.

    • brucedawson says:

      I assume that was supposed to end with “order of magnitude slower than X.”?

      I do use that phrasing for huge changes. For instance, if something is a thousand times as fast I’ll say it’s ten orders of magnitude faster (base-2 orders of magnitude make all of your optimizations seem better).

  10. Luís Avelino Relógio says:

    If the new plain is 10 times faster (or 11 times as fast) its speed is 90% of the former but 1100%.
    Where the hell is the confusion?

  11. Pingback: What We Talk About When We Talk About Performance | Random ASCII – Amalgamated Contemplation

  12. wg says:

    You are making your point too often in this blog post.

    • brucedawson says:

      You’re not wrong, and it used to be even more redundant. My only comment in my defense is that the more I talked to people about this the more conflicting-and-contradictory viewpoints I found. I felt forced to refute some beliefs that you may not have encountered.

      That is, the whole thing seems unnecessary, and yet, my search for “90% faster” found more misuses than correct uses.

  13. Ksec says:

    What baffle me is most of these shouldn’t even be a problem in the first place.
    As pointed out “two times faster” and “two times as fast” is another worrying example.

    Seriously, what went wrong? It is not so much about English or Maths, but basic logic error.

    Personally, anything that has to do with time, it is much better to write something like X times faster, the bigger the number, the better.

    However for memory, storage space, or anything that has less to do with time, % reduction makes much more sense. E.g 80% less memory being used.

    • brucedawson says:

      Instead of “X times faster” I would recommend “X times as fast” to avoid the 1.5/0.5 ambiguity. I agree that the logic used in some of these cases is surprising, but “human languages” and “precise statements” are often in opposition.

  14. Marc C says:

    Thanks Bruce!

    There is also a problem around language regarding improvement and degradation when you are referring to absolute quantities of time (not rates). I’m a performance engineer and I’ve tried to lay down the law at work regarding this; it isn’t easy. Suppose a benchmark normally takes 5s but now we experience a regression and the test takes 10s. One can say:

    * The test takes “2x (or twice) as long”, which most people would understand as definitely bad.
    * The test is “half as fast”, definitely nonstandard and just weird.
    * The test is “2x slower”, technically wrong but most people would understand as bad (it’s only 1x “slower”). Using “2x (or twice) as slow” would probably work also but the word “slow” sort of implies that the normal execution time is not good either!
    * We have a performance degradation of 100%. Yikes. It’s technically correct (10s – 5s)/5s x 100%. But as you’ve mentioned, percentages can cause more confusion.

    Everything you and others have mentioned also applies to performance improvements, with one caveat that is unlikely to cause much worry for most of us: whereas the amount of degradation can increase practically without bound, the amount of improvement can only increase up to our ability to measure it, i.e., infinity is a lot further away than zero when you deal in micro- or nanoseconds resolution. The reason I say it’s not a problem in practice is that in most cases we can just add more input to make the tests run longer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.