Commute Challenge 2018

Bruce Dawson in wet suit with commute methods, copyright 2018 425 Business MagazineI’ll bet I had more fun commuting during September 2018 than you did.

In April 2017 I gave myself the challenge of commuting to work using a different method every workday for a month – twenty ways in twenty days! The write up and video are here. It was great fun and it also served as a joyous celebration of the many ways to make commuting more fun than sitting alone in traffic.

In September 2018 I did the same thing, with nineteen new methods. It was a lot of work but a huge amount of fun, and I’ve got the video to prove it. I got assistance from friends, coworkers, and strangers in ways that I would not have thought possible, and the second commute challenge actually worked.

Continue reading

Posted in Commuting, Environment, Unicycling | 11 Comments

24-core CPU and I can’t type an email (part two)

In my last post I promised to give more details about some rabbit holes that I went down during the investigation, including page tables, locks, WMI, and a vmmap bug. Those details are here, along with updated code samples. But first, a really quick summary of the original issue:

In the last post I talked about how every time a CFG-enabled process allocates executable memory some Control Flow Guard (CFG) memory is allocated as well. Windows never frees the CFG memory so if you keep allocating and freeing executable memory at different addresses then your process can accumulate an arbitrary amount of CFG memory. Chrome was doing this and that was leading to an essentially unbounded waste of memory, and hangs on some machines.

And, I have to say, hangs are hard to avoid if VirtualAlloc starts running more than a million times slower than normal.

Continue reading

Posted in Investigative Reporting, memory, Performance | Tagged , , , | 26 Comments

24-core CPU and I can’t type an email (part one)

I wasn’t looking for trouble. I wasn’t trying to compile a huge project in the background (24-core CPU and I can’t move my mouse), I was just engaging in that most mundane of 21st century tasks, writing an email at 10:30 am. And suddenly gmail hung. I kept typing but for several seconds but no characters were appearing on screen. Then, suddenly gmail caught up and I resumed my very important email. Then it happened again, only this time gmail went unresponsive for even longer. Well that’s funny

CFG ended up being a big part of this issue, and eight months later I hit a completely unrelated CFG problem, written up here.

I have trouble resisting a good performance mystery but in this case the draw was particularly strong. I work at Google, making Chrome, for Windows, focused on performance. Investigating this hang was actually my job. And after a lot of false starts and some hard work I figured out how Chrome, gmail, Windows, and our IT department were working together to prevent me from typing an email, and in the process I found a way to save a significant amount of memory for some web pages in Chrome.

Continue reading

Posted in Investigative Reporting, memory, Performance | Tagged , , , | 55 Comments

Making Windows Slower Part 1: File Access

Tortoises are slowWindows has long had a reputation for slow file operations and slow process creation. Have you ever wanted to make these operations even slower? This weeks’ blog post covers a technique you can use to make all file operations on Windows run at one tenth their normal speed (or slower), in a way that will be untraceable for most users!

And, of course, this post will also cover how to detect and avoid this problem.

Continue reading

Posted in Investigative Reporting, Performance, Programming, xperf | Tagged , , | 25 Comments

Compiler bug? Linker bug? Windows Kernel bug.

See the end of the post for an October 2018 bug fix update, or read the whole story:

imageFlaky failures are the worst. In this particular investigation, which spanned twenty months, we suspected hardware failure, compiler bugs, linker bugs, and other possibilities. Jumping too quickly to blaming hardware or build tools is a classic mistake, but in this case the mistake was that we weren’t thinking big enough. Yes, there was a linker bug, but we were also lucky enough to have hit a Windows kernel bug which is triggered by linkers!

Continue reading

Posted in Debugging, Investigative Reporting, Programming | Tagged , , , | 65 Comments

Zombie Processes are Eating your Memory

Zombies probably won’t consume 32 GB of your memory like they did to me, but zombie processes do exist, and I can help you find them and make sure that developers fix them. Tool source link is at the bottom.

Is it just me, or do Windows machines that have been up for a while seem to lose memory? After a few weeks of use (or a long weekend of building Chrome over 300 times) I kept noticing that Task Manager showed me running low on memory, but it didn’t show the memory being used by anything. In the example below task manager shows 49.8 GB of RAM in use, plus 4.4 GB of compressed memory, and yet only 5.8 GB of page/non-paged pool, few processes running, and no process using anywhere near enough to explain where the memory had gone:


My machine has 96 GB of RAM – lucky me – and when I don’t have any programs running I think it’s reasonable to hope that I’d have at least half of it available.

Continue reading

Posted in Debugging, Investigative Reporting, Performance, Programming, Rants | Tagged , , | 74 Comments

What We Talk About When We Talk About Performance

TL;DR – the phrase “90% faster” is interpreted by people in two different ways:

  1. 90% less time
  2. The speed (ops/sec or km/h) is 90% greater

I strongly prefer #2 but due to the ambiguity this phrase should never be used.

Describing performance improvements exists at the intersection of mathematics and linguistics. It is quite common to use incorrect math to describe performance improvements, and it is possible to use incorrect, misleading, or just sub-optimal rhetoric to describe your math.

Consider this hypothetical press release:

AirTrain Inc. is proud to announce the new AirTrain-8000. This revolutionary new plane can fly from London to Seattle at an average speed of 7,700 km/h – a huge improvement over the 770 km/h of other jets. This drops the travel time from ten hours to just one, making the AirTrain-8000 90% faster than our competitors.

AirTrain-8000 takes 1 hour, other jets take 10 hoursA press release like this would never be released. The new plane is ten times as fast as previous planes (7,700 km/h divided by 770 km/h), and no marketing team would allow this improvement to be summarized as “90% faster”, which to many people – probably most people – means “almost twice as fast.” And yet, when talking about computers – where ten-times speedups happen fairly often – this mistake is made quite frequently.

This abuse of percentages has made them meaningless for describing optimizations – we need to stop using them. The AirTrain-8000 is ten times as fast, full stop.

Continue reading

Posted in Math, Performance, Rants | Tagged , , | 25 Comments