I use Visual Studio to develop some fairly large projects. When I’m testing a change to a global header file then I load up a solution that contains 183 projects and build them all. Since many of the projects are building in parallel and since I build most of them with /MP there can be a lot of compiler processes simultaneously:
C:\>tlist | find /c “cl.exe”
Normally there are just a few dozen compiler instances, but as the command above shows it can get up pretty high.
My Windows 7 development machine has six hyperthreaded cores (twelve threads total), 24 GB of RAM, and an SSD main drive. But somehow either VS or Windows has trouble dealing with that much compilation going on simultaneously. When doing these massive builds VS hangs. A lot.
I recently installed PerfWatson which means that all of the hangs are recorded. You can then go to %localappdata%\perfwatson and see all of the accumulated reports:
type *.maxdelay | find /i “<Duration>”
During a recent eleven minute period while I was building this massive solution I found that PerfWatson had reported 52 hangs ranging from 2.1 to 23.5 seconds (the Duration field is presumed to be milliseconds). More impressive was the total hang time – 369.3 seconds, or roughly six minutes of hangs over an eleven minute period.
My guess is that this is actually a Windows bug. I think it is giving those many compiler instances too much CPU time and thus starving Visual Studio, despite the fact that the scheduler is supposed to favor foreground applications. Or maybe Visual Studio is waiting on results from a particular compiler instance and thus hitting a self-induced priority inversion. Or maybe I don’t know.
Anyway, if you use Visual Studio you should install PerfWatson. And PerfWatson Monitor, because it is cool to see Visual Studio reporting exactly how long it has been hung for.
Update: almost three years later I blogged about a very specific explanation for why Visual Studio hangs when doing builds with /analyze. In 2011 I was only able to observe that these hangs were happening, but in 2014 I had the skills to understand and report on exactly why the hangs were happening. I think the explanation is funny.