Analyzing xperf traces is virtually impossible if you don’t have symbols. Unfortunately the most recent version of the xperf toolkit makes it easy to end up with symbols that takes hours to load, or don’t load at all.
After two people contacted me the same day (one asking why symbol loading didn’t work, the other asking why it was too slow) I decided it was time to document this issue.
Update, November 2014
It’s still important to copy known-good copies of to the WPT install directory, since otherwise you can’t be sure which DLL versions you will get (if any). The VS 2013 versions seem to be good. But, for some PDBs a new problem has surfaced. It shows up when profiling the Chrome web browser. You can read about this new problem and its solution at Slow Symbol Loading in Microsoft’s Profiler, Take Two. And, this new episode shows the importance of profiling your profiler, if only to see whether time is being spent in dbghelp.dll, msdia120.dll, or somewhere else.
Now we return you to your regularly scheduled blog post.
Symbol loading is handled by two DLLs – symsrv.dll (which retrieves PDB files from your symbol server) and dbghelp.dll (which scans through the PDBs). Unfortunately, the xperf toolkit no longer ships with these DLLs. That means that some people will be able to load symbols, and some people won’t – it all depends on what happens to be in your path.
With xperfview it could be very hard to diagnose symbol loading problems. WPA has a Diagnostic Console that gives you some hope of figuring out what is going on. It was the diagnostic console that first told me that my colleague couldn’t load symbols because he didn’t have symsrv.dll, and it is the diagnostic console that tells you what symbols are really being loaded – ignore the inaccurate progress meter.
Foreshadowing: newer is not better
The obvious place to get symsrv.dll and dbghelp.dll is from the latest version of windbg. After all, newer is shinier, and shinier is better, right? And doesn’t it make sense to get symsrv.dll and dbghelp.dll from the same Windows 8 SDK that you got the xperf toolkit from?
The way xperf toolkit uses PDBs is very different from how a debugger uses PDBs. Xperf (whether you are using xperfview or WPA or GPUView) scans through all of the symbols in the PDBs and stores the transcoded results in .symcache files in the directory specified by the _NT_SYMCACHE_PATH environment variable. This is great because these transcoded .symcache files load really quickly, and they are archived in a compact format so that if you trace a temporary build you can still analyze it a few days later when the PDB files are long gone. So far so good.
However, for some reason the most recent versions of dbghelp.dll do this scanning extremely slowly. One of our PDBs took 2.5 hours to load. That’s 150 minutes. I’m a patient man, but if symbols don’t load while I’m making coffee then I get annoyed.
Yes, I recorded a trace of xperfview analyzing a trace. How recursive. But not very helpful. The trace (once its symbols loaded) showed me that symbol decoding is single threaded (boo) and CPU bound in dbghelp.dll, but it did not directly lead to a solution.
When I found a PDB which took 150 minutes to transcode I used it as a test case. I deleted the associated .symcache file so that it would have to be transcoded again and then I switched to using an older version of dbghelp.dll
It was faster. A lot faster. It transcoded the file in one minute. For those of you watching from home that is:
2.5 hours * 60 min/hr / 1.0 minutes = 150 hr*min/hr*min = 150
If I’ve done my math correctly then the old version of dbghelp.dll was 150 times faster, which in metric is ‘awesome’. It makes the difference between xperf being usable, and not.
DLL hell, solved
I recommend making an install image for the xperf toolkit. Put it on a file server and run everything from there so that xperf can be easily updated. And in that install image put a copy of dbghelp.dll and symsrv.dll. Make sure they match the bitness of the xperf toolkit – which should mean 64-bit versions, since all right-thinking developers are running 64-bit Windows and 64-bit xperf.
You can get old copies of dbghelp.dll and symsrv.dll from old installs of windbg, or from Visual Studio 2010.
February 2010 was a good month.