WPA Symbol Loading is Much Faster, but Broken for Chrome

There’s good news, and there’s bad news.

The good news is that the latest Windows Performance Analyzer (WPA), the visualization tool for ETW (Event Tracing for Windows) traces, can now load symbols faster than ever before – it’s multi-threaded, and it scans huge PDBs about eight times faster.

The bad news is that it utterly fails to download Chrome’s symbols.

Update: the Creators Update version of WPA fixes this bug and the latest UIforETW automatically installs it.

Oops. Luckily I was able to diagnose and work around the problem.

I first reported problems with slow symbol loading in 2012 when I noticed that the newest versions of dbghelp.dll took two and a half hours to load some symbol files into WPA. For a while I hacked around this problem by using an older version of dbghelp.dll (150 times faster!), and this bug was eventually fixed.

Then, in 2014, I hit a problem where the large symbol files created for Chrome’s large DLLs were giving WPA heartburn – it was taking about 25 minutes to load these symbols, even with the ‘fast’ version of dbghelp.dll. I hacked around this problem by writing a script (StripChromeSymbols.py) which downloaded Chrome’s symbols, used pdbcopy to strip the ‘private’ data from the symbol files, and then got xperf to generate WPA-compatible .symcache files using the (much smaller) stripped symbols. UIforETW automatically uses this script if the Chrome developer setting is checked, saving about 23 minutes each time I profile a new version of Chrome.

Loading symbols into WPA is a multi-step process. First WPA needs to find the PDB file, downloading it from a symbol server if necessary. Then it translates the PDB to a smaller and more efficient “.symcache” format, stored in c:\symcache, that is sufficient for WPA’s purposes. Then it loads the symbols from the .symcache files. The translation step is where I’ve seen slowdowns and problems in the past. Luckily this step is only needed the first time a particular PDB is seen. However, if you are a developer who is frequently profiling new builds the translation step is needed often.

So far so good. Then, when the 10.0.14393 (Windows 10 Anniversary Edition) version of WPA came out I tested to see whether these old hacks were still needed. The good news is that WPA can now translate Chrome’s full symbols from PDBs to symcache files much faster – about three minutes instead of 25 minutes.

But the bad news is that WPA cannot download symbols from Chrome’s symbol server.

I didn’t notice this at first because StripChromeSymbols.py was automatically downloading and translating Chrome’s two biggest symbol files, so WPA didn’t need to download those files. But eventually I noticed that symbols for chrome.exe and some other Chrome specific DLLs were always missing in WPA. I investigated by using Fiddler to monitor the symbol downloads. I saw URLs like this when WPA tried downloading Chrome’s PDBs from Google’s symbol server:

https://chromium-browser-symsrv.commondatastorage.googleapis.com/chrome.exe.pdb/0AE31EAB5FE04EBC906260D62B7FC1FB1//chrome.exe.pd_

That URL is a bit of a mouthful but if we simplify the server address and the hexadecimal string that identifies which PDB is being downloaded then we get this:

https://<serveraddress>/chrome.exe.pdb/<hexgoop>//chrome.exe.pd_

Now it is easier to see the problem, which is that there are two slashes in front of chrome.exe.pd_, and there is only supposed to be one. That seemed suspicious so I then modified Fiddler’s OnBeforeRequest function (thanks for the suggestion Eric!) to fix the URL and see if it helped:

static function OnBeforeRequest(oSession: Session)
{
// Fix double-slashes that WPA unintentionally uses when downloading
// symbols. This fix lets WPA download Chrome’s symbols.
if (oSession.uriContains(“http://chromium-browser-symsrv.commondatastorage.googleapis.com”)) {
oSession.url = oSession.url.Replace(“//”, “/”);

With this change, and with Fiddler running, WPA was able to download the missing symbols.

So, the bug is that WPA creates incorrect symbol download URLs, and Google storage refuses to accept them. What is a Chrome developer to do?

It’s times like this when I am really glad that I wrote UIforETW. It gives me a place to put in workarounds for bugs like this. I had been hoping to retire StripChromeSymbols.py but instead I ended up modifying it. Instead of just downloading the two big PDBs it now downloads symbols for all of Chrome’s executables and DLLs. It also translates all of them to .symcache files, although that step is less crucial now.

If you get the latest version of UIforETW then you will get this fix automatically. If you don’t profile Chrome then this bug will probably not affect you, because Microsoft’s symbol server quietly ignores the extra slash. However other symbol servers could be affected, and if so adjustments to the symbol stripping script will be needed.

So, grab the latest version of UIforETW by downloading a pre-built version from here. The release notes explain what other improvements are available.

Note: it turns out that there’s no actual directory structure in Google Cloud Storage file names. The names of the files contain slashes in order to simulate that structure:

https://cloud.google.com/storage/docs/naming#objectnames

We thought about uploading Chrome’s symbols twice – once with the extra slash – but eventually decided that that was an ugly and wasteful hack, and not necessary once the UIforETW fix was in place. The UIforETW hack is also better in some ways because it can work around the problem in traces of old versions of Chrome.

The bug was reported in September 2016 and should be fixed in the next release of WPA.

Meanwhile, if WPA seems to be having problems downloading symbols from Microsoft’s symbol server – well – there have been a lot of outages and slowdowns and missing files on those servers this year. There’s nothing that WPA can do about that – contact windbgfb@microsoft.com if you hit problems with Microsoft’s symbol server.