Analyzing xperf traces is virtually impossible if you don’t have symbols. Unfortunately the xperf toolkit makes it easy to end up with symbols that takes hours to load, or don’t load at all.
Update, July 2020: this blog post is obsolete. The hacks described here are not needed and should not be used.
After two people contacted me the same day (one asking why symbol loading didn’t work, the other asking why it was too slow) I decided it was time to document this issue.
Update, November 2014
It’s still important to copy known-good copies of to the WPT install directory, since otherwise you can’t be sure which DLL versions you will get (if any). The VS 2013 versions seem to be good. But, for some PDBs a new problem has surfaced. It shows up when profiling the Chrome web browser. You can read about this new problem and its solution at Slow Symbol Loading in Microsoft’s Profiler, Take Two. And, this new episode shows the importance of profiling your profiler, if only to see whether time is being spent in dbghelp.dll, msdia120.dll, or somewhere else.
Update, October 2015
UIforETW ships a workaround for this problem. From the UIforETW Settings dialog you just need to click the Copy symbol DLLs button to copy known-good DLLs to the WPT install directory which should avoid this problem. UIforETW also ships with a workaround for the new problem found with Chrome’s symbols – see the link in the pargraph above.
Now we return you to your regularly scheduled blog post.
DLL hell
Symbol loading is handled by two DLLs – symsrv.dll (which retrieves PDB files from your symbol server) and dbghelp.dll (which scans through the PDBs). Unfortunately, the xperf toolkit no longer ships with these DLLs. That means that some people will be able to load symbols, and some people won’t – it all depends on what happens to be in your path.
With xperfview it could be very hard to diagnose symbol loading problems. WPA has a Diagnostic Console that gives you some hope of figuring out what is going on. It was the diagnostic console that first told me that my colleague couldn’t load symbols because he didn’t have symsrv.dll, and it is the diagnostic console that tells you what symbols are really being loaded – ignore the inaccurate progress meter.
Foreshadowing: newer is not better
The obvious place to get symsrv.dll and dbghelp.dll is from the latest version of windbg. After all, newer is shinier, and shinier is better, right? And doesn’t it make sense to get symsrv.dll and dbghelp.dll from the same Windows 8 SDK that you got the xperf toolkit from?
Bad idea.
The way xperf toolkit uses PDBs is very different from how a debugger uses PDBs. Xperf (whether you are using xperfview or WPA or GPUView) scans through all of the symbols in the PDBs and stores the transcoded results in .symcache files in the directory specified by the _NT_SYMCACHE_PATH environment variable. This is great because these transcoded .symcache files load really quickly, and they are archived in a compact format so that if you trace a temporary build you can still analyze it a few days later when the PDB files are long gone. So far so good.
However, for some reason the most recent versions of dbghelp.dll do this scanning extremely slowly. One of our PDBs took 2.5 hours to load. That’s 150 minutes. I’m a patient man, but if symbols don’t load while I’m making coffee then I get annoyed.
Yes, I recorded a trace of WPA analyzing a trace. How recursive. But not very helpful. The trace (once its symbols loaded) showed me that symbol decoding is single threaded (boo) and CPU bound in dbghelp.dll, but it did not directly lead to a solution.
When I found a PDB which took 150 minutes to transcode I used it as a test case. I deleted the associated .symcache file so that it would have to be transcoded again and then I switched to using an older version of dbghelp.dll
It was faster. A lot faster. It transcoded the file in one minute. For those of you watching from home that is:
2.5 hours * 60 min/hr / 1.0 minutes = 150 hr*min/hr*min = 150
If I’ve done my math correctly then the old version of dbghelp.dll was 150 times faster, which in metric is ‘awesome’. It makes the difference between ETW being usable, and not.
DLL hell, solved
If you use UIforETW then the problem is easily solved by clicking Copy symbol DLLs from the UIforETW Settings dialog.
Random ASCII - tech blog of Bruce Dawson 
The new version of XPerf for Windows 8 still ships dbghelp.dll and symsrv.dll – they get installed to C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64. Copying installed software from one machine to another without using the installer is always asking for trouble. The ADK installer even has an option to download an individual .msi for XPerf exactly so you don’t have to do this.
It’s also worth noting that the .symcache files generated from the two versions of dbghelp.dll are different. For better or worse, you’ll have to figure out, but it’s worth keeping in the back of your mind.
If you run the MSI for xperf (WPTx64-x86_en-us.msi) then you don’t get dbghelp.dll and symsrv.dll. You are then forced to scrounge them from some other source. I know this because I ran the MSI on other people’s machines, and often they couldn’t load symbols. True story. You are getting a version from the Debugging Tools for Windows install, which is often but not necessarily installed with xperf.
And, if the Debugging Tools are installed there is no guarantee they will be first in the path.
Also, if you make the mistake of using the DLLs from the 8.0\Debuggers directory then you will be sad. Xperf symbol loading will be unusably slow.
Copying dbghelp.dll and symsrv.dll from other locations may be dodgy and dangerous, but it has one redeeming quality:
It works.
In my case, I installed xperf/gpuview (via the windows SDK) and ended up without the ability to load symbols at all. Copying the DLLs from VS2010 fixed it.
It might be ‘asking for trouble’, but I’d rather have trouble than not have working symbols 🙂
Sometimes it’s depressing that Microsoft can make such great developer tools and then make them nigh-unusable like this.
Pingback: Symbols on Linux Part Three: Linux versus Windows | Random ASCII
Pingback: Xperf Basics: Recording a Trace (the easy way) | Random ASCII
Just to make things clear, am I supposed to copy the two dlls from C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools to C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit ?
You need the 64-bit versions (assuming you are running 64-bit Windows) so you should copy from “C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Remote Debugger\x64”, Don’t copy from “Performance Tools\x64” because that is the 32-bit version.
You can look at the Diagnostic Console to see if things seem to be running smoothly, or use something like procexp to see what DLLs loaded.
Ok, thank you
Hello Bruce!
Did you ever encounter an extremely weird problem with “-heap” and “-stackwalk HeapAlloc+HeapRealloc”, where if you use “-pidNewProcess”, then stacks are not being shown in xperfview, while using “-pids PID” options works fine, all other things being absolutely equal. This was driving me up the wall last few days – I tried different versions for xperf toolkit in several different environments, but the difference still happens (with a side note that SDK 8.0 version of xperf absolutely sucks when displaying stacks in the summary table, just hangs forever if the trace is very big. SDK 7.1 version is way better)
I use -Pids 0 and use Image File Execution Options to enable heap tracing for a particular executable. This works for me so I haven’t tried anything else. This is on Windows 7.
I rarely use xperfview anymore — I use wpa for trace analysis. The 64-bit version on a machine with lots of memory works pretty well. I wouldn’t go back to earlier versions.
By “heap tracing” you are referring to “Create user mode stack trace database” in gflags tool? is “-pids 0” the same thing as omitting the “-pid*” completely?
Side question – when you are using WPA, do you still have to replace SDK’s dbghelp.dll and symsrv.dll from Visual Studio 2010?
By heap tracing I’m referring to ETW heap tracing, not gflags.
I think -pids 0 is required when using the Image File Execution Options to record a heap trace. See how UIforETW does it – https://github.com/google/UIforETW/releases – or read my documentation here: https://randomascii.wordpress.com/2015/04/27/etw-heap-tracingevery-allocation-recorded/
Yes, copying the VS 2010 versions of dbghelp.dll and symsrv.dll is necessary for WPA as well. WPA and xperfview use exactly the same symbol decoding pipeline so they suffer exactly the same horrible slowdowns with the latest/greatest symbol DLLs.
(note that copying the DLLs is now a bad idea and will break WPA – updated November 2018)
Pingback: The New WPA Xperf Trace Viewer–New Bugs and Old | Random ASCII
Pingback: New Version of Xperf–Upgrade Now | Random ASCII
Important: Windows 8.1 didn’t show the symbols at all. Upgrading to the newest version of XPerf (included in Windows ADK) solved the problem.
I’m glad that worked. If I understand correctly then the DLLs are put in C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64. Whether xperf/wpa uses those versions depends on your path environment variable. If you have any problems with symbols not loading or loading slowly then you have to figure out how to get the right versions to load.
i can see my pdb files in ./symcahe but in summary figure there is also ? in stack why?
Pingback: Slow Symbol Loading in Microsoft’s Profiler, Take Two | Random ASCII
Pingback: Profiling the profiler: working around a six minute xperf hang | Random ASCII
Pingback: UIforETW – Windows Performance Made Easier | Random ASCII
Pingback: ETW Central | Random ASCII
I got a copy of the files for UIforETW .
I ran UIforETW but saw no no copy symbol DLLS from the settings menu.
From the bin directory I copied the symbol dlls in there to
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit
I had en .etl I captured with xperf so I ran WPA, opened the etl and then could not find (as instructed in the “Getting Started” section) the option from the File menu to Configure Symbols.
The ‘load symbols’ and ‘configure symbol paths’ from the Trace menu were grayed out.
Any idea?
This blog post is eight years old now and the DLLs involved have been changed many times in-between. I just updated the post to deprecate the hacks that were once useful. That’s why the Copy Symbol DLLs button is gone.
You should undo the customizations that you made (you may need to uninstall/reinstall, or recopy files from another machine) and see if that fixes the Load Symbols and Configure Symbol Paths options in the Trace menu. I have not seen them grayed out before.