Xperf Symbol Loading Pitfalls

Analyzing xperf traces is virtually impossible if you don’t have symbols. Unfortunately the xperf toolkit makes it easy to end up with symbols that takes hours to load, or don’t load at all.

Update, July 2020: this blog post is obsolete. The hacks described here are not needed and should not be used.

After two people contacted me the same day (one asking why symbol loading didn’t work, the other asking why it was too slow) I decided it was time to document this issue.

Update, November 2014

It’s still important to copy known-good copies of to the WPT install directory, since otherwise you can’t be sure which DLL versions you will get (if any). The VS 2013 versions seem to be good. But, for some PDBs a new problem has surfaced. It shows up when profiling the Chrome web browser. You can read about this new problem and its solution at Slow Symbol Loading in Microsoft’s Profiler, Take Two. And, this new episode shows the importance of profiling your profiler, if only to see whether time is being spent in dbghelp.dll, msdia120.dll, or somewhere else.

Update, October 2015

UIforETW ships a workaround for this problem. From the UIforETW Settings dialog you just need to click the Copy symbol DLLs button to copy known-good DLLs to the WPT install directory which should avoid this problem. UIforETW also ships with a workaround for the new problem found with Chrome’s symbols – see the link in the pargraph above.

Now we return you to your regularly scheduled blog post.

DLL hell

Symbol loading is handled by two DLLs – symsrv.dll (which retrieves PDB files from your symbol server) and dbghelp.dll (which scans through the PDBs). Unfortunately, the xperf toolkit no longer ships with these DLLs. That means that some people will be able to load symbols, and some people won’t – it all depends on what happens to be in your path.

imageWith xperfview it could be very hard to diagnose symbol loading problems. WPA has a Diagnostic Console that gives you some hope of figuring out what is going on. It was the diagnostic console that first told me that my colleague couldn’t load symbols because he didn’t have symsrv.dll, and it is the diagnostic console that tells you what symbols are really being loaded – ignore the inaccurate progress meter.

Foreshadowing: newer is not better

The obvious place to get symsrv.dll and dbghelp.dll is from the latest version of windbg. After all, newer is shinier, and shinier is better, right? And doesn’t it make sense to get symsrv.dll and dbghelp.dll from the same Windows 8 SDK that you got the xperf toolkit from?

Bad idea.

The way xperf toolkit uses PDBs is very different from how a debugger uses PDBs. Xperf (whether you are using xperfview or WPA or GPUView) scans through all of the symbols in the PDBs and stores the transcoded results in .symcache files in the directory specified by the _NT_SYMCACHE_PATH environment variable. This is great because these transcoded .symcache files load really quickly, and they are archived in a compact format so that if you trace a temporary build you can still analyze it a few days later when the PDB files are long gone. So far so good.

imageHowever, for some reason the most recent versions of dbghelp.dll do this scanning extremely slowly. One of our PDBs took 2.5 hours to load. That’s 150 minutes. I’m a patient man, but if symbols don’t load while I’m making coffee then I get annoyed.

Yes, I recorded a trace of WPA analyzing a trace. How recursive. But not very helpful. The trace (once its symbols loaded) showed me that symbol decoding is single threaded (boo) and CPU bound in dbghelp.dll, but it did not directly lead to a solution.

When I found a PDB which took 150 minutes to transcode I used it as a test case. I deleted the associated .symcache file so that it would have to be transcoded again and then I switched to using an older version of dbghelp.dll

It was faster. A lot faster. It transcoded the file in one minute. For those of you watching from home that is:

2.5 hours * 60 min/hr / 1.0 minutes = 150 hr*min/hr*min = 150

If I’ve done my math correctly then the old version of dbghelp.dll was 150 times faster, which in metric is ‘awesome’. It makes the difference between ETW being usable, and not.

DLL hell, solved

If you use UIforETW then the problem is easily solved by clicking Copy symbol DLLs from the UIforETW Settings dialog.

About brucedawson

I'm a programmer, working for Google, focusing on optimization and reliability. Nothing's more fun than making code run 10x as fast. Unless it's eliminating large numbers of bugs. I also unicycle. And play (ice) hockey. And sled hockey. And juggle. And worry about whether this blog should have been called randomutf-8. 2010s in review tells more: https://twitter.com/BruceDawson0xB/status/1212101533015298048
This entry was posted in Symbols, xperf and tagged , , , , . Bookmark the permalink.

24 Responses to Xperf Symbol Loading Pitfalls

  1. Kyle Sabo says:

    The new version of XPerf for Windows 8 still ships dbghelp.dll and symsrv.dll – they get installed to C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64. Copying installed software from one machine to another without using the installer is always asking for trouble. The ADK installer even has an option to download an individual .msi for XPerf exactly so you don’t have to do this.

    It’s also worth noting that the .symcache files generated from the two versions of dbghelp.dll are different. For better or worse, you’ll have to figure out, but it’s worth keeping in the back of your mind.

    • brucedawson says:

      If you run the MSI for xperf (WPTx64-x86_en-us.msi) then you don’t get dbghelp.dll and symsrv.dll. You are then forced to scrounge them from some other source. I know this because I ran the MSI on other people’s machines, and often they couldn’t load symbols. True story. You are getting a version from the Debugging Tools for Windows install, which is often but not necessarily installed with xperf.

      And, if the Debugging Tools are installed there is no guarantee they will be first in the path.

      Also, if you make the mistake of using the DLLs from the 8.0\Debuggers directory then you will be sad. Xperf symbol loading will be unusably slow.

      Copying dbghelp.dll and symsrv.dll from other locations may be dodgy and dangerous, but it has one redeeming quality:

      It works.

    • Kevin Gadd says:

      In my case, I installed xperf/gpuview (via the windows SDK) and ended up without the ability to load symbols at all. Copying the DLLs from VS2010 fixed it.

      It might be ‘asking for trouble’, but I’d rather have trouble than not have working symbols 🙂

  2. tedmielczarek says:

    Sometimes it’s depressing that Microsoft can make such great developer tools and then make them nigh-unusable like this.

  3. Pingback: Symbols on Linux Part Three: Linux versus Windows | Random ASCII

  4. Pingback: Xperf Basics: Recording a Trace (the easy way) | Random ASCII

  5. Patrik Osgnach says:

    Just to make things clear, am I supposed to copy the two dlls from C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools to C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit ?

    • brucedawson says:

      You need the 64-bit versions (assuming you are running 64-bit Windows) so you should copy from “C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Remote Debugger\x64”, Don’t copy from “Performance Tools\x64” because that is the 32-bit version.

      You can look at the Diagnostic Console to see if things seem to be running smoothly, or use something like procexp to see what DLLs loaded.

  6. Alexander Safronov says:

    Hello Bruce!
    Did you ever encounter an extremely weird problem with “-heap” and “-stackwalk HeapAlloc+HeapRealloc”, where if you use “-pidNewProcess”, then stacks are not being shown in xperfview, while using “-pids PID” options works fine, all other things being absolutely equal. This was driving me up the wall last few days – I tried different versions for xperf toolkit in several different environments, but the difference still happens (with a side note that SDK 8.0 version of xperf absolutely sucks when displaying stacks in the summary table, just hangs forever if the trace is very big. SDK 7.1 version is way better)

    • brucedawson says:

      I use -Pids 0 and use Image File Execution Options to enable heap tracing for a particular executable. This works for me so I haven’t tried anything else. This is on Windows 7.

      I rarely use xperfview anymore — I use wpa for trace analysis. The 64-bit version on a machine with lots of memory works pretty well. I wouldn’t go back to earlier versions.

      • Alexander Safronov says:

        By “heap tracing” you are referring to “Create user mode stack trace database” in gflags tool? is “-pids 0” the same thing as omitting the “-pid*” completely?

        Side question – when you are using WPA, do you still have to replace SDK’s dbghelp.dll and symsrv.dll from Visual Studio 2010?

  7. Pingback: The New WPA Xperf Trace Viewer–New Bugs and Old | Random ASCII

  8. Pingback: New Version of Xperf–Upgrade Now | Random ASCII

  9. Giovanni says:

    Important: Windows 8.1 didn’t show the symbols at all. Upgrading to the newest version of XPerf (included in Windows ADK) solved the problem.

    • brucedawson says:

      I’m glad that worked. If I understand correctly then the DLLs are put in C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64. Whether xperf/wpa uses those versions depends on your path environment variable. If you have any problems with symbols not loading or loading slowly then you have to figure out how to get the right versions to load.

  10. 123456789 says:

    i can see my pdb files in ./symcahe but in summary figure there is also ? in stack why?

  11. Pingback: Slow Symbol Loading in Microsoft’s Profiler, Take Two | Random ASCII

  12. Pingback: Profiling the profiler: working around a six minute xperf hang | Random ASCII

  13. Pingback: UIforETW – Windows Performance Made Easier | Random ASCII

  14. Pingback: ETW Central | Random ASCII

  15. rob18767 says:

    I got a copy of the files for UIforETW .

    I ran UIforETW but saw no no copy symbol DLLS from the settings menu.

    From the bin directory I copied the symbol dlls in there to

    C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit

    I had en .etl I captured with xperf so I ran WPA, opened the etl and then could not find (as instructed in the “Getting Started” section) the option from the File menu to Configure Symbols.

    The ‘load symbols’ and ‘configure symbol paths’ from the Trace menu were grayed out.

    Any idea?

    • brucedawson says:

      This blog post is eight years old now and the DLLs involved have been changed many times in-between. I just updated the post to deprecate the hacks that were once useful. That’s why the Copy Symbol DLLs button is gone.

      You should undo the customizations that you made (you may need to uninstall/reinstall, or recopy files from another machine) and see if that fixes the Load Symbols and Configure Symbol Paths options in the Trace menu. I have not seen them grayed out before.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.