Xperf Symbol Loading Pitfalls

Analyzing xperf traces is virtually impossible if you don’t have symbols. Unfortunately the most recent version of the xperf toolkit makes it easy to end up with symbols that takes hours to load, or don’t load at all.

After two people contacted me the same day (one asking why symbol loading didn’t work, the other asking why it was too slow) I decided it was time to document this issue.

DLL hell

Symbol loading is handled by two DLLs – symsrv.dll (which retrieves PDB files from your symbol server) and dbghelp.dll (which scans through the PDBs). Unfortunately, the xperf toolkit no longer ships with these DLLs. That means that some people will be able to load symbols, and some people won’t – it all depends on what happens to be in your path.

imageWith xperfview it could be very hard to diagnose symbol loading problems. WPA has a Diagnostic Console that gives you some hope of figuring out what is going on. It was the diagnostic console that first told me that my colleague couldn’t load symbols because he didn’t have symsrv.dll, and it is the diagnostic console that tells you what symbols are really being loaded – ignore the inaccurate progress meter.

Foreshadowing: newer is not better

The obvious place to get symsrv.dll and dbghelp.dll is from the latest version of windbg. After all, newer is shinier, and shinier is better, right? And doesn’t it make sense to get symsrv.dll and dbghelp.dll from the same Windows 8 SDK that you got the xperf toolkit from?

Bad idea.

The way xperf toolkit uses PDBs is very different from how a debugger uses PDBs. Xperf (whether you are using xperfview or WPA or GPUView) scans through all of the symbols in the PDBs and stores the transcoded results in .symcache files in the directory specified by the _NT_SYMCACHE_PATH environment variable. This is great because these transcoded .symcache files load really quickly, and they are archived in a compact format so that if you trace a temporary build you can still analyze it a few days later when the PDB files are long gone. So far so good.

imageHowever, for some reason the most recent versions of dbghelp.dll do this scanning extremely slowly. One of our PDBs took 2.5 hours to load. That’s 150 minutes. I’m a patient man, but if symbols don’t load while I’m making coffee then I get annoyed.

Yes, I recorded a trace of xperfview analyzing a trace. How recursive. But not very helpful. The trace (once its symbols loaded) showed me that symbol decoding is single threaded (boo) and CPU bound in dbghelp.dll, but it did not directly lead to a solution.

When I found a PDB which took 150 minutes to transcode I used it as a test case. I deleted the associated .symcache file so that it would have to be transcoded again and then I switched to using an older version of dbghelp.dll

It was faster. A lot faster. It transcoded the file in one minute. For those of you watching from home that is:

2.5 hours * 60 min/hr / 1.0 minutes = 150 hr*min/hr*min = 150

If I’ve done my math correctly then the old version of dbghelp.dll was 150 times faster, which in metric is ‘awesome’. It makes the difference between xperf being usable, and not.

DLL hell, solved

I recommend making an install image for the xperf toolkit. Put it on a file server and run everything from there so that xperf can be easily updated. And in that install image put a copy of dbghelp.dll and symsrv.dll. Make sure they match the bitness of the xperf toolkit – which should mean 64-bit versions, since all right-thinking developers are running 64-bit Windows and 64-bit xperf.

You can get old copies of dbghelp.dll and symsrv.dll from old installs of windbg, or from Visual Studio 2010.

February 2010 was a good month.

About these ads

About brucedawson

I'm a programmer, working for Valve (http://www.valvesoftware.com/), focusing on optimization and reliability. Nothing's more fun than making code run 5x faster. Unless it's eliminating large numbers of bugs. I also unicycle. And play (ice) hockey. And juggle.
This entry was posted in Symbols, xperf and tagged , , , , . Bookmark the permalink.

20 Responses to Xperf Symbol Loading Pitfalls

  1. Kyle Sabo says:

    The new version of XPerf for Windows 8 still ships dbghelp.dll and symsrv.dll – they get installed to C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64. Copying installed software from one machine to another without using the installer is always asking for trouble. The ADK installer even has an option to download an individual .msi for XPerf exactly so you don’t have to do this.

    It’s also worth noting that the .symcache files generated from the two versions of dbghelp.dll are different. For better or worse, you’ll have to figure out, but it’s worth keeping in the back of your mind.

    • brucedawson says:

      If you run the MSI for xperf (WPTx64-x86_en-us.msi) then you don’t get dbghelp.dll and symsrv.dll. You are then forced to scrounge them from some other source. I know this because I ran the MSI on other people’s machines, and often they couldn’t load symbols. True story. You are getting a version from the Debugging Tools for Windows install, which is often butnot necessarily installed with xperf.

      If you make the mistake of using the DLLs from the 8.0\Debuggers directory then you will be sad. Xperf symbol loading will be unusably slow.

      Copying dbghelp.dll and symsrv.dll from other locations may be dodgy and dangerous, but it has one redeeming quality:

      It works.

    • Kevin Gadd says:

      In my case, I installed xperf/gpuview (via the windows SDK) and ended up without the ability to load symbols at all. Copying the DLLs from VS2010 fixed it.

      It might be ‘asking for trouble’, but I’d rather have trouble than not have working symbols :)

  2. Pingback: xperf – Show me the performance! | dickyjim

  3. tedmielczarek says:

    Sometimes it’s depressing that Microsoft can make such great developer tools and then make them nigh-unusable like this.

  4. Pingback: Symbols on Linux Part Three: Linux versus Windows | Random ASCII

  5. Pingback: Xperf Basics: Recording a Trace (the easy way) | Random ASCII

  6. Patrik Osgnach says:

    Just to make things clear, am I supposed to copy the two dlls from C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools to C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit ?

    • brucedawson says:

      You need the 64-bit versions (assuming you are running 64-bit Windows) so you should copy from “C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Remote Debugger\x64″, Don’t copy from “Performance Tools\x64″ because that is the 32-bit version.

      You can look at the Diagnostic Console to see if things seem to be running smoothly, or use something like procexp to see what DLLs loaded.

  7. Alexander Safronov says:

    Hello Bruce!
    Did you ever encounter an extremely weird problem with “-heap” and “-stackwalk HeapAlloc+HeapRealloc”, where if you use “-pidNewProcess”, then stacks are not being shown in xperfview, while using “-pids PID” options works fine, all other things being absolutely equal. This was driving me up the wall last few days – I tried different versions for xperf toolkit in several different environments, but the difference still happens (with a side note that SDK 8.0 version of xperf absolutely sucks when displaying stacks in the summary table, just hangs forever if the trace is very big. SDK 7.1 version is way better)

    • brucedawson says:

      I use -Pids 0 and use Image File Execution Options to enable heap tracing for a particular executable. This works for me so I haven’t tried anything else. This is on Windows 7.

      I rarely use xperfview anymore — I use wpa for trace analysis. The 64-bit version on a machine with lots of memory works pretty well. I wouldn’t go back to earlier versions.

      • Alexander Safronov says:

        By “heap tracing” you are referring to “Create user mode stack trace database” in gflags tool? is “-pids 0″ the same thing as omitting the “-pid*” completely?

        Side question – when you are using WPA, do you still have to replace SDK’s dbghelp.dll and symsrv.dll from Visual Studio 2010?

  8. Pingback: The New WPA Xperf Trace Viewer–New Bugs and Old | Random ASCII

  9. Pingback: Why does Explorer hang when I right-click on a folder? | Needle in a Thread Stack

  10. Pingback: New Version of Xperf–Upgrade Now | Random ASCII

  11. Giovanni says:

    Important: Windows 8.1 didn’t show the symbols at all. Upgrading to the newest version of XPerf (included in Windows ADK) solved the problem.

    • brucedawson says:

      I’m glad that worked. If I understand correctly then the DLLs are put in C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64. Whether xperf/wpa uses those versions depends on your path environment variable. If you have any problems with symbols not loading or loading slowly then you have to figure out how to get the right versions to load.

  12. 123456789 says:

    i can see my pdb files in ./symcahe but in summary figure there is also ? in stack why?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s