Symbols the Microsoft Way

Symbol servers allow developer tools on Windows to automatically find symbols. They do this so well that most developers never have to worry about the internal mechanisms. However when things go wrong it can be helpful to understand how they work, and it turns out that it is all very simple.

This article should serve as a good comparison to the process of getting symbols, especially for crashes on customer machines, for Linux. I documented that process in a four-part series:

• Symbols on Linux Part One: g++ Library Symbols – learning how to get symbols for my Linux machine
• Symbols on Linux Part Two: Symbols for Other Versions – struggling to get symbols for customer machines
• Symbols on Linux Part Three: Linux versus Windows – the promise of build IDs
• Symbols on Linux update: Fedora Fixes – build ID progress

My discussion of Windows symbol servers make use of the symbol server that I have on my laptop, for my own personal projects. Whenever I release a new version of Fractal eXtreme (64-bit optimized, multi-core, fast and fluid exploration of fractals, demo version here) I put the symbols and binaries on my symbol server so that I can trivially investigate any crash reports that I receive. This may seem like overkill for a home project, but in fact a local symbol server is just a copy of the files, arranged in a specific way for easy retrieval, and it is trivial to set up.

Finding PE files

Symbol servers store not just symbols but also PE files (DLLs and EXEs). If these aren’t already available (they will be absent when looking at a minidump or a xperf profile) then they must be retrieved first, before the symbols.

There are three pieces of information that are needed in order to retrieve a PE file from a symbol server: the file name, link time stamp, and the image size. In order to see if the latest version of FractalX.exe made it into my symbol server I would extract the link time stamp and the image size from the executable like this:

dumpbin FractalX.exe /headers | find “date stamp”
        4FFD0109 time date stamp Tue Jul 10 21:28:57 2012
dumpbin FractalX.exe /headers | find “size of image”
          147000 size of image

The format for the path to a PE file in a symbol server share is:

“%s\%s\%s%s\%s” % (serverName, peName, timeStamp, imageSize, peName)

My symbol server is in c:\MySyms (normally it would be on a shared server, but this is my personal laptop) so the full path for the file examined above is:

c:\MySyms\fractalx.exe\4FFD0109147000\FractalX.exe

Simple enough. In my case I use symstore.exe’s /compress option (it saves a lot of space) when I add the files. Compressed files are indicated by replacing the last character with an underscore, so the actual path is this:

c:\MySyms\fractalx.exe\4FFD0109147000\FractalX.ex_

This is a good test to make sure that your PE files have been correctly added to your symbol server but it’s not a very realistic use case since we used the PE file to obtain the values needed to retrieve the PE file. The more common scenario is that you would have a minidump or an xperf ETL file and this file would contain a series of module name, link time stamp, image size triplets and these would be used at analysis time to retrieve the PE files. In the case of minidump files there is an array of MINIDUMP_MODULE structures which contain the relevant data.

Note that the layout of symbol server shares can be much more complex. You should use the APIs (discussed later) to retrieve PE files – the technique above is purely for troubleshooting.

Finding PDB files

Finding PE files is handy when analyzing customer crash dumps in order to have the assembly instructions but it’s actually more important than that. Minidump files and most profile files do not actually record enough information to retrieve PDB files. Instead the tools retrieve the PE files and then look in the PE files to get the information needed to retrieve the PDB files. Once again we can extract this information from a PE file using dumpbin:

dumpbin FractalX.exe /headers | find “Format:”
    4FFD0109 cv           56 000B9308    B7B08    Format: RSDS, {6143E0D1-9975-4456-AC8E-F24C8777336D}, 1, FractalX.pdb

The long hexadecimal number after RSDS is a GUID, and the number after that (a 32-bit hexadecimal number, but in this case just ‘1’) is called the ‘age’. The PDB file name is also listed here. Together these uniquely identify a particular version of a PDB file. The format for the path to a PDB file in a symbol server share is:

“%s\%s\%s%s\%s” % (serverPath, pdbName, guid, age, pdbName)

As with the PE files a final underscore indicates when a file is compressed by symstore.exe. The path on my symbol server for the PDB listed above looks like this:

c:\MySyms\FractalX.pdb\6143E0D199754456AC8EF24C8777336D1\FractalX.pd_

Simple enough.

The algorithm for generating the GUID and age is that whenever you do a rebuild – whenever a fresh PDB is generated – a new GUID is created and the age is set to one. Whenever you do an incremental build the PDB is updated with new debug information and the age is incremented.

That’s it – use the PE name, link time stamp, and image size to find the PE (if it isn’t already loaded) and then use the GUID, age, and PDB file name to find the PDB file.

Note that the layout of symbol server shares can be much more complex. You should use the APIs (discussed later) to retrieve PDB files – the technique above is purely for troubleshooting.

Adding to a symbol server

If you ship software on Windows then you should have a symbol server. That symbol server should contain the PE files and PDB files for every product you ship. If you don’t do this then you are doing either yourself or your customers a disservice.

You should also have a symbol server for all internal builds that anybody at the company might end up running. If a program might crash, and if you want to be able to investigate the crash then put the symbols on the symbol server. If you’re worried about internal builds using up too much space then put them on a separate symbol server and purge the old files occasionally.

You should also make sure that your build machines are running source indexing so that when you’re debugging a crash in an old version of your software you will automatically get the right source files. Luckily I wrote about that already.

Adding files to a symbol server is the height of simplicity. Set sourcedir to point at a directory containing files to add and set dest to your symbol server directory, which should be accessible to all who need the symbols. Then run these commands:

symstore add /f %sourcedir%\*.dll /s %dest% /t prodname /compress
symstore add /f %sourcedir%\*.exe /s %dest% /t prodname /compress
symstore add /f %sourcedir%\*.pdb /s %dest% /t prodname /compress

That’s it. Use /r if you want the files recursively added, and see the help for more information.

You can download the compress.exe program if you want to compress an existing symbol server – we did this at work and saved many TB of space.

http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=17657

Update: a reader followed my recommendation of using compress.exe and found that it is dangerously unreliable. He shared his tests with me and I confirmed that in some cases the files created by compress.exe are corrupted. They also don’t compress as well as using the /compress option of symstore.exe. If you do use compress.exe use -ZX as the compression option as this produces the smallest files and appears to avoid the corruption problem. But be careful. Another alternative is to extract .pdb files from your existing symbol server and then resubmit them with symstore.exe /compress.

You can also make your symbol server available through http if you want, but I know nothing about how to set this up.

Using a symbol server

The precise details of how to get your development tools to use your symbol server vary, but one almost universal method is to set the _NT_SYMBOL_PATH environment variable (advanced usage here and here), to something like this:

_NT_SYMBOL_PATH=SRV*c:\symbols*c:\MySyms;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

This tells tools to first look in the local cache (c:\symbols) and then look in the symbol server c:\MySyms. If symbols are found in c:\MySyms then they are copied (and decompressed) to c:\symbols. If none of that works then the same process (including the same cache directory) is followed for Microsoft’s web based symbol cache.

Note that a local symbol cache is required when dealing with compressed symbols.

Programmatically retrieving symbols

Usually the debuggers and profilers that you use will know how to use symbol servers, but occasionally you may need to write code to download symbols – perhaps you are writing a debugger or profiler. In my case I had a web page that listed GUIDs, ages, and PDB names for dozens of Microsoft DLLs from dozens of versions of Windows for which we needed symbols. Writing code to download all of these symbols was trivial – several orders of magnitude easier than getting symbols for other versions of Linux.

The short explanation of what I needed to do was “call SymFindFileInPath”.

In order to demonstrate how easy it was I decided to give a slightly longer explanation. The code below takes a GUID, age, and pdb name and downloads the symbols from Microsoft’s symbol server. The biggest chunk of code is for parsing the GUID – the actual PDB downloading is trivial.

// Symbol downloading demonstration code.
// No warranty, retain this notice, no other restrictions.
// For more information see:
// http://randomascii.wordpress.com/2013/03/09/symbols-the-microsoft-way/

#include <Windows.h>
#include <DbgHelp.h>
#include <string>

// Link with the dbghelp import library
#pragma comment(lib, “dbghelp.lib”)

#define TESTING

int main(int argc, char* argv[])
{
    // Tell dbghelp to print diagnostics to the debugger output.
    SymSetOptions(SYMOPT_DEBUG);

    // Initialize dbghelp
    const HANDLE fakeProcess = (HANDLE)1;
    BOOL result = SymInitialize(fakeProcess, NULL, FALSE);

    // Set a search path and cache directory. If this isn’t set
    // then _NT_SYMBOL_PATH will be used instead.
    SymSetSearchPath(fakeProcess, “SRV*c:\\symbolstest*http://msdl.microsoft.com/download/symbols”);

#ifdef TESTING
    // Valid PDB data to test the code.
    std::string gText = “072FF0EB54D24DFAAE9D13885486EE09″;
    const char* ageText = “2″;
    const char* pdbName = “kernel32.pdb”;
#else
    #error Fill in code to get filename, GUID, and age
#endif

    // Parse the GUID and age from the text
    GUID g;
    int count = sscanf(gText.substr(0, 8).c_str(), “%x”, &g.Data1);
    DWORD temp;
    count += sscanf(gText.substr(8, 4).c_str(), “%x”, &temp);
    g.Data2 = (unsigned short)temp;
    count += sscanf(gText.substr(12, 4).c_str(), “%x”, &temp);
    g.Data3 = (unsigned short)temp;
    for (auto i = 0; i < ARRAYSIZE(g.Data4); ++i)
    {
        count += sscanf(gText.substr(16 + i * 2, 2).c_str(), “%x”, &temp);
        g.Data4[i] = (unsigned char)temp;
    }
    DWORD age = 0;
    count += sscanf(ageText, “%x”, &age);

    if (count != 12)
    {
        printf(“Couldn’t parse the GUID/age string. Sorry.\n”);
        return 10;
    }

    char filePath[MAX_PATH] = {};
    void* id = &g;
    DWORD two = age;
    DWORD three = 0;
    DWORD flags = SSRVOPT_GUIDPTR;
    if (SymFindFileInPath(fakeProcess, NULL, pdbName, id, two, three,
                flags, filePath, NULL, NULL))
    {
        printf(“Found symbol file – placed it in %s.\n”, filePath);
    }
    else
    {
        printf(“Symbols not found – error %u. Are dbghelp.dll and “
                “symsrv.dll in the same directory as this executable?\n”,
                GetLastError());
    }

    SymCleanup(fakeProcess);

    return 0;
}

The one gotcha is that dbghelp.dll and symsrv.dll have to be in the same directory as your tool – having them in your path does not work reliably.

Diagnosing symbol problems with windbg

If you have a minidump and its symbols are not loading then I recommend loading the minidump into windbg and using its diagnostics:

• !sym noisy – print verbose information about attempts to get symbols
• lmv m MyModule – print a record from the crash dump’s module list including its name, time stamp, image size, and where the PDB is located if it was found
• !lmi MyModule – print a module’s header information – this only works if the PE file has been loaded, which is a prerequisite for having symbols load

Dumpbin summary

• “%VS100COMNTOOLS%..\..\VC\vcvarsall.bat” – this adds dumpbin’s directory to the path
• dumpbin FX.exe /headers | find “date stamp” – find the link time stamp of a PE file
  
• dumpbin FX.exe /headers | find “size of image” – find the image size of a PE file
• dumpbin FractalX.exe /headers | find “Format:” – find the GUID, age, and file name of a PE file’s PDB file

About these ads