Creating a Public Symbol Server, Easily

I’ve been a big fan of symbol servers for years. They are a part of the Microsoft/Windows ecosystem that is far better than anything I have seen for other operating systems. With Microsoft’s and Chrome’s symbol servers configured I can download a user’s Chrome crash and start analyzing it immediately – with code bytes and symbols – without knowing or caring what OS or Chrome version that user was running. Since Chrome’s symbols are source-indexed I will even get source-code popping up automatically, and all of this is available to anyone who is interested – no special Google privileges required.

So, symbol servers are magical and wonderful. And yet, I’ve never set up a publicly visible one. I’ve landed some fixes to the symbol servers at my last two jobs, but I’ve never created one from scratch to make symbols to my own tools and samples available.

Since I’m quarantined (to varying degrees) at home in Kirkland I’m spending this sunny Saturday setting up a symbol server and documenting the process. It was really easy.

A symbol server is basically just a file or HTTPS server with the contents arranged in a specific way. There are lots of ways to set up such a server but I decided to use the one that I am most familiar with, the one used by Chrome’s symbol server, which is Google storage. I work at Google, so the obvious bias is obvious, make of that what you will.

Seven simple steps

The first step was to create a Google Cloud account. There’s a free trial that gives you $300 of credit in your first year, so publishing my symbols to the world shouldn’t cost me anything for a while. I plugged a few numbers into the cost estimator (it shows up when you go to create a bucket) and it’s hard to see how it could end up costing me as much as a few cups of coffee a year (class B operations are downloads, 640K should be enough for anyone):

The next step was to create a storage bucket. The only challenging part was choosing a name. Bucket names are globally unique – grab your favorites now. I chose randomascii-symbols. I’m glad that I still like my blog name.

I changed access control to uniform, but this probably wasn’t necessary. Otherwise I took all of the default settings.

It wasn’t entirely obvious how to make the symbols world-readable but a bit of searching found the answer. In the bucket you go to Permissions, click Add members, then set the New members field to allUsers, and the Role to Storage Object Viewer.

I then needed to get gsutil.py set up so that I could command-line upload files. I already had gsutil installed as part of Chromium’s depot_tools so I just had to run gsutil.py config to set up my credentials.

The next step was to get my symbols (PDB files and PE files, actually) into symbol server format. The simplest way to do that is to create an empty directory and then use symstore.exe to add the files to it, as shown here.

I have notes in my original Symbols the Microsoft Way post that PE file names had to be lower-cased when uploading to a case-sensitive symbol server. Well, Google Storage is case sensitive and my testing shows that the case of PE files has to be preserved. I don’t know what I’m doing differently, but test your PE-file downloading carefully, perhaps using RetrieveSymbols, and windbg with !sym noisy.

I already had UIforETW’s symbols in this format so all I had to do was run this command, from my local symbol server directory:

python c:\src\depot_tools\gsutil.py cp -Z -R . gs://randomascii-symbols

The -R says to copy recursively. The -Z says to upload the files with gzip compression. You can also use the /compress option to symstore but we got burned by that when chrome.dll.pdb broke through 2 GiB so I prefer to let gsutil do the compression now. Either way works.

That’s it.

Existing public symbol servers:

This seems like as good a place as any to put a list of publicly available symbol servers. Note that some of these “symbol” servers only host PE files – .dlls and .exes. That may seem odd, but the metadata in PE files is essential for stack-walking in 64-bit processes. Since companies like NVIDIA and AMD will not release their PDBs we should be thankful that they at least release their PE files. In all cases the usual way to specify these in _NT_SYMBOL_PATH or windbg is with a SRV*c:\symbols*https://symbols.mozilla.org/ entry in a semi-colon separated list. Visual Studio and other tools usually just have a place to paste them in directly.

Here’s the list:

• Microsoft – https://msdl.microsoft.com/download/symbols
• Google (Chrome, clang-cl) – https://chromium-browser-symsrv.commondatastorage.googleapis.com
• randomascii (UIforETW and blog post resources) – https://randomascii-symbols.commondatastorage.googleapis.com
• Mozilla – https://symbols.mozilla.org/
• Unity (the game engine) – https://symbolserver.unity3d.com/
• Citrix – https://ctxsym.citrix.com/symbols
• Intel (PE only) – https://software.intel.com/sites/downloads/symbols/
• NVIDIA (PE only) – https://driver-symbols.nvidia.com/
• AMD (PE only) – https://download.amd.com/dir/bin and https://download.amd.com/dir/bin_2018 (having two is missing the point…)
• NuGet (portable PDBs) – https://symbols.nuget.org/download/symbols

Coda

I updated my UIforETW packaging script to do this for all future releases and published the UIforETW symbols for the last three releases to my symbol server.

My new symbol server path, usable for debugging and profiling UIforETW, and for future blog posts, is this:

https://randomascii-symbols.commondatastorage.googleapis.com

Caveat: only use symbol servers that you trust. These are the ones that I trust:

In short, the steps were:

1. Sign up for Google Storage
2. Create a storage bucket – maybe make access control uniform?
3. Make the symbols world-readable
4. Install gsutil (I had it already)
5. Run gsutil config to get command-line access to your project
6. Use symstore.exe to create a local symbol server directory structure
7. Use gsutil.py -Z -R to upload your local symbol server

See this blog post for another perspective on symbol servers.

That’s all folks.