I’ve been a big fan of symbol servers for years. They are a part of the Microsoft/Windows ecosystem that is far better than anything I have seen for other operating systems. With Microsoft’s and Chrome’s symbol servers configured I can download a user’s Chrome crash and start analyzing it immediately – with code bytes and symbols – without knowing or caring what OS or Chrome version that user was running. Since Chrome’s symbols are source-indexed I will even get source-code popping up automatically, and all of this is available to anyone who is interested – no special Google privileges required.
So, symbol servers are magical and wonderful. And yet, I’ve never set up a publicly visible one. I’ve landed some fixes to the symbol servers at my last two jobs, but I’ve never created one from scratch to make symbols to my own tools and samples available.
Since I’m quarantined (to varying degrees) at home in Kirkland I’m spending this sunny Saturday setting up a symbol server and documenting the process. It was really easy.
A symbol server is basically just a file or HTTPS server with the contents arranged in a specific way. There are lots of ways to set up such a server but I decided to use the one that I am most familiar with, the one used by Chrome’s symbol server, which is Google storage. I work at Google, so the obvious bias is obvious, make of that what you will.
Seven simple steps
The first step was to create a Google Cloud account. There’s a free trial that gives you $300 of credit in your first year, so publishing my symbols to the world shouldn’t cost me anything for a while. I plugged a few numbers into the cost estimator (it shows up when you go to create a bucket) and it’s hard to see how it could end up costing me as much as a few cups of coffee a year (class B operations are downloads, 640K should be enough for anyone):
The next step was to create a storage bucket. The only challenging part was choosing a name. Bucket names are globally unique – grab your favorites now. I chose randomascii-symbols. I’m glad that I still like my blog name.
I changed access control to uniform, but this probably wasn’t necessary. Otherwise I took all of the default settings.
It wasn’t entirely obvious how to make the symbols world-readable but a bit of searching found the answer. In the bucket you go to Permissions, click Add members, then set the New members field to allUsers, and the Role to Storage Object Viewer.
I then needed to get gsutil.py set up so that I could command-line upload files. I already had gsutil installed as part of Chromium’s depot_tools so I just had to run gsutil.py config to set up my credentials.
The next step was to get my symbols (PDB files and PE files, actually) into symbol server format. The simplest way to do that is to create an empty directory and then use symstore.exe to add the files to it, as shown here.
I have notes in my original Symbols the Microsoft Way post that PE file names had to be lower-cased when uploading to a case-sensitive symbol server. Well, Google Storage is case sensitive and my testing shows that the case of PE files has to be preserved. I don’t know what I’m doing differently, but test your PE-file downloading carefully, perhaps using RetrieveSymbols, and windbg with !sym noisy.
I already had UIforETW’s symbols in this format so all I had to do was run this command, from my local symbol server directory:
python c:\src\depot_tools\gsutil.py cp -Z -R . gs://randomascii-symbols
The -R says to copy recursively. The -Z says to upload the files with gzip compression. You can also use the /compress option to symstore but we got burned by that when chrome.dll.pdb broke through 2 GiB so I prefer to let gsutil do the compression now. Either way works.
Existing public symbol servers:
This seems like as good a place as any to put a list of publicly available symbol servers. Note that some of these “symbol” servers only host PE files – .dlls and .exes. That may seem odd, but the metadata in PE files is essential for stack-walking in 64-bit processes. Since companies like NVIDIA and AMD will not release their PDBs we should be thankful that they at least release their PE files. In all cases the usual way to specify these in _NT_SYMBOL_PATH or windbg is with a SRV*c:\symbols*https://symbols.mozilla.org/ entry in a semi-colon separated list. Visual Studio and other tools usually just have a place to paste them in directly.
Here’s the list:
- Microsoft – https://msdl.microsoft.com/download/symbols
- Google (Chrome, clang-cl) – https://chromium-browser-symsrv.commondatastorage.googleapis.com
- randomascii (UIforETW and blog post resources) – https://randomascii-symbols.commondatastorage.googleapis.com
- Mozilla – https://symbols.mozilla.org/
- Unity (the game engine) – https://symbolserver.unity3d.com/
- Citrix – https://ctxsym.citrix.com/symbols
- Intel (PE only) – https://software.intel.com/sites/downloads/symbols/
- NVIDIA (PE only) – https://driver-symbols.nvidia.com/
- AMD (PE only) – https://download.amd.com/dir/bin and https://download.amd.com/dir/bin_2018 (having two is missing the point…)
- NuGet (portable PDBs) – https://symbols.nuget.org/download/symbols
I updated my UIforETW packaging script to do this for all future releases and published the UIforETW symbols for the last three releases to my symbol server.
My new symbol server path, usable for debugging and profiling UIforETW, and for future blog posts, is this:
Caveat: only use symbol servers that you trust. These are the ones that I trust:
In short, the steps were:
- Sign up for Google Storage
- Create a storage bucket – maybe make access control uniform?
- Make the symbols world-readable
- Install gsutil (I had it already)
- Run gsutil config to get command-line access to your project
- Use symstore.exe to create a local symbol server directory structure
- Use gsutil.py -Z -R to upload your local symbol server
See this blog post for another perspective on symbol servers.
That’s all folks.
I had no idea how to workaround symstore’s 2gib limit problem. Thanks! 🙂
If you look at the history of the bug you’ll see that we explored quite a few options. It was anticlimactic when we found that adding -Z was all that was needed.
The CAB format supports 4 GiB files but most of the tools and samples use a signed long instead of unsigned. pigz is another option, with modifications to handle the CAB format, but Microsoft advised against pushing our luck.
It’s also quite sad that makecab doesn’t fail until it gets to the 2 GiB limit. That is, it spends ~twenty minutes reading and compressing and *then* fails.
I believe the best code is the least code. Your patch looks elegant. Good job!
Apparently Mozilla does run a public symbol server for nightly and release Firefox builds: https://developer.mozilla.org/en-US/docs/Mozilla/Using_the_Mozilla_symbol_server
Available at https://symbols.mozilla.org/
I’m creating a list at the end of the post and I’ll put that there. Thanks.
Some other noteworthy public Symbol Servers:
Unity (the game engine): https://symbolserver.unity3d.com/
I’m creating a list at the end of the post and I’ll put these there. Thanks.
Seems like the link for step 6 is pointing to the same link as step 5. I guess that is a mistake?
Thanks for the report. Fixed (once I figured out what my intention had been)
I wonder how about private symbol servers? I was looking for a solution to serve NuGet packages form a linux server and I only found a server allowing you to limit who can publish a package but not who can subscribe (LiGet: https://github.com/ai-traders/liget). Do you happen to know any solution that would require authorisation for subscribing to the symbol server?
I don’t know how an Internet accessible private symbol server could be implemented. It’s easy enough to implement access controls, but the symbol server DLLs do not, to my knowledge, no anything about these permissions.
You can, however, use a local file share as a symbol server. If this is only accessible from the Intranet then it is automatically private and the files will be retrieved using normal file access APIs instead of https.That’s what I use for my private symbols, as discussed in the original post:
Thanks for your reply. I’m slowly getting to the same conclusion that only Intranet would be an option here.
Great post – I’m saving it for later! Can you possibly make a post on how to set up a caching symbol server? That is, a symbol server that proxies for the microsoft (or other) server? I’d love to provide access to the symbols in our lab, but I don’t want general internet access in that environment.
I’m not sure how to do that, and I’m not sure what restrictions you have. What I have seen done is crash-processing infrastructure that keeps track of what symbols it encounters that aren’t in its local symbol store. This list is then processed once a day on another machine to try to find the symbols. Those symbols then get copied to the symbol store used by the crash-processing system. So I guess that’s roughly what you want?