After many years programming solely on Windows I have recently started working on Linux. This is the third post in a series of tutorials that will share what I have learned about handling Linux symbols. This third post explains how to more easily get symbols, and compares Linux to the situation on Windows.
One advantage of working on multiple platforms is the opportunity to realize what each one is missing. There are parts of Linux that I miss when I am working on Windows and (heresy!) there are things about Windows that I miss when I am working on Linux. Symbol handling is one of those.
Here’s a recap of the posts on this topic to date:
- Getting symbols for the C and C++ libraries on your machine
- Getting symbols for other Linux distributions and other versions of files
There are also some posts on this topic from the future:
Windows
After working on Windows for many years I have grown accustomed to having symbols show up automatically. Debuggers and profilers on Windows use symbol servers to automatically retrieve symbols. Typically they are configured to retrieve symbols for Windows and other Microsoft products, and perhaps other external products such as Mozilla or SourceMod. Symbol servers should also be used to store and retrieve your company’s own private symbols – if you are a Windows developer and you aren’t doing this then you should fix that right now.
Symbol files are retrieved from the configured symbol stores based on unique identifiers embedded in DLLs and EXEs. When doing post-mortem debugging even the DLLs and EXEs can be automatically retrieved from the symbol store based on a few identifiers recorded in the crash dump. The mechanism is described in more detail here.
This combination is powerful. I can take a crash dump file from a customer running an arbitrary version of our games, running on an arbitrary version of Windows, load it into the debugger and have all of the symbols automatically show up. I don’t need to know or care what version of the game or what service pack of Windows the customer was running, and I don’t need to think about package names or new repositories. The symbols and binaries Just. Show. Up.
The convenience of this is hard to overstate. No special knowledge is required, and no time is wasted tracking down symbols. They just appear, like magic. Anything that requires more knowledge or more time is, by definition, less efficient.
It is certainly true that the symbols you get, from Microsoft anyway, are missing source information and some other details. However the depth of the symbols is a separate issue from how you get the symbols.
Automatic symbol finding is critical for batch processing. When I have a collection of Windows crashes I can run a batch file (batch files are like shell scripts only much less powerful) to do bulk analysis of the crashes, including generating call stacks for all of them. The value of this fully automated system should be obvious.
Occasionally symbol resolution on Windows goes awry – I’ve seen it be unusably slow, and it’s frustrating that proprietary graphics driver symbols are never available – but normally it works perfectly.
Linux as it is
Acquiring symbols on Linux could be equally easy. But it isn’t. It’s taken me months of learning and two blog posts in order to document some ad-hoc methods that will usually allow you to track down the symbols that you need. It’s time consuming, and inconvenient, it raises the barrier to entry, and I know that it can be easier.
In my previous posts I described a series of ad-hoc methods for getting symbols. To get symbols for crashes on your machine or your distribution of Linux this will usually work:
- Add the ddebs repository if you haven’t already (Ubuntu specific)
- For each shared object find out what package it comes from
- Add -dbgsym to that package name and install that, if it exists
If the crash occurred on a different Linux distribution then the steps are different:
- Install the distribution where the crash occurred
- Note that all the following steps are implemented differently for Debian versus Red Hat style package systems
- For each shared object find out what package it comes from
- For each package find out what URL its debug version can be downloaded from, if it exists
- Download the debug package
- Extract the contents
- Put the symbols where you can use them
In short, tracking down symbols on Linux is time-consuming, requires significant expert knowledge, and sometimes doesn’t work. However it doesn’t have to be this way.
Linux as it could be
In 2007 a proposal was made to embed build IDs in all Linux binaries. These build IDs – typically 40 character hex numbers – were designed to be used as a search key to find a particular version of a particular binary, and its symbols. Since that initial proposal most of the work to implement it has been done. The gcc toolchain defaults to putting build IDs in ELF files, most Linux distributions put build IDs in core files (except Ubuntu, but I’ve filed a bug to get that fixed), and gdb supports finding symbols on the local file system using build IDs.
However what is still missing is a way to download symbols to your local system, based purely on a build ID. And that is the part that could really make build IDs worthwhile.
Fedora has started trying to solve this with their darkserver project. The idea is that you just append the build ID that you are interested in to a URL and you can then download full package details. This works for their canonical sample, shown here:
https://darkserver.fedoraproject.org/buildids/0d0669e4ce89ffb335e36d41eacf3dfd04072e17
However it failed when I used it to find symbols that I actually wanted, such as the symbols for libc-2.16.so for Fedora Core 18 (as discussed on the previous installment of this series). The Build ID for libc-2.16.so is cf7bdd994de74c7d4a0cff6a0293d96b64681e06, but as of today nothing is to be found at this address:
https://darkserver.fedoraproject.org/buildids/cf7bdd994de74c7d4a0cff6a0293d96b64681e06
Update: darkserver has been fixed and is now (as of March 2013) being kept up to date with Fedora releases, with plans to support finding Ubuntu symbols also. See Fedora Fixes for details.
Another Fedora option is this command that asks yum to install the file that will be referenced through the standard build ID derived symlink (note the slash after ‘8b’):
yum –enablerepo=fedora-debuginfo install /usr/lib/debug/.build-id/8b/d8064a80f57906f7e21504f13a86110cdb4535.debug
This only works for the latest/updated release and only for the primary architecture and I have not tested it. A final Fedora option is this option, which doesn’t require root access:
echo 8bd8064a80f57906f7e21504f13a86110cdb4535 \
| abrt-action-install-debuginfo –ids=- –cache=/tmp
I assume that the latest/updated release and primary architecture restrictions still apply, making these techniques good for local debugging, but not for analysis of crashes from remote machines, especially if the developer is running a different distribution.
A solution for Ubuntu symbols
Anybody can come along and say that something is broken, but that’s no way to make friends. So, I spent some extra time to create a solution, for Ubuntu anyway.
The Ubuntu symbol packages can be found at http://ddebs.ubuntu.com. Within the various sections of this web site there are Packages files that list all of the packages for various versions of Ubuntu – precise, precise-updates, quantal-security, etc. Those Packages files are almost what I need, but they lack build IDs. All that is needed was a Python script to parse the Packages files, download each .ddeb file listed, unpack it, get build IDs for the installed files, and append the data to an enhanced Packages file. Something like this pseudo-code:
for packagesURL in packagesURLs:
wget packagesURL
for ddebName in Packages:
wget ddebName
ar -x ddebName
tar -xf data.tar.*z
for file in unpackedFiles:
buildID = GetBuildID()
AppendToPackagesFile(buildID, file, ddebName)
After running this over precise and precise-updates I have an enhanced Packages file that I can query using grep to trivially find what file and package are associated with any build ID. Just run “readelf -n” on the file of interest – or look at the debug identifier in a breakpad crash report – and then run grep on the enhanced Packages file.
To get the download URL for the C++ runtime symbols, libstdc++.so.6 just do this (output word wrapped for readability):
$ grep 96b9cb6b542dac65f995ff4b2a68213653b86f02 Packages
BuildID:
96b9cb6b542dac65f995ff4b2a68213653b86f02
/usr/lib/debug/usr/lib/i386-linux-gnu/libstdc++.so.6.0.16
http://ddebs.ubuntu.com/pool/main/g/gcc-4.6/libstdc++6-dbgsym_4.6.3-1ubuntu5_i386.ddeb
In my last episode I needed symbols for libX11 for a customer crash. I only had a 32-character breakpad debug identifier but that is plenty for searching the enhanced Packages file. Just remember to fix the endianness of the breakpad debug identifier before grepping:
$ grep 2aba5398f777cdafba76dfaf3900bd79 Packages
BuildID:
2aba5398f777cdafba76dfaf3900bd794b39bf34
/usr/lib/debug/usr/lib/i386-linux-gnu/libX11.so.6.3.0
http://ddebs.ubuntu.com/pool/main/libx/libx11/libx11-6-dbgsym_1.4.99.1-0ubuntu2_i386.ddeb
BuildID:
2aba5398f777cdafba76dfaf3900bd794b39bf34
/usr/lib/debug/usr/lib/libX11.so.6.3.0
http://ddebs.ubuntu.com/pool/main/libx/libx11/libx11-6-udeb-dbgsym_1.4.99.1-0ubuntu2_i386.ddeb
Curiously this same technique fails when used to find symbols for the latest libc6 on Ubuntu 12.04. Apparently those symbols are not available in the ddebs repository, and I didn’t index the main repository, so this query returns nothing.
$ grep bc99bb8745130f34a31106951ceccfd9dc3295b4 Packages
It would be quite trivial to automate the process of downloading, extracting, and using the symbols that are found using this build ID technique. If the major Linux distributions made build ID information available in this form then the symbol problem would be solved – developers would just need to download Packages files from the distributions that they cared about. Until then, in order to save every Linux developer from using my script to download every package from Canonical’s site, I made my enhanced Packages file available here but it hasn’t been updated in many years so it is useless.
Meanwhile Debian has rolled out their debuginfod service which I believe is basically a symbol server (with source packages?) for Debian. It’s progress.
Any other thoughts or ideas on this topic are appreciated – put your suggestions in the comments.
Windbg for Linux developers
Windbg/cdb, the low-level Windows debuggers, have acquired the ability to load a variety of Linux dumps. You can even configure the symbol server paths in the debuggers so that they will automatically retrieve the necessary DWARF/ELF symbol files from symbol servers, following this convention for the keys used to identify the files. The Microsoft PE-timestamp/filesize and PDB-Signature/Age keys are listed in there along with ELF-buildid and Mach-uuid. If the binaries and symbols that you need are on a symbol server – or if you can push them there – then windbg and friends can automatically retrieve them, and you too can enjoy the magic of having symbols and binaries Just. Show. Up.
Conclusion
There is no fundamental reason why symbols can’t be downloaded either automatically or effortlessly on Linux. There is no fundamental reason why this can’t work even when you’re analyzing a core file that was recorded on a different distribution. Linux is gradually moving in this direction, but right now Microsoft wins the “convenience” prize hands down.
I hope that my demonstration of how Linux symbol finding could be enhanced is useful as a proof of concept, or as an immediately useful way of locating symbol files on Ubuntu.
Random ASCII - tech blog of Bruce Dawson 
As someone who works at Red Hat on Fedora and other projects like this, let me first give a heartfelt “Thank You”. Being in the Linux development business for a long time, it’s easy to lose track of what would make development for ISVs easier, and this sort of feedback about where our platform falls down is extremely helpful for us.
I’ll ask a few coworkers tomorrow about the issues you were having with Fedora 18 debug symbols, and forward them this post. Hopefully we can work together on implementing something that’s useful for you.
Thanks Jasper.
My bias would be towards a solution that could work in the same way for all Linux distributions. darkserver seems like the right idea, but isn’t up to date. Ubuntu’s Packages files are always up to date but don’t have build IDs. Fixing either one of these — and convincing the other distribution(s) to follow suit — would do the trick.
Sorry for the trouble with darkserver. I am the upstream author and maintainer of the project. I had some initial difficulties with the koji-plugins to import data on time.
Now the whole system is redesigned and within 7-10 days the production server will be in sync with current koji builds.
I’m glad to hear it should be keeping up better in the future. Is that 7-10 days the time needed to catch up, or will there always be 7-10 days of lag? It turns out to be important to have very little lag. If a new libc comes out then there will be crashes in it immediately (there are always programs misusing it) and there is an instantaneous need for symbols. Simultaneous availability of symbols and binaries is important.
7-10 days is only for initial catch up, from then it should be in few minutes.
That’s great news.
I hope Fedora considers doing the Packages file technique (with build IDs) as well, as that can also be a good option, especially when you have breakpad debug identifiers (32 characters) instead of full debug IDs.
“There is no fundamental reason why symbols can’t be downloaded either automatically or effortlessly on Linux”
Gentoo. LFS. apt-get source && dpkg-buildpackage.
Also, there’s a lot of other distributions, many with a rolling release cycle and a dozen of architectures, and potentially fast-updating repositories like Debian Experimental, that make the number of different versions grow quickly.
I’d be happy if the top few distributions made their symbols available via build-id even for those who don’t have that version installed. Ubuntu and Fedora would be a good start. Gentoo is trickier to solve perfectly, but it is entirely possible to have stable ways of finding symbols for many packages (libc6 anyone?) that change relatively rarely, even if individual users are building them from source.
I use Gentoo and while a symbols repo is a non-solution for similar distributions it is also the case that Gentoo users that want a piece of software to work are more likely to get you the information you need even if it means rebuilding a package with some flags enabled and sending the symbols file to you. Obviously this is will not be the case for everyone but if I had to guess between distros like Arch and Gentoo there are probably less than 10,000 users with Steam installed. If there are some packages that are routinely causing problems it might be possible to get a USE flag added to the steam-games-meta ebuild (or the like) that would make things simpler, but since it isn’t always possible to predict where bugs will come from this might only have limited utility.
I would think that a Gentoo symbols repository is still possible because any user could create one and, if packages are built in a standardized chroot environment, the symbols that any user publishes should be representative of most Gentoo users. The majority of the crashes I’ve looked at were in libc or libc++ so just having symbols for those easily available would be a huge step.
Steam is mostly solving these problems now by shipping most of the packages that we depend on — it’s easy to have the symbols when we’re building the packages.
Bruce, I need to echo Jasper here as well: many thanks for writing about this. I have also spent an inordinate amount of time digging through the Linux debug symbol mess, and I hope that the distributions find some sort of consistent way to handle this problem.
Though partly exhibiting the same buildID issue, I do have to give Centos/RHEL credit for having generally reliable debug symbols available (accessible by either enabling the debuginfo repository or directly visiting http://debuginfo.centos.org).
While still early in development, I’ve seen automatic debuginfo pulling work with http://linux.fenrus.org/ solution (under All Debug Information, All The Time for a little more info, the git repo is available). It is just one distro but the technique can be applied to work across distros but of course the infrastructure needs to be in place…
The Fenrus solution (and their understanding of the importance of the problem) sounds great. I hope some aspects of the solution allow developers on other distros who have customers on Fenrus to get symbols.
Look at it selfishly: the easier it is for developers to get symbols for your distro (without having to install it) the more likely they are to investigate and fix bugs. Of course that motivation only applies if you run cross-platform binaries on your distro.
Reblogged this on anthonyvenable110.
A very interesting blog entry, bookmarked!
It seems like the Arch Linux guys are working on a basic debug package integration through their makepkg program but the changes have not made it into the stable version yet:
https://projects.archlinux.org/pacman.git/log/?qt=grep&q=debug
meh. Or we could just include symbols (and maybe includes/headers…). Obsessing over saving few gigs when I have multiple terabytes of disk in my 201x desktops+servers. Come to think of it, I didn’t agree with the newfangled -dev separation of rpms/devs back in the day, and I certainly had less than 1/10th the space I do now…
Its amazing that the Linux community is still figuring this out when the BSDs have had this solved for years. Throw in OPTIONS_SET=DEBUG to /etc/make.conf and walla, you have debug symbols.
Setting OPTIONS_SET=DEBUG is, if I understand correctly, helpful for code that you are compiling. Is that correct?
My concern is with symbols for precompiled code that you have installed on your machine, and for precompiled code installed on customer’s machines. If OPTIONS_SET=DEBUG helps solve the problem of analyzing crashes from Fedora and Ubuntu 12.10 crashes on my Ubuntu 12.04 machine then I’d love to hear how it does it.
It doesn’t solve the problem in the way you’ve proposed your ideal solution to work.
What Bob is suggesting is basically the same as setting “USE += debug” in gentoo
(or whatever the exact invocation is) and then recompiling, installing the -dbg package on
debian-based distros, or invoking the ABS on archlinux.
So no, I don’t think the BSDs really are ahead on this front…
Please have a second look at the darkserver, now it is syncing with latest koji builds.
It seems to work perfectly. I tried my test build ID for libc:
https://darkserver.fedoraproject.org/buildids/cf7bdd994de74c7d4a0cff6a0293d96b64681e06
and this gave me details about the package. I then followed the instructions at https://darkserver.fedoraproject.org and appended the package name to https://darkserver.fedoraproject.org/package. This gave me a download URL for the actual package file. So, three calls to wget and a bit of simple parsing and we have an automatic way to convert a build ID into a package file for Fedora.
Most excellent.
You can even use a command line client for the same, https://github.com/kushaldas/darkclient It provides the output in much more machine readable format so that you can parse them easily in any language.
Pingback: Symbols on Linux update: Fedora Fixes | Random ASCII
Pingback: Symbols the Microsoft Way | Random ASCII
Pingback: Symbols on Linux Part One: g++ Library Symbols | Random ASCII
Pingback: Symbols on Linux Part Two: Symbols for Other Versions | Random ASCII
Pingback: Counting to Ten on Linux | Random ASCII
I wanted to leave a note here for anyone who might find this useful in the future. I took Bruce’s scanpackages.py and reworked it a bit to scrape *every* ddeb on ddebs.ubuntu.com, and output a JSON file of ddeb URL -> list of filename, build id. The source is here:
https://github.com/luser/linux-symbol-scraping
I also ran it for a few days on an EC2 instance to generate a full JSON file, that output is here:
https://s3-us-west-2.amazonaws.com/ubuntu-build-ids/ddebs.json
I’d like to set up a cron job to update that regularly for my own use, but perhaps someone else might find it useful as well. You can see an example of using the data in scrapedebs.py in the same repository, which tries to fetch symbols that were missing from Firefox crash reports using ddebs.json to look them up by Build ID.
Thanks Ted. That’s great! I haven’t been doing Linux work lately so I’ve let this fall by the wayside and it’s great to see you running with it.
It is possibly quite overdue, but I think the coming version of Debian will improve the situation quite a bit due to changes in debhelper (the most widely used tool for building debian packages in Debian):
* Debian now have debug symbols from all debhelper based packages. They are available from http://debug.mirrors.debian.org/ and (by extension) http://snapshot.debian.org/archive/debian-debug/
* The Packages files have the Build-Ids listed in the package providing them (when the dbg symbols were handled by debhelper
– Not quite compatible with your syntax/packages files, but a bit of grep-dctrl can easily give the package name of the package containing a given build-id[1]
My own estimate suggests that these changes will affect up to 99% of all Debian packages[2]. A few items that are missing/up in the air:
* We do not have a service like the Fedora “darkserver”. Though I think all the pieces needed to build it is now there.
* I don’t know if/when Ubuntu will migrate to this method.
* I doubt we will have 100% coverage any time soon.
* There are some TODO items left to have dbgsym packages supported in Debian testing (and by extension the next stable release).
[1] grep-dctrl -sPackage -FBuild-Ids /path/to/Packages1 […]
[2] https://nthykier.wordpress.com/2016/01/02/dput-change-all-of-debian-changes/
Thanks for the update. That sounds quite encouraging.
Thanks for this article. I wonder if now, almost 10 years later, there are some more updates regarding the situation on Linux.
https://archlinux.org/news/debug-packages-and-debuginfod/ It looks like that at least on the Archlinux distribution they are moving towards a server-based solution to provide debug symbols as well
Thanks for the reply Silvan. I stopped working on Linux a while ago so I’m not up-to-date on the latest changes. I get the impression that things are improving, but given the intrinsically fractured nature of Linux distributions I would guess that it will never be as simple as on Windows (if you are shipping software for multiple Linux distributions)