Symbols on Linux Part Two: Symbols for Other Versions

After many years programming solely on Windows I have recently started working on Linux – Ubuntu to be precise. This is the second post in a series of tutorials that will share what I have learned about handling Linux symbols. This second post will cover how to get symbols for gcc C and C++ library versions that are not installed on your machine, which can be important when investigating crashes from customers’ machines.

The previous post explained how to get symbols for the C and C++ libraries on your machine. That’s a good start, but now we’re going to go to the next level.

Future posts on this topic include:

• Symbols on Linux versus on Windows – the promise of build IDs
• Symbols on Linux update: Fedora Fixes
• Symbols the Microsoft Way

Crash Analysis

The unfortunate reality of shipping a product to thousands or millions of people is that, despite all of your careful testing, it will crash on some users’ machines. Often this is due to some confluence of factors – different hardware, different software, different usage patterns – that was not anticipated. Sometimes it is due to bugs in the user’s hardware or other software, but no matter what the cause, if enough customers are hitting a crash you need to be able to investigate and fix it, even if the crashes are on a distribution which you don’t normally use.

If you are only interested in symbols for your install of Linux then this post has nothing to offer you – sorry.

At Valve we use google-breakpad to record crashes on Linux, Windows, and Mac OS. These crashes are uploaded to our servers where we can generate reports and then investigate the crashes, typically in order of frequency. In many cases we can fix these bugs without ever reproducing them on our machines – it’s amazing how much debugging can be done with just a call stack, with occasional reference to the raw stack memory and a list of loaded modules.

Not surprisingly this post-mortem crash analysis works best if we have symbols, both for our code and for every other shared object in our process. It is quite common for software to crash deep inside the C/C++ libraries and if we don’t have accurate function names then our investigation is unnecessarily complicated. In addition to function names, having full unwind data is also important – otherwise the stack walking may misbehave and we will be looking up incorrect addresses. This is particularly important on 64-bit platforms where there is typically no linked list of stack frames. Finally, it is best to have file name and line number information to allow zeroing in on the precise source line where a crash occurred.

Our development machines are mostly running fully patched Ubuntu 12.04 (Precise Pangolin). When we processed the libc6 symbols from our machines to breakpad format and pushed them to the crash dump analysis server this gave us C/C++ library symbols for most of our crash dumps. But not all.

Unpatched

We get a significant number of crashes from older versions of Ubuntu 12.04. The crash reports show that these customers are running an earlier kernel and a different version of the C/C++ libraries. The challenge then was how to get symbols for these older C/C++ library versions.

One obvious way to get older symbols would be to install the original version of Ubuntu 12.04 and then incrementally update it, publishing symbols along the way. This option did not make me happy. Luckily I found a better option.

Installing old symbol versions, a failed attempt

Normally when you install a package you get the latest version, but you can request a specific version. The option to do this can be seen if you run man apt-get and look in the help for install. It says:

A specific version of a package can be selected for installation by following the package name with an equals and the version of the package to select.

You can see the version of libc6 that you have installed by running dpkg -s libc6 or, alternately, running libc6 as a program, like this: /lib/i386-linux-gnu/libc.so.6

Either way I find that on my machine I am running version 2.15-0ubuntu10.3. I can verify this and the version-install syntax by running sudo apt-get install libc6=2.15-0ubuntu10.3, in which case I will be told that I already have that version. However if I make a guess as to the previous version and try installing its debug information I will hit problems:

$ sudo -E apt-get install libc6-dbg=2.15-0ubuntu10.2

…
The following packages have unmet dependencies:
libc6-dbg : Depends: libc6 (= 2.15-0ubuntu10.2) but 2.15-0ubuntu10.3 is to be installed
E: Unable to correct problems, you have held broken packages.

In short, I have version 10.3 of libc6 installed and apt-get, in order to save me from myself, refuses to install the incompatible 10.2 symbols. That’s a good thing in general, because incompatible packages can lead to serious problems, but in this case I just want to temporarily install some old symbols so the limitation is annoying.

Installation is optional

Thankfully we don’t really want to install these old versions of the symbols – we just want to extract the files so that we can make them available to breakpad or other crash-dump analysis tools. It turns out that extracting the contents of a package is far easier than installing it. These are the steps that I used:

$ cd $(mktemp -d)
$ apt-get download libc6-dbg=2.15-0ubuntu10.2
$ ar -x libc6-dbg_2.15-0ubuntu10.2_i386.deb
$ mkdir contents && cd contents
$ tar -xf ../data.tar.*z

These steps create and cd to a temporary directory, download the package, extract the archive contents, and then unpack the data.tar.*z file to the contents directory, giving us access to all of the symbol files.

Now we’re making progress. The next step is to find all the packages we might need.

Finding available versions

Probably the easiest way to find what packages are available is to find out what URL a particular debug package is downloaded from and then browse to the containing web page to see what other versions are there. The following command will get the URL for libc6-dbg:

$ apt-get download –print-uris libc6-dbg

‘http://us.archive.ubuntu.com/ubuntu/pool/main/e/eglibc/libc6-dbg_2.15-0ubuntu10.3_i386.deb’ libc6-dbg_2.15-0ubuntu10.3_i386.deb 2575218 sha256:a94d04814b464ae24b7cff7ba59465147dd45a8917a9d61a6e5cb5deb3131b9e

If we browse to http://us.archive.ubuntu.com/ubuntu/pool/main/e/eglibc/ and search for libc6-dbg we will see symbols for about a dozen versions of libc6.

The same process can be repeated for libstdc++6.

$ apt-get download –print-uris libstdc++6-4.4-dbg

‘http://us.archive.ubuntu.com/ubuntu/pool/main/g/gcc-4.4/libstdc++6-4.4-dbg_4.4.7-1ubuntu2_i386.deb’ libstdc++6-4.4-dbg_4.4.7-1ubuntu2_i386.deb 5122718 sha256:4ac0dda2d607bddddb4cdcd11595b197ed588d30094ec9bb397612b40e2914b5

We can then navigate to http://us.archive.ubuntu.com/ubuntu/pool/main/g/gcc-4.4/ (and gcc-4.5 and gcc-4.6) and search on libstdc++6-4.4-dbg (and 6-4.5 and 6-4.6).

Finally, the libgcc1-dbg package (gcc support library) can be found in the same web directories as the libstdc++ symbols, with names like libgcc1-dbg_4.6.3-1ubuntu5_i386.deb.

With these four web directories and the names of the debug packages in hand we can construct a simple shell script that will download symbols for 32 different libc6, libstdc++6, and libgcc1 packages.

$ BASE=http://us.archive.ubuntu.com/ubuntu/pool/main
$ SUFFIX=i386.deb
$ wget -r -l1 -nd -nc -A “libc6*dbg*$SUFFIX” $BASE/e/eglibc
$ wget -r -l1 -nd -nc -A “libstdc*dbg*$SUFFIX” $BASE/g/gcc-4.4
$ wget -r -l1 -nd -nc -A “libstdc*dbg*$SUFFIX” $BASE/g/gcc-4.5
$ wget -r -l1 -nd -nc -A “libstdc*dbg*$SUFFIX” $BASE/g/gcc-4.6
$ wget -r -l1 -nd -nc -A “libgcc1-dbg*$SUFFIX” $BASE/g/gcc-4.4
$ wget -r -l1 -nd -nc -A “libgcc1-dbg*$SUFFIX” $BASE/g/gcc-4.5
$ wget -r -l1 -nd -nc -A “libgcc1-dbg*$SUFFIX” $BASE/g/gcc-4.6

Warning: I can’t get rid of the “smart quotes” in the script above so you’ll probably have to replace both quotes on each wget line with ‘real’ quotes. Sorry.

Depending on your needs this might be excessive or insufficient. You can download just a few packages as needed, or download every -dbg package you can find.

With slight modifications this script can download amd64.deb packages or others.

Other Ubuntu debug packages

The libc6 and libstdc++ debug packages are an anomaly in the land of Ubuntu because they can be found in the normal repositories. For the majority of debug packages we need to go to the ddebs repositories, but this is actually quite simple. Let’s say we want to get the symbols for libX11.so.6. First we execute these two commands to find out what package contains libX11.so.6, and what URL contains that package:

$ dpkg -S libX11.so.6

libx11-6: /usr/lib/i386-linux-gnu/libX11.so.6
$ apt-get download –print-uris libx11-6

‘http://us.archive.ubuntu.com/ubuntu/pool/main/libx/libx11/libx11-6_1.4.99.1-0ubuntu2_i386.deb’

Now we take that URL and replace “.*ubuntu.com/ubuntu” with http://ddebs.ubuntu.com – if you’re feeling lazy you can do this with sed:

$ apt-get download –print-uris libx11-6 | sed “s#.*ubuntu.com/ubuntu#http://ddebs.ubuntu.com#”

http://ddebs.ubuntu.com/pool/main/libx/libx11/libx11-6_1.4.99.1-0ubuntu2_i386.deb’

Now navigate to the resulting web directory (ignore the file portion) and download whatever -dbgsym file seems promising. In my case I chose libx11-xcb1-dbgsym_1.4.99.1-0ubuntu2_i386.ddeb because I guessed it would match the customer’s machine.

In this case we were looking at a customer crash so we had a breakpad debug identifier of:

9853ba2a77f7afcdba76dfaf3900bd79

After fixing the byte ordering of the first 32-bit chunk and the two subsequent 16-bit chunks we are left with a partial build ID which looks like this:

2aba5398f777cdafba76dfaf3900bd79

I could then compare this to the result of readelf -n on the libX11.so.6 symbols that I had downloaded and I could see that they were a match:

2aba5398f777cdafba76dfaf3900bd794b39bf34

As with all of the advice in this particular post this technique only makes sense when tracking down symbols for crashes happening on other machines. For development on your own machine you generally want to add the ddebs repository and then go:

sudo apt-get install libx11-6-dbgsym

Other Linux versions method one: guess and go

After I retrieved symbols for Ubuntu’s versions of libc6 and published them to our breakpad symbol server I was still seeing crashes with no symbols. It turned out these were from people running our code on other Linux distributions. It was time to find out how to get symbols from Linux versions that I didn’t have installed. A large chunk of crashes were on Arch Linux with libc-2.17.so on the stack so I tried that first.

A bit of web searching found a description of the official Arch Linux Repositories. From there I went to the mirrors page, which took me to the mirror status page, which gave me a list of repository mirrors. I chose a conveniently located mirror and navigated down until I found a glibc package. This wasn’t actually a debug package, but I downloaded it and hoped that it would be helpful.

$ cd $(mktemp -d)
$ wget http://mirror.us.leaseweb.net/archlinux/core/os/i686/glibc-2.17-1-i686.pkg.tar.xz
$ mkdir contents && cd contents
$ tar -xf ../glibc-2.17-1-i686.pkg.tar.xz

This is similar to the process to unpack a .deb file except that the step to unpack the .tar.xz from the .deb file can be skipped.

Before proceeding I wanted to make sure that I had downloaded the correct libc version. Most ELF files contain a build ID which uniquely identifies the file:

$ readelf -n usr/lib/libc-2.17.so

Notes at offset 0x00000174 with length 0x00000024:
  Owner                 Data size    Description
  GNU                  0x00000014    NT_GNU_BUILD_ID
    Build ID: ca7f7b19456bdf0460ec93f7c3e92e19d439e96e

I compared that to breakpad’s module information for libc-2.17.so on the Arch Linux crashes:

libc-2.17.so  197B7FCA6B4504DF60EC93F7C3E92E190

At first glance it may seem like a mismatch, but breakpad treats build IDs as GUIDs and one consequence of this is that the first four bytes are endian-swapped, so that ca7f7b19 becomes 197B7FCA, so in fact this was the correct shared object.

This package doesn’t have full debug information – the filename and line number information is missing – but it does have symbols and unwind information. A bit more research showed that this embedded debug information was probably as good as I was going to get. It appears that, despite the efforts of some contributors, Arch Linux still doesn’t generally have debug packages. Which is a shame.

Other Linux versions method two: install ‘em

The web search method of finding symbols is unreliable, so if you really need symbols for a particular Linux version you may just have to install it. I’ve got a collection of virtual machines running various distributions of Linux. My usual pattern is to install, then use apt-get download –print-uris to find where the symbols for that version are, and then hope that a bit of exploring of the server will find other versions of the symbols. The actual downloading and unpacking can be done on my Linux development machine – in some cases I just run a couple of shell commands on the VM and then never use it again.

Fedora fun

When tracking down symbols the first thing you need to know is what version of Linux a crash is coming from. The breakpad reports include an OS field to help with this. The Ubuntu OS description is pretty clear:

Linux 0.0.0 Linux 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64

ARCH is similarly pretty obvious:

Linux 0.0.0 Linux 3.6.11-1-ARCH #1 SMP PREEMPT Tue Dec 18 08:57:15 CET 2012 x86_64

However this one had me confused for a while:

Linux 0.0.0 Linux 3.6.11-3.fc18.x86_64 #1 SMP Mon Dec 17 21:35:39 UTC 2012 x86_64

It took a bit of random searching before I realized that fc18 stood for Fedora Core version 18.

Within fc18 I knew from the crash reports that I wanted libc-2.16.so  with breakpad debug identifier 99DD7BCFE74D7D4C4A0CFF6A0293D96B0. I then installed Fedora in order to try to find these symbols, but this brought some extra challenges. Fedora uses yum/rpm for package management instead of apt-get/dpkg, so all my tricks would have to be reworked. Here’s the path that I ended up taking:

I used ldd /bin/ls to find where libc-2.16.so is installed, and then rpm -qf /lib/libc-2.16.so to find what package installs libc-2.16.so – it’s glibc-2.16-24.fc18.1686.

Then I used cat /etc/yum.repos.d/* to get a list of respositories, including http://mirror.chpc.utah.edu/pub/fedora/linux/.

I wanted the latest version of libc-2.16.so so I drilled down into the updates directory, version 18, i386, and eventually found this: http://mirror.chpc.utah.edu/pub/fedora/linux/updates/18/i386/glibc-2.16-28.fc18.i686.rpm.

The .rpm packages used by Fedora are not standard archive files so I did some searching and found that rpm2cpio and cpio could be used to extract their contents, and those tools are available on Ubuntu.

At this point I could abandon my Fedora VM and resume working on my physical Ubuntu machine. So, on my Ubuntu machine I did the following dance:

$ sudo apt-get install rpm2cpio
$ wget http://mirror.chpc.utah.edu/pub/fedora/linux/updates/18/i386/glibc-2.16-28.fc18.i686.rpm
$ mkdir contents && cd contents
$ rpm2cpio ../glibc-2.16-28.fc18.i686.rpm | cpio -idmv
$ readelf -n lib/libc.so.6 | grep Build
    Build ID: cf7bdd994de74c7d4a0cff6a0293d96b64681e06
$ readelf -s lib/libc.so.6 | wc -l
11005

The readelf -n step confirmed that the build ID matched the debug identifier I was looking, and the readelf -s step showed that there were enough symbols to be useful.

This system is still a bit ad-hoc as I haven’t yet found a way to directly find what URL a package comes from, but it worked relatively cleanly. It does appear that there should be debuginfo packages for glibc here but I could not find them, and running sudo yum install glibc-debuginfo-2.16-28.fc18.i686.rpm also got me nothing. I think the embedded symbols are sufficient so I am not looking further.

Using downloaded symbols

The question of what to do with all of these symbol files that have been downloaded is a topic for another post.

Summary

For reference purposes here are the new commands (not covered last time) covered in this post:

• sudo apt-get install libc6-dbg=2.15-0ubuntu10.2: installs a specific version of a package
• dpkg -s libc6: see what version of a package you have installed
• apt-get download libc6-dbg=2.15-0ubuntu10.2: downloads a package by name
• ar -x libc6-dbg_2.15-0ubuntu10.2_i386.deb: extract the contents of a package
• tar -xf data.tar.*z: extracts the contents of the data file from a package
• apt-get download –print-uris libc6-dbg: gets the download URL for a package
• cd $(mktemp -d): makes a temporary directory and sets it as the current directory
• wget $URL: download a package from a specific URL
• wget -r -l1 -nd -nc -A “libc6*dbg*$SUFFIX” $BASE/e/eglibc: download a set of files
• readelf -n usr/lib/libc-2.17.so: read the build ID from an ELF file
• readelf -s lib/libc.so.6 | wc -l: count how many symbol names are in a shared object

Here are the commands used for dealing with yum/rpm based package management tools:

• rpm -qf /lib/libc-2.16.so: find what package installs a particular file
• cat /etc/yum.repos.d/*: get a list of respositories
• rpm2cpio ../glibc-2.16-28.fc18.i686.rpm | cpio -idmv: unpack a .rpm file