When I’m describing what I do for a living to non-programmers I sometimes say that I solve puzzles. I solve fascinating puzzles that are different every day, and there’s no answer key, and very often nobody else knows the solution. Whether it’s figuring out why code is slow, or why it is crashing, or how to make code simpler and better, it’s all puzzles, and I love it.
This past weekend I worked on a puzzle which I thought was interesting, especially in the ways that it was different from ‘normal’ puzzles such as jigsaw or crossword puzzles.
Note: VS 2015 RTM has a bug in /DEBUG:FASTLINK which corrupts some debug information. It will be fixed in SP1.
Finding the puzzle
When most people solve puzzles it is because they are asked to on Facebook, or because they bought a book of puzzles, or they are in a math class pondering a train from Chicago…They are given a puzzle to do. In this case my first challenge was realizing that there was a puzzle to solve. I knew, from talking to the Visual Studio team at Microsoft and from reading their blog, that VC++ 2015 had a new /DEBUG:FASTLINK switch that could make PDB generation run faster. The puzzle was to enable this for Chromium in order to get faster builds. I decided that this was a worthwhile task that probably nobody else was working on. Challenge accepted!
Figuring out when you’ve won
Most puzzles have clearly defined winning conditions – fill in the Sudoku or finish the jigsaw puzzle. It’s pretty obvious when you’re done. However with FASTLINK this wasn’t as obvious. I could try timing the builds as I adjusted the build settings but build timings are notoriously fickle – affected by the disk cache, power settings, and more. However I knew that FASTLINK worked by not copying debug information from object files, so the PDB should be smaller when FASTLINK was working and that would be a much more stable metric than build time.
However, if you alternate between builds with and without FASTLINK you’ll find that the PDB size does not change. That’s because VC++ never shrinks PDBs, it just shifts data around inside the container, and increases its size when necessary. Therefore, whenever doing PDB size tests you always need to delete the PDB between runs. This is unrelated to incremental linking – it’s more related to PDB ages.
Figuring out the rules
The rules of most games are clearly stated, but in many programming puzzles they are not. In this case I adjusted the Chromium build settings to add FASTLINK, confirmed that it was being passed to the linker, and observed that the PDB size was unchanged.
I then tried a separate project – a ‘normal’ Visual Studio project (UIforETW, as it turns out) and verified that converting that to VC++ 2015 and setting FASTLINK worked as expected – the PDB got smaller.
Given one case that worked and another case that didn’t the next step is to understand what the difference is. I captured the .rsp file used by Chromium’s ninja build system and then called the linker directly. That let me modify the .rsp file in order to quickly iterate.
I tried rearranging command-line options and then I tried deleting command-line options. A bit of binary search and a bit of intuition led to the solution.
Winning!
It turns out that FASTLINK is incompatible with /PROFILE. The /PROFILE switch is passed to the linker by default in Chromium builds which means that, by default, FASTLINK doesn’t work in Chromium. With that detail understood it was relatively easy to create a patch that, once it lands, will allow easy access to the VC++ 2015 FASTLINK feature.
We don’t need no stinkin’ documentation!
It would be nice if the /DEBUG:FASTLINK documentation mentioned this restriction, but at the time that I am writing this blog post there is no such documentation. There are a couple of blog posts that mention the FASTLINK feature, but the official VC++ documentation on VC++ 2015’s /DEBUG switch does not mention FASTLINK, and certainly doesn’t mention its restrictions. Such is the joy of working on the bleeding edge, being among the first to use a new feature.
To the victor, the spoils
My limited tests suggest that FASTLINK reduces elapsed link time by 33-55%, and reduces peak working set of the linker by 65-90%. PDB sizes are reduced by 65-72%. These are all worthwhile gains that will make switching Chromium to VC++ 2015 more worthwhile.
The PDB files created by FASTLINK are inherently machine local. That’s a reasonable tradeoff, but sometimes it is useful to be able to convert machine-local PDBs to portable PDBs. I’ve filed a bug requesting a tool to do this.
Caveats
- FASTLINK apparently doesn’t work if you must debug in Mixed Mode
- FASTLINK has no effect on LTCG builds
Random ASCII - tech blog of Bruce Dawson 
Have you seen the TIS-100 game? It’s both a fun time-waster for programmers, and a great way to show around to non-programmers. 😉
I’ve heard about TIS-100 but have resisted it so far, at least partially because I get enough cool puzzles to solve in my regular work. And because I fear getting sucked in.
Binary search comes in handy more than I probably would have thought in school.
Neat! Apple’s linker has been doing this forever (hence the need to run dsymutil for a lot of things), and the GNU toolchain added -gsplit-dwarf a while back ( https://gcc.gnu.org/wiki/DebugFission ) which is also similar. I’ve actually seen people trying to build Firefox who had to disable debug symbols to get their builds to work because linking with debug info requires so much more memory. It’s a nice workaround, and if they can produce a tool to link a distributable PDB file it’d be even better. (Not particularly useful for our official builds which are using PGO anyway.) Thanks for doing the research here!
Will these FASTLINK PDBs work like original ones when stored in Symbol Servers?
Would they work when served up to multiple users?
FASTLINK PDBs contain references to local files and therefore they will not work on symbol servers or when server up to multiple users. FASTLINK should be used for local builds, not for build-machine builds.
Not entirely a puzzle, but have you noticed copies of pdb’s named lnk{GUID}.tmp linger in your temp directory when using /debug:fastlink? In my specific environment, my %temp% directory is on a different drive than the one for the build output directory, but I don’t know entirely how relevant that is. I haven’t been able to get a good command line only repro, so it’s possible that in Chromium builds you wouldn’t see it, but an IDE or msbuild of a fairly vanilla vcxproj seems to reproduce it, so I would think you would see it with UIforETW. It’s possibly tied with the /manifest options, since the mt, rc, and cvtres programs all seem to write to the files that linger.
I haven’t noticed that but I’ll watch for it.
Thanks for checking and looking for it in the future. I was just able to reproduce it on another install with a “default” project, so I don’t think it’s specific to my specific install. I’ve reported the issue on Connect, but sometimes it seems like the only two ways to get a quick response on Connect for C++ items is if “boost” is in the issue or if it was submitted by Bruce Dawson (hopefully Bruce Dawson2 gets a similar treatment eventually :).
I was able to get it to reproduce with UIforETW by making some changes in the Manifest Tool Section: 1) getting rid of the Additional Manifest Files file, and 2) changing the DPI Awareness setting to “None”. Both are seemingly required, which is confusing since with “my” projects, the DPI awareness setting is also the inherited default option of “High DPI Aware” and adding the CompatibilityManifest.man to the project does not fix the issue. I know I’m flailing around the in the dark, I just wish I could find some sensible combination of settings to get it to reproduce reliably.
Ah, and I see that you’ve also hit this in UIforETW: https://connect.microsoft.com/VisualStudio/feedback/details/806379/midl-compiler-defaultvalue-bstrs-and-the-vs2013-strictstrings-setting
Sadly, generating both invalid BSTRs and invalid C++ for a C++ specific header is better than potentially introducing new compiler errors on SDK upgrades. Though, if the group in charge of the midl compiler wanted to be “tricky” they could avoid the compiler errors and fix the strictStrings incompatibility.
I just checked and I found that I have 3.7 GB of lnk{GUID}.tmp files in my %temp% directory. Most of them are small but there are two that are 1.9 and 1.1 GB each.
They date back to April – I’m not sure if I was experimenting with fastlink back then or if they can be created in other ways.
It may well be that there is some race condition that causes these temporary files to not always get cleaned up. I’ll file a bug but meanwhile I recommend occasionally deleting.
Bug filed: https://connect.microsoft.com/VisualStudio/feedback/details/1628016