The Guardian is one of my favorite news sources. I’m a subscriber (support news organizations!) and I read it daily. But it is not immune to errors, as this headline shows:
68 °F above average is a lot. For a tropical country it is not credible for temperatures to be that much warmer than average because the average is too high to give enough headroom. So what gives?
Memory is a relatively scarce resource on many consumer computers, so a feature to limit how much memory a process uses seems like a good idea, and Microsoft did indeed implement such a feature. However:
This feature works by limiting the working set of a process – the amount of memory mapped into the address-space of the process – to 32 MiB. Before reading any farther take a moment to guess what the maximum slowdown might be from this feature. That is, if a process repeatedly touched more than 32 MiB of memory – let’s say 64 MiB of memory – then how much longer could these memory operations take compared to if the working set was not limited? Take a moment and write down your guess. The answer is later in this post.
TL;DR – upgrade your tools, including Visual Studio, windbg, and Windows Performance Toolkit, if you want to handle Chromium’s symbol files.
Details:
Death, taxes, and browser engines relentlessly growing – those are the three things that you can really be certain of. And so it was in early 2020 when I realized that Chromium’s inexorable growth meant that we were eventually going to produce PDB (Windows debug symbol) files that exceeded the PDB format’s 4 GiB limit.
I filed a Visual Studio bug in February 2020 requesting that the limit be raised, and three years and three days later we flipped the switch so that Chromium can produce larger PDBs. At that point the PDB for Chrome was at 95% of 4 GiB, and several test binaries had already crossed over the threshold, so it was just in time.
I tend to launch most programs on my Windows 10 laptop by typing the <Win> key, then a few letters of the program name, and then hitting enter. On my powerful laptop (SSD and 32 GB of RAM) this process usually takes as long as it takes me to type these characters, just a fraction of a second.
Usually. Continue reading
Last month I wrote about an odd crash that was hitting a few Chrome users. Something was corrupting the XMM7 register and that was causing Chrome to crash. We fixed a couple of bugs in Chrome and we were able to contact the third-party company whose software was causing the problems. They released a fixed version, and I assumed that my work was done.
However, instead of a gradual decline in the rates of this crash I saw a gradual increase. Apparently enterprise software updates roll out extremely slowly, and users were installing the old buggy version faster than they were updating to the fixed version. This situation will resolve itself eventually, but a lot of crashes were going to happen in the meantime. With the proper fix moving slowly through the pipelines I decided to try to hack in a decidedly improper fix.
“Hey, you. Yes you, that function over there. When you’re cleaning up please remember to restore all of my registers. Yes, that one too – what do you think this is, Linux?”
That’s the problem I was dealing with in a nutshell. Functions are required by a platform’s ABI (Application Binary Interface) to preserve certain registers – restoring them if they were used – but the set of registers that must be restored varies between platforms, and the rules on Linux are different from those on Windows. That may be why I encountered register corruption in Chrome on Windows. But let’s take a step back.
I apologize for this title because there are many things that can make modern software slow. Blindly applying one explanation without a bit of investigation is the software equivalent of a cargo cult. That said, this post describes one example of why modern software can be painfully slow.
All I wanted was to record a forty-second voiceover for a throw-away video, so I fired up the Windows Voice Recorder app and hit the record button. Nothing seemed to happen.
While investigating some performance mysteries in Chrome I discovered that Microsoft had parallelized how they zero memory, and in some cases this was making it a lot slower. This slowdown may be mitigated in Windows 11 but in the latest Windows Server editions – where it matters most – this bug is alive and well.
The good news is that this issue seems to only apply to machines with a lot of processors. By “a lot” of processors I mean probably 96 or more. So, your laptop is fine. And, even the 96-processor machines may not hit this problem all the time. But I’ve now found four different ways to trigger this inefficiency and when it is hit – oh my. The CPU power wasted is impressive – I estimate that memory zeroing is using about 24x the CPU time it should.
Okay – time for the details.
In 2004 I was working for Microsoft in the Xbox group, and a new console was being created. I got a copy of the detailed descriptions of the Xbox 360 CPU and I read it through multiple times and suddenly I’d learned enough to become the official CPU expert. That meant I started having regular meetings with the hardware engineers who were working with IBM on the CPU which gave me even more expertise on this CPU, which was critical in helping me discover a design flaw in one of its instructions, and in helping game developers master this finicky beast.
It was literally the day after I cracked the __FILE__ determinism bug that I hit a completely different build determinism issue. I was asked to investigate why the Chrome build number reported for Chrome crashes on Windows 11 was lagging behind what was reported by winver. For example, Chrome crashes on 10.0.22000.376 were being reported as happening on 10.0.22000.318. After some code spelunking I found that crashpad retrieves the Windows version number from kernel32.dll, so I focused on that.
Aside: crashpad grabs the Windows version number from kernel32.dll instead of using GetVersionExW (which is deprecated, BTW) because the GetVersion* functions will frequently lie about the Windows version for compatibility reasons. For crash reporting we really want the actual-no-lies-we-can-handle-the-truth version number, and kernel32.dll used to be the best way to get this.
That’s when things got weird.
Random ASCII - tech blog of Bruce Dawson 