Category Archives: Debugging

Please Restore Our Registers When You’re Done With Them

“Hey, you. Yes you, that function over there. When you’re cleaning up please remember to restore all of my registers. Yes, that one too – what do you think this is, Linux?” That’s the problem I was dealing with in … Continue reading

Posted in Chromium, Debugging, Investigative Reporting, Symbols | Tagged , , | 23 Comments

Two Deterministic Build Bugs

‘Twas the week before Christmas and I ran across a deterministic-build bug. And then another one. One was in Chromium, and the other was in Microsoft Windows. It seemed like a weird coincidence so I thought I’d write about both … Continue reading

Posted in Bugs, Chromium, Computers and Internet, Debugging, Programming | Tagged , , | 14 Comments

The Easy Ones – Three Bugs Hiding in the Open

I write a lot about investigations into tricky bugs – CPU defects, kernel bugs, transient 4-GB memory allocations – but most bugs are not that esoteric. Sometimes tracking down a bug is as simple as paying attention to server dashboards, … Continue reading

Posted in Bugs, Code analysis, Code Reliability, Debugging, Floating Point, Linux, Performance | Tagged | 17 Comments

GDI leaks and the importance of luck

In May 2019 I was asked to look at a potentially serious Chrome bug. I initially misdiagnosed it as unimportant, thus wasting two valuable weeks, and when I rejoined the investigation it was the number one browser-process crash in Chrome’s … Continue reading

Posted in Bugs, Chromium, Debugging, Programming | Tagged | 25 Comments

Creating a Public Symbol Server, Easily

I’ve been a big fan of symbol servers for years. They are a part of the Microsoft/Windows ecosystem that is far better than anything I have seen for other operating systems. With Microsoft’s and Chrome’s symbol servers configured I can … Continue reading

Posted in Debugging, Programming, Symbols, uiforetw | Tagged | 15 Comments

Compiler bug? Linker bug? Windows Kernel bug.

See the end of the post for an October 2018 bug fix update, or read the whole story: Flaky failures are the worst. In this particular investigation, which spanned twenty months, we suspected hardware failure, compiler bugs, linker bugs, and … Continue reading

Posted in Debugging, Investigative Reporting, Programming | Tagged , , , | 57 Comments

Zombie Processes are Eating your Memory

Zombies probably won’t consume 32 GB of your memory like they did to me, but zombie processes do exist, and I can help you find them and make sure that developers fix them. Tool source link is at the bottom. … Continue reading

Posted in Debugging, Investigative Reporting, Performance, Programming, Rants | Tagged , , | 74 Comments

Finding a CPU Design Bug in the Xbox 360

The recent reveal of Meltdown and Spectre reminded me of the time I found a related design bug in the Xbox 360 CPU – a newly added instruction whose mere existence was dangerous. Back in 2005 I was the Xbox … Continue reading

Posted in Debugging | Tagged , , , , | 49 Comments

Analyzing a Confusing Crash–Stack Walks Gone Bad

Part of my job always seems to include crash analysis. A program crashes on a customer’s machine, a minidump is uploaded to the cloud, and it might be my desk that it appears on when Monday morning rolls around. The … Continue reading

Posted in Debugging | Tagged , | 8 Comments

Everything Old is New Again, and a Compiler Bug

“What’s an EXCEPTION_FLT_STACK_CHECK exception?” one of my coworkers said. I said “It’s a weird and rare crash. Why do you ask?” It turns out that one of these weird and rare crashes had started showing up in Chrome (M54 branch, … Continue reading

Posted in Debugging, Floating Point | Tagged | 26 Comments