I created a series of training videos that cover Event Tracing for Windows, also known as xperf or the Windows Performance Toolkit. This set of videos, available on WintellectNow, should be enough to teach any experienced programmer how to use this amazing set of tools to investigate tricky performance problems on Microsoft Windows.
One way to get access to these videos is to check out Visual Studio Dev Essentials, a free Microsoft program that includes Wintellect. https://www.visualstudio.com/products/visual-studio-dev-essentials-vs and https://twitter.com/BruceDawson0xB/status/668318302309998592
I’m currently watching John Robbins’ excellent WinDBG training video (slightly condensed from Tolstoy’s original version).
My ETW talks are based, to some extent, on articles I have posted to this blog, and those posts are still available for free. But I think that video works well for demonstrating effective use of Windows Performance Analyzer – a quick demonstration is often more effective than a paragraph of explanation, and makes it easier to convey the joy of exploration. Plus, the videos all demonstrate using the latest trace analysis tools and techniques, including what I’ve learned in the last couple of years.
I tried to make use of the flexibility afforded by video editing and multiple takes to get the demos to flow as smoothly as possible. I think the end result should inform (and entertain) without wasting the viewer’s time. And, as always, I learned a few new things about ETW from the process. Take a look at the videos and let me know if you have any feedback. I’m particularly proud of the first five minutes of the second video.
Here are descriptions of the videos, extended from what can be found on the WintellectNow site:
Video 1: Introduction to Profiling with ETW
Event Tracing for Windows (ETW) allows investigation of performance problems on Windows to a greater depth than any other system. ETW can be intimidating to use at first but this talk explains how to get started with recording and analyzing ETW traces. The talk covers essential trace analysis techniques and concepts, with an emphasis on investigation of CPU bound performance problems. After viewing this talk you will be able to confidently use the free Windows Performance Toolkit to find CPU slowdowns, and you will be prepared to learn additional ETW investigation techniques.
Note that this video was created before UIforETW was created. The steps on how to get the Windows Performance Toolkit and record traces, from 4:09 to 7:10 in the video, can be skipped over, and instead just grab UIforETW and use it instead. If you want a video description of how to install and use UIforETW, take a look at this free video. The description of recording a trace that starts around 7:10 is sufficiently similar to UIforETW to still be applicable, and the rest of the video is as relevant as ever.
This talk is designed to take the viewer from zero to analyzing ETW traces in less than an hour. It covers installing the Windows Performance Toolkit and recording and analyzing a trace. The demo focuses on poor performance in PowerPoint, which requires diving into many graphs and tables, including CPU Usage (Sampled) data. After watching this video you should understand the Zen of WPA tables – how to fearlessly configure them to mine for the information that you need. The talk also explains how to work around this common PowerPoint problem.
Spoiler alert: a cache for decoded and scaled background images might be a good idea.
This video explains how to use Event Tracing for Windows (ETW) to easily find why a thread is not running – to find what it is waiting on, and who wakes it up. Additionally this talk explains how to easily use custom ETW events to annotate traces and make them easier to investigate, and how to customize WPRUI to keep trace sizes manageable. The WPA tables and graphs for viewing Generic Events, CPU Usage (Precise), and File I/O are explained and demonstrated.
Note that this video was created before UIforETW was created. File I/O activity, keyboard events, and other custom events are automatically recorded in UIforETW, and UIforETW defaults to using a reasonable amount of memory. Therefore, the section of video from 7:00 to 14:00 can be skipped. And, since UIforETW ships with WPA startup profiles the first use of WPA will be less intimidating than the video shows.
This talk is built around the analysis of a hang in Visual Studio. The additional resources include an ETW key logger which is both useful by itself (now built in to UIforETW) and as an example of how to enrich your ETW traces with custom events.
The main focus of this talk is the subtle but critical skill of wait analysis – finding why a thread is not running. This ability (missing from most profilers) is one of the most important aspects of ETW and is used in many performance investigations, including the third video in this series.
Spoiler alert: treating Perforce paths as UNC paths can lead to UI hangs.
Video 3: ETW Disk I/O and Machine Information
This video, the third in the ETW training series, covers a wide range of topics built around the analysis of a hang in Windows Live Photo Gallery. Starting with the simplest way to locate a hang, moving through some multi-threaded wait analysis to find the badly behaved thread, it then moves to a deep dive on the differences between file I/O and disk I/O. Then, the talk explains how to work around the Windows Live Photo Gallery hang that was being investigated, and finishes with a few tricks on how to find information about the machine the trace was recorded on, and other bonus tips.
Note that this video was created before UIforETW was created. Therefore you can ignore the discussion of adding the Microsoft-Windows-Win32k provider because UIforETW always records data from this provider.
Graphs and tables that are used include Process Lifetimes, Generic Events, UI Delays, CPU Usage (Precise), DPC/ISR, File I/O, Disk Usage, the totally cool Disk Usage Offset graph, System Configuration, Images, and Marks. The usually awesome Window in Focus and CPU Usage (Sampled) graphs are cruelly ignored. The View Editor’s ability to have duplicated columns and save custom views are demonstrated. Some important details of the architecture of Windows (Deferred Procedure Calls and the system cache) are also explained.
This video ties up the loose ends and should make the three-video series a complete explanation of how to do excellent ETW trace analysis.
Spoiler alert: doing thousands of random 4-KB reads from your disk is not efficient.
Finding the videos
In addition to using the links above to find the videos you can go to http://www.wintellectnow.com and then search on the author Bruce Dawson to find them.