Event Tracing for Windows (ETW) has always recorded a rich set of data and allowed graphing it all on the same timeline. With the creation of UIforETW (which records more data) and the new* ETW trace viewer (which can graph custom data) the ability to visualize important patterns is better than ever before.
And, what better way to visualize how to create graphs than a video demonstrating what is discussed in this post.
Unlike Microsoft’s wprui which is a passive recorder of trace information UIforETW is also a source of ETW trace data. UIforETW uses the ETWProviders.dll that it ships with to emit events containing information about CPU temperature, frequency, and power draw**, along with battery discharge information, process working sets***, the Windows timer frequency, and user input. This serves as a good demonstration of how to emit custom events using ETWProviders.dll, but most importantly this extra data helps with trace analysis.
With earlier versions of Windows Performance Analyzer (WPA) this extra data existed only as numbers. If you wanted graphs you’d have to paste the data into Google Sheets (or Excel – is that still a thing?). With WPT 10 there is now the ability to graph this data. The WPA Analysis Assistant does a good job of describing this new feature:
The WPA 10 screenshot below shows the sort of information that we can now graph. At the top is the normal CPU Usage (Precise) graph. At six seconds in it shows that the Idle thread (purple) stops running and FractalX64 (blue, and a great fractal explorer) starts consuming all CPU time. At the same time the CPU temperature (TempStatus) starts rising, from 90 degrees up to 120. This makes sense. Then the CPU temperature anomalously drops to 96 degrees before going back up to 116. Curious…
The PowerStatus (third) graph shows that the processor power draw jumps from 4.9 W to 53 W, before dropping to 9.9 W and then rising back up to 41 W. This is consistent with the temperature data, but unexpected given the still 100% CPU usage (the slight drops in FractalX64 CPU usage are all from other processes stealing some cycles).
This peculiar behavior makes somewhat more sense when the CPU FrequencyStatus (fourth) graph from UIforETW is shown since it shows the CPU frequency going from 800 MHz to 3 GHz and then back down to 800 MHz before recovering to 2.6-2.7 GHz.
Meanwhile the ‘official’ CPU frequency graph just shows the CPU going from 800 MHz to 2.2 GHz – missing all of the variations after the six second mark.
The data that finally makes sense of all of this is the Battery Status information in the table at the bottom. This shows that my laptop went from On AC power to Discharging at around 10.18 seconds. Apparently I unplugged my laptop and it took two seconds to adjust to the new state of affairs before it continued on in a slightly less Turbo Boosted state. The last On AC power sample is selected and its location in the timeline shows up on all of the graphs.
Frequency measurement frailty
The Frequency by Cpu graph is supplied by Windows, through the POWER provider, but it has some problems. Apparently there are some times of CPU frequency changes that the CPU knows about but the OS doesn’t. Either that or the POWER provider is buggy on Windows 7. Either way it cannot be completely trusted.
Another problem with the Windows POWER provider is that it fails completely on my work machine. It’s probably my fault for having more than thirty-two logical CPUs, but I still want this first-world problem fixed.
The final problem with the Windows POWER provider is that it knows nothing about thermal throttling. Unfortunately, I don’t think that the Intel Power Gadget frequency data reflects that either, so thermal throttling must still be inferred from CPU temperature and other factors (at least until I rewrite my actual-CPU-frequency-monitor™). Thermally throttling appears to be an increasingly serious problem. I think it is shocking that some OEMs are selling machines with cooling solutions that are insufficient to let their CPUs run at their rated speeds. If you buy one of these defective machines you should return it – try RealTemp or one of the other programs I suggested in my original thermal throttling blog post. Or try Intel Extreme Tuning Utility which has a graph dedicated to thermal throttling (thanks to jaimemmoreno for the tip).
There are a few limits on the graphing. There doesn’t appear to be a way to adjust the range of the graphs, so the CPU temperature graphs are hard to read, with most of the dynamic range squished into the top of the graph.
The graphing also can’t handle negative numbers, which is why I didn’t graph the battery discharge data.
The default chart type of ‘Line’ works poorly for me so I always change it to Stacked Bars. The Select Chart Type option appears after you put the data column to be graphed on the far right (and remove the Time column.)
WPA’s default aggregation mode for the generic events data fields is None, which won’t graph. The Generic Events presets that ship with WPA 10 have been changed to have Average as the aggregation type for easier graphing. You can change this to Maximum or some other setting as needed.
With the v1.12 release of UIforETW there are new presets to make this much simpler – just filter to the desired data and then select Randomascii Graph Field 2 to graph field 2 (presets for graphs 1 to 6 are available). To see this in action watch this video.
Disclaimers and addendums
The trace that I recorded can be found in the bigfiles repo in the ETWTraces directory – look for “2015-07-30_08-29-33 FX battery and power test.*”.
* The new ETW trace viewer is the Windows Performance Toolkit (WPT) 10 version of WPA, sub-version 10*2^10. It shipped in its final version on July 29, 2015 as part of the Windows 10 SDK. There are also redistributable installers that will install just WPT.
** CPU temperature, frequency, and power draw all require having Intel Power Gadget 3.0 installed. A reboot may be required to ensure that UIforETW sees the environment variable that indicates where to find the Intel Power Gadget binaries.See PowerStatus.cpp for details. I’d be happy to add support for AMD processor monitoring – call me @amd.
*** Working set monitoring only occurs for a specified set of processes – see the UIforETW settings dialog for details. Note that ‘*’ will monitor all processes but this is not recommended when expensive working set monitoring (PSS and private working set) is enabled. See WorkingSet.cpp for details.
For instruction on how to do ETW trace analysis take a look at the series of training videos I created.