This post can be found through https://tinyurl.com/etwtracing, and an alternate version of these instructions can be found here.
If your Windows computer is running slowly – if a program takes a long time to launch, if a game has a poor frame rate, or if an idle application uses too much CPU time – the best way to investigate is to record an Event Tracing for Windows (ETW) trace. An ETW trace records a wealth of information (CPU sampling, context switches, disk I/O, custom data, and much more) that allows most performance problems to be understood by a trained expert. If you want to analyze traces, go here. If you’re not a trained expert then you can still record an ETW trace, and then share it with somebody who is.
If a particular program is being slow or inefficient then you may be able to record an ETW trace and share it with the authors of that program. Quite often they can figure out what is going wrong, whether it is a bug on their side or an overheating CPU on your side. Tell them I sent you. They may be grateful for receiving an actionable report instead of vague complaints about slowness which they cannot reproduce.
Not all developers are equipped to analyze ETW traces, for technical and practical reasons – ask first.
Recording and sharing ETW traces has never been easier. Here are the three steps.
Recording ETW traces
If you prefer to learn through watching instead of reading you can view this brief video that demonstrates installing UIforETW and recording a trace. For written instructions, read on:
- Get the latest release of UIforETW. This is an open source tool for recording and managing ETW traces. It makes recording traces easier while adding additional information such as input events to make analysis easier. Download etwpackage.zip, extract the contents, and run etwpackage\bin\UIforETW.exe. This will install the necessary versions of the Windows Performance Toolkit. Wait for the installations to finish.
- Now click Start Tracing. ETW tracing will begin. By default it goes to in-memory circular buffers and can be left running indefinitely, recording the last 10-60 seconds (actual duration varies) of activity. When you have reproduced the slowdown type Ctrl+Win+R from wherever you are (you don’t need to switch to UIforETW) to save the trace buffers to disk. You should enter a description of what happened in the Trace information field associated with your trace. Detailed descriptions are ideal, as they tell the analyst what the problem is and where in the trace it occurred.
- Right-click on the list of traces and select Browse folder to open the documents\etwtraces folder containing the traces. There will be a .etl file and a .txt file for each trace. Upload them to your favorite file-sharing service to share with someone who can analyze the traces.
Be aware, however, that ETW traces can contain personal information. ETW traces record information about all processes on your system. Typically this include the names of files being read and written, so an analyst may be able to tell what document you were editing, or what music you were listening to. However the traces will not include the contents of the files or the names of files that are on-disk but not referenced. The Input tracing information is very important for a successful analysis but it defaults to Private mode, where all letters are recorded as ‘A’ and all numbers are recorded as ‘1’, to avoid being a key-logger. Full mode input tracing can be useful, but enable it with caution, for obvious reasons. And, be thoughtful about who you share ETW traces with.
ETW traces also include full information about your hardware, and version numbers of any software that is running when the trace is recorded
That’s it. That’s all it takes.
Extra bonus steps
- If you install Intel’s Power Gadget (and launch UIforETW after the Power Gadget install completes) then additional information about CPU frequencies, power draw, and temperature will be recorded. Sometimes this is vital, other times it doesn’t matter, and sometimes it just causes crashes (thanks Intel).
- If you are reporting problems in Chrome or Edge then, starting with version 74, it is possible to get some browser tracing events to show up in ETW traces. This may include additional information such as URLs from any of your tabs so be mindful about the privacy implications. To use this feature you select which Chrome tracing events are exported by selecting categories from UIforETW’s Settings dialog – the “latency” category is a good first choice. This feature is best used in cooperation with a Chrome developer who is investigating your issue and can recommend categories. https://crbug.com is the best place to start these discussions. Note that in many cases a Chrome trace may be a better option than an ETW trace.
Recording great traces
Some traces are better than others. If your description of your problem is vague, or if your trace doesn’t capture the critical moment, then the analyst may not be able to identify your problem. Here is an example of a bad description:
your program sucks and its always slow why cant u make it work better bruce said send a trace lol
And here is a good description, from a trace that was well recorded:
I started tracing in UIforETW and then clicked the foo-bar widget and it took about ten seconds to update during which time WizzyFuzz was hung. I saved the trace about two seconds after WizzyFuzz started responding again. This hang happens about 10% of the time when I click the foo-bar widget. I’m using the default settings and fuzzing a two-hundred cubit Wizzy.
You don’t, however, have to describe your hardware. ETW traces already contain this information.
Analyzing ETW traces
If you are given the job of analyzing one of these ETW traces look for blog posts in the xperf category. In particular, look at the ETW Training Videos I created to help new analysts get started – just ignore the information on recording traces – or go to https://tinyurl.com/etwcentral.
I thought you might like to know that Smartscreen prevents running the application initially. You have to click through More Options. Not a big deal, but just in case you whitelisted your own stuff eons ago…
Thanks for letting me know. Chrome doesn’t object to the download and I haven’t used IE for a while so I didn’t realize.
I do think SmartScreen is a good idea but it is always a shame when it blocks something I know is good.
I just tested now with IE – downloading etwpackage.zip from the v1.16 release – and it had no objections. SmartScreen is enabled.
Ditto. Smartscreen popup on Win10 downloaded zip via Firefox unzipped and when I clicked on extracted UIforETW get Smartscreen popup.
Yes, this is what I did, too. Firefox downloaded it without prompt. Launching in Windows 10 for the first time produced a big blue bar in front of everything listed as “Windows Smartscreen.” I was able to get through it. No big deal.
What’s a good primer on analyzing the output from UIforETW?
Analyzing traces is a big topic. I’ve written a lot of posts on this topic over the years, but it can be hard to know which ones to start with.
Probably your best bet is the training videos that I created last year. The recording part is now different (easier!) but the rest of the analysis techniques are unchanged. See the “Analyzing ETW traces” section that I just added for links.
Thank you much. I actually have two logged issues I need to look into. I’m curious if Analyzing is going to help here. One is an Outlook 2013 client that takes a very long time to open. The second is WMI errors after three days of running successfully.
ETW should help understand why Outlook 2013 takes so long to open.
WMI errors – probably not.
Actually, I don’t see that section you’re referring to. I first thought you meant a “Links” section on the side-bar. But I don’t see any “Analyzing” on this page or the main page either. Sorry, I missing it! 🙂 I imagine I can find your YouTube videos.
I suspect you are seeing a cached older version of the post, somehow. This is the text that I added:
“If you are given the job of analyzing one of these ETW traces look for blog posts in the xperf category. In particular, look at the ETW Training Videos I created to help new analysts get started – just ignore the information on recording traces.”
xperf category: https://randomascii.wordpress.com/category/xperf/
Training videos: https://randomascii.wordpress.com/2014/08/19/etw-training-videos-available-now/
Thanks. I think I misunderstood what you were saying. I expected some sort of link in the sidebar of the WordPress design. I have it now! Thanks a bunch. 🙂
Reblogged this on Jarek Przygódzki. Blog programisty.
This is great stuff especially the power consumption ETW provider is interesting. I stil wonder how many people are really capable to analyze ETW traces. Even Microsoft support seems not to have too many people beeing used to it. Perhaps you should open an ETW consulting agency 😉
Pingback: The Lost Xperf Documentation–CPU sampling | Random ASCII
Pingback: UIforETW – Windows Performance Made Easier | Random ASCII
Pingback: Xperf Basics: Recording a Trace | Random ASCII
Pingback: Xperf Resources | Random ASCII
Pingback: ETW Training Videos Available Now | Random ASCII
Pingback: New Version of Xperf–Upgrade Now | Random ASCII
Pingback: The New Xperf is Here! | Random ASCII
Pingback: ETW Central | Random ASCII
Pingback: The New WPA Xperf Trace Viewer–New Bugs and Old | Random ASCII
Pingback: Xperf for Excess CPU Consumption: WPA edition | Random ASCII
Pingback: Xperf and Visual Studio: The Case of the Breakpoint Hangs | Random ASCII
Probably a security thing but the chrome://flags link doesn’t work.
trace-export brilliant! Didn’t know about this.
Just had a tiny play with UIforETW and it didn’t load chrome.pdb initially, I assume it is meant to?
Thanks for this! Chrome has a few issues on Windows for me and this will help me figure out what it is, much more easily!
Probably a security thing indeed – I’ll make chrome://flags not be a link to avoid confusion.
WPA will load Chrome symbols if Chrome’s symbol server is in _NT_SYMBOL_PATH. If you have Chrome developer checked in the Settings dialog *and* if you don’t have _NT_SYMBOL_PATH set yourself then UIforETW will set it to include the Chrome symbol server.
If these conditions are true when you record a trace then UIforETW will also strip the chrome symbols to dramatically (50:1 !!!) speed up loading of Chrome’s symbols. So, if you check Chrome developer after recording a trace (or if you receive a trace from someone else with Chrome in it) you should right-click, Scripts, Strip Chrome Symbols to do this symbol optimization step.
If you record traces that show Chrome problems then please share them with me at my brucedawson address at Google.
I’m in awe! Thank you for developing this tool and all the blog posting you have been sharing. I just found your site and am pouring over it.
I don’t code, but do perform Win performance troubleshooting and your site and utility should really help expand my knowledge and skill set for trace captures and analysis when I’m trying to figure out what process(s) are causing performance issues on a system.
A humble post of mine at my amateur attempts at performance tracing from a few years back for some context; http://grandstreamdreams.blogspot.com/2012/04/case-of-unexplained-donut-of-death.html
I carry “portable” (to the degree they can be) packages of the Win 10 and 8.1 WPTs on a portable USB stick so I don’t have to install/impact the systems I’m troubleshooting.
I’ve found I can then fire off my traces and do analysis (capture details obviously dependent on OS of system limitations) with the xperf version I am using (ie: using the WPT 10 on a Win 7 system).
Is there a way to configure/use UIforETW with one of these “portable” sets instead of going through the “install” routine and putting the WPT bits on the system.
Right now most of the support work I do remains for Win 7 (x64) systems so being able to use a tool like this with one of Win 8.1/10 WPT sets (without installing it) would be helpful.
You make a good point. Since WPT can run without being installed it is technically easy to make a completely stand-alone UIforETW package that contains its own local WPT copies.
There are legal reason why I can’t redistribute UIforETW that way, but here’s what could be done:
Step 1) Write a simple batch file that copies the installed WPT files to directories inside an etwpackage directory.
Step 2) Tweak UIforETW so it looks for WPT locally first and uses the local copy if found.
Then you would just need to run the batch file and copy the result to your thumb drive.
Simple enough. Feel free to submit a pull request, or I might get to it some day.
Thanks for the great tool.
Is it possible to limit the ETW trace output to the data from one process, or even one thread within one process?
The reason I want to do this (besides .etl file size reduction) is that when I use WPA to analyze the ETW trace and I “Filter To Selection” for the necessary process/Thread ID, the column “% Weight” never seems to be re-rooted to the current filtered output. Perhaps I’m doing something wrong here – I’m only very new to this.
Sorry, not possible, as far as I know. ETW is inherently a whole-system profiling system. This is *usually* what you want (whether you know it or not) because, for instance, the system process does page-zeroing and disk I/O on your behalf, and other processes compete for resources. Anyway, regardless, not possible as far as I know.
As for the % Weight and other CPU usage percentages, those are always relative to the total CPU power available. Making them relative to a single core would be nice, but that is orthogonal to how many processes are recorded or viewed.
Ah, ok. Thanks for the quick reply!
I totally misunderstood the basis for that % Weight column, but I suppose as long as all values are relative it’s intuitive enough to work with.
I think I’ll have to pause and actually read the docs/watch your video.
I was finally able to save a trace (looks like Antivirus software may have complicated matters after all) but there was no txt file saved along with the etl.
The text file is not always automatically created. If you have “Chrome Developer” checked in the settings dialog then a summary of Chrome processes will be put into a .txt file. Otherwise nothing will be put there until you start typing notes.
Is there an easy way to automate the capture based on certain events? I.e. something like “configure in PerfMon a trigger where content of xperf circular buffer is dumped to file with unique name if CPU consumption of process X exceeds 95%”. I mean, I am sure it is possible, especially when using xperf directly, but wanted to see if there is a cheat available. Thanks!
I have not created a system to do that. I hacked up a custom event for that purpose in a previous job, but nothing generic, and nothing that works with UIforETW. It would be very helpful.
Hello, I have been trying to use this tool, but I can’t get the executables in the path “etwpackage\bin\UIforETW.exe” I am downloading the wrong package.
From https://github.com/google/UIforETW/releases you need to download etwpackage1.56.zip and extract it somewhere. The files will be there.
The post says
> When you have reproduced the slowdown type Ctrl+Win+R from wherever you are (you don’t need to switch to UIforETW) to save the trace buffers to disk.
Shouldn’t that be Ctrl+Win+C (C not R) to save?
The screenshot is out of date. It’s Ctrl+Win+R to start and stop tracing.