Xperf Basics: Recording a Trace (the ultimate easy way)

imageIf your Windows computer is running slowly – if a program takes a long time to launch, if a game has a poor frame rate, or if an idle application uses too much CPU time – the best way to investigate is to record an Event Tracing for Windows (ETW) trace. An ETW trace records a wealth of information (CPU sampling, context switches, disk I/O, custom data, and much more) that allows most performance problems to be understood by a trained expert. If you’re not a trained expert then you can still record an ETW trace, and then share it with somebody who is.

If a particular program is being slow or inefficient then you may be able to record an ETW trace and share it with the authors of that program. Quite often they can figure out what is going wrong, whether it is a bug on their side or an overheating CPU on your side. Tell them I sent you. They may be grateful for receiving an actionable report instead of vague complaints about slowness which they cannot reproduce.

Not all developers are equipped to analyze ETW traces, for technical and practical reasons – ask first.

Recording and sharing ETW traces has never been easier. Here are the three steps.

Recording ETW traces

If you prefer to learn through watching instead of reading you can view this brief video that demonstrates installing UIforETW and recording a trace. For written instructions, read on:

  1. Get the latest release of UIforETW. This is an open source tool for recording and managing ETW traces. It makes recording traces easier while adding additional information such as input events to make analysis easier. Download etwpackage.zip, extract the contents, and run etwpackage\bin\UIforETW.exe. This will install the necessary versions of the Windows Performance Toolkit. Wait for the installations to finish.
  2. imageNow click Start Tracing. ETW tracing will begin. By default it goes to in-memory circular buffers and can be left running indefinitely, recording the last 10-60 seconds (actual duration varies) of activity. When you have reproduced the slowdown type Ctrl+Win+C from wherever you are (you don’t need to switch to UIforETW) to save the trace buffers to disk. You should enter a description of what happened in the Trace information field associated with your trace. Detailed descriptions are ideal, as they tell the analyst what the problem is and where in the trace it occurred.
  3. Right-click on the list of traces and select Browse folder to open the documents\etwtraces folder containing the traces. There will be a .etl file and a .txt file for each trace. Upload them to your favorite file-sharing service to share with someone who can analyze the traces.

Be aware, however, that ETW traces can contain personal information. ETW traces record information about all processes on your system. Typically this include the names of files being read and written, so an analyst may be able to tell what document you were editing, or what music you were listening to. However the traces will not include the contents of the files or the names of files that are on-disk but not referenced. The Input tracing information is very important for a successful analysis but it defaults to Private mode, where all letters are recorded as ‘A’ and all numbers are recorded as ‘1’, to avoid being a key-logger. Full mode input tracing can be useful, but enable it with caution, for obvious reasons. And, be thoughtful about who you share ETW traces with.

ETW traces also include full information about your hardware, and version numbers of any software that is running when the trace is recorded

That’s it. That’s all it takes.

Extra bonus steps

  • If you install Intel’s Power Gadget (and launch UIforETW after the Power Gadget install completes) then additional information about CPU frequencies, power draw, and temperature will be recorded. Sometimes this is vital, and other times it doesn’t matter.
  • If you are reporting problems in Chrome then, starting with version 46 (beta, as of September 2015), it is possible to imageget some of Chrome’s tracing events to show up in ETW traces. This may include additional information such as URLs from any of your tabs so be mindful about the privacy implications. To use this feature you have to enable it in Chrome by going to chrome://flags/, searching for “trace-export”, enabling “Enable exporting of tracing events to ETW”, and then relaunching imageChrome. You can then select which Chrome tracing events are exported by selecting categories from UIforETW’s Settings dialog. This feature is best used in cooperation with a Chrome developer who is investigating your issue and can recommend categories. https://crbug.com is the best place to start these discussions. Note that in many cases a Chrome trace may be a better option than an ETW trace.

Recording great traces

Some traces are better than others. If your description of your problem is vague, or if your trace doesn’t capture the critical moment, then the analyst may not be able to identify your problem. Here is an example of a bad description:

your program sucks and its always slow why cant u make it work better bruce said send a trace lol

And here is a good description, from a trace that was well recorded:

I clicked the foo-bar widget and it took about ten seconds to update during which time WizzyFuzz was hung. I saved the trace about two seconds after WizzyFuzz started responding again. This hang happens about 10% of the time when I click the foo-bar widget. I’m using the default settings and fuzzing a two-hundred cubit Wizzy.

You don’t, however, have to describe your hardware. ETW traces already contain this information.

Some history

For the last four years I have been writing about how to record and analyze ETW traces. I may be obsessed.

My first attempt used a series of batch files to call xperf. It worked, and it helped lots of people record traces without having to learn the peculiar xperf syntax for recording traces. But batch files are horrible. This was not good.

Then Microsoft released wprui. This was a point-and-click UI for recording trace. Wprui was a huge improvement, but it was not easily extensible. I briefly promoted it but it is ultimately missing too many features that I need.

Then I decided to write my own UI. The first versions of UIforETW worked well and had imagethe features that I needed. UIforETW also worked around a few xperf bugs, but the initial versions were too hard to use. Initially you had to build UIforETW from source, and you had to track down the Windows Performance Toolkit installers yourself. Who has time for that? The most recent UIforEW releases, from v1.11 and beyond, are turnkey. They include pre-built binaries and the WPT installers. So, if you just need to record or analyze a trace, grab the latest release and you are set.

If you want to customize UIforETW then you can grab the source code. Pull requests are welcome (see the CONTRIBUTING file for legal details).

Analyzing ETW traces

If you are given the job of analyzing one of these ETW traces look for blog posts in the xperf category. In particular, look at the ETW Training Videos I created to help new analysts get started – just ignore the information on recording traces.

About brucedawson

I'm a programmer, working for Google, focusing on optimization and reliability. Nothing's more fun than making code run 10x faster. Unless it's eliminating large numbers of bugs. I also unicycle. And play (ice) hockey. And juggle.
This entry was posted in Performance, Programming, uiforetw, xperf and tagged , , . Bookmark the permalink.

37 Responses to Xperf Basics: Recording a Trace (the ultimate easy way)

  1. ultramagnustcv says:

    I thought you might like to know that Smartscreen prevents running the application initially. You have to click through More Options. Not a big deal, but just in case you whitelisted your own stuff eons ago…

    • brucedawson says:

      Thanks for letting me know. Chrome doesn’t object to the download and I haven’t used IE for a while so I didn’t realize.
      I do think SmartScreen is a good idea but it is always a shame when it blocks something I know is good.
      I just tested now with IE – downloading etwpackage.zip from the v1.16 release – and it had no objections. SmartScreen is enabled.

    • jaimemmoreno says:

      Ditto. Smartscreen popup on Win10 downloaded zip via Firefox unzipped and when I clicked on extracted UIforETW get Smartscreen popup.

      • ultramagnustcv says:

        Yes, this is what I did, too. Firefox downloaded it without prompt. Launching in Windows 10 for the first time produced a big blue bar in front of everything listed as “Windows Smartscreen.” I was able to get through it. No big deal.

  2. ultramagnustcv says:

    What’s a good primer on analyzing the output from UIforETW?

    • brucedawson says:

      Analyzing traces is a big topic. I’ve written a lot of posts on this topic over the years, but it can be hard to know which ones to start with.

      Probably your best bet is the training videos that I created last year. The recording part is now different (easier!) but the rest of the analysis techniques are unchanged. See the “Analyzing ETW traces” section that I just added for links.

  3. Alois Kraus says:

    This is great stuff especially the power consumption ETW provider is interesting. I stil wonder how many people are really capable to analyze ETW traces. Even Microsoft support seems not to have too many people beeing used to it. Perhaps you should open an ETW consulting agency 😉

  4. Pingback: The Lost Xperf Documentation–CPU sampling | Random ASCII

  5. Pingback: UIforETW – Windows Performance Made Easier | Random ASCII

  6. Pingback: Xperf Basics: Recording a Trace | Random ASCII

  7. Pingback: Xperf Resources | Random ASCII

  8. Pingback: ETW Training Videos Available Now | Random ASCII

  9. Pingback: New Version of Xperf–Upgrade Now | Random ASCII

  10. Pingback: The New Xperf is Here! | Random ASCII

  11. Pingback: ETW Central | Random ASCII

  12. Pingback: The New WPA Xperf Trace Viewer–New Bugs and Old | Random ASCII

  13. Pingback: Xperf for Excess CPU Consumption: WPA edition | Random ASCII

  14. Pingback: Xperf and Visual Studio: The Case of the Breakpoint Hangs | Random ASCII

  15. Pingback: Visual Studio Single Step Performance Fixes | Random ASCII

  16. Pingback: You Got Your Web Browser in my Compiler! | Random ASCII

  17. Pingback: Self Inflicted Denial of Service in Visual Studio Search | Random ASCII

  18. Pingback: Xperf Symbol Loading Pitfalls | Random ASCII

  19. Pingback: Windows Slowdown, Investigated and Identified | Random ASCII

  20. Pingback: PowerPoint Poor Performance Problem | Random ASCII

  21. David Poole says:

    Probably a security thing but the chrome://flags link doesn’t work.

    trace-export brilliant! Didn’t know about this.

    Just had a tiny play with UIforETW and it didn’t load chrome.pdb initially, I assume it is meant to?

    Thanks for this! Chrome has a few issues on Windows for me and this will help me figure out what it is, much more easily!

    • brucedawson says:

      Probably a security thing indeed – I’ll make chrome://flags not be a link to avoid confusion.

      WPA will load Chrome symbols if Chrome’s symbol server is in _NT_SYMBOL_PATH. If you have Chrome developer checked in the Settings dialog *and* if you don’t have _NT_SYMBOL_PATH set yourself then UIforETW will set it to include the Chrome symbol server.

      If these conditions are true when you record a trace then UIforETW will also strip the chrome symbols to dramatically (50:1 !!!) speed up loading of Chrome’s symbols. So, if you check Chrome developer after recording a trace (or if you receive a trace from someone else with Chrome in it) you should right-click, Scripts, Strip Chrome Symbols to do this symbol optimization step.

      If you record traces that show Chrome problems then please share them with me at my brucedawson address at Google.

  22. Claus Valca says:

    I’m in awe! Thank you for developing this tool and all the blog posting you have been sharing. I just found your site and am pouring over it.

    I don’t code, but do perform Win performance troubleshooting and your site and utility should really help expand my knowledge and skill set for trace captures and analysis when I’m trying to figure out what process(s) are causing performance issues on a system.

    A humble post of mine at my amateur attempts at performance tracing from a few years back for some context; http://grandstreamdreams.blogspot.com/2012/04/case-of-unexplained-donut-of-death.html

    I carry “portable” (to the degree they can be) packages of the Win 10 and 8.1 WPTs on a portable USB stick so I don’t have to install/impact the systems I’m troubleshooting.

    I’ve found I can then fire off my traces and do analysis (capture details obviously dependent on OS of system limitations) with the xperf version I am using (ie: using the WPT 10 on a Win 7 system).

    Question please.

    Is there a way to configure/use UIforETW with one of these “portable” sets instead of going through the “install” routine and putting the WPT bits on the system.

    Right now most of the support work I do remains for Win 7 (x64) systems so being able to use a tool like this with one of Win 8.1/10 WPT sets (without installing it) would be helpful.



    Claus V.

    • brucedawson says:

      You make a good point. Since WPT can run without being installed it is technically easy to make a completely stand-alone UIforETW package that contains its own local WPT copies.

      There are legal reason why I can’t redistribute UIforETW that way, but here’s what could be done:
      Step 1) Write a simple batch file that copies the installed WPT files to directories inside an etwpackage directory.
      Step 2) Tweak UIforETW so it looks for WPT locally first and uses the local copy if found.

      Then you would just need to run the batch file and copy the result to your thumb drive.

      Simple enough. Feel free to submit a pull request, or I might get to it some day.

  23. richiek says:

    Thanks for the great tool.

    Is it possible to limit the ETW trace output to the data from one process, or even one thread within one process?

    The reason I want to do this (besides .etl file size reduction) is that when I use WPA to analyze the ETW trace and I “Filter To Selection” for the necessary process/Thread ID, the column “% Weight” never seems to be re-rooted to the current filtered output. Perhaps I’m doing something wrong here – I’m only very new to this.

    • brucedawson says:

      Sorry, not possible, as far as I know. ETW is inherently a whole-system profiling system. This is *usually* what you want (whether you know it or not) because, for instance, the system process does page-zeroing and disk I/O on your behalf, and other processes compete for resources. Anyway, regardless, not possible as far as I know.

      As for the % Weight and other CPU usage percentages, those are always relative to the total CPU power available. Making them relative to a single core would be nice, but that is orthogonal to how many processes are recorded or viewed.

      • richiek says:

        Ah, ok. Thanks for the quick reply!
        I totally misunderstood the basis for that % Weight column, but I suppose as long as all values are relative it’s intuitive enough to work with.

        I think I’ll have to pause and actually read the docs/watch your video.
        Thanks again.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s