The 8.1 and above versions of xperf/WPA/WPT comes with a tool called wpaexporter. This tool works as promised and lets you export arbitrary summary tables to .csv files, thus allowing for automated analysis of xperf traces.
The documentation for wpaexporter is pretty light – it just explains the command line options. It doesn’t bother to explain how the profile file that you provide is translated into a .csv file. In particular, it doesn’t say how grouping affects the results. I had to experiment for a little while to figure out how it worked.
(for an example of using wpaexporter see Process Tree from an Xperf Trace)
It turns out that the design is quite simple. Set up your summary table and look at it in WPA without expanding any of the grouping columns. This is what your .csv file will look like. It’s exactly the same as doing a copy/paste of the entire unexpanded summary table. This means that if you have more than one column to the left of the orange grouping bar then those extra columns are not used for any sort of grouping. It also means that if you have any grouping columns then any data columns which cannot be summarized will not be displayed.
In short, you can group by one column and get summarized data for each row, or you can do no grouping and get raw data. Those are your two options.
Update: see Kyle Sabo’s comment below for how to expand particular columns in order to gain more data display options.
And filtering is good.
Getting a data summary
One use for wpaexporter would be to get a summary of the CPU time consumed by every process on the system. The most accurate CPU usage data comes from the context switch information that is shown in the CPU Usage (Precise) table.
(for documentation of the columns in this graph see The Lost Xperf Documentation–CPU Scheduling)
You can get per-process data by adjusting the summary table to have New Process as the only column to the left of the orange grouping bar. Or you could have New Process Name instead if you want multiple processes with the same name collected together. Then, to the right of the grouping bar put your data columns. Columns like New Thread Id don’t make sense because they can’t be summarized – the sum or average of thread IDs isn’t meaningful. Here’s one possible layout:
In this layout the first data column records the number of context switch events per process, and the last column records how many ms of CPU time were consumed. Note that I have two TimeSinceLast (us) columns – one is the sum of all of the switched out time, and the other is the maximum switched out time. When using a column which has multiple options for how they will be summarized you need to choose the one that makes most sense for your scenario.
Now you can export this profile using the Profiles menu, Export… command. I saved it as:
Then you run wpaexporter like this:
wpaexporter test.etl -profile CPUUsageByProcess.wpaProfile
The name of the file created depends on the name of the summary table and the name of the view you created. To set the name of your view open the View Editor (Ctrl+E), click on Manage…, and type in a name for your preset. In my case I named the view ByProcess so this created a file called CPU_Usage_(Precise)_ByProcess.csv which I could then load in to Excel. The data is identical to the unexpanded data that was shown in WPA – except for formatting differences and the loss of the sum and max subscripts on some columns.
Not bad. It’s not hard to imagine writing some tools to parse this and issue warnings about excessive background CPU usage.
Getting all the data
Sometimes you don’t want a summary, you want all of the data of a particular type. That might mean all context switches, all CPU samples, all file I/O, all disk I/O, etc. Let’s take CPU sampling data as an example.
(for documentation of the columns in this graph see The Lost Xperf Documentation–CPU sampling)
Normally you would group by process, thread, and stack and then drill down in the UI, but with wpaexporter that doesn’t work. Those grouping columns will all be collapsed and you’ll just get a summary. Instead you have to drag the orange grouping bar to the left and you end up with something like this:
This is quite useless in the UI because without grouping the volume of data is too great – and most of the samples are from the idle thread! You could export this data but it would probably contain more information than you really need. So, you need to filter. Start by opening the View Editor (Ctrl+E), then click Advanced. You’ll be on the Filter tab and you can arbitrarily add to the filter by clicking the green plus sign. You could filter out all rows where the Process Name equals Idle:
Note that for this to work you have to manually type in “OR” in front of the new expression that WPA adds.
Or, you could filter out all rows where the Process Name does not equal WPRUI.exe:
Again, don’t forget to manually type in “OR” or “AND” or whatever – I’ve circled where you need to put it in the picture above.
With the filter in place the raw data looks like this:
I removed the Count column because with this layout it will always be one. The weight column is still useful because if you change the sampling frequency then the weight (length in ms) of each sample will change.
In this case I named my view WPRUI_Stacks. After saving the profile you just have to run wpaexporter again – this time adding “-symbols” to the end so that the stacks are exported with symbols.
wpaexporter test.etl -profile CPUSamplesWPRUI.wpaProfile -symbols
Again, just load the resulting .csv file into Excel and you can see the raw data:
Loading the data into Excel isn’t how you will normally use the data – the whole point is to write scripts to process it and look for common patterns – however loading it into Excel or a text editor is useful for confirming that you have exported the data that you are interested in.
When parsing the data be aware that some symbols, especially for template functions, contain commas. These names will be properly quoted in the .csv file, just be aware that you have to handle this case.
That’s all folks
That’s really all there is to it. The feature works and it means that you can now export any data that you want from an ETL file. I can think of a couple of uses for this right now. I should probably update my flamegraph scripts – let me know if you do that.
I’m pleased that this feature exists now. However I’m always greedy so here is how I think this feature could be improved.
- There should be an option to send the results to stdout. This would allow the output to be processed and filtered before going to disk and it would simplify the command line options. Learn from Unix.
- wpaexporter should print the name of the file that it saves the data to. It’s a little thing but it would get rid of another possible source of confusion for new users.
- wpaeporter should explicitly handle the case where it’s output file is locked – right now it displays the raw .Net call stack when the CreateFile operation fails. Sloppy.
- The rules for how a .profile file is mapped to a .csv file should be documented. Maybe I’m too dim to instantly figure it out, or maybe I’m too smart to discard the many other ways that the translation could have been done, but either way it would be nice to have it explained (for those who don’t find this blog post).
But hey, cool feature. It handily beats the previously available alternatives.