High Interrupt CPU Time Troubleshooting with XPerf

Recently our System Center Operations Manager reported high Interrupt CPU time on one of our Windows 2008 Domain Controllers. Even after a couple of hours the CPU time would remain high, so I started investigating this. I decided to use XPERF (download from Microsoft) for this.

I first confirmed that the Interrupt CPU time was indeed high by taking a quick look at “Process Explorer”. It showed that it was constantly at around 15%, which is definitely not common and often indicates a problem with hardware.

XPerf can be quite complicated and confusing, but it’s also extremely powerful and helpful in cases like this one. To get started, I had XPerf display a list of Kernel providers:

xperf -providers K

From the list returned I thought the group “DiagEasy” sounded promising. It includes the most important providers, so I started the trace by typing:

xperf -on DiagEasy

Since my CPU time was high all the time all I needed to do now was wait a little while and then stop the trace and write the output to a file:

xperf -d int.etl

To view the contents of the trace:

xperfview int.etl

This will display a graphical summary containing information about all providers in the DiagEasy group. The obvious start is to scroll down to the “Interrupt CPU Usage” graph. Just like Process Explorer had shown it displayed a constant usage of about 15%:

 

 

That much we knew already
To dig deeper and hopefully find the culprit, select a random time frame in the graph, right-click and click “Summary Table”:

 

This will bring up this list:

Line 1 immediately looks suspicious since this driver is not only on top of the list, but also had used by far more CPU time than anything else. From the name hpqilo2.sys I knew this was related to the HP Insight Lights out board, but I am sure a quick search on the Internet would have revealed that information, too. Since iLos can be restarted I decided to give this a shot first (e.g. go to Network settings and click Apply – even without any changes this will trigger a restart of the iLo). I waited a few seconds and watched the CPU usage in Process Explorer. Sure enough, after about half a minute my CPU usage went back to normal. I waited a little longer but the system remained stable, so for now I will leave this system alone, hoping it was a one-time fluke. If it happens again we will see if there are any updated drivers, and if this doesn’t help there might be a problem with the actual hardware.