BL680c high CPU usage, interrupts on VMware ESX 3.5 due to USB

[Joseph Arauzo]

Folks,

Has anyone experienced this before?

We have ten ProLiant BL680c servers in two C7000 enclosures.

Onboard Administrator FW version : 2.40
BL680C system ROM version : 09/23/2008, Family 386I17, Type 03
Smart Array P400i FW version : 5.22
ILO2 FW version : 1.70

VMware ESX 3.5 Update 3 + all current patches


Problem:
All ten of our hosts are experiencing very high CPU utilization on only the first CPU (there are 16 cores). This adversely impacts the VMware Service Console. VMware Support has been working on this issue for over a week. We were surprised when we ran the following command.

less -s /proc/interrupts

CPU0
0: 8784379 vmnix-edge timer
1: 3 vmnix-edge keyboard
2: 4965963 vmnix-edge VMnix interrupt
16: 1080207399 vmnix-level usb-uhci, ehci_hcd
17: 1096889075 vmnix-level usb-uhci
18: 1078010075 vmnix-level usb-uhci
19: 0 vmnix-level usb-uhci
21: 213598 vmnix-level usb-uhci
NMI: 0
LOC: 0
ERR: 0
MIS: 0
interrupts (END)

It seems that USB is responsible for the numerous CPU interrupts. We rebooted the server and disabled USB on the system board. Running the same command revealed the following.

CPU0
0: 350064 vmnix-edge timer
1: 3 vmnix-edge keyboard
2: 624957 vmnix-edge VMnix interrupt
20: 168156 vmnix-level ipmi_si
21: 25778 vmnix-level usb-uhci, cpqci
NMI: 0
LOC: 0
ERR: 0
MIS: 0
interrupts (END)

As you can see, it is much better now.

Has anyone seen this problem before? We obviously don't want to disable USB on all our servers as this will disable advanced ILO features, such as Virtual Media. Since ILO uses USB to provide connectivity to virtual media, could this be a problem with ILO firmware?

Any help is much appreciated.

Thanks,

Joe
________________________________________________
Joseph M. Arauzo | Sr. Systems Engineer, VCP
Information Technology | Infrastructure Operations
Morrison & Foerster LLP, San Francisco
www.mofo.com

[Tim Puszynski]

Interestingly I found this thread and am experiencing this on a BL460c. It is in a C7000 chassis as well. I am running ESX 3.5 U4. I just saw this on a single server after vmotioning some servers around. Guests show 15 Mhz cpu each (I only had 2 servers on single host) yet host shows 3000 Mhz in use. After a vmotion to get guests off this host and a reboot, cpu goes back to "normal." A vmotion to get all guests off host did not resolve issue. I am looking in to this as well.

[UK-Blr]

Please refer this KB article on Disbaling USB. I am sure this will solve/reduce the IO conflicts.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003710

--Uday