System Process using 25% CPU constantly traced to CPQFCAC.SYS thread

[Vinnie D'Alessandro]

Please accept my apologies if this is not the correct forum, but I'm really not sure where to drop this one...

Hi,
We have a system where the system process is using 25% CPU utilisation. We have an identical system that averages 1-2% for the system process. The system is a dual CPU HPDL360 running hyperthreading, win2k Server Sp4, and has been running without problems for 3 years. It is connected to a SAN via a MSA1000.There have been no recent changes and this problem appeared to manifest itself after the site was recently closed down for a few hours to allow some UPS reconfiguration.
I used Process explorer and found that the thread CPQFCAC.SYS was utilising 25% CPU and performing an awful lot of context switches. I later discovered taht this is the Fibre Channel notification driver. The adapter in question is an emulex adapter. I talked to HP and they got me to update the driver - no change. Then they came out, and to cut a long story short, the engineer proved it wasn't a hardware issue.
What I'm looking for is if anyone has any pointers to any documentation or tools that will allow me to understand what this driver does and so pick up some clues as to where to go from here.

Any help would be greatly appreciated

Cheers
Steve

[Andrey A. Khlebnikov]

Hi.
Recently I got the same situation.
I have the MSA1500 with v7.00 A/A firmware, four MSA30 enclosures, two SAN 4/16 switches operating in redundant configuration and a pack of HP DL380G4 and BL20p servers with Windows 2003 Server (Standard and Enterprise SP1) installed on them.

This problem appeared right after new firmware was flashed to the MSA and CPQFCAC.sys driver v5.32.0.32 was installed on all of these servers.

Because each server has four redundand paths to the MSA, there are four "StorageWorks MSA1000/1500" devices in the System Devices section of Device Manager. Each device corresponds to a thread in System process (PID 4). Two of these four threads are constantly use CPU resources (25% per thread on a 2 CPU server with HT) and make a great number of context switches.

The only workaround I found is to disable CPQFCAC.sys driver in registry and reboot the servers.

If anyone has a clue on this problem, please talk back to me.

[Michael Richter]

Anybody know if this issue is Emulex based?

cheers
mike

[Mike Celone]

It appears I am not the only one with this problem either. I noticed also after I installed the latest MSA1500 firmware and having CPQFCAC with version 5.32.0.32 installed. I have a case open with HP and we are currently working on the issue.

Has anyone tried going back to CPQFCAC.sys version 5.30.0.32 yet?

I have used Process Explorer to suspend the CPQFCAC.sys files from loading on the affected servers for now.

[Mike Celone]

Michael,

I don't believe this is an Emulex problem because I have a QLogic controller and I'm having the same issue.

[Andrey A. Khlebnikov]

This problem is definitely not an Emulex-based. All of my Fibre Channel cards are Qlogic (HP FCA2214DC on a PCI-X bus and HP Blade p-class mezzanine cards for HP BL20pG3, BL25p and BL35p servers)

As for a CPQFCAC version downgrade to v5.30.0.32, I've tried to do that. No luck. It does not solve the problem.

[Mike Celone]

On one machine that has a QLA200 controller in it downgrading to CPQFCAC.SYS 5.30.0.32 has fixed the problem. I have not tried this on one of our machines with a FCA2214 controller in it yet.

[B Contario]

I will add another hat in the ring. We have a brand-new DL380 G5 with a QLogic PCI-X (not express) card connected to an MSA1500cs active/passive through the HP/Brocade 4/8 FC Switch. Previously we had a DL380 (G1) with a QLogic card connected directly to the MSA1500cs with only one (active) controller. Old config was rock solid, new one is taking up all CPU time on one core of a dual core processor (50% cpu time).

So we changed a lot of variables, and we also are getting random temporary "hangs" where the system stops responding to EVERYTHING but pings for 3-4 minutes and then it returns to normal, with NOTHING showing up in the event log, but our monitoring software shows everything is down but Ping and our servers lose their SMB network share connections and Backup Exec Continuous Protection job connection.

However, the one thing I did not notice is if we had the CPU % issue back before we upgraded the MSA controller firmware from v4.xx to v5.20 Active/Passive. We definitely had the "temporary hang" problem before we upgraded to the v5.20 firmware on the MSA1500, but I don't know about the CPU %.

Since this just happened when introducing this whole new DL380 G5 server I can't say if if is related to a Windows update, a driver update, or a firmware update, but hopefully some others can shed some light on when this popped up for them.

It sounds as if the FC HBA card is not important, but just reply if anyone wants the model of the QLogic card

Also, my CPQFCAC.SYS file is dated 09-Sep-2005 and shows version 5.32.0.32 Build 3 (x86).

Thanks!

[pasquino bocci_1]

Hi,

What version of PSP are you running?

A lot of problems with V7.40 are now solved by 7.70.


http://209.85.135.104/search?q=cache:nKg8kj7WiZgJ:h20000.www2.hp.com/bc/docs/support/SupportManual/c00600028/c00600028.pdf+%2Bcpqfcac+%2Bcpu+%2Butilization&hl=it&ct=clnk&cd=1&gl=it#18
Pasquino

[B Contario]

I just installed the 7.70 Product Support Pack on Sunday, and the high CPU usage continues and the "temporary pause" happened again about an hour after my posting overnight. So 7.70 did not help my situation.

For others that are having the high CPU issue, are you connecting direct fiber from HBA to MSA, or are you going through a FC switch?

[Mike Celone]

We are connecting through an HP Storageworks 2/24 Edge switch. We have 3 MSA1500s on our SAN all at 5.20 firmware.

[pasquino bocci_1]

Hi,

it seems this problem happens on MSA with ACTIVE/ACTIVE config. Try to downgrade the fw to 5.20 or 5.10.
Pasquino

[Andrey A. Khlebnikov]

Is it real to downgrade firmware from v7.00 to v5.xx? The manual says that this procedure will clear all MSA configuration and all data on the disk array will be lost.

Installation of Support Pack v7.70 does not solve the problem. Moreover, the installer of this pack hungs for approx. 2 hours if installation is performed while CPQFCAC.sys threads in the System process are suspended by ProcessExplorer. The installer hungs while it tries to reinstall CPQFCAC.sys and (a strange thing) network card drivers (HP NC7781 Gigabit Server Adapters on HP BL20pG3 server). After 2 hours the installer shows the installation log with timeout error messages for these drivers installation procedures.

[B Contario]

Hey, don't bother downgrading. I'm on 5.20 Active/Passive with the problem. It is totally something else.

I have a case open with HP support now, too, so I will post any results I get. I will be changing from the QLogic STORport to the SCSIport driver this weekend, although since the original post is on Win2K which doesn't support STORport I am sure it will not help, but will try it just to keep support from stalling.

-Brian

[B Contario]

Another question to try to establish a commonality and eliminate possible causes:

Is there ANYONE here with the problem that is connected directly from the HBA to the MSA FC port with NO FC switch?

Our original good config was directly connected like that, but the config with the problem has the HP/Brocade FC switch in between. Maybe this is an issue provoked by using a FC switch?

[Andrey A. Khlebnikov]

My good configuration initially contsisted of two HP/Brocade SAN 4/16 switches and MSA1500cs with firmware v5.10 Active/Passive. The problem appeared right after MSA firmware was upgraded to v7.00 Active/Active and no changes in SAN topology were made.

[B Contario]

Ok, here is another possible variable.

I have Diskeeper 2007 installed and Backup Exec 11d (with Continuous Protection Server). I've tried disabling both, and it didn't seem to affect it, but am about to try uninstalling them to see.

-Brian

[John Kufrovich]

With this high of utilization, what does your system event logs show.

jk

[Michael Richter]

Hi Guys,
we solved the issue!

At first we tried to update with the Storport Hotfix from M$ http://support.microsoft.com/kb/916048 and updated to latest FW from Fiber HBA's 2.70a5 but the issue exist further ... After installing the latest ACU / ADU Versions 7.80.60 via Repository Manager (also available at HP Driver Section) the CPU load is at 0% when its idle. Seems the Tools obtain new Agent/Drivers ..

May anybody can confirm this too!?

rg, Mike

[B Contario]

We did a reboot a few days ago after no apparent real change and had 0% usage at idle. We then also uninstalled all of our Backup Exec and Diskeeper packages and installed the abovementioned MS Storport fix and I do not believe we have seen the high CPU usage. I will set up a special monitor to watch for the high 100% usage on one CPU.

We did, however, still see the temporary hang issue I wrote about earlier in this thread, but I am suspecting the two issues are not directly related. But we have only had it up for around 33 hours, so I will watch closely to make sure we are seeing the same thing for sure.

HP support is suggesting swapping the QLogic FC cards, so that may be next during our weekend maintenance window.

[Michael Richter]

Same Solution another configuration!

Blade P-Class Center with Bl20p G3 Blade and QLogic FC HBA, 14 MSA 1500 attached.

The CPU utilisation was about 100% when creating new raid arrays on the MSA Shelf's.

Every MSA Controller has its own Thread/Handle in CPQFCAC.SYS and u could see that the Thread which belongs to the controller which perform a raid creation uses about 25% .. when u count the threads u can imagine the load from the CPU.

In Detail we could find out that installing the ADU in Version 7.80.60 fix the issue ... its not clear what causes the issue ..

good to know is if you're using Secure Path or MPIO

rg, mike

[CharlieChoate]

Hello - Thanks for the info. I just discovered this issue today on our newly installed SQL cluster. Our configuration is similar as deascribed above: 2 DL380 G5s (Win 2K3 SP2) running through two MSA1000 controllers (recently upgraded firmware running in active/active mode) and dual fiber brocade switches.

I noticed this morning that the system process was running at 25% referencing the CPQFCAC.sys file. As noted above I upgraded the ADC/ACU to ver 7.80.6 & this helped. Now the process runs at 13% instead of 25% on both nodes. Any ideas what might be the final piece in getting this resolved? Is this an SP2-realted issue?

thanks
Charlie

[Glenn Strafford]

I am having this issue as well. I recently moved to the 5.20 A/P firmware for my MSA1000. I just happened to notice today while moving the SQL database that the CPU was always at or above 25%. This was with no users on the system and no apps running. System was taking up 25% of the total CPU on a Quad processor system.
I tried the aforementioned suggestions of installing ACU/ADU 7.80 and that did not help. Has anyone else found a work around? And what exactly is the CPQFCAC.sys file for? I am guessing Compaq Fiber Channel Array Controller. So I am cautious to delete/disable it.

[Glenn Strafford]

Fibre Channel Array Notification driver 6.0.0.32 seemed to fix the problem. There was an issue loading the update initially. It would peg the cpu at 100% during the update and never finish. After a reboot it showed that the new version was installed; however the problem was still there with the "system" process taking up 25% CPU. A re-install and reboot of this driver fixed the problem for me.
The re-install went through smoothly as opposed to the initial install.

[Andrey A. Khlebnikov]

I have installed the cpqfcac.sys driver v6.0.0.32 (PSP 7.80) on one of the problem servers. It seems it's a solution, but to completely solve the problem I need to install new version of the driver on other servers. This can be done at the next maintenance window (approx. in two weeks).