About to Fire My P2000 G3 SAN Out of a Cannon - PFU Loop

[VirtualJordo]

Hello, 

I'll come right out and admit I am an HP SAN noob. My background is in NetApp and I am new to my current organization which had implemented these SANs prior to my arrival. Yesterday my monitoring solution started kicking off alerts related to inaccessibility of the management portal on either controller (http monitor). At the same time the SAN itself generated an alert that the MC had restarted due to error correction. 

I restarted the MC on both controllers and gained access to the portal only to find two windows constantly cycling, one with the title "progress of PFU" and a big blue "i" with a circle around it and another with the same title that says transfer in progress and shows a progress bar that never indicates progress (images attached). 

It's absurd to me that this SAN would try and kick off a firmware update automatically, on hours, and without notification. But it is what it is. Now the management portal on both controllers is unusable, these two windows just keep cycling. 

I have...

- disabled PFU

- restarted the MC on both controllers multiple times

- reseated both controllers 

absolutely no change. 

I was hoping the community could point me in the right direction as I would rather light myself on fire then deal with HPs atrocious support who probably would not even bother to answer my inquiry till 2015 (as has been my consistent experience). 

I have attached images of the error windows for reference and am happy to generate any output the community may need.

Any help is genuinely appreciated. 

Regards, 

J 

[Torsten.]

>> It's absurd to me that this SAN would try and kick off a firmware update automatically, on hours, and without notification

Partner firmware update will occur if a controller came up with another firmware and PFU is enabled.

So this cannot happen on its own, unless someone insert another controller or the controllers run different firmware and somebody enables PFU.

You could try to access the controller by telnet/ssh and run

# show controllers

# show version

# show system

to start to investigate whats going on.

[VirtualJordo]

Torsten, 

Thank you for the speedy reply, it's much appreciated. 

I am a one man shop and I have applied no configurations to this SAN since the start of my tenure two months ago. It is using the configuration that was in place when I arrived. I am the only individual with authorization to access the management capabilities of either controller. When I received the alert I was in a WebEx, as I had been all morning. 

I have attempted to do my due diligence in terms of research, and I have run the suggested commands prior to posting here but could not glean anything from the output that I found useful to the issue at hand. 

I have attached the output of those commands for your review (IP, MAC, SN, excluded) and included the original events I received at the onset of the challenge.

Again thank you very much for your input. It's all worth it if I can learn something!  

First Event:

Event:

The Storage Controller is not receiving data from the Management Controller. (This is normal during firmware update.)

EVENT ID:#B1331

EVENT CODE:152

EVENT SEVERITY:Warning

EVENT TIME:2014-09-25 17:14:09

System Name:FS_SAN

System Location:

System Contact:

IP Address:

Additional Information:

  The Management Controller (MC) has not communicated with the Storage Controller (SC) for 15 minutes and may have failed.

  This event is initially logged as Informational severity. If the problem persists, this event is logged a second time as Warning severity and the MC is automatically restarted in an attempt to recover from the problem. Event 156 is then logged.

Second Event:

Event:

The Management Controller was restarted automatically by the Storage Controller for the purpose of error recovery.

EVENT ID:#B1332

EVENT CODE:156

EVENT SEVERITY:Warning

EVENT TIME:2014-09-25 17:14:09

System Name:FS_SAN

System Location:

System Contact:

IP Address:

Additional Information:

Recommended Action:

  - See the recommended actions for event 152, which is logged at approximately the same time.

[StephenWagner7]

It sounds like there may be something wrong with this unit.

Do you have any history on the unit, any information on previous hardware replacements from warranty, etc...?

Do you have any information going back further in the logs? Have you seen any other information in the logs going back far other than typical vDisk scans?

These units won't update their firmware automaticaly as someone else mentioned. The partner firmware update feature is used for when only one controller has a management interface on the network. The network connected controller will update, then update the other controller using the SANs internals. This is why it was suggested earlier that someone may have kicked off a firmware update.

[Torsten.]

>> The Management Controller (MC) has not communicated with the Storage Controller (SC) for 15 minutes and may have failed.

I think it was NOT a firmware update, but really a controller hang. Maybe bad hardware, maybe bad firmware. Consider to update the firmware now.

>> The partner firmware update feature is used for when only one controller has a management interface on the network.

This is not true.

PFU kicks in when you insert a controller with other (newer or older) firmware or when the first controller is updated. Then it will update the second controller that is running the older firmware.