EVA 4400: Controller 2 down, cannot be restarted due to Error: Status code: 6003

[sam bell]

Hi,

about an hour ago I received a message from WEBES that one controller (controller 2) in our EVA 4400 (XCS 09522000) has begun booting.

So...I checked Command View and can see that controller 2 is not available anymore. When looking at the WOCP's it shows me that the Management Link to controller 2 has failed. Controller 1 is working fine, all paths to controller 2 however are down.

When trying to reboot the controller using WOCP I get the error message "Command execution failed. Error: Status code: 6003".

The controller event log in Command View shows the following error message:

"HSV300 controller operation was terminated due to an unrecoverable event detected by either software or hardware or due to an action initiated via the Storage System Management Interface".

As I'm not on site anymore I haven't tried removing power from the controller yet.

The system is not in production yet. The only things I changed today was setting the management controller for four LUNs to controller 2 (the one that failed).

Any suggestions? Would you try removing power of the failing controller and see what happens or rather recommend to leave everything as it currently is and have HP investigate further? It's my first EVA so I'm not sure what to do.

Thanks

[Víctor Cespón]

The only way to know what's going on is to let an HP technician connect to the controller serial port and see what messages it does give during boot.

The 09522000 firmware was installed recently? I had a case like this on a customer right after firmware upgrade. The new firmware on controller 2 was not starting correctly and we had to remove and reinsert it.

[sam bell]

Thanks for your reply. The firmware was installed about four weeks ago by the HP technican while we were initially setting up the EVA. During the last 4 weeks I performed one planned shutdown and back then both controllers booted just fine.

The real question to me, however, is why the reboot occured in the first place. Have you been able to find the cause in your case at the customer?

As I've written above I changed the managing controller of four Vdisks to controller 2. Controller 2 was not used before, so maybe this is in indicator that prolly something is wrong with controller 2 (though it is not really in production yet there is already some load existing due to testing).

Regarding the controller logs - the controller 2 messale log in WOCP ends (if the bottom is the end, not the beginnig - there are no time indicators) with the following error message:

RTC (TOY clock) test (BIST TE 0xbb)
RTC Memory test
*** boot error report - 0000bb02,f4000000,00000000,000000ff,bb,05,1

Would you recommend keeping everything as it currently is and get HP to it, or would you try unplugging / powering off controller 2 first and see what happens?

[sam bell]

So, I've had a call with HP and after reviewing the EVA configuration and controller logs they asked me to reseat the affected controller.

After reseating, the controller automatically booted up fine. Command View, however, still reported "Soft Diagnostic Failure" as Operational State for the affected controller - but after manually rebooting the affected controller using WOCP it went to "Good" again.

So far everything looks fine again. After the reseat WEBES however reported the message "A Controller Diagnostic Operation Failed" (which seems to include information about the first error) to HP so it might be possible that HP is getting back to me once more.

[sam bell]

Accordint to HP the error was caused because the affected controller was temporarily overloaded with host I/O. As to HP the error is already known to engineering and will be fixed in the next XCS release scheduled for February 2010.