8.5 -> 9.0 upgrade hanging waiting for reboot

[Matthew D]

I have two P4300 G2 clusters, geographically seperated, running version 8.5.  I paused all remote copy schedules, upgraded to the latest 9.0 CMC and then started the automated upgrade process about 80 minutes ago.  Everything ran okay until 30 minutes ago when it started hanging at 30% on "Waiting on trsan2 to reboot in order to proceed" and "Searching for 1 system via TCP".  I've never had to wait 30 minutes for an NSM to reboot so it seems to me that something is wrong.  Also, when I hover over the NSM graphic it says "Applying package 9.0.00.3561.0" and I can no longer ping the NSM in question.  Luckily my cluster is still online.  Unfortunately the NSM in question is an hour drive away!

How long should it take to reboot after applying this package?  Is this normal?  Should I abort the installation and/or manually power cycle the NSM?  Thanks.

Matt

[Matthew D]

As it turns out the storage node is completely hosed.  It is in an endless cycle of reboots, each time the RAID controller complaining about a different set of disks being bad.  I'm thinking it's a bad RAID controller because what are the odds that 2-3 random disks keep failing when they were all working just hours ago??  Anyway... I will probably need to rebuild this storage node and I have never done that, nor do I know how to rebuild a node mid-upgrade.  I put a ticket in with HP 2.5 hours ago and am waiting for a response from support.  I've had really good experiences with support until recently so hopefully my last experience was a fluke and not the new trend.  Anyone have a similar issue?  How brutal is it to restore a storage node while an upgrade is running?  Will the storage cluster be in a usable state if I abort the installation?

[Emilo]

You have a couple of options here.

#1) You will need to re-init the controller,

You will neet to power off the San then  disengage all the drives from the cages, don't remove them just make sure you release them so that they are not active.

Restart the SAN and let the controller re-initialize, it will probably re-boot

Power it down and re-engage the drives.

Power it up see if the all the lights are green

If this doesn't work insert the smartstart CD and run all the firmware upgrades

Make sure you check the fw upgrades list and you stay within those guidelines

Firmware matrix
http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_iscsi.html&lang=en&cc=us&hpappid=hppcf
The document name is "P4300 & P4500 Firmware Compatibility Matrix'
Once this is complete it is quite possible you have multiple drive failures 

Check in the CMC and look at the storage tab and see the status of the drives

If some show offline you will to replace those drives