P4500 - Should volumes go offline during a resync & rebalance of the cluster?

[CS23]

Hi, excuse my lack of knowledge on this prodcut, I'm completely new to it.

One of our HP P4500  storage nodes went offline last night and came back online a few minutes later. It looks like this was due to it crashing.

When it came back online it showed that it had come back into the cluster with status Not Ready  followed by status UP. The production volumes then were resyncing & the staging volume was back online.  The SAN cluster then rebalanced before coming back online 10 minutes later.

Can someone confirm that its normal for the volumes to go offline when re-syncing and re-balancing the cluster? Seems strange and not very High Availablity-like.

[oikjn]

if you are running network raid 0 then yes that is normal, if you are running network raid 10, then no, that is not normal and typically means you haven't setup your MPIO correctly on your servers.

[CS23]

Thanks for your reply.

We have raid 10 but from what I can see its nothing to do with MPIO. When the node went offline we still had access to the data from what I can see.

The issue seemed to occur when the node came back online and the HP SAN was performing tasks to re-sync the disks. The actual logs show the volumes were taken offline by the SAN not that the network to the SAN was a problem.

[Gediminas Vilutis]

---

@CS23 wrote:

Thanks for your reply.

We have raid 10 but from what I can see its nothing to do with MPIO. When the node went offline we still had access to the data from what I can see.

The issue seemed to occur when the node came back online and the HP SAN was performing tasks to re-sync the disks. The actual logs show the volumes were taken offline by the SAN not that the network to the SAN was a problem.

---

What LH OS version do you have at the moment?

[CS23]

Looks like v9, I think HP will just tell me to update to the latest version.

Anyone have any experience of upgrading these SANs?

[oikjn]

Thats strange. I have never seen that as an issue before. I would talk to support before upgrading as if the problem happens when the node comes back online and not when it goes offline, you are doing to experience the same problem during the upgrade process as upgrades require a reboot for each node (which usually is not a problem for your servers, but apparently that is a problem for you). I have been running since at least v8.5 and haven't had a problem like that. Is there anything in the log that gives you a clue for why its going offline?

[Gediminas Vilutis]

---

@CS23 wrote:

Looks like v9, I think HP will just tell me to update to the latest version.

Anyone have any experience of upgrading these SANs?

---

LH OS versions prior to 10.5 had very painfull node resync process with big impact to normal operations. If cluster load is above 10-15% of its theoretical max performance (e.g. above 600-800 iops for 4 node 4500 G2 with 12x15 krpm disks), then during resync time we also had experience with disconnected volumes, iscsi timeouts, kernel panics on VM's, etc. On 10.5 this seems to be improved a lot. So go upgrade, but do it in off peak time :) 

Upgrade to 10.5 is straightforward - upgrade CMC to latest version first, then go with cluster upgrade. And after LH OS upgrade you can update firmware of hardware components (iLO, backplane firmware, etc) with HP SPP, LH OS CMC does not update these.