9.5 Upgrade Experiences?

[Steve Burkett]

Anyone else got any tales of woe with 9.5?   Having to upgrade our 3 x P4500 G1's this weekend from 8.1 and not sure to aim for 9.5 or a previous version.

[Emilo]

One of the changes that they made in 9.5 had to do with the way the implemented flow control.

Previously on 9.0 flow control was only implmented as recieve on. It was recommended that you also enable flow control on the switch as well. Now with 9.5 flow control is implemented recieve and transmit. Which can adversly impact perfomance if your network is not setup correctly. Anyway without getting to much in depth, you can always tell if this is impacting you is to check your switch counters and see if you are seeing alot of pause frames and drops packets errors ect. If you are there are several things you can do to get back to pre 9.5 configuration. You can turn off flow control on the switch and turn it on the san as transmit only. You will need to clear you switch counters and then mounitor.

This will take a little time and some configuration changes on your San as long as you are using 2 way replication (network raid 10)  it shouldnt impact prodcution. You will need to break the bonds and set flow control on each NIC (you will have to assign the 2nd NIC a temporary address) once you set flow control you can recreate the bonds.

Switches - Most switches only support flow - control recieve, however newer models are starting to support the transmit also.

Bottom line its going to take some monitoring and clearing counters to find out if this is whats impacting your perfomance.  So check your switch counters

[Steve Burkett]

Well we took the plunge and went for 9.5 on our three P4500 G1 (SANiQ 8.1) nodes. Things went through without a hitch. I'd previously got them up to date with firmwares as per the SPOCK documentation, and had updated our DSM installs to the 9.5 version.

First thing it did was upload the software to the devices and then installed a small patch on each in turn before continuing with the main SANiQ 9.5 install. This took each node offline for about 5 minutes while it installed (very quick, less then a minute) and rebooted (not so quick). As the device came back online it proceeded with the next one.

After it had done all three it then banged through a few quick patches on each in turn and that was it. Total time about one hour (probably would be quicker if you had a 1Gbps link to your nodes from your CMC, I've only got a 100Mbps link so the initial software upload took a good 20 minutes).

Only problem I did have was with upgrading the VSS Provider on our SQL Server as part of the post-upgrade tasks, the installer seemed to hang trying to stop some services and the server became unresponsive. That required a reset of the VM. Once it came back up it allowed the uninstall of the 8.1 VSS Provider and the install of the new 9.5 version.

Did notice that the 9.5 nodes had 'Auto - Receive On, Transmit On' on them under the TCP Status tab of TCP/IP Network, I'm guessing they've auto-negotiated with our 3750G's to be able to do both. Under 8.1 it didn't indicate if Transmit Flow Control was enabled or not, just that Flow Control was enabled. Not currently seeing any errors on our Cisco ports.

One concern we had was how much space we needed available to be able to complete the upgrade successfully. We were indicating about 677GB free at the start and it looks like it used hardly anything to upgrade, so this fear was unfounded.

While it was installing part of the upgrade it was throwing up the infamous 'There was a problem getting the alarm list from .... ' warning, but this cleared within a few seconds and continued no problem.

So all in all, thumbs up from me, performance is fine, I now need to work out how to enable the VAAI support on our datastores as only some are showing as Supported currently, the others are showing as 'Unknown'. Not sure why that is as yet.

After we'd upgraded to 9.5, our new 9.5 P4500 G2 could be added in to the management group no worries.

[Russell Couch]

Hi Emilo

I'm having some similar issues with 9.5 after our upgrade from 8.5. I currently have a case open with HP but they are denying any changes with flow control from 8.5 through to 9.5. I'm tempted to go through with your suggestions but I'd rather get the go ahead from HP before doing so. Have you got any reference at all to the changes? Was this information direct from HP?

Thanks
Russ..

[ryan_1212]

If you download the file hist.ethtool.log from each NSM on Version 9.0, you will notice this:

Fri Nov 16 16:40:17 GMT 2012

Pause parameters for eth0:
Autonegotiate:    on
RX:        on
TX:        on

Pause parameters for eth1:
Autonegotiate:    on
RX:        on
TX:        on

This tells me that in 9.0, it was turned on for send and receive.

[Jonna85]

We had an issue when upgrading a management group with a mixture of SANIQ 8.5 through 9.5 nodes. One cluster got stuck in a VIP lock during the upgrade and this required a senior HP engineer to resolve. Not a nice situation and at one point HP were suggesting that we would need to rebuild the cluster.

The cause was determined to be a manager was running on an 8.5 node and this caused the upgrade to fail.