BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

VC Firmware Upgrades - What's Your Experience?

Julian Stenning
Frequent Advisor

VC Firmware Upgrades - What's Your Experience?

Hi -

Whilst I've been using VC modules for quite some time now (VC-Enet 1/10's and Flex-10's) and find them very good pieces of equipment, I still feel that performing a firmware upgrade can be a bit hit-and-miss.

Certainly we've had problems caused by bad config (uplinks spanning vc's in the early installs, poor cluster heartbeat setups etc), but even recently on a update we seem to experience firstly a failure in the auto update process (it went through the motions, but the revision was not updated) and then on the retry it taking both vc's in bays 1 and 2 down for a short period (causing VM hosts to think they'd lost cluster members & restarting vm's).

I'm curious to hear of others experiences & and nuggets of info you may have. From what I read HP say it should be possible to update the modules without outages and I can see that it should work, and I really want to a place when I can consider performing updates without having to schedule large outages.

Additionally, does stacking enclosures and staggering uplinks across enclosures provide additional benefits during firmware updates?

Regards,
Julian.
4 REPLIES
The Brit
Honored Contributor

Re: VC Firmware Upgrades - What's Your Experience?

I have never had any issues upgrading VC ENet modules (1/10G or Flex10) however I have had many problems upgrading FC modules.

I determined that most of the upgrade issues with FC modules are related to firewalls and Anti-Virus software.

One thing to note. The failover of the VC manager between modules takes ~25-30 seconds.
This is normally more than cluster heartbeats can handle. It is important to have your heartbeats using a pair of teamed NIC's (which of course, failover virtually immediately).

If you have your heartbeats and Network connectivity properly configured for failover, then there is no reason why your servers should be affected during upgrade (that I am aware of).

Having said that, at my site we do not perform VC upgrades during uptime, we always schedule a PM window. (primarily because my cluster communications is not on a teamed pair (at the moment))

HTH

Dave.
Julian Stenning
Frequent Advisor

Re: VC Firmware Upgrades - What's Your Experience?


Thanks Dave.

On the subject of teamed heartbeats for clusters (Microsoft Win2003 in this case), I read that the best practice is to not team the heartbeat interfaces. I've certainly had trouble in the past with these teamed and have since found that if you have a 'heartbeat A' and a 'heartbeat B' nic using different IP ranges this works better.

I need to look further at timeout values to make things a bit more tolerant to the failovers, hopefully this will help.

Julian.
HEM_2
Honored Contributor

Re: VC Firmware Upgrades - What's Your Experience?

Julian:

I would recommend using VCSU instead of the GUI. VCSU can run a non-intrusive healthcheck on your VC domain prior to performing an update.

If you have a fault-tolerant setup adding an activation delay of a minute or two to give the module that is getting activated a little more time to come back up before the next module is activated...

DELAY = OPTIONAL. Delay in minutes between activations of bays
-we = VC-Enet Delay
-wf = VC-FC Delay
Default is 0 or no delay. Maximum is 60 minutes

in case you aren't familiar with VCSU:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=3794423&prodNameId=3794431&swEnvOID=1005&swLang=13&mode=2&taskId=135&swItem=MTX-15cda0bebbbc4d8c861a7898f7


cjb_1
Trusted Contributor

Re: VC Firmware Upgrades - What's Your Experience?

I agree, vcsu is the best way I've seen to update the firmware.
As a rule the updates we've run have all gone ok first time. Occasionally one module will fail to update but you can usually get around this by resetting/reseating the offending module or simply re-running the update tool. There are a number of threads in this section which describe these issues and how they can be resolved.

As far as any outages go we haven't had any problems yet. Servers have resilient configs and handle individual port disconnections. Then again, it's more likely to be OS/application issues that trip you up.

That said, I'd always run updates at quiet times on business critical servers.