Data Centre Move - lift and shift - VMS HPrx8640 / 7640

[The Gunners]

Hi There , we are moving systems (HP rx8640 (integrity) and a HP rx7640 (They are a 2 node VMS cluster) to a new data centre. Apart from having the right cables / space etc in the new data centre , has anyone had issues with moving/powering off/ back on at remore site these type of servers. Just putting the question out there in case someone has or if someone has any experience of such a move and what issues came up. We dont envisage any issues (fingers crossed)

[Robert Gezelter · Accepted solution]

Davey,

The operative (pardon the pun) issue is whether one can tolerate the downtime while the systems are relocated.

Often, a move of this type is a good opportunity to expand the cluster to an additional member, which makes it possible maintain operations without interruption.

"Powering on/of" is the least of the problems. As I noted, it is the downtime that is the most serious obstacle. Also, what are the current arrangements for mass storage?

- Bob Gezelter, http://www.rlgsc.com

[Ian Miller.]

The issues with moves are mostly political - e.g. getting all interested parties to agree to the downtime.

However, the usual hardware issues are that the storage refuses to work after the move (ensure plenty of backups and test your restore procedure), or that memory, cpu, option cards dislodged and require reseating.

I've always used HP/Digital Relocation Services and had Field Service lined up at both sites.

[Andy Bustamante]

>>>I've always used HP/Digital Relocation Services and had Field Service lined up at both sites.

For a system that needs to back in service, having the hardware vendor handle the move is insurance. The most common issue is going to be disk failure after the move. Make sure you have current back ups.

I once moved a 5 node cluster with minimal downtime. By adding a new storage array at the delivery site, I was able to move 2 nodes over a weekend. Once installed and restored, pulling incremental data from the production site let me migrate users to the new location group by group. Once everyone was at the new site weeks later, I shipped the remaining nodes and the older storage array. My problem would have been in handling all users in the event of a hardware issue.

My worst case was having a new system rammed by an 18 wheeler within site of the final customer. Bits of bytes scattered all down the road, and a pushed scheduled.

[John Gillings]

Davey,

Depending on distance, and how the system is currently configured, this can be done without downtime. Start by extending the cluster to the new site with one system. Once it's established and all the storage has shadowed across, start shutting down and moving the other nodes.

Talk to Jan van den Ende, he's done this a few times. Continuous uptime with at least 2 complete cycles of new hardware and physical data centre moves.

[Steve Reece_3]

I wouldn't expect anything specific about these servers compared with any others when you power them down, move them, then power them back up again.

Issues I've experienced previously are power supplies failing, CPUs failing and disks failing. The most likely time for things to fail is when they get powered up again (think of a light bulb - it doesn't fail whilst it's lit, it fails when you power it on!)

Remember that you may need to repoint the EFI boot device if the path to the boot disk changes.

My approach would always be to get the maintenance provider (be that HP or others) to do such a move since any issues would be their problem rather than mine.

Steve

[The Gunners]

Hi Guys,
Bob/Ian/Andy/John/Steve,
Thanks a million for your inputs. This gives me a good scop to start discussing with the business what way we should best do this. I really just wanted to get a rough idea as this is the first time I have undertaken such a project.
FAIL TO PLAN.......PLAN TO FAIL will deffo be in practice in this one :)
Points on the way :)

[Bob Blunt]

Davey, there are a thousand more common issues that I've hit when moving gear from site A to B. The most frequent, and the items that are usually considered the most simple continue to trip up my customers and me are:

misunderstandings about required power
incorrect outlets
underestimates about HVAC requirements
lack of communication about moving systems into a building (no loading dock, elevator restrictions, etc)
manpower requirements differing between sites
raised floor loading limits

I've had in-data center moves fail miserably because we were moving from free-standing systems to rack-mounting and noone bothered to compare the power requirements of the racks versus the individual systems. Yes, we had ample power for 20 workstations until we put them into a rack that used 3-phase power instead of 120V single phase. The excuse at the time was "we gave you what you had!" Although it was fixed with a quick trip to Radio Shack for surge-protected power strips the problem could have been solved before we moved. Every now and then we did have an incorrectly wired outlet that would cause panic because plugging in a rack would shoot sparks and take out a power cabinet.

In general, given the age of your hardware, you should be able to follow standard shutdown and startup procedures. In general the older gear gets more and more sensitive to being slammed around. In general you might want to get spares of your more common disks and carefully make (or take) backups or snapshots of your data when quiesced. The main failure I've seen have been disks and the second-most common has been power supplies on older systems and racks. It never failed that we'd cautiously move something and at the destination we'd carefully and sequentially power up a storage rack and find that a stack of SCSI disks would be failed and without spares or tape backup we'd be effectively down until we could somehow get replacements and rebuild.

bob

[abrsvc]

I second John G.'s suggestion. I have implemented the extended cluster option a few times with total success each time.

Dan

[Dave Sullivan_3]

Not knowing Davey's luxury of downtime can send us in many directions. Also not knowing how his cluster is setup to run is another. If extending a cluster is not an option IE: 8.4 not installed, I never like to mix a move with an upgrade, too many moving parts (there's a nice pun). I don't know if the storage is moving as well or if there is new storage because the plan to move the data will be much more important and time consuming then moving systems. To strip this down to the simplest form, Davey, let me know if your exisiting storage is moving or not and if you can run your business with only one node operational. The answer to these questions will allow us to point you in a better direction.

Thanks,

-Dave Sullivan

[The Gunners]

Hi Dave, We are holding a meeting next week to try nail down what way we want to do this. I will update more info after that. I just wanted to get a general idea of DCM's I guess , and I have plenty to go on so far thanks to all of you. I will post more detail when I get the ok from the business and hopefully get some more invaluable ideas and help from you guys.

[Jan van den Ende]

John G wrote

>>>
Talk to Jan van den Ende, he's done this a few times. Continuous uptime with at least 2 complete cycles of new hardware and physical data centre moves.
<<<

Thnx John!

Yes, we did.

As long as you got connectivity between the site (geographically redundant!!) just ADD a (maybe rental extra) node, configure it as homologueously as possible. INCLUDE shadowing, with the extra node as serving node. Exercise a reboot. TEST the user functionality!
At a convenient, (as defined by relative quietness, NOT human convenience) shutdown the extra node, tranport it, reboot, catch up shadowing (with minimerge nowadays seconds instead of hours) and TEST,TEST.
Test out user functionality from the new site.
Remove user activity from one of the nodes at the original site.
Repeat the moving.
And the last node idem.
If all works well at the new site, shut down the rental and return it.

DO HAVE hardware support involved, as in, HAVE THEM on-site.

And be prepared to backstep EVERY step, maybe 2 or 3 steps back.

Even after a number of such operations, be aware of and prepared for freak accidents.
(we DID have one of our intersite links disrupted by groundworks we had no idea of until afterwards).

But, all in all, it is quite doable.
The keywords are "be prepared" and "maintain redundancy".

Success!

Proost.

Have one on me.

jpe

[Andy Bustamante]

Adding another note. Schedule enough downtime so your users will be prepared. Make sure the notice is out early and you have time to rebuild RAID sets, restore from tape and still make your target.

I found out after one move that my customer's help desk was running a pool for "system back online" with times up to 4 hours over due. The winner was the guy who bet my scheduled.