HPUX server planning and move

[Raghunandhan Srinivasan]

Hi
This is the best forum I have ever come across so I am hoping to get some good advice here.
I need to come up with a project plan to move 4 HPUX servers (HP-UX B.11.00 U 9000/800 R390) running Oracle 11.03 from East coast to West Coast of the US. Once it comes to West coast, its name, network config, nfs mounts, backups, monitoring etc will change. These are mission critical 24x7 servers, so I need to plan thoroughly. I will be working with a DBA. From sysadmin point,
1) Does anyone have a template or a project plan that goes through a comprehensive list of things to do when a server migrates.
2) Does anyone have a checklist of what config. files need to be changed when the network configuration, name etc changes.
TIA for your time
Ragu

[Steven E. Protter]

You have a number of issues to prepare for.

Let me give a general outline.

1) Prepare for a complete loss of the servers. They are going to be on a truck. Trucks usually but do not always reach their destination. Have Ignite make_tape_recovery tapes ready and complete hardware specifications on hand in case you have to repladce the server.

2) Your basic configurations are all in the /etc directory. /etc/rc.config.d/netconf is important and will be changes due to the move.

Here is a basic list.

/etc/exports NFS
/etc/rc.config.d/netconf
/etc/hosts
/etc/lvmrc
/etc/fstab
contents of /etc/lvmconf directory
/etc/nsswitch.conf
/etc/resolv.conf

All will probably have to change.

Depending on the extent of the changes, many other changes to other files are possible.

SEP

[Raghunandhan Srinivasan]

Steve
Thx for your advice. To make matters complicated, I will be using Legato while the other company is using Veritas to backup. The DBA was paranoid on hearing that FQDN name will also change. He asked me if it is possible to retain the same FQDN name so that he does not have to go through every line of Oracle code that references this name. I said it is impossible. Would you agree? Thx
Ragu

[Steve Lewis]

We had to move 12 servers 200 miles a couple of years ago. We got HP to do it for us who subcontracted a specialist. All the went wrong was we lost a single disk in an old K class that hadn't been powered down in a while. Its the disks which are most likely to stop working, so it may be worth getting HP on standby with some spares just in case.

I will let others focus on networking changes and tell you about the other stuff.

The key issues we faced were these:
Getting the power details of all the existing servers and organising connections. We went from screwed-in power to commando plugs. You may need an electrical consultant to help with the peak output current etc. The new machines will put extra load on your HVACs, UPSs and generators so you will need to verify that you have the required facilities.
Checking whether you will need UPSs or whether they aren't allowed -we had to ditch ours.
Check the floor strength and ramps.
Check the a/c temp and humidity. Adding more servers could put additional load on your unit.
Verifying that you have sufficient network ports the other end.
Verifying that all the *other* network connections and lines had been installed beforehand and work, stuff like remote support lines, NTP, modems etc.
Pre-pack your machines, take an inventory of all the parts and label them up clearly for everyone. The removal guys just like to shift a large box, so it would help if you did some of the cable unplugging, coiling and tying-up. Label all the connections because if you plug something back in the wrong port it could get very confusing.
DNS changes and propogation.
Changes to all of your client machines that connect to the servers. Hopefully you won't have any hard-coded IPs in 10000 PCs across the country.
Change equipment address on your HP support contract.
Get SLAs in place for the new data centre.
Book hotels well in advance.
One interesting problem I once had was an E44 machine cabinet was too tall to fit into the elevator. Nobody thought of that one.

[Raghunandhan Srinivasan]

Thx, Steve for giving these useful tips. I will document the same. With uptimes of 700+ days,I am now worried of the servers will come up after a reboot leave alone move. Any more responses/tips. Thx
Ragu

[Devender Khatana]

Hi,

We also shifted aroud 10 servers couple of months back. This also included services being hired from HP experts. The most sensitive part which can be affected out of this is the Hard disks. I would suggest to get system packed in perfect packing material, getting this loaded / unloaded in your presence and guide them to handle boxes with care while loading /unloading.

HTH,
Devender

[Robert Bennett_3]

I just finished moving our data center.

Here are a few things to think about:

1. Label everything!! I used colored electrical tape to color code all cables/connections. It's very simple, red-to-red, blue-to-blue...It worked real well for me. Visio drawings are a life saver.

2. Ignite tapes right before moving.

3. Change networking (IP address, name,subnet, gateway ... )before shutting down servers so they come up as needed in new location.

4. Coordinate network connections as needed (lan speed and duplex settings) to be exactly the same - These seem to always cause headaches.

5. Un-manage them from ITO or whatever flavor of management you are using before shutting them down.

We only had a few glitches - and they all had to do with disk failure. Take an inventory of Disks and have a couple of each flavor on hand when you boot up just in case. Mirror your OS and have Ignite tapes on hand. We Mirror/alternate_root and ignite.

Reboot the servers while they are in the old environment, servers that have been up that long may have gremlins that will rear their head after a reboot, better to take care of them before the move. Servers that have been up that long haven't been patched in a while either, this may be the opportunity to bring them up-to-date.

Our worst nightmares were our 12H AutoRaids. Not one of them moved without losing a disk and losing two of them can turn into a challenge.

Those D's are bullet-proof, they were the least problem of any of our servers.

Good luck!

B

[Raghunandhan Srinivasan]

Thx, Robert. I will remember these important tips. All of you have responded kindly but it looks I might have to add up all of your suggestions together to come up with a complete project document. It may not still not be complete. If any of you is willing to send their document (with sensitive information removed), I can modify the same and present (with due acknowledgements:)) the same to my manager. Thx, again.
Ragu

[Florian Heigl (new acc)]

1) If You worry about the uptime, hardware status etc., I'd try to do get a 45-minute downtime (will be well enough for R-Classes) a week before the move. If You have a support contract, get a technician on-call or on-site for that time.

Shutdown and Power-off&unplug the systems, give them 10 minutes to cool of, turn them on again.
A harddisk, PSU or fan that want's to die will do now.
(Ok, to be honest, it might also die after cooling off a bit more, so You might also to have HP ready when You poweron the systems at Your west coast site.)

Things to expect:
S/W had undocumented, unsaved changes and fails to come up
H/W breaks at power-up
H/W breaks shortly after powerup
Everything of the above.

Also, have someone the electricity parameters at the new site - are there any means filtering power spikes, PSUs, etc.
You can't install any of these afterwards, and power spikes are great at killing harddrive motors ...

Also DO demand the q/a protocols of all LAN cabling at the new site, and take care to have some extra 30% of fibre cabling.

(At our D/R site, every 3rd fc or gbit cable was mismatched tx/rx, and there also were some broken cables...)

I'd even consider to break vg00 mirrors and
send one half of the disks via courier, same idea for a set of ignite tapes.


2)

/etc/rc.config.d/netconf
/etc/hosts
/etc/issue
/etc/resolv.conf

oracle tns config!

using set_parms is possible, but better don't use it on multihomed hosts, and I'd consider it a waste of time.

GSP lan address


We used about 100 hosts in one weekend with months of planning.
To be honest, check every physical parameter, have everything planned and in good shape, everything documented, be in good mood and expect a total disaster.

[Raghunandhan Srinivasan]

Thx, Florian. I will remember these tips as well. In a nutshell, I need to prepare for a complete disaster recover and therefore should plan as meticulously as possible. Thx, everyone for responding. Again, I repeat "this is the best forum I have seen on the Net".
Ragu