conver a production server to SG cluster

[Hanry Zhou]

We are going to implement SG to the existing production server. Following are a few issues that we will have to address first before we can implement it. I’m concern that I might overlook something, so, wanted to hear any advice or suggestion from you.

Dependency
We have a list of File systems which will be candiates to be configured on the EMC shared storage. Although majority of software has aleady resides on the storage, there are some others there is not on any one of such file systems. For the purpose of avoiding too much space as the shared, we plan to move this part of data over to the file systems, of cause, because of that, I understand that any references to these moved files/directories will be modified.

Any data inputs to the system
We will have to make sure of all daily processes on the system or data inputs to the current system also be working on the adoptive node. For that, we will just need to modify all references to the current system, and rename them to the package name (pkg ip).

Batch process
make sure same batch jobs will be configured and being able to be running on adoptive node.

Please let me know if there is anything we are missing, and should add/change/correct to what I have just listed above. Thanks!

[Mel Burslan]

Hanry,

Keep in mind that, cron jobs, or as you referenced them as the batch jobs, are not too easy to be moved with the package, as the cron system runs on the physical server, not in a virtual package environment. So, while developing batch jobs, it is best to build them with this logic construct :

MYLOGFILE="/path/log_file_name"
MYPKG="your_package_name_here"
NODE=$(cmviewcl | grep $MYPKG | awk {'print $5'})
if [ "$NODE" = "$(hostname)" ]
then
call my_job_script.sh
else
echo "$(date) -- package is not running on this node" >> $MYLOGFILE"
fi

and run this script from cron on both nodes.

And of course, whenever you modify the my_job_script.sh, do not forget to copy it to the other partner node, as well as not forgetting to add any new cron jobs to each of the nodes at the same time. The best is to run a cron job to compare crontabs on either node and send a report out to the admins if there are any differences discovered.

Hope this helps a bit.

[Rita C Workman]

The best thing is to carefully review the documents under docs.hp.com for Managing MC/SG under High Availability.

Here's a couple points too:
1. Resolution - you mention pointing to the pkg. We point to the pkg name (not the IP). Make sure you can do solid reverse lookups. So DNS becomes critical in Oracle apps, because your clients must have the same info (pkg name) in their tnsnames file. And make sure your /etc/host file is solid for both too.
2. passwd & group files. You must remember that users must be the same uid/gid in both servers !
3. What goes inside the pkg......
This is up to you, but my preference is to get EVERYTHING I can inside the pkg. I only leave out what I can't get in there...period.
[TIP - because of multiple instances, and then multiple environment settings, I find it helpful to set /home directories to be unique and then within the pkg for everything except that which must remain on the server. Sounds confusing, but it keeps everyone's stuff clean, unique and always there on any box in a failover. So I might have /home or /lhome or /ehome directories all on the same box, each one belonging to a unique instance with it's respective users.]
4. If you have automated start/stop scripts (like for Oracle) make sure you copy these over to the other boxes too.
5. Make sure your boxes are software loaded and patched accordingly so you can run, without any problems on the other node.
6. Test,test,test failing over and back. Make sure you can mount, start & stop things via automated tools.

Last Tip...even though you are starting with what sounds like a simple 2 node cluster, plan it out for growth. Set standards. Couple years ago I had one 3 node cluster. Today I have CC cluster and shortly my 4 node cluster on one side will become a 10 node cluster. Make sure your ready to grow. Set your standards for things, and make everyone stick to them !

Best Wishes,
Rita

[Mark Nieuwboer]

Hi Hanry,

When making a serviceguard cluster make sure al the right patches are inplace. There are some issues with 11.16. Futher more;
1. put the heartbeat in a seperate vlan see attached document.
2. reserve a number of ip-adresses for the packages you want to configure also in the futhere in the document a configured it for 16 packages.
3. make your data lan in the configuration also a heartbeat lan if your heartbeat fails you have a backup heartbeat lan.
4. make sure your mknod of the volume groups are the same on both nodes.
5. make a volume group that is cluster aware and shareble over both nodes.
6. check network and volumegroups before you begin it saves downtime. most problems i encountered with serviceguard are network problems.

grtz. Mark

[Hanry Zhou]

Thanks all for the response. More inputs will be appreciated.