Can we upgrade one node to 11.23 and not inhibit the SG Cluster

[EJ Stremler]

We presently have a Servie Guard cluster with 2 nodes, both are running HP-UX 11.11. If we move all the packages to one node, Can we upgrade the other node to 11.23?, probably by a cold install.. Can we do a cmdeletenode on the system we plan to upgrade to get it out of the cluster? then add it back in when we are finshed with the install? Is there SG version imcompatabilities between 11.11 and 11.23? Any help with this would be appreciated...thanks, Ed

[Ivan Krastev · Accepted solution]

What is your SG version ?

See support matrix - http://docs.hp.com/en/5971/SG-SGeRAC-EMS-Support.pdf


regards,
ivan

[EJ Stremler]

$ /usr/sbin/swlist | grep -i servi
B3935DA A.11.16.00 Serviceguard

[Geoff Wild]

Yes - I just did this.

I had 11.16 in the cluster - upgraded 1 node to HP-UX 11.23 and SG 11.18.

The following weekend, I upgraded the other node.

Rgds...Geoff

[Steven E. Protter]

Shalom,

You can easily, if the cluster is configured correctly pull one node out of the cluster and then upgrade the OS and then everything else.

It will not be able to re-join the cluster.

What I would do is configure and get a second cluster with all the same services on the updated node. Then you will need a short downtime to halt the old cluster and start the new one.

The concept of a rolling upgrade applies to a SG only upgrade. I believe it is impossible to run a mixed OS SG cluster.

I think since one node will be out of the cluster for some time its a good opportunity to upgrade to a newer, better supported version of SG.

SEP

[Geoff Wild]

When I did mine - I did a cmhaltnode, upgradeded it (yes - update-ux rocks - and anyone who says you shouldn't upgrade hasn't tried it in a while). Then after the upgrade - I did a cmrunnode and the cluster successfully formed. That is how a "rolling" upgrade should be performed.

I can post my detailed plan if you like...

Rgds...Geoff

[EJ Stremler]

Geoff, if you can post your detailed plan that would be incredibly helpful, all you did was a cmhaltnode, then cmrunnode to bring the server back into the cluster? did you copy the /etc/cmcluster contents to the upgraded server? You didn't have to take the cluster down and do a cmapplconf to re-initialize the cluster lock??.thanks all for your help, ed

[Geoff Wild]

Because I did an upgrade - I didn't have to cmapply anything - it just worked. We created depots of the dvd's and installed over the network.

Rgds...Geoff


SAP Prod Cluster OS Upgrade

Ensure vg00 has enough disk space

On svr003 and svr004:

lvextend -L 5120 /dev/vg00/lvol8
fsadm -b 5120M /var
lvextend -L 5120 /dev/vg00/lvol7
fsadm -b 5120M /usr
lvextend -L 5120 /dev/vg00/lvol6
fsadm -b 5120M /opt
bdf |grep vg00
lvextend -L 1024 /dev/vg00/lvol5
fsadm -b 1024M /home
umount /ops
mount /dev/vg00/lvol10 /zmnt
cp -p -R /zmnt/* /ops/
rm -rf /ops/lost+found
lvremove /dev/vg00/lvol10
umount /zmnt
lvremove /dev/vg00/lvol10
vi /etc/fstab and remove /ops


Ensure Ignite Servers have latest revision:

swinstall -x autoreboot=false -s svr090:/var/software/hp/Ignite-UX-11-ALL_C.7.2.94.depot \* @`uname -n`


Start of outage

Call OPS to let them know you are starting

On svr004 go into sam and backup the spooler.



Upgrade svr003:

cmhaltnode svr003


Stop NFS

/sbin/init.d/nfs.client stop

Split vg00 Mirror and uninstall PowerPath


for lv in `vgdisplay -v /dev/vg00 | grep "LV Name" | awk '{print $3}'`
do
lvreduce -m 0 $lv /dev/dsk/c20t5d0
done

swremove -x autoreboot=true EMCpower

After reboot - Upgrade OS

/sbin/init.d/nfs.client stop

swinstall -s svr090:/var/software/hp/HP11iV2.MC.Jun2007 Update-UX

/usr/sbin/update-ux -s svr090:/var/software/hp/HP11iV2.MC.Jun2007 HPUX11i-OE-MC




Apply Latest Patches:

/sbin/init.d/nfs.client stop

swinstall -x autoreboot=true -x mount_all_filesystems=false -x autoselect_patches=TRUE -x patch_match_target=TRUE -x patch_filter=*.* -s svr090:/var/software/hp/patch/11.23.pa.aug2007 \* @`uname -n`

Re-install PowerPath

swinstall -x ask=true -x autoreboot=true -x mount_all_filesystems=false -s svr090:/var/software/EMC/pp.11.23 \* @`uname -n`

You see the following prompt:
Install for cluster environment?[y,n,q,?](default: n):
5. Enter y and press ENTER.
You are prompted for a major number that can be assigned to the
PowerPath device driver (emcp) on every host in the cluster:
Specify major number for the device driver [#,q,?]: 77
6. Enter a major number and press ENTER. You must specify a major
number that is not used by any host in the cluster. (Refer to
Prepare for a Clustered Environment on page 1-5.)
You are prompted to confirm the major number:
Assign major number number to the device driver?
[y,n,q,?] (default: y):
7. Press ENTER to confirm the major number.

After Reboot:

Configure PowerPath devices. Enter:

powermt config

Enable load balancing and failover using one of the following
commands:


powermt set policy=so dev=all



Upgrade Ignite:

swinstall -x autoreboot=false -s svr090:/var/software/hp/Ignite-UX-11-ALL_C.7.2.94.depot Ignite-UX-11-23 @`uname -n`



Failover prd to svr003 - mount only:

Call OPS to let you know you are starting - and that they do NOT start backup yet.

Disable FTP and start prd mount only:

cp -p /etc/inetd.conf /etc/inetd.conf.bak
cp -p /etc/inetd.conf.noftp /etc/inetd.conf
inetd -c

Note: may or may not have to reapply the cluster: cmapplyconf -C pcha.ascii

cd /etc/cmcluster/prdDBCI
cp -p prddbci.cntl.mountonly prddbci.cntl

cmhaltpkg wrkpkg
cmhaltpkg prdprtpkg
cmhaltpkg prddbcipkg
cmhaltpkg pctranspkg

cmrunpkg -n svr003 prddbcipkg

Call DBA - to start DB manually to test:



After DBA finished, restart package full:

cmhaltpkg prddbcipkg
cp -p prddbci.cntl.full prddbci.cntl
cmrunpkg pctranspkg
cmrunpkg prddbcipkg
cmrunpkg prdprtpkg
cmrunpkg wrkpkg


Restore Spooler and FTP:

touch /tmp/prdprt.lock

Go into sam, printers, restore spooler.
cp -p /etc/inetd.conf.bak /etc/inetd.conf
inetd -c

Note: after a spooler restore, the /var/spool/lp/receive/XEBEC will be wiped out
mkdir /var/spool/lp/receive/XEBEC
chown lp:lp /var/spool/lp/receive/XEBEC
chmod 775 /var/spool/lp/receive/XEBEC

chmod 775 /var/spool/lp
chmod 775 /var/spool/lp/receive
chown lp:lp /var/spool/lp
chown lp:lp /var/spool/lp/receive

rm /tmp/prdprt.lock




Ask OPS to start prd backup



Repeat steps 3 - 13 on svr004


Failover prd back to svr004

Call OPS to let you know you are starting - and that they do NOT start backup yet.


cmhaltpkg wrkpkg
cmhaltpkg prdprtpkg
cmhaltpkg prddbcipkg
cmhaltpkg pctranspkg

cmmodpkg -e -n svr004 pctranspkg
cmmodpkg -e -n svr004 prddbcipkg
cmmodpkg -e -n svr004 wrkpkg
cmmodpkg -e -n svr004 prdprtpkg

Check logs:

cd /etc/cmcluster/prdDBCI

Check prddbci.cntl.log

If all okay, and SAP is up and running (either sign in yourself or ask OPS) then ask OPS to run the prd cold.


Restore Spooler (note - may or may not have to do this)

Test printing:

pp W052

That should send a test job to W052.

If not, touch /tmp/prdprt.lock and go into sam, printers, restore spooler. Then re - test.
Then remove the lock file /tmp/prdprt.lock


Call OPS and let them know change is done.


Post

Re-mirror vg00
for lv in `vgdisplay -v /dev/vg00 | grep "LV Name" | awk '{print $3}'`
do
lvextend -m 1 $lv /dev/dsk/c3t6d0
done
lvlnboot -b /dev/vg00/lvol1
lvlnboot -r /dev/vg00/lvol3

[EJ Stremler]

Thanks Geoff - we use ignite-ux to blast new images down on our servers.. Security auditors continue to tell us to lock down our images more and more, so we are constantly creating new ones, and installing them.. I guess in our case, if we vgexport the cluster lock and other volume groups, take the node out of the cluster (cmhaltnode), then from a 2 node it will become a 1 node cluster.. upgrade it, copy /etc/cmcluster over, vgimport the cluster lock and all the other volume groups. and bring the other node back in (cmrunnode). But wondering if the OS version difference will cause any problems?

[Geoff Wild]

If you are "blasting" a new image - then you may have to re-create the cluster - and you will have to execute all commands on the node with the latest OS and ServiceGuard.

You can try your method of just copying the files over - but it would be great to test that scenario first - prior to messing with your production cluster.


http://docs.hp.com/en/B3936-90117/apes02.html


Rgds...Geoff

[Stephen Doud]

HP does not support the technique that you are proposing - namely, removing a node from the cluster, updating it's O/S and re-adding it to the cluster. The reason being that mixed version O/S are not supported with the same version of Serviceguard. That means that if your procedure does not work, HP will advise that you use HP-supported procedures to make it work. A rolling upgrade is the only supported procedure that allows for a mixed-hpux/same-SGversion or mixed-hpux/mixed-SGversion migration.

Be aware of the following is you decide to go it a different way...

For HPUX-> Serviceguard compatibility questions, refer to the Support matrix at http://docs.hp.com/en/5971/SG-SGeRAC-EMS-Support.pdf

To remove a node from a cluster, you must edit all package configuration files and the cluster configuration file, commenting out the departing node, and performing cmapplyconf on each file (package config file before, or simultaneous to cluster file).

Serviceguard will not allow you to add a node to a cluster, where the node runs a different version of Serviceguard. Therefore, after updating the O/S on the node, install the same version of SG.

If your intent is to update both HP-UX AND Serviceguard, the departing node will not be able to rejoin the old cluster. A rolling upgrade is the only method suited for this. A rolling upgrade does not allow for a reinstall of a server.

Some customers try to trick Serviceguard by leaving the node in the cluster, installing a newer version of HP-UX and Serviceguard and then running /usr/bin/convert on the cluster binary file on the updated node ('convert' is used in the post-install script for Serviceguard swinstall). Though it is feasable to use this approach it may not work due to the chance that the names of the LAN NICs may have changed - a change which will disagree with the cluster binary file (use cmviewconf to see this) and prevent the node from rejoining the cluster.

[EJ Stremler]

Thank you everyone and I appreciate all your input.. This made us more aware that a cold install of a node in an SG cluster is not supported. We are also going to investigate the option of using upgrade-ux in our development environment and try to perform a rolling upgrade instead. ed