Approved cluster procedures

[Andrei Magureanu]

Hi everyone,

I wasn't able to find anywhere anything about "approved cluster procedures", so I am asking for your help.
We have a HPUX cluster that's composed from 2 servers (cmax1 and cmax2). We recently ran out of space on the /backup/prod (production database directory) directory and were unable to perform anymore database backups to this location. The directory itself has 500GB of allocated space, but our full backup is bigger than that. So, we allocated 100GB from the SAN server. We tried to append those 2 new disks to the vg05 volume group, but we were unable to do so because of a message saying: "The volume group you have selected, "vg05", must exist and be deactivated before it can be undistributed. Please deactivate this volume group using approved cluster procedures before attempting to undistribute it.". What are the cluster procedures? We tried to deactivate the volume group using command:
vgchange -a n /dev/vg05
That showed a "device busy" error. We killed several processes that we believed are using the disk and still no luck.

Has anyone experienced this before? Any input is appreciated. Also, see attached image for SAM screenshots of the steps we took in trying to add the available disks to the volume group.

[A. Clay Stephenson · Accepted solution]

Since you don't seem familiar with ServiceGuard commands, I strongly suggest that you find some who does have SG experience. In any event, you need to execute a cmhaltpkg command, the VG will then be deactived. After you do the vgextend, you have to export and vgimport the VG to all the nodes. Using SAM for SG voluime group distribution is a far from ideal method.

At the very least, read the SG Documentation.

[A. Clay Stephenson]

By the way, the reason you are having trouble finding "approved cluster procedures" is that the procedures are highly cluster and package dependent so what would be perfectly acceptable for one cluster might be a disaster for another.

[Andrei Magureanu]

Thank you Clay for your quick response. Indeed, I am not familiar with ServiceGuard but I will try to get better at it next time. My question now, is how much will the "cmhaltpkg" affect the cluster availability and performance. We have about 1000 users at any given time, and taking the service (oracle) that runs on this cluster offline is not an option.

[melvyn burnard]

cmhaltpkg does NOT affect the cluster, just the package you halt, and any performance impact would be from the application performing it's shutdown procedures.

your attempt at deactivating vg05 is due to teh fact that the package is up and running, and using the VG. Halting the package would (or should!) automatically deactivate the vg

[melvyn burnard]

RE-reading your initial post I get the feeling you are not just running Serviceguard, but possibly using Oracle RAC with the SGeRAC add-on.
Is this so?
use swlist to check for Serviceguard Extensions for RAC
If you are, then this has a different connotation, and you shoul drefer to page 184 in the following:
http://docs.hp.com/en/T1859-90038/T1859-90038.pdf

[Andrei Magureanu]

Thank you melvyn.

I did some research and ran the swlist -i command. This returned the following packages:

MirrorDisk/UX
HP OnLineJFS
MC / ServiceGuard
HP-UX Installation Utilities (Ignite-UX)
Ximian GNOME 1.4 GTK+ Libraries for HP-UX 11.00 and 11i
Java 2 RTE for HP-UX (700/800), PA1.1 + PA2.0 Add On
HP Cluster Object Manager
ServiceGuard Manager
Java 2 Plugin for HP-UX (700/800)
Java2 1.3 RTE for HP-UX
Required Patch Bundle for HP-UX 11i, June 2003
Base VERITAS Volume Manager Bundle 3.5 for HP-UX
Consistency Monitoring Tools
English CDE Environment
Debugging Support Tools
PCI FDDI;Supptd HW=A3739A/A3739B;SW=J3626AA
Feature Enablement Patches for HP-UX 11i, Sept 2002
PCI/HSC FibreChannel;Supptd HW=A6684A,A6685A,A5158A,A6795A
PCI-X FibreChannel;Supptd HW=A6826A,A9782A,A9784A
Gold Applications Patches for HP-UX 11i, June 2003
Gold Base Patches for HP-UX 11i, June 2003
PCI/HSC GigEther;Supptd HW=A4926A/A4929A/A4924A/A4925A;SW=J1642AA
PCI/PCI-X GigEther;Supptd HW=A6794A/A6825A/A6847A/A9782A/A9784A/A7109A
HP-UX 11i Operating Environment Component
HP-UX 64-bit Base OS
HP-UX Base OS Auxiliary
Hardware Enablement Patches for HP-UX 11i, September 2003
PCI Ethernet;Supptd HW=A6974A
HP-UX Installation Utilities for Installing 11.11 Systems
Mozilla 1.2 for HP-UX
Mozilla 1.2 Source distribution
HPUX 11.11 Support Tools Bundle, Sep 2003
PCI RAID; Supptd HW=A5856A
Java2 1.3 Netscape Plugin for HP-UX
Java2 1.4 RTE for HP-UX
Java2 1.4 Netscape Plugin for HP-UX
IBM Tivoli Storage Manager Client
HP Wildebeest (HP WDB) Debugger
GUI for the HP WDB Debugger
Ignite-UX Manufacturing Supplement (Reference Only)
HP-UX Apache-based Web Server
HP-UX Tomcat-based Servlet Engine
HP-UX Webmin-based Admin
HP-UX XML Web Server Tools
Perl Programming Language
tusc

It looks like we are running "HP OnLineJFS", so this may change completely the situation by not having to halt packages. Also, our Oracle install is not RAC, so nothing to worry on this.

[A. Clay Stephenson]

OnlineJFS is not germane to your problem (although no box running SG should be without it) and since you are not running RAC, halting the package will indeed kill your Oracle sessions. If you were simply extending existing LVOL's then you could do this "on the fly" and OnlineJFS would certainly help you BUT you need to add additional PV's to an existing VG and then do your lvextend's (to grow the LVOL(s)) and fsadm's (to grow the filesystem(s)). You will need to schedule a maintenance window to do this task.

As a temporary fix, you could create a new VG and use it for your backups and let it be a standalone VG (ie not shared (yet)) but this works only if your package does not switch to another node. You could later import this VG and make it cluster aware and part of the package during a maintenance window.

At the moment, you are being asked to fly an airplane without first having gone to flight school. You really need to get some SG training before doing any of this.

[Andrei Magureanu]

Well... thank you gentlemen. I will do some learning and if I figure out the way to fix this issue, I will all let you know in this thread. I will leave it open for right now.