Activating VG in systemstart in a ServiceGuard Cluster without Cluster software started

[Andreas Tuerk]

In a ServiceGuard Cluster I need to activate several volume groups as part of a "package" during system start, while no cluster services are available.
Depending on a parameter AUTOSTART_CMCLD in /etc/rc.config.d/cmcluster the packages on a two-node cluster should be started as part of a ServiceGuard cluster or without ServiceGuard.
The difficulty is, that the VGs cannot be activated exclusive (vgchange -a e) without cluster services running and cannot be activated normally (vgchange -a y) when activated for cluster (vgchange -c y).
Once activated for cluster use (vgchange -c y during configration for cluster) the VGs can only be activated via vgchange -a e. But when booting the servers without ServiceGuard cluster services are not available and VGs can only be activated via vgchange -a y, but this is not allowed, because the disks are still activated for cluster use.

[Rainer von Bongartz]

so why dont't you simply start your cluster services ??

You don't need to start your packages just use cmruncl

[Andreas Tuerk]

I cannot start the cluster, because some parts of the hardware are not available and the cluster need 100% of the hardware to start after reboot.

[Robin Wakefield · Accepted solution]

Hi,

vgchange -c n /dev/vgwhatever
vgchange -a y /dev/vgwhatever

should allow you to work on the vg outside of the cluster.

Robin

[John Waller]

Andreas, Unfortunatly I can't quite understand what you are asking , but why can't you vgchange -c n to stop the VG from been cluster aware then vgchange -a y to activate. Later you can alsways vgchage -a n and vgchange -c y ro remake the VG cluster aware.

[James R. Ferguson]

Hi ANdreas:

In that case, you could:

Set AUTOSTART_CMCLD=0 in /etc/rc.config.d/cmcluster.

'vgexport' the volume groups in question on the node where they will *not* exist/be used.
(For safety, physically disconnect the connections on the node that will not be using these volume groups so that there is no possibility of access).

On the node that you want to activate the volume groups:

# vgchange -c n /dev/vgXX

Then proceed to active the volume group as needed.

...JRF...

[Andreas Tuerk]

Thanks. Deactivating VGs for cluster use is fine, but activating VGs for cluster use can be done only with cluster services running.
When and where do I have to active VGs for cluster use in system startup??
Remember, the only manual interaction is editing the parameter AUTOSTART_CMCLD in /etc/rc.config.d/cmcluster.

[James R. Ferguson]

Hi Andreas:

See: /etc/lvmrc

...JRF...

[Andreas Tuerk]

Hi James,
in /etc/lvmrc the VGs are activated via vgchange -a y, but I need to activate them for cluster use via vgchange -c y and for that the cluster services have to run. In system boot they are started nearly at the end, long after activating vg.

[James R. Ferguson]

Hi Andreas:

Is this a "cake-and-eat-it-too" situation? If you want cluster activation then you need to run cluster services and let it be done there.

I presumed from your original comments that the absence of 100% of your hardware was a temporary situtation that you were attempting to remedy.

Is this *not* the case?

...JRF...

[Robin Wakefield]

With some logic applied to your /sbin/init.d/cmcluster script, you should be able to determine the state of the cluster on the node (primary or alternate). Then issue the "cmruncl" & "vgchange -c y" commands if you know the cluster is not running on either server, else just cmrunnode.

Robin.

[Andreas Tuerk]

Hi James:

Let me explain this "special" situation.
We have a two-node ServiceGuard cluster. In case of a hardware failure followed by a server reboot of both servers the cluster won't start. Our customer doesn't want a single node cluster, but needs the application started with minimum manual interference. So we copied the package control scripts and modified them. Depending on the parameter AUTOSTART_CMCLD the server boots with the cluster starting the application (AUTOSTART_CMCLD = 1) or starting the application with the modified package control script.
The change from cluster to single node is working fine.
But after repair of the hardware, the change back to cluster is the difficulty. How can I activate the Volume groups during system boot, after the cluster has been enabled via editing /etc/rc.config.d/cmcluster AUTOSTART_CMCLD = 1.

[James R. Ferguson]

Hi Andreas:

I don't think this is feasible. To me, this sounds like you are attempting to defeat the controls on volume groups that MC/ServiceGuard offers to prevent catastrophic dual access.

If the "customer doesn't want a single node cluster, but needs the application started with minimum manual interference" then what is the purpose of a two-node cluster in the first place?

...JRF...

[Andreas Tuerk]

Hi James:

It sounds weired to me too, but he's the customer and he's got the money.
I would always use cluster tools, even in the case of a hardware failure. I tried to change his mind, but wasn't successfull. So what?
What about activating the volume groups for cluster use in the package control script and deactivating in the modified script?

[A. Clay Stephenson]

Hi Andreas,

It appears to me after reading all the postings that you need to rethink your hardware configuration. There should never be a hardware failure that requires a reboot of both nodes in a cluster. If there is then the configuration is not robust enough. The only time that I've seen this to be a problem was when someone 'saved' money and did not purchase in-line SCSI terminators. The other gotcha that will kill a two-node cluster after a reboot of both is failure of the lock disk -- but again in this case both nodes should have not been shutdown.

I think you need to take a fresh look at the problem from the standpoint of always having at least one node running and with the cluster always up and with no single failure being able to harm anything.

If the customer's method of fixing any problem is to reboot everything then I don't think that is a very realistic SG model.

My 2 cents, Clay

[James R. Ferguson]

Hi (again) Andreas:

I think at this point I would respectfully suggest to the customer that he/she bear with you while you provide a little technical education! There's never a free lunch :-)

...JRF...

[A. Clay Stephenson]

Hi again Andreas,

One method I've found to be very convincing to even the most hard-nosed skeptic is to invite the guy/gal to the data-center and tell them you can break/cut/disconnect/remove any ONE thing and let's see what happens. If you've done your job well (and this is an excellent test), the package/cluster handles this well.


Of course, you are next asked : "Well, what if TWO things break?". You then tell him/her that the price of that goes up approximately as the square of the number of tolerated failures.

I've never has this little demonstration to fail me yet. Of course, long before I let anyone try to crash the cluster I've already done my most devious.

Regards, Clay

[James R. Ferguson]

Hi (yet again):

One more thought: There are failure scenarious where you can observe that cluster services will not start until AUTO_START_TIMEOUT (usually 10-minutes) has expired. During this period, neither 'cmruncl' nor 'cmrunnode' will work.

Perhaps this too is useful.

...JRF...

[Mladen Despic]

Andreas,

When you tried to re-enable the cluster, did you ensure that the package volume groups are prevented from being activated at system boot time by the /etc/lvmrc file? You need to set AUGO_VG_ACTIVATE to 0, then include all the volume groups that are not cluster bound in the custom_vg_activation function. (note: the root volume group does not need to be included in the custom_vg_activation function).

Mladen

[Andreas Tuerk]

Hi all,
thanks for your support. I foung a solution now.
Kill the customer - just a joke?!
In the package control script I enable the volume groups via vgchange -c y before activating them in exlusive mode. In the modified control script I disable the volume groups via vgchange -c n before activating in normal mode vgchange -a y. The modified control script is started during system boot after cluster activation and is checking the parameter AUTOSTART_CMCLD in /etc/rc.config.d/cmcluster. The volume groups are not automatically activated in /etc/lvmrc.
It's working fine.
To Clay.
We had the customer in our data center an showed him all the scenarios for switching. Everything worked fine. But he replied, that they have a bad network and a bad power supply. The whole network often goes down for a few moments and he doesn't want the cluster to restart the packages because of that. Also the power supply is weak. His problem with that is, what happens if some redundant hardware fails and the two servers reboot because of a power failure.
I know that it's not a good thing to handle network and power problems by server software, but it's the customer who wants me to do that.
Again, thank's a lot for your help.

Regards Andreas