RedHat Cluster -- VG does not get taged

[Ralf Seefeldt]

Hello all, I have a testcluster that can not start services containing filesystems becuce it does not try to addtag or deltag the accociated VGs. I do not, where to fix that problem. When I addtag manuallyto the VG, the service is able to start. When I stop the service, the tag remains set. What did I miss in configuring the cluster? Any help will be greatly aprediated. Bye Ralf The details are: Two virtual Servers on one VMware ESX server running RedHat 6.3 (2.6.39-300.17.3.el6uek.x86_64) luci-0.26.0-13.0.1.el6.x86_64 ricci-0.16.2-55.el6.x86_64 cman-3.0.12.1-32.el6.x86_64 @ol6_latest ccs-0.16.2-55.el6.x86_64 @ol6_latest omping-0.0.4-1.el6.x86_64 @ol6_latest rgmanager-3.0.12.1-12.el6.x86_64 @ol6_latest cluster-cim-0.16.2-18.el6.x86_64 ol6_latest cluster-glue-libs-devel-1.0.5-6.el6.i686 ol6_latest cluster-glue-libs-devel-1.0.5-6.el6.x86_64 ol6_latest cluster-snmp-0.16.2-18.el6.x86_64 ol6_latest clusterlib-devel-3.0.12.1-32.el6.i686 ol6_latest clusterlib-devel-3.0.12.1-32.el6.x86_64 ol6_latest corosynclib-devel-1.4.1-7.el6.i686 ol6_latest corosynclib-devel-1.4.1-7.el6.x86_64 ol6_latest fence-virtd-checkpoint-0.2.3-9.el6.x86_64 ol6_latest foghorn-0.1.2-1.el6.x86_64 ol6_latest libesmtp-devel-1.0.4-15.el6.i686 ol6_latest libesmtp-devel-1.0.4-15.el6.x86_64 ol6_latest openaislib-devel-1.1.1-7.el6.i686 ol6_latest openaislib-devel-1.1.1-7.el6.x86_64 ol6_latest pacemaker-1.1.7-6.el6.x86_64 ol6_latest pacemaker-libs-devel-1.1.7-6.el6.i686 ol6_latest pacemaker-libs-devel-1.1.7-6.el6.x86_64 ol6_latest python-repoze-what-quickstart-1.0.1-1.el6.noarch ol6_latest The Servers names are ifbseef1.x.de and ifbseef2.x.de I have some VGs and LVS: [root@ifbseef2 ~]# vgs VG #PV #LV #SN Attr VSize VFree vg_ifbseef2 1 2 0 wz--n- 24.51g 0 vgprod 1 2 0 wz--n- 30.00g 28.00g vgquorum 1 1 0 wz--n- 1020.00m 0 vgtest 1 2 0 wz--n- 5.00g 3.00g [root@ifbseef2 ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert lv_root vg_ifbseef2 -wi-ao-- 20.63g lv_swap vg_ifbseef2 -wi-ao-- 3.88g lvbackupp vgprod -wi-a--- 1.00g lvdatap vgprod -wi-a--- 1.00g lvquorum vgquorum -wi-ao-- 1020.00m lvol0 vgtest -wi----- 1.00g lvol1 vgtest -wi----- 1.00g The cluster was fine before I added services. The same cluster.conf was fine using Redhat 5 before reinstallation. As I have only some months of experience with RedHat, I may have forgotten some OS configuration. lvm.conf contains: filter = [ "a|^/dev/vgprod|", "a|^/dev/vgprod/.*|" , "a|^/dev/vgtest/.*|" , "a|^/dev/vgquorum/.*|" , "a/.*/" ] volume_list = [ "vg_ifbseef2", "vgquorum", "@ifbseef2.x.de" ] #of course the other server has equivalent but not identical settings There is no different behavior with filter = [ "a/.*/" ] The cluster.conf is attatched

[Matti_Kurkela]

Using LVM tags for cluster VG protection is the old (pre-RHEL 4.5) way. It is still available, but if you use it, you should make sure that the "locking_type" parameter in /etc/lvm/lvm.conf is set to 1 (= no CLVM locking, only single-host LVM locks in effect).

If you set up a RHEL 6 cluster using Conga/Luci, it automatically sets the LVM locking_type to 3 if you check "enable shared storage support" in node parameters. So you may have to undo this modification if the cluster management tool has already done it for you.

The new way to protect cluster VGs uses clvmd and allows cluster VG activation in exclusive mode (very much like in HP-UX Serviceguard, although the command is slightly different: "vgchange -a ey <VG_name>" instead of HP-UX style "vgchange -a e <VG_name>").  If clvmd is not running, cluster-mode VGs cannot be activated at all.

Please see:

https://access.redhat.com/knowledge/solutions/3067

[Ralf Seefeldt]

Hello Matti,

thank you for your fast response.

As I did not use lucci for configuration, bots server's lvm.conf has lockint-type set to "1".

The OS does not try to vgchange -a y <VGNAME>

Is there any other place to configure? I would prevere configuring the servers using vi instead of lucci to make sure, I know and understand every step required.

Bye

Ralf

[Ralf Seefeldt]

Hi,

meanwhile I ecreated the cluster using luci / GUI.

I use clvmd (I believe).

the porblem is still the same.

When stopping a service, the tag(s) will be removed from the LVs.

Then, the vg will be disabled, which removes /dev/vgprod

When starting the service. It is missing the devicefiles and fails:

Jan 16 17:17:05 rgmanager Starting disabled service service:prod
Jan 16 17:17:05 rgmanager #68: Failed to start service:prod; return value: 1
Jan 16 17:17:05 rgmanager Stopping failed service service:prod
...

Can someone please send me a link to a documentation of this VG stuff for RH 6.

I'm fluent with that in HP-UX, but not in LINUX.

Bye

Ralf

[Matti_Kurkela]

Your original post in this thread seems to have been a victim of a certain intermittent bug(?) in the community system, and all the formatting has been stripped off it. You probably can edit the post to restore the formatting, to make it more readable. The bug has been reported, and hopefully will be fixed.

Your cluster.conf attachment seems to be missing too. If you attach it again, please add a .txt suffix to the filename: the software that runs the community forums seems to be somewhat picky about attachment filenames and types of files you can attach.

So, if you (re)created the cluster with luci, the LVM "locking_type" parameter should now be set to 3 in all cluster nodes. To use CLVM-based locking in an existing VG, you must set the VG to cluster mode. Just as in HP-UX: "vgchange -c y <vgname>" while the VG is inactive on all cluster nodes. Unlike HP-UX, Linux allows the activation of a cluster VG with "vgchange -a y <vgname>" in multiple nodes if the VG has not yet been activated exclusively by any node: this is to allow the use of the GFS cluster filesystem. But if (and *only* if) no other node already has the VG active, a cluster VG can be activated in exclusive mode with "vgchange -a ey <vgname>", and then all subsequent activation attempts by other nodes will fail.

(In HP-UX, you would have to explicitly mark the cluster VG as shareable and then activate it in shared mode to get something similar to Linux cluster VG "vgchange -a y" behavior. Yet HP-UX VG in shared mode will not allow LV resizing; Linux LVM handles it just fine.)

Your problem with the cluster VG might be because Linux LVM normally activates all non-cluster VGs it sees at boot time, unless special steps have been taken to prevent it (e.g. the VG tags). But when clvmd is running and "locking_type" is set to 3, RedHat 6.x cluster tries to use exclusive activation instead of VG tags... and if the VG is in non-cluster mode, it is probably already activated on all nodes, so exclusive activation cannot succeed. If you activate the VG manually, the automatic VG deactivation by the cluster apparently always includes the removal of the old-style VG tags, and an actual VG deactivation with "vgchange -a n <vgname>"... which, like in HP-UX, works regardless of whether the VG is in cluster mode or not.

To fix this, deactivate the VG manually on all cluster nodes, then set it to cluster mode if it's not already set.

Then try running the service again.

Note: a side effect of setting "locking_type" to 3 is that all new VGs created after that will be created in cluster mode by default, unless you use "vgcreate -c n ..." to override it on VG creation. This may surprise you later.

And for Linux LVM documentation, please see: http://docs.redhat.com -> RedHat Enterprise Linux, and then find a "Logical Volume Manager Administration" guide for RHEL version 5 or 6.

[Ralf Seefeldt · Accepted solution]

Now I got it working.

Using clvmd was quite straight forward. Manually, only  locking_type = “3”  had to be set in /etc/lvm/lvm.conf.

Using HA LVM required, that the resources of filesystems where definded as children of their LMV ressources.

In addition, after installing and patching, the initrd file had not been created automaically for the new kernel. I had to create it manually.

Now, the cluster is working well. No matter whether using clvmd or HA LVM.

Problem solved