Fenceing in RHEL6

[rhel6_cluster]

Hi..

My server is BL460 G7, with iLO 3  and i use IPMI LAN as fencing device. I have added fencing method on both the nodes and the status shows that both the nodes are using fencing device.

But i am not sure how to test this?

Can you help me how to configure it?

Thanks!

[Matti_Kurkela]

When fencing is properly configured, you should be able to use the "fence_node" command to manually fence any node from any other node.

As you're using ipmilan fencing, that means e.g. running "fence_node node2" on node1 should cause node2 to immediately power-off and reboot. If that happens, fencing is configured correctly for node2. Likewise for node 1.

[rhel6_cluster]

Now i tried with hp ilo but still doesn't work :(

# fence_node PDC-PIC-PL-02
fence PDC-PIC-PL-02 failed

But it says that status are ok..
What are the values do i need to set for it to work?

<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="PDC-PIC-PL-01" nodeid="1" votes="1">
                        <fence>
                                <method name="Fencing">
                                        <device name="HPiloFence1"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="PDC-PIC-PL-02" nodeid="2" votes="1">
                        <fence>
                                <method name="Fencing">
                                        <device name="HPiloFence2"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="3"/>
        <fencedevices>
                <fencedevice agent="fence_ilo" ipaddr="PDC-PIC-PW-01" login="hpilologin" name="HPiloFence1" passwd="hpilopword"/>
                <fencedevice agent="fence_ilo" ipaddr="PDC-PIC-PW-02" login="hpilologin" name="HPiloFence2" passwd="hpilopword"/>
        </fencedevices>

Thanks! much appriciated!

[Matti_Kurkela]

fence_ilo is for iLO and iLO2 only.

For iLO3, you will need fence_ipmilan.

The RHEL6.3 fence_ipmilan man page says:

To use fence_ipmilan with HP iLO 3  you have to enable lanplus (lanplus / -P) and increase wait after operation to 4 seconds (power_wait=4 / -T 4)

If you have configured the iLO3 user "hpilologin" to have access to the "Virtual Power and Reset" only, you'll also have to specify the IPMI privilege level as USER.

So your <fencedevices> cluster.conf section should be like this:

        <fencedevices>
                <fencedevice agent="fence_ipmilan" ipaddr="PDC-PIC-PW-01" login="hpilologin" name="HPiloFence1" passwd="hpilopword" lanplus power_wait="4" privlvl="USER"/>
                <fencedevice agent="fence_ipmiln" ipaddr="PDC-PIC-PW-02" login="hpilologin" name="HPiloFence2" passwd="hpilopword" lanplus power_wait="4" privlvl="USER"/>
        </fencedevices>

Some tests you can do:

1.) Query the power state of a node using ipmitool (tests connectivity and username/password validity only, does not cause a reboot):

ipmitool -H PDC-PIC-PW-01 -I lanplus -U hpilologin -L USER -P hpilopword chassis power status

2.) Query the power state of a node using fence_ipmilan directly from the command line:

fence_ipmilan -a PDC-PIC-PW-01 -P -T 4 -l hpilologin -p hpilopword -L USER -v -o status

3.) Fence a node using fence_ipmilan directly from the command line:

fence_ipmilan -a PDC-PIC_PW-01 -P -T 4 -l hpilologin -p hpilopwd -L USER -v -o reboot

[rhel6_cluster]

<fencedevices>
                <fencedevice agent="fence_ipmilan" auth="password" ipaddr="192.168.25.18" lanplus="on" login="hpadmin" name="HPiloFence1" passwd="hpinvent"/>
                <fencedevice agent="fence_ipmilan" auth="password" ipaddr="192.168.25.26" lanplus="on" login="hpadmin" name="HPiloFence2" passwd="hpinvent"/>

[root@PDC-PIC-PL-01 cluster]# fence_ipmilan -a 192.168.25.18 -l hpadmin -p hpinvent -o status
Getting status of IPMI:192.168.25.18...Chassis power = Unknown
Failed

I think it is not contacting my ilo. how to contact hp ilo from the OS?

My OA IP is 192.168.25.11 where as the ilo device bay for servers addreses are : .25.18 and .25.26, login to OA are same which i include during device adding, but i don't know what is missing here.

I am editing from GUI, but when i try to manually edit cluster.conf then there are lots of error and i need to reboot both the nodes.

Thanks!

[tyronefrance]

This looks like a 2 node cluster but it lookes like you are missing some variables making it so/ With a 2 node cluseter you do not have a quaram disk so you need to add. <cman expected_votes="1" two_node="1" />

Please note you can test connectivity to your ILO ports below, I have added a test to see if you can reach each node:

fence_ipmilan -a PDC-PIC-PW-01 -P -l hpilologin -p hpilopword -o status
fence_ipmilan -a PDC-PIC-PW-02 -P -l hpilologin -p hpilopword -o status

Can you try this one:

But dont forget to set the CLUSTER NAME to be the same as the CLUSTER-NAME you define when you format your GFS partitions if you want to share GFS partitions across the nodes:

Example of formatting the partition with 4 x journals: (only do this if you have GFS2 partitions)

gfs_mkfs -p lock_dlm -t gfs-cluster:gfs -j 4 /dev/vg_repo/lv_gfs_mydrive

Example of /etc/cluster/cluster.conf:

<?xml version="1.0"?>
<cluster name="gfs-cluster" config_version="30">
<fence_daemon clean_start="0" post_fail_delay="1" post_join_delay="3"/>
<clusternodes>
<clusternode name="PDC-PIC-PL-01" nodeid="1" votes="1">
<fence>
<method name="1">
<device lanplus="1" name="device_node1"/>
</method>
</fence>
</clusternode>
<clusternode name="PDC-PIC-PL-02" nodeid="2" votes="1">
<fence>
<method name="1">
<device lanplus="1" name="device_node2"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1" />
<fencedevices>
<fencedevice agent="fence_ipmilan" ipaddr="PDC-PIC-PW-01" login="hpilologin" name="HPiloFence1" passwd="hpilopword" power_wait="4"/>
<fencedevice agent="fence_ipmilan" ipaddr="PDC-PIC-PW-02" login="hpilologin" name="HPiloFence2" passwd="hpilopword" power_wait="4"/>
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>

Once you are happy that your config is ok... Simply run (ccs_config_validate): to make sure that the config is valid:

[root@test~]# ccs_config_validate
Configuration validates

Next, we need to start cman service, on both nodes (at same time):
/etc/init.d/cman start

Afterwards, we can start the other services, on both nodes:
/etc/init.d/clvmd start
/etc/init.d/gfs2 start (only do this if you have GFS2 partitions)

chkconfig --level 35 cman on
chkconfig --level 35 clvmd on
chkconfig --level 35 gfs2 on (only do this if you have GFS2 partitions)