syslog LLT messages

[sysadm_1]

We are using Service Guard cluster (2 nodes) with cluster filesystem for oracle RAC (HP Serviceguard Cluster File System for RAC).
we get the following error message when i connect the redundant standby LAN (LAN 2) interface to the switch.Both the active (LAN 1)and standby LAN interfaces are on same VLAN.

___________________________________
Aug 9 08:12:10 clnode1 vmunix: LLT INFO V-14-1-10023 lost 6 hb seq 304589 from 2 link 2 (lan2)
Aug 9 08:12:11 clnode1 vmunix: LLT INFO V-14-1-10023 lost -2 hb seq 304588 from 2 link 2 (lan2)
Aug 9 08:12:17 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 650 ticks from 2 link 2 (lan2)
Aug 9 08:12:17 clnode1 vmunix: LLT INFO V-14-1-10023 lost 14 hb seq 304603 from 2 link 2 (lan2)
Aug 9 08:12:18 clnode1 vmunix: LLT INFO V-14-1-10023 lost -2 hb seq 304602 from 2 link 2 (lan2)
Aug 9 08:12:45 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 2751 ticks from 2 link 2 (lan2)
Aug 9 08:12:45 clnode1 vmunix: LLT INFO V-14-1-10023 lost 56 hb seq 304659 from 2 link 2 (lan2)
Aug 9 08:12:46 clnode1 vmunix: LLT INFO V-14-1-10023 lost -2 hb seq 304658 from 2 link 2 (lan2)
Aug 9 08:13:01 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 1550 ticks from 2 link 2 (lan2)
Aug 9 08:13:01 clnode1 vmunix: LLT INFO V-14-1-10023 lost 32 hb seq 304691 from 2 link 2 (lan2)
Aug 9 08:13:02 clnode1 vmunix: LLT INFO V-14-1-10023 lost -2 hb seq 304690 from 2 link 2 (lan2)
Aug 9 08:13:17 clnode1 vmunix: LLT INFO V-14-1-10023 lost 32 hb seq 304723 from 2 link 2 (lan2)
Aug 9 08:13:55 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 3700 ticks from 2 link 2 (lan2)
Aug 9 08:13:55 clnode1 vmunix: LLT INFO V-14-1-10023 lost 75 hb seq 304799 from 2 link 2 (lan2)
Aug 9 08:13:17 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 1550 ticks from 2 link 2 (lan2)
Aug 9 08:13:56 clnode1 vmunix: LLT INFO V-14-1-10023 lost -2 hb seq 304798 from 2 link 2 (lan2)
Aug 9 08:14:07 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 1153 ticks from 2 link 2 (lan2)
Aug 9 08:14:07 clnode1 vmunix: LLT INFO V-14-1-10023 lost 24 hb seq 304823 from 2 link 2 (lan2)
Aug 9 08:14:08 clnode1 vmunix: LLT INFO V-14-1-10023 lost -2 hb seq 304822 from 2 link 2 (lan2)
Aug 9 08:14:14 clnode1 vmunix: LLT INFO V-14-1-10019 delayed hb 600 ticks from 2 link 2 (lan2)
____________________________________________

why this error message appears ??

[Steven E. Protter]

Shalom,

There is a problem with heartbeat in the cluster. I'm not familiar with the actual problem, but its possible there is a network issue on one of the heartbeat LANS.

If this is not dealt with, your cluster could go into split brain syndrom and then one node will TOC, Transfer of Control.

I suggest first, checking out network connectivity both on and off the machine on the heartbeat lans specified in the cluster configuration.

SEP

[sysadm_1]

Salam Steven,

There is no issue with network.I have tested the LAN connectivity on all interfaces and working very well.I have done the redundancy test by removing the primary LAN interfaces on the nodes and cluster works perfectly.

[Patrice Le Guyader]

Hello,

Take a look at your /etc/llttab file.
We've got some troubles with veritas clusters and llt links on the same vlan.
Try to put these two llt link on separate vlan.

Hope this help
Pat

[sysadm_1]

/etc/llttab shows all the interfaces are configured for sending LLT signals.
We cannot modify /etc/llttab manually since veritas CFS is bundled with HP service guard and Service guard CFS scripts are managing this.I havent seen any option to change LLT settings in service guard CFS package config files.

I can not keep active and standby lan interfaces in seperate VLAN since same subnet from active LAN will be switching to standby LAN in case of any LAN failure.

[Patrice Le Guyader]

Hello,

Ok, I don't work with MC/SG and CFS but with VCS and CVM over CFS. For us there are 2 private links for VCS cluster in Gb. We don't use the lan/Standby card for heartbeat only these two privates links, nothing else on it. It's not like MCSG where you can setup heartbeat to go thru all configured lan card. Do you know that llt/gab traffic is not based on tcp/ip so you can set it to use lan port which don't have any ip adress or even plumbed ?

Have you try to take a look at the lltstat and gabconfig commands result ?

Hope this helps
Kenavo
Pat.

[sysadm_1]

Hi Patrice,

see /etc/llttab and gabtab files.

------------------------------------------------
#
# cat /etc/llttab
# WARNING - Do not edit! This file has been automatically generated
# and any edits will be overwitten
#
# Create time: Mon Aug 7 13:53:03 WAT 2006
#
set-cluster 26751
set-node 1
link lan0 /dev/lan:0 - ether - -
link lan1 /dev/lan:1 - ether - -
link lan2 /dev/lan:2 - ether - -
set-timer peerinact:17900
start
#
# _---------------------------------------------

#
# cat /etc/gabtab
# WARNING - Do not edit! This file has been automatically generated
# and any edits will be overwitten
#
# Create time: Mon Aug 7 13:53:03 WAT 2006
#
/sbin/gabconfig -c -n2
#
#
_------------------------------------------------

lltstat -l
LLT link information:
link 0 lan0 on etherfp hipri
mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
txpkts 3639024
rxpkts 2760796
latehb 452 errors 0
link 1 lan1 on etherfp hipri
mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
txpkts 3637893
rxpkts 2732550
latehb 438 errors 28
link 2 lan2 on etherfp hipri
mtu 1500, sap 0xcafe, broadcast FF:FF:FF:FF:FF:FF, addrlen 6
txpkts 3555016
rxpkts 2702895
latehb 442 errors 16815
#

----------------------------------------------------


All the three interfaces are used for LLT messages.Out of three two interfaces (LAN1 adn LAN2 are in same VLAN).

Since Service guard package maintains these files,we can not make any manual changes.Each time service guard restart the CFS package,it over write the file with new one.
Otherwise i would have added option in llttab "link-lowpri" for one of the interface in same VLAN.Then this interface will be used for LLT messages only when all other interfaces are down.