HP-UX ServiceGuard LAN Question

[cheesytime]

I have a cluster where cmviewcl -v shows the following output:

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan1
PRIMARY up 0/6/1/0/6/1 lan4
STANDBY down 0/5/2/0/6/0 lan5

PACKAGE STATUS STATE AUTO_RUN NODE
rox up running enabled roxsco

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

I wonder why the STANDBY interfase shows as down on both nodes, is this a normal behavior? why is it maked as down?

ifconfig for the standby interface on one node shows

lan3: flags=1842
inet 0.0.0.0 netmask 0


Thanks

[Steven E. Protter · Accepted solution]

Shalom,

Standby is down for both nodes, because its not the production or primary lan for either one.

It should be tested to make sure it will work.

SEP

[cheesytime]

I dont understand what you mean, could you explain?

[Raj D.]

thathotspot,
It is obvious from the cmviewcl ouput that the standby lan is down!.
But you have mentioned that the stdby lan is down on other node as well. This is not a normal behavior. Looks like something broke or something wrongly configured.


- Was it working earlier (stdby lan ).
- If the stdby lan went down recently , you will see it in the log.

Do you see anything with:
# cat /var/adm/syslog/syslog.log | grep -i lan

- If you have a log rotation of the syslog , you have to check other syslog files.


- also check:
1. #lanadmin -sx 5
2. If 1 is UP, check the lan5 connectivity with linkloop command.


Hth,
Raj.

[cheesytime]

Raj:

May 22 22:59:26 rox vmunix: igelan3: INITIALIZING HP A9784-60002 PCI/PCI-X 1000Base-T FC/GigE Combo Adapter at hardware path 0/5/2/0/6/0
May 22 23:00:35 rox cmcld: lan5 0x0015605e6121 standby bridged net:1
May 22 23:00:35 roxb cmcld: lan5 failed

it says it failed, what could be wrong?

I just cat /var/adm/syslog/syslog.log | grep -i lan5

[Raj D.]

> May 22 23:00:35 roxb cmcld: lan5 failed

So that means lan5 is down since May22, 11.00pm night. Almost 4 months .
(assuming it is 2009 log)



- What is the output of :
# lanadmin -sx 5

[ is it showing connected or DOWN. ]
- If it is down , you have to ask your network support guys to verify the cable and the connectivity.


Hth,
Raj.

[cheesytime]

Raj:

lanadmin -sx 5

Speed = 1000000000
The link is down. The speed and other link state information
are undefined. The configuration specified for this card is:
Speed = 1000 or 100 or 10 Mbps.
Autonegotiation = On.

What else could I look at as far as testing connectivity?

[Raj D.]

thathotspot,

> The link is down.
> What else could I look at as far as testing connectivity?




-What is the model of the server.?

- You can find out the slot number to the lan5 . (using # olrad -q )
- go to the server if possible and take a look at the slot , check if the 'LINK LED' is up , and glowing constatnly.

-If not it could be a network issue or card issue, most probably it looks like a network issue as you mentioned that the other server also having the same problem. The switch port may be bad or disconnected ..something ..you have to figure out step by step with the help of network team. You can ask your network support tem to check the connectivity .


- If the connectivity made up ,you will see the light in the slot, and lanadmin will show it is UP, and the standby lan will come up automatically with a message in syslog (lan 5 recovered).



Cheers,
Raj.



( * Remember to assign the pending points for this thread. )
- Your profile says: I have assigned points to 26 of 149 responses to my questions.
:: make it look 100%, when get a chance)

[Michael Steele_2]

Hi

a) "...Serviceguard does not support using networking tools such as ifconfig or the configuration file /etc/rc.config.d/netconf..."

b) verify current cluster configuration

cmgetconf -c clustername config.ascii

MC/SG exchanges the MAC addresses of the nics to the nodes in the cluster via the cluster binary file. YOU DO NOT USE STAND LAN UTILITIES LIKE IFCONFIG OR LANSCAN WITH MC/SG

c) cmgetconf will reply with an error if it finds one. If error, verify the cluster configuration in /etc/cmcluster/cluster.ascii

[Stephen Doud]

1) unplumb and re-plumb the standby NIC:

# ifconfig lan5 unplumb
# ifconfig lan5 plumb

Repeat on the other nodes' standby NIC

2) Wait a few seconds and check syslog to see if he network interface has recovered. If it has not recovered, then Serviceguard cannot see a DLPI (link-level) connection between the local standby and it's partner primary NIC, or the standby NIC on the other server. Though this is not a definitive test, use the 'linkloop -a ' command to test NIC-to-NIC tranmission capability at the link level. If that succeeds, the switch in between may be the source of the problem. It may be blocking protocols, or the VPN may not be working as expected.

[cheesytime]

Are you sure that if i unplumb and plumb that standby interfase (on both nodes) nothing wrong will happen?

[cheesytime]

The mac address of lan5 on one of the nodes is 0x001560048930 so from the other node I ran linkloop -i 0 0x001560048930 and it gave me the following results

Link connectivity to LAN station: 0x001560048930
-- OK


I assume connectivity is OK now. is it safe to unplumb/plumb up that particular standby interfase again?

Please advice what else could be done.

P.D: I already submitted the points.

[Michael Steele_2]

run

cmscancl -s -o /tmp/file

vi /tmp/file
search for LAN section near bottom

[cheesytime]

------ Checking LOCAL network connections (node2) ------


(The linkloop command will test for link level connections between all LAN
hardware displayed by lanscan. A -- OK after the line means those two
devices can talk to each other. A (NO CONNECTION) after a line means
the two devices can not talk at the link (MAC) level. Network connectivity
check will not be performed for non-LAN hardware (HyperFabric, ATM. etc),
if any, since linkloop command is supported only for LAN hardware.)

------ lan0 to lan1 ------
PPA 0 link test to 0x001560048911 (NO CONNECTION)

------ lan0 to lan2 ------
PPA 0 link test to 0x0015605E811C (NO CONNECTION)

------ lan0 to lan3 ------
PPA 0 link test to 0x0015605E7185 -- OK

------ lan0 to lan4 ------
PPA 0 link test to 0x00110A42BE58 (NO CONNECTION)

------ lan0 to lan5 ------
PPA 0 link test to 0x00110A42BE59 (NO CONNECTION)

------ lan1 to lan0 ------
PPA 1 link test to 0x001560048910 (NO CONNECTION)

------ lan1 to lan2 ------
PPA 1 link test to 0x0015605E811C (NO CONNECTION)

------ lan1 to lan3 ------
PPA 1 link test to 0x0015605E7185 (NO CONNECTION)

------ lan1 to lan4 ------
PPA 1 link test to 0x00110A42BE58 (NO CONNECTION)

------ lan1 to lan5 ------
PPA 1 link test to 0x00110A42BE59 (NO CONNECTION)

------ lan2 to lan0 ------
PPA 2 link test to 0x001560048910 (NO CONNECTION)

------ lan2 to lan1 ------
PPA 2 link test to 0x001560048911 (NO CONNECTION)

------ lan2 to lan3 ------
PPA 2 link test to 0x0015605E7185 (NO CONNECTION)

------ lan2 to lan4 ------
PPA 2 link test to 0x00110A42BE58 (NO CONNECTION)

------ lan2 to lan5 ------
PPA 2 link test to 0x00110A42BE59 (NO CONNECTION)

------ lan3 to lan0 ------
PPA 3 link test to 0x001560048910 -- OK

------ lan3 to lan1 ------
PPA 3 link test to 0x001560048911 (NO CONNECTION)

------ lan3 to lan2 ------
PPA 3 link test to 0x0015605E811C (NO CONNECTION)

------ lan3 to lan4 ------
PPA 3 link test to 0x00110A42BE58 (NO CONNECTION)

------ lan3 to lan5 ------
PPA 3 link test to 0x00110A42BE59 (NO CONNECTION)

------ lan4 to lan0 ------
PPA 4 link test to 0x001560048910 (NO CONNECTION)

------ lan4 to lan1 ------
PPA 4 link test to 0x001560048911 (NO CONNECTION)

------ lan4 to lan2 ------
PPA 4 link test to 0x0015605E811C (NO CONNECTION)

------ lan4 to lan3 ------
PPA 4 link test to 0x0015605E7185 (NO CONNECTION)

------ lan4 to lan5 ------
PPA 4 link test to 0x00110A42BE59 (NO CONNECTION)

------ lan5 to lan0 ------
PPA 5 link test to 0x001560048910 (NO CONNECTION)

------ lan5 to lan1 ------
PPA 5 link test to 0x001560048911 (NO CONNECTION)

------ lan5 to lan2 ------
PPA 5 link test to 0x0015605E811C (NO CONNECTION)

------ lan5 to lan3 ------
PPA 5 link test to 0x0015605E7185 (NO CONNECTION)

------ lan5 to lan4 ------
PPA 5 link test to 0x00110A42BE58 (NO CONNECTION)


------ Comparing node2 binary configuration with node1 ------


(The cluster configuration files matched.)


###### Checking REMOTE network connections (node2 to node1) ######

------ lan0 on node node2 to lan0 on node node1 ------
PPA 0 link test to 0x001560048930 -- OK

------ lan0 on node node2 to lan1 on node node1 ------
PPA 0 link test to 0x001560048931 (NO CONNECTION)

------ lan0 on node node2 to lan2 on node node1 ------
PPA 0 link test to 0x0015605E6132 (NO CONNECTION)

------ lan0 on node node2 to lan3 on node node1 ------
PPA 0 link test to 0x0015605E6121 (NO CONNECTION)

------ lan0 on node node2 to lan4 on node node1 ------
PPA 0 link test to 0x00110A42DEB0 (NO CONNECTION)

------ lan0 on node node2 to lan5 on node node1 ------
PPA 0 link test to 0x00110A42DEB1 (NO CONNECTION)

------ lan1 on node node2 to lan0 on node node1 ------
PPA 1 link test to 0x001560048930 (NO CONNECTION)

------ lan1 on node node2 to lan1 on node node1 ------
PPA 1 link test to 0x001560048931 (NO CONNECTION)

------ lan1 on node node2 to lan2 on node node1 ------
PPA 1 link test to 0x0015605E6132 (NO CONNECTION)

------ lan1 on node node2 to lan3 on node node1 ------
PPA 1 link test to 0x0015605E6121 (NO CONNECTION)

------ lan1 on node node2 to lan4 on node node1 ------
PPA 1 link test to 0x00110A42DEB0 (NO CONNECTION)

------ lan1 on node node2 to lan5 on node node1 ------
PPA 1 link test to 0x00110A42DEB1 (NO CONNECTION)

------ lan2 on node node2 to lan0 on node node1 ------
PPA 2 link test to 0x001560048930 (NO CONNECTION)

------ lan2 on node node2 to lan1 on node node1 ------
PPA 2 link test to 0x001560048931 (NO CONNECTION)

------ lan2 on node node2 to lan2 on node node1 ------
PPA 2 link test to 0x0015605E6132 -- OK

------ lan2 on node node2 to lan3 on node node1 ------
PPA 2 link test to 0x0015605E6121 (NO CONNECTION)

------ lan2 on node node2 to lan4 on node node1 ------
PPA 2 link test to 0x00110A42DEB0 (NO CONNECTION)

------ lan2 on node node2 to lan5 on node node1 ------
PPA 2 link test to 0x00110A42DEB1 (NO CONNECTION)

------ lan3 on node node2 to lan0 on node node1 ------
PPA 3 link test to 0x001560048930 -- OK

------ lan3 on node node2 to lan1 on node node1 ------
PPA 3 link test to 0x001560048931 (NO CONNECTION)

------ lan3 on node node2 to lan2 on node node1 ------
PPA 3 link test to 0x0015605E6132 (NO CONNECTION)

------ lan3 on node node2 to lan3 on node node1 ------
PPA 3 link test to 0x0015605E6121 (NO CONNECTION)

------ lan3 on node node2 to lan4 on node node1 ------
PPA 3 link test to 0x00110A42DEB0 (NO CONNECTION)

------ lan3 on node node2 to lan5 on node node1 ------
PPA 3 link test to 0x00110A42DEB1 (NO CONNECTION)

------ lan4 on node node2 to lan0 on node node1 ------
PPA 4 link test to 0x001560048930 (NO CONNECTION)

------ lan4 on node node2 to lan1 on node node1 ------
PPA 4 link test to 0x001560048931 (NO CONNECTION)

------ lan4 on node node2 to lan2 on node node1 ------
PPA 4 link test to 0x0015605E6132 (NO CONNECTION)

------ lan4 on node node2 to lan3 on node node1 ------
PPA 4 link test to 0x0015605E6121 (NO CONNECTION)

------ lan4 on node node2 to lan4 on node node1 ------
PPA 4 link test to 0x00110A42DEB0 (NO CONNECTION)

------ lan4 on node node2 to lan5 on node node1 ------
PPA 4 link test to 0x00110A42DEB1 (NO CONNECTION)

------ lan5 on node node2 to lan0 on node node1 ------
PPA 5 link test to 0x001560048930 (NO CONNECTION)

------ lan5 on node node2 to lan1 on node node1 ------
PPA 5 link test to 0x001560048931 (NO CONNECTION)

------ lan5 on node node2 to lan2 on node node1 ------
PPA 5 link test to 0x0015605E6132 (NO CONNECTION)

------ lan5 on node node2 to lan3 on node node1 ------
PPA 5 link test to 0x0015605E6121 (NO CONNECTION)

------ lan5 on node node2 to lan4 on node node1 ------
PPA 5 link test to 0x00110A42DEB0 (NO CONNECTION)

------ lan5 on node node2 to lan5 on node node1 ------
PPA 5 link test to 0x00110A42DEB1 -- OK

I apologize, STANDBY on both nodes is LAN3 NOT lan5. Thats the output from cmscancl.

[cheesytime]

http://docs.hp.com/en/5991-1125/5991-1125.pdf

Issue
If you unplumb a standby interface configured in a running Serviceguard cluster
(ifconfig unplumb) and then Serviceguard tries to fail over from the
primary interface to the standby interface, the interface failover will fail as expected. If
the standby interface is replumbed after the failover fails, further attempts by the
primary interface to fail over to the standby interface can fail. This behavior is seen only
on systems with the compartments feature enabled.
Workaround
TIP HP recommends that you not unplumb standby interfaces, whether or notcompartments are enabled.

Are you sure unplumb on a running cluster is OK? Please, ADVICE.

[Michael Steele_2]

HI

Well you have to go with the manufacturer, always. There is no higher source. And this source has repeatedly stated not to use the standard HP-UX Networking commands with MC/SG.

So don't unplumb. There are other MC/SG commands for this. Read this paragraph again carefully, its from the replacement NIC procedure for MC/SG:

After the on-line or off-line replacement of LAN cards has been done,
Serviceguard will detect that the MAC address (LLA) of the card has
changed from the value stored in the cluster binary configuration file,
and it will notify the other nodes in the cluster of the new MAC address.
The cluster will operate normally after this.
It is also recommended that you update the new MAC address in the
cluster binary configuration file by re-applying the cluster configuration

Use the following steps for on-line reconfiguration:
1. Use the cmgetconf command to obtain a fresh ASCII configuration
file, as follows:
cmgetconf -c clustername config.ascii
2. Use the cmapplyconf command to apply the configuration and copy
the new binary file to all cluster nodes:
cmapplyconf -C config.ascii
This procedure updates the binary file with the new MAC address and
thus avoids data inconsistency between the outputs of the cmviewconf
and lanscan commands.

[Michael Steele_2]

Hi

In reviewing your cmscancl output I now have a concern:

node1 to node2
lan3 to lan1

node2 to node2
lan0 on node node2 to lan4 on node node1

Please verify that you have the same versions of MC/SG on these nodes.