One network interface is down in all cluster

[gsingh]

Hi,

OS: HPUX- 11.23
Service Guard: 11.17.00
Hardware: rx6600
1 day back one switch was rebooted by network guys. In that network we have 3 clusters of 2 node each. After this reboot of switch some of network interfaces are showing 'down' in output of 'cmviewcl'. Below is the output from on of cluster:

redbw2 / > cmviewcl -v

CLUSTER STATUS
cluster1 up

NODE STATUS STATE
redbw1 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vglock /dev/dsk/c13t0d5 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY down 0/4/2/0 lan2 <---DOWN
PRIMARY up 0/2/1/0 lan0
STANDBY down 0/2/1/1 lan1 <---DOWN
STANDBY up 0/4/2/1 lan3

PACKAGE STATUS STATE AUTO_RUN NODE
dbciRBP up running enabled redbw1

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled redbw1 (current)
Alternate up enabled redbw2

NODE STATUS STATE
redbw2 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vglock /dev/dsk/c14t0d5 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/4/2/0 lan2
PRIMARY up 0/2/1/0 lan0
STANDBY up 0/2/1/1 lan1
STANDBY down 0/4/2/1 lan3 <---DOWN
redbw2 / >

Before reboot of switch all interfaces were up. While I was cehcking the cables one node got rebooted of another cluster and after that reboot all interfaces are up for that NODE ONLY.
I wanted to know how the status of lan3 could be changed. Do I need to reboot all systems or any other way around.

[sujit kumar singh]

hi

can you also provide from both the nodes the following O/pS?

lanscan
netstat -rn
lanadmin -s 2
lanadmin -s 3
ifconfig lan2
ifconfig lan2
and the /var/adm/syslog/syslog.log from both the nodes?

regards
sujit

[Duncan Edmonstone]

This might be down to speed negotiation between your LAN carde and your switch - if one end is set to AUTO and the other to a hard speed, then this sort of thing can happen.

Make sure that for 100BaseT cards you hard set youyr end and the switch port to 100FD, and for GbE cards you leave both ends as AUTO

ALternatively this could be your network admins failing to save their switch configurations before the switches are rebooted. Cisco switches for instance will not save any changes the admin makes to take effect afetr reboot unless the admin specifically issues a command to save the configuration...

HTH

Duncan

[gsingh]

Attached are outputs asked.

[sujit kumar singh]

Hi,


The LAN Switching of the subnets on the nodes is taking place as this should in case of the LAN failures, that is OK with the nodes.

But there are LAN Switchings in the syslogs that might be due to the switch reboots.

yes as said in the earlier reply please properly set the Switch ports and do save the Switch Configurations on the switches.
The lan0,1,2,3 seem to be Gigabit cards so adjust the port settings on teh Swicth side.

Also in case of the I/Fs showing Down you can confirm with the linkloop to the down LAN I/Fs the LAN cards as such:

#linkloop -i 0 0x
#linkloop -i 2 0x

regards
Sujit

[Stephen Doud]

Before trying a reboot, try resetting the down NICs via lanadmin.

Example:
# lanadmin


LOCAL AREA NETWORK ONLINE ADMINISTRATION, Version 1.0
Thu, Apr 9,2009 08:02:51

Copyright 1994 Hewlett Packard Company.
All rights are reserved.

Test Selection mode.

lan = LAN Interface Administration
menu = Display this menu
quit = Terminate the Administration
terse = Do not display command menu
verbose = Display command menu

Enter command: lan

LAN Interface test mode. LAN Interface PPA Number = 0

clear = Clear statistics registers
display = Display LAN Interface status and statistics registers
end = End LAN Interface Administration, return to Test Selection
menu = Display this menu
ppa = PPA Number of the LAN Interface
quit = Terminate the Administration, return to shell
reset = Reset LAN Interface to execute its selftest
specific = Go to Driver specific menu

Enter command: ppa
Enter PPA Number. Currently 0: 1

LAN Interface test mode. LAN Interface PPA Number = 1

clear = Clear statistics registers
display = Display LAN Interface status and statistics registers
end = End LAN Interface Administration, return to Test Selection
menu = Display this menu
ppa = PPA Number of the LAN Interface
quit = Terminate the Administration, return to shell
reset = Reset LAN Interface to execute its selftest
specific = Go to Driver specific menu

Enter command: reset
Resetting LAN Interface to run selftest.

LAN Interface test mode. LAN Interface PPA Number = 1

clear = Clear statistics registers
display = Display LAN Interface status and statistics registers
end = End LAN Interface Administration, return to Test Selection
menu = Display this menu
ppa = PPA Number of the LAN Interface
quit = Terminate the Administration, return to shell
reset = Reset LAN Interface to execute its selftest
specific = Go to Driver specific menu

Enter command: q