link aggregation config issues

[Jason W. Neiss]

Hi, all;

I recently installed HP-APA on my HPUX 11.11i A-class system in order to gang the four ports on the installed A5506B four-port ethernet adapter and increase the available bandwidth to the server. However, when I enable the aggregated port, it works for about 60 seconds, then appears to shut down. There are no error messages that I can find, and everything appears to still be in place, but the network connection simply stops working. Rebooting causes the same behavior: works for 60 seconds or so after init completes, then appears to shut down. If I enable APA by hand, the aggregated port exhibits the same behavior. The ports are connected to a Cisco switch which my network admin has configured for APA. Any ideas where I might start looking?

I'm including my config files just as a check; hostnames and IP addresses have been changed to protect the guilty.

Thanks;
Jason

/*--/etc/rc.config.d/hp_apaconf--*/

HP_APA_START_LA_PPA=900
HP_APA_DEFAULT_PORT_MODE=MANUAL

HP_APA_INTERFACE_NAME[0]=lan900
HP_APA_LOAD_BALANCE_MODE[0]=LB_MAC
HP_APA_GROUP_CAPABILITY[0]=5
HP_APA_HOT_STANDBY[0]=off
HP_APA_MANUAL_LA[0]="1,2,3,4"

HP_APA_INIT_ARGS="HP_APA_LOAD_BALANCE_MODE HP_APA_GROUP_CAPABILITY HP_APA_HOT_STANDBY HP_APA_MANUAL_LA HP_APA_INIT HP_APA_KEY"

/*--/etc/rc.config.d/hp_apaportconf--*/

HP_APAPORT_INTERFACE_NAME[0]=lan1
HP_APAPORT_GROUP_CAPABILITY[0]=5
HP_APAPORT_CONFIG_MODE[0]=MANUAL

HP_APAPORT_INTERFACE_NAME[1]=lan2
HP_APAPORT_GROUP_CAPABILITY[1]=5
HP_APAPORT_CONFIG_MODE[1]=MANUAL

HP_APAPORT_INTERFACE_NAME[2]=lan3
HP_APAPORT_GROUP_CAPABILITY[2]=5
HP_APAPORT_CONFIG_MODE[2]=MANUAL

HP_APAPORT_INTERFACE_NAME[3]=lan4
HP_APAPORT_GROUP_CAPABILITY[3]=5
HP_APAPORT_CONFIG_MODE[3]=MANUAL

HP_APAPORT_INIT_ARGS="HP_APAPORT_GROUP_CAPABILITY HP_APAPORT_PRIORITY HP_APAPORT_CONFIG_MODE HP_APAPORT_KEY HP_APAPORT_SYSTEM_PRIORITY"

/*--/etc/rc.config.d/hpbtlanconf--*/

HP_BTLAN_INTERFACE_NAME[0]="lan0"
HP_BTLAN_STATION_ADDRESS[0]=
HP_BTLAN_SPEED[0]=AUTO_ON

HP_BTLAN_INTERFACE_NAME[1]="lan1"
HP_BTLAN_STATION_ADDRESS[1]=
HP_BTLAN_SPEED[1]=100FD

HP_BTLAN_INTERFACE_NAME[2]="lan2"
HP_BTLAN_STATION_ADDRESS[2]=
HP_BTLAN_SPEED[2]=100FD

HP_BTLAN_INTERFACE_NAME[3]="lan3"
HP_BTLAN_STATION_ADDRESS[3]=
HP_BTLAN_SPEED[3]=100FD

HP_BTLAN_INTERFACE_NAME[4]="lan4"
HP_BTLAN_STATION_ADDRESS[4]=
HP_BTLAN_SPEED[4]=100FD

HP_BTLAN_INIT_ARGS="HP_BTLAN_STATION_ADDRESS HP_BTLAN_SPEED"

/*--/etc/rc.config.d/netconf--*/

HOSTNAME="backup-server.domain.com"
OPERATING_SYSTEM=HP-UX
LOOPBACK_ADDRESS=127.0.0.1

INTERFACE_NAME[0]=lan900
IP_ADDRESS[0]=10.10.10.10
SUBNET_MASK[0]=255.255.254.0
BROADCAST_ADDRESS[0]=""
INTERFACE_STATE[0]=""
DHCP_ENABLE[0]=0

ROUTE_DESTINATION[0]=default
ROUTE_MASK[0]=""
ROUTE_GATEWAY[0]=10.10.10.1
ROUTE_COUNT[0]=1
ROUTE_ARGS[0]=""

[TwoProc]

Try LB_IP for the following section...

for the file...
/*--/etc/rc.config.d/hp_apaconf--*/

HP_APA_INTERFACE_NAME[0]=lan900
HP_APA_LOAD_BALANCE_MODE[0]=LB_MAC

would be...
HP_APA_INTERFACE_NAME[0]=lan900
HP_APA_LOAD_BALANCE_MODE[0]=LB_IP


Also, regarding the /etc/rc.config.d/hp_apaportconf file...
try renaming it to hp_apaportconf.try_without

See if that works...

[Jason W. Neiss]

According to the docs, LB_MAC determines routing over the aggregate by using a portion of the destination's MAC address, and is recommended in a server-to-switch configuration. LB_IP determines routing over the aggregate using a portion of the destination's IP address, and is recommended in a server-to-router configuration. My server is connected to a Cisco C2980 switch. I'm going to try it, but I'm curious as to the theory behind this recommendation.

Thanks;
Jason

[TwoProc]

Jason, I agree with your specs re: the docs. However, we're also on Cisco switches and I'm just giving you what we did to make it work for us. It's only a suggestion to try based on empirical evidence - not built to withstand good arguments for discussion... :-)

[TwoProc]

Also, I see in my testing over here, that you're going to need to hp_apaportconf file back. Apparently, defaults are not gonna fly...

We've got the following in each group in the ap_apaportconf file... I don't know if this is going to drop off your proc or not...
HP_APAPORT_PRIORITY[0]=0

So, our entries are more like (adjusted to be your example lan names) ...
HP_APAPORT_INTERFACE_NAME[0]=lan1
HP_APAPORT_GROUP_CAPABILITY[0]=5
HP_APAPORT_PRIORITY[0]=0
HP_APAPORT_CONFIG_MODE[0]=MANUAL

I was able to test the apaportconf file switcheroo in a planned downtime of a test server a while ago - but didn't get a chance to make a test with the AP_APA_PRIORITY[0] array uninitialized... probably works without it though...

[Jason W. Neiss]

Thanks for the responses Jim. The "it worked for us" argument is certainly valid, and I've used that many times myself. Unfortunately, in this case, it didn't work for me.


This is my Data Protector cell server in a test environment, so I can do whatever reconfig and testing I like during the day. So, I was able to test your recommendations.

The hp_apaportconf file does need to be in place. The HP_APAPORT_PRIORITY config option is used for hot backup port configs, and when I had it in place, I saw noticeable errors (dropped packets and such), so I removed it. However, dropped packets is better than no packets, so I may put it back in place to test further tomorrow.

What I find especially odd, though, is that after ping stops responding, if I run netstat -in repeatedly, both Ipkts and Opkts keep increasing. I assume I'm reading this correctly, and that means "incoming packets" and "outgoing packets".

Thanks,
Jason

[TwoProc]

Sorry Jason, just pointing out the diffs b/w what you had and what I have - and seeing if that would fix it. Hope you get some *real* help from someone else who knows more.

[Dave Hutton]

I'm not sure about A-Class servers. But do you have any of these entries in your syslog at the top where it's booting?:

Mar 14 07:58:10 servername vmunix: igelan3: APA NOT SUPPORTED on HP A6825-60101 PCI 1000Base-T Adapter at hardware path
1/8/0/0
Mar 14 07:58:10 servername vmunix: igelan3: INITIALIZING HP A6825-60101 PCI 1000Base
Mar 14 07:58:12 servername vmunix: -T Adapter at hardware path 1/8/0/0

This is an L class server. It looks like some slots APA isn't supported in. By chance do you have a card in a slot it doesn't support?

We don't have APA on this server. So this error is more of a OS error.

[Xianjie Zhang]

Hi, Jason,

I think the problem is due to mismatch of switch and server configuration. Please check with the switch admininstrator what fec (Fast EtherChannel of Cisco)mode the switch ports are set. I suspect it is auto, which means the switch ports are running PAgP aggregation protocol.
To match that, you should set the port mode of server to FEC_AUTO in hp_apaportconf file.
Please change all lines of
"HP_APAPORT_CONFIG_MODE[{1,2,3,4}]=MANUAL" to
"HP_APAPORT_CONFIG_MODE[{1,2,3,4}1]=FEC_AUTO".
Important note: do not change the line for lan0( HP_APAPORT_CONFIG_MODE[1]=MANUAL), "otherwise your core lan may be aggregated, and you may lose site-lan connection.

In hp_apaconf file, you should remove the following line:
HP_APA_MANUAL_LA[0]="1,2,3,4".

Load balance alogrithm is NOT releavant to your problem.

HTH,

Xianjie


P.S.: Cisco switches use different names for port mode. "On" is equivalent to "MANUAL".
"Desireable" is equivalent to "FEC_AUTO".

If you are using new Cisco switches, it also supports LACP. Ask your switch administrator if LACP is used for those ports.
IF LACP is used, then the mode of lan1, 2, 3 and 4 should be set to "LACP_AUTO".

[Jason W. Neiss]

Dave, there are no errors of any kind in syslog.

Xianjie, thanks for the input. I spoke with my network admin, and he said we're not using Fast Etherchannel, because it is incompatible with the private vlans we use. The only configuration done on the APA ports at the switch end is to force 100FD and set the ports to promiscuous mode (because these are being used by a backup server on an admin LAN).

Is it possible that APA is incompatible with private vlans?

Thanks;
Jason

[Mitra Rath]

Hi jason,
Firstly i can suggest you to change your "HP_APA_GROUP_CAPABILITY[0]=5"
to 0....and check if it is working....

And one more thing we have vlans in my cisco catalyst switch also.And we have Fast etherchannel enabled on our switch...Cos without Fast etherchannel we cannot configure APA for load balancing,if i am not wrong.....

Mitra

[Jason W. Neiss]

Thanks for your input Mitra, but no luck.

Something else weird that I just noticed: even though the aggregated interface does not appear to let any traffic through-no ping response from either the server itself or another machine pinging that IP, no other access--in addition to the netstat -in behavior I noted above, Openview monitoring appears to be working. When I have the aggregated port active, after it fails 60 seconds in, Openview does not alert on the system. As soon as I reboot, OV generates a heartbeat polling alert ("failed to contact it using ping").

Now why would an OV ping work, but not an explicit ping from a command line?

This is starting to be very frustrating.

Thanks;
Jason

[Xianjie Zhang]

Hi, Jason,

Did you try 'linkloop' to verify the physical connection if OK? If linkloop works, then there is no problem for the aggregation part.

Can you elaborate on 'private VLAN' on switch? Is it just port-base VLAN? What is the purpose of enabling promiscuous mode on the switch?

What is the output of 'lanscan -q' on the computer?

Can you check the link status of each link? (Using 'lanadmin -g 1').

I suspect that the forced "100FD" may cause problem with the link. Did you try to set lan1 to lan4 using "Auto sense" (on both the computer and switch.)

To isolate the problem, you may want to try to use a single link (say lan1) to see if it works before try to aggregate it.

Xianjie

[Jason W. Neiss]

After using the troubleshooting information above and doing further research, I've come to the conclusion that APA is incompatible with our switch configuration. It's not stated anywhere explicitly, but I've come to believe that Fast Etherchannel is required on the switch for APA, and since our switch configuration cannot use it, APA won't work for me.

Thanks;
Jason