HP NIC Teaming and 3Com Switches

[Jeffry A Spain]

There is a scarcity of documentation regarding the specific configuration of HP NIC Teaming with 3Com switches. Here are my experiences.

I tested three models of HP Proliant servers (a DL380 G3, DL380 G4, and DL385 G1) all of which have dual Broadcom gigabit Ethernet NICs built in (NC7781 in the DL380 G3, NC7782 in the others). I used version 11.7.1.0 of HPs NC-series NIC driver and version 9.40 of the HP Network Configuration Utility (teaming software), both current as of this writing. All systems were running Windows Server 2008 Enterprise SP-1, x86 for the DL380s, x64 for the DL385.

The Proliant servers were connected to a 3Com 5500G SuperStack 4 switch running enhanced software version 3.03.00s168p01, also current as of this writing.

There are two ways to configure NIC Teaming: using either a static link aggregation group on the switch, which uses the 802.3ad LACP protocol, or a manual link aggregation group, which does not use LACP. Both can be made to work, but after testing, I found that the manual group initializes network connectivity 5 to 10 seconds faster, which is a good thing for Windows 2008. I also believe it to be less susceptible to link aggregation failure due to LACP communication problems.

[Jeffry A Spain]

To configure a manual link aggregation group on the 3Com switch, the prototypical configuration is as follows:

[Switch] link-aggregation group 1 mode manual
[Switch] interface GigabitEthernet1/0/1
[Switch-GigabitEthernet1/0/1] stp edged-port enable
[Switch-GigabitEthernet1/0/1] port link-aggregation group 1
[Switch-GigabitEthernet1/0/1] interface GigabitEthernet1/0/2
[Switch-GigabitEthernet1/0/2] shutdown
[Switch-GigabitEthernet1/0/2] stp edged-port enable
[Switch-GigabitEthernet1/0/2] port link-aggregation group 1

Display the switch configuration and be sure that â lacp enableâ does not appear in the configuration of either of these interfaces, as it might if you have previously configured a static link aggregation group. Enabling these interfaces as stp edged ports makes them go into forwarding mode more rapidly when the server is rebooted or the interfaces are brought up. Shutting down the second interface prevents networking problems on the server while NIC Teaming is being set up.

To set up NIC teaming, I suggest a console or iLO connection to the server to avoid the intermittent or lost connectivity that will occur using RDP. Be sure that any static IP addresses are removed from the individual Local Area Connections before setting up the team. Now using the Network Configuration Utility, team the two NICs. Open the properties of the Team. On the Teaming Controls tab, set the Team Type Selection to â Switch-assisted Load Balancing with Faulty Tolerance (SLB).â Set the Transmit Load Balancing Method to â Destination IP Address.â Click OK to accept the configuration changes. The Network Configuration Utility will create a new miniport visible in the Network Connections control panel (accessible from the Network and Sharing Center). HP recommends configuring static IP addressing on this miniport, again to reduce the delay before network connectivity is available to the operating system when the server is restarted.

Now undo shutdown on interface GigabitEthernet1/0/2 and the team should become active. In the Network Configuration Utility, the team icon should change color from yellow to green. On the switch, the command â display link-aggregation verbose 1â should indicate that the Loadsharing Type is â Sharâ (Loadsharing). Both local ports should have status â Sâ (Selected) and an identical numeric key.

One problem that I found occasionally, mainly with the use of static link aggregation groups, was that the second port in the group would have status â Uâ (Unselected) and the Loadsharing Type would be â NonSâ (Non-Loadharing). In these cases the numeric keys of the two ports would invariably be different. The only way I found to fix this was to shut down the second interface, remove all of the link aggregation configuration commands, i.e. â undo link-aggregation group 1,â and, if necessary, on the interfaces â undo lacp enable,â and then start over with the link aggregation configuration.

[Jeffry A Spain]

The configuration procedure for a static aggregation group on the 3Com switch is similar to the foregoing. The prototypical configuration is as follows:

[Switch] link-aggregation group 1 mode static
[Switch] interface GigabitEthernet1/0/1
[Switch-GigabitEthernet1/0/1] stp edged-port enable
[Switch-GigabitEthernet1/0/1] port link-aggregation group 1
[Switch-GigabitEthernet1/0/1] interface GigabitEthernet1/0/2
[Switch-GigabitEthernet1/0/2] shutdown
[Switch-GigabitEthernet1/0/2] stp edged-port enable
[Switch-GigabitEthernet1/0/2] port link-aggregation group 1

Display the configuration and be sure that â lacp enableâ does appear in the configuration of both of these interfaces. It should have been entered automatically for you as a result of the above commands.

The NIC Teaming configuration is also similar. Set both the Team Type Selection and Transmit Load Balancing Method to â Automatic (Recommended).â

Again the â undo shutdownâ command on the second interface should make the team icon go green, and the command â display link-aggregation verbose 1â should again indicate that both links are Selected and that Loadsharing is in effect.

On rebooting the servers, I found that it took 5 to 10 seconds longer for networking to come up with a static link aggregation group compared to a manual group, which is potentially problematic for the Windows operating system. Thus I have chosen to use manual groups in all cases. I have not been able to test what happens when the switch itself is rebooted. Here again I would be concerned about link aggregation failures and possible connectivity loss with the use of static groups.

Although this has been pointed out by others, itâ s worth mentioning that even with link aggregation, the maximum connection bandwidth between the server and any other individual host is limited to 1 GBPS or the bandwidth of a single one of the aggregated links. With either a manual or a static link aggregation group, the server chooses its outgoing link based on a hash of the destination IP address. The switch chooses its outgoing link to the server based on a hash of the source and destination IP addresses. Thus for communication with any given host, the switch and the server will always use the same, single link to transmit data.

[Jeffry A Spain]

In the above, the curly double-quote character appears as â . Sorry for the legibility problems.

[RicN]

Thank you for sharing this!

[Jeffry A Spain]

One problem that occurs in connection with the configuration of link aggregation groups is the disappearance of the command "apply qos-profile default" from interfaces that are made members of a link aggregation group. Attempting to add back this command generates an error. Based on a support case with 3Com, this is apparently a bug in the 5500G software. It has been fixed in version 3.03.01s168p07.