Shared uplink set with Map VLAN tags and Multiple Networks

[Chris House]

I have multiple networks that I need a Windows blade server to be on, but the server only has 2 nics.

The blade server is in a c7000 enclosure with 2 x 1/10Gb VC-Enet Modules. It does not have a Flex-10 NIC or modules.

Port 1 on each module is connected to a Cisco switch that has multiple VLANs trunked on it.

Port 2 on each module is connected to another Cisco switch but with no vlan tagging - only one network (heartbeat) is available on this link.

I plan to create a shared uplink set of the Port 1 links for the VLAN tunnel link and define the VLANs inside it.

I will also create a regular ethernet network using the Port 2 links for the heartbeat network.

Then I will set the Server VLAN Tagging Support to Map VLAN Tags, and for the server NICs, set those to Multiple Networks and choose the VLANs that I need from the trunk as well as the Heartbeat network.

Then in the HP NIC Teaming software on the Windows blade server, define the VLANs coming in from Virtual Connect.

How will the server distinguish the Heartbeat network? Is there an option to tag that in Virtual Connect as a VLAN?

Will this work?

[WFHC-WI]

Hi Christopher,

Your Windows blade only has two physical NICs, non-Flex. This means it can only connect to (up to) two Virtual Connect networks.

When you create a shared uplink set, you then create virtual networks for each VLAN presented to the SUS. Effectively you have now limited your Windows box to seeing no more than two VLANs.

Instead, I think Tunneling VLAN Tags is a better choice. That way you create two virtual networks, VCnetMultiple and VCnetHeartbeat. Virtual Connect will pass through all tagged packets without modification, and it will be the responsibility of the teaming software to strip/add tags.

So NIC1 is assigned to VCnetMultiple and NIC2 is assigned to VCnetHeartbeat. You have redundancy, just not at the server NIC level. You can't team your NICs since they are now seeing different VLANs. If either NIC goes out (however unlikely), you have problems. If possible, present the heartbeat VLAN to the trunked port... then assign both NIC1 and NIC2 to use VCnetMultiple, and team them within your OS.

I realize best practices often dictate not teaming a heartbeat connection. I believe this is more due to load balancing then it is the nature of a team; if that is the case just use "network fault tolerance only" in your team!

good luck, if you appreciate the answers we appreciate the points!

[Chris House]

Thanks very much WFH, I have been trying to get to a point where I could test my hypothesis but I am unable to save a setting so I ended up posting here.

What you have written makes complete sense. I guess ultimately I am trying to get my two NICs teamed with the HP NIC teaming software and then split out the different networks using VLAN tagging. The problem is that my Heartbeat network does not come in on the VLAN trunk that I already have, and it itself is not coming in under a VLAN on its own port. It is a completely separate network from our datacenter network.

I want to achieve NIC teaming at the server level so that firmware updates to Virtual Connect are non-disruptive to the host since the NIC will go down while the module reboots.

What you have outlined is exactly what we are doing now - one NIC has the trunk network and in the HP NIC teaming software, the VLANs that we need are split out and then the other NIC has the heartbeat network.

So with Map VLAN Tags set, those networks are presented to the server NIC without VLAN tags, and therefore the NIC can only have one network?

[WFHC-WI]

That's right. If you have available mezzanine slots in your blades you might want to consider adding NICs (which also necessitates extra VC-Enet modules). That way you can get redundancy at the NIC level.

[Chris House]

Honestly, I have just implemented this as I initially described it and it does indeed work.

NIC teaming in the blade OS with VLANs that come in to the Virtual Connect module over a trunk, PLUS a regular network (heartbeat) that comes in on a separate link.

Here are the settings:
Ethernet Settings | Advanced settings - Map VLAN Tags (enabled, no other settings here set)

Shared Uplink set defined using the uplink ports that carry the trunked VLANs. Each vlan that it carries is defined in the shared uplink set. Smart link enabled. Native & Private network disabled.

Regular vc ethernet network using the heartbeat uplinks which do not carry any VLANs, with smart link enabled. No Private network or VLAN Tunneling enabled.

Server profiles - each server NIC set to Multiple Networks, and then those networks are the two VLANs that I need from the shared uplink set with the Server VLAN Id set to the same as what they are in the Shared Uplink Set for consistency's sake (but NOT checking "Force same VLAN mappings as Shared Uplink Sets") and then also the Heartbeat network which is Server VLAN Id 0. For "Untagged Network", Heartbeat was selected.

Server NIC teaming - Windows 2003 server using the HP Network Configuration Utility. A team was defined consisting of the two physical nics. Transmit Load Balancing with Fault Tolerance was selected. Transmit Load Balancing Method was Automatic. Nothing was changed in the Settings tab. In the VLAN tab, VLANs were added for the two that come in via the Shared Uplink Set, using the same VLAN Ids as what they are defined as in the uplink set, and then also a VLAN Id of 1 for the Heartbeat network. Default/Native VLAN Id was set to 1, and Receive Path Validation VLAN Id was set to one of the other VLANs.

After applying these changes, 3 network adapters were created (joining the 2 physical ones). I then assigned valid IPs on these new adapters for each VLAN. I am able to send/receive traffic to valid hosts on each VLAN, accomplishing exactly what I hoped to do which was have a trunked set of VLANs available to the server as well as a regular network.

[WFHC-WI]

I'm glad to hear you got things working! I would caution others to first verify that they have external devices capable of VLAN stacking (tagging packets multiple times) before implementing this config.

[Chris House]

Is it really tagging packets multiple times? My understanding was that VC was instead rewriting the VLAN ID in the frame between the shared uplink set and the server NIC - which is why you can use externally tagged VLAN networks and present them to the server under a different VLAN ID.

I'm doubtful that any external equipment of mine is aware of what is going on inside virtual connect - it just sends/receives tagged packets and continues to do so without any configuration change, so it wouldn't know about any VLAN stacking.

I have also implemented this config in HP-UX 11iv3 on a BL860c blade in the same enclosure. The blade has 4 NICs but for this exercise I'm only use two, both of which again are set to "Multiple Networks". These multiple networks are two VLANs from the shared uplink set: 157, 3020 (set the the same server VLAN ID to match what they are in the uplink set), as well as the heartbeat network which comes in on a different uplink, which is set to Server VLAN ID 0 and is the "Untagged network".

In HP-UX, these two NICs that we are using for this are lan2 and lan3. I wish to use auto port aggregation to team them in failover mode only (no link aggregation).

I set this up using system management homepage but recorded the commands that it used.

Creating lan900 which is an aggregate of lan2 and lan3 was done via the following:

nwmgr -a -S apa -c lan900 -A links=2,3 -A mode=LAN_MONITOR
nwmgr -s -S apa -A all --saved --from cu

assign heartbeat IP to lan900
/usr/sbin/ifconfig lan900 192.168.214.41 netmask 255.255.255.0 broadcast 192.168.214.255 up

create vlan 157 on lan900:
nwmgr -a -S vlan -A vlanid=157,name=157,ppa=900
nwmgr -s -A all --sa --from current -c lan

-- this created lan5000


create vlan 3020 on lan900:
nwmgr -a -S vlan -A vlanid=3020,name=3020,ppa=900
nwmgr -s -A all --sa --from current -c lan

-- this created lan5001

ifconfig was then used to assign a valid IP on the 157 vlan to lan5000 and a valid IP on the 3020 vlan to lan5001.

So now I can reliably use lan900 (heartbeat), lan5000 (157 vlan) and lan5001 (3020 vlan) for my serviceguard config and not have to worry about firmware updates to VC or a failed VC module affecting my servers or clusters.