Flex-10 VC and Network Redundancy

[Chas Martin]

We are fairly new to utilzing Flex-10 NIC technology and I'm concerned about the best way to provide redundant network connections at the server level. My manager is adamant about not utilizing NIC Teaming due to what he deems complexity and SCOM/Alerting issues.

I've read a ton of stuff on Flex-10 including the various cookbooks. Here is my current configuration.

I have two Flex 10 Modules Installed one in Bay #1 and One in Bay #2. I currently have one fiber uplink running from each VC module into a Cisco Nexus switch. Currently it's set up in an active/standy mode and both ports are in a shared uplink set (SUS).

I only utilize one NIC interface in our standard server configuration. If I lose an uplink (pull the cable) or lose a VC module (pull the whole VC). On the active side, ping to the server fails and never comes back. Obviously if I pull the stand by side nothing happens. If I software disable the active VC module, the inactive module will become active and I only lose pings for a fraction of a second.

The other thing is i tested our older style 1 Gig flex modules and of course the configuration is slightly different with 1 cable for each Vlan running to Module #1 and to Module #2. You can pull either cable and you lose pings to a server only momentarily.

I guess my main question is the only way to provide redundancy to the server level to use NIC teaming?

I'm well aware that I can provide another uplink port to each of the VC and then create a shared uplink set on the same VC providing redundancy on the uplink side.

Any ideas are welcome..

Thank.

[The Brit]

Chas,
It really depends on the OS, and the definition of "NIC Teaming". On some OS' NIC Teaming is an "Active" NIC and a "Standby" NIC, with the Active configured with the IP, and the Standby taking over the IP in the event that the Active NIC fails. I normally think of this as IP Failover, and it may be limited to failing over if the physical NIC is defective, and not if the path is down. It may not be available to your OS.

Other OS's create a Virtual Device which sits in front of two (or more) physical devices. The IP is assigned to the Virtual device, and communication is maintained as long as one of the physical devices has a functioning path. The failover between physical NICs is automatic. This is what I think of as NIC teaming.

Another option (which is used on OpenVMS) servers is called LOAD_BROKER. In this scenario several servers watch the NICs on other servers looking for failures and reacting accordingly.

Anyway, I would invest the real time in trying to educate your management. Bottom line, the only real redundancy which does not incur outages is NIC Teaming, and is so widely used, on just about every OS, that it should almost be the default.

The redundancy characteristics of the c-class blade technology has an underlying, tacit assumption that NIC teaming will be employed.

HTH

Dave.

[Chas Martin]

Thanks,

I should have been more explicit on OS type, primarily we use Windows 2003, 2008R2, Clustering and very few linux servers.

I was afraid that was going to be your answer regarding redundancy. So essentially without NIC teaming there is no way to just have the network connection failover if the active uplink fails or the entire active VC fails?

[WFHC-WI]

There is a hard internal mapping from half of the NICs that goes to the VC modules on one side of the enclosure, and an identical mapping for the remaining NICs to go to the other side.

e.g. For a full-height BL680 G5, NIC1 & NIC3 will always send traffic through the Flex 10 module in bay 1.

You can configure a virtual network to span the modules, or even to cross them so all traffic from 1/3 would travel through uplink ports on VC module 2. This requires the module in bay 1 to be online however; the traffic moves through an internal crosslink joining the modules.

So the short answer is no, for redundancy you must either team or use multi-homing and configure your servers with more than one IP (which has its own issues).

I agree with Dave about discussing this with management. Teaming is often thought of as load-balancing, but the two are separate terms. If you configure the team to use only fault tolerance without load balancing, you can be less concerned about false alerts and complex data paths.

good luck!

[shocko]

yuo should aslo enable smartlink on your vc switches for uplink failure detection

[The Brit]

Hi again Chas,
Since this perculated back to the top again, I thought I would add one more comment.

There is a "sub-"optimal solution which involves setting up dual-independent paths from two different NICs (i.e. two different IP's maybe linked by some kind of aliasing).
One path would go to VC Module 1, and one would go to Module 2. I dont know if these could be set up as Active-Active (probably can) In this case, this may be a way to "stay up" when a module fails.

As I mentioned earlier, the VC failover (between modules) takes 20-25 seconds. Some OS' (particularly clusters) can be configured to wait a prespecified interval before considering communication "down", and therefore, by increasing the setting to 30-35 seconds, you can survive a VC module failover (On my openvms clusters I set my failover wait to 40 secs, although with NIC teaming it is irrelevant).

Anyway HTH

Dave.