Cisco UCS Management Interface configuration (1 or 2?)

[alex_goltz]

I'm about to go live with my Cisco UCS blades, and I'm looking for some community feedback on a configuration question.

Part of this challenge, is that I haven't seen UCS Fabric Interconnects fail-over yet to see what impacts there are.  And I also don't know what the best practice is, since UCS architectures have been a so-called "game changer".
Nimble's documentation in KB-000202 has shown them using 1 Management interface in failover mode.
HOWEVER, this article (which has been extremely helpful), shows 2 management interfaces, each on a fabric, in failover mode.

What are users protecting themselves against, having two management interfaces in failover mode?
(I do plan on having my vMotion traffic on the management interface(s).)

[rugby0134 · Accepted solution]

the MGMT interfaces failover between themselves to phone home. If an FI was to drop, the MGMT ports failover - but the CTRL does not failover.  Best practice is to put port eth1 into FI0 and port ETH2 into FI1.  If you choose not to put eth1 into the same FI - it works fine.  But a loss of the ethernet port on eth1 would the array to failover CTRL after several heartbeat failures and if the other eth0 port was up.

most small nimble deployments don't attached eth1 and eth2 up for failover.

[alex_goltz]

If you were building a fresh UCS setup, what would you do?
I have a huge problem right now with Nimble's KB-000202 document.  This is apparently THE "Best Practice" document for a UCS build, and they only do one management interface in "Failover" mode, aside from other errors like MTU settings and upstream switch settings.  What I really need to know, is what is going to happen when I have to reboot an FI?  Are my host heartbeats going to freak out my HA?

[rugby0134]

No. The reboot of an FI will only cause the some ports to drop on both sides of the array. the array will see that as a switch failure and stay on the same CTRL since a failover will not fix the issue.

[sgonzalez126]

One thing to keep in mind though.  If you are using iSCSI boot from SAN volumes with VMWare hosts the FI failover will raise alerts on the ESXi hosts that require restarting the management agents to cleanup.  It puts the boot volumes in RO mode until you do this but the VM's continue to run normally in memory the whole time.

This is the alert we get when we update firmware on the FI's and fail over to the second one with our iSCSI mounted storage.

Lost network connectivity on virtual switch

"iScsiBootvSwitch". Physical NIC vmnic2 is down.

Affected portgroups:"iSCSI-B".

error

9/17/2015 8:31:14 PM

servervm-2

Restarting the agents using one of the methods in this article will bring everything back to normal in VMware and no downtime or issue on the hosts.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003490

Took us a while to track this down.

Steve