Oversubscription of VC Flexfabric SAN connections?

[chuckk281]

Dae Yong had a question on Virtual Connect FC SAN links:

**************

Greetings from Korea,

VC Flexfabric Module is being just installed and configured, four Uplink Ports (X1-X4) are configured as FC (8Gb supported).

Below picture shows 16 servers oversubscription of legacy VC-FC module uplinks.

Now there are 8 servers in C7000 with VC Flexfabric and just TWO N-ports (NPIV+x enabled) can be found in SAN switch which is connected with VC Flexfabric, even though 4 uplink ports are attached.

**********

A lively discussion ensued.

From Dave:

Do all servers have an OS loaded and powered on?

They will activate uplinks as the drivers are powered on.

First server will use Uplink1, second to power up and login will use Uplink2 and so on.

Reply from Dae Yong:

Yes, all servers (Ubuntu OS loaded) are powered on and of course, inbox driver is used. Do you mean that ‘they will activate uplinks as the drivers are powered on’ is normal behavior with ‘manual login re-distribution’ setting?  You know, ‘auto login re-distribution’ setting will balance it dymamically. Now configured as default setting, manual login re-distribution.

From Kelly:

Did you have all cables connected and remained cabled since the OS’ were booted?    Do you see the same on both fabrics?  (left and right)

If I remember correctly, as an example, if only two uplink X ports were connected at boot, only those TWO X ports will be used and will remained used even if the other cables are connected/configured until logins are re-distributed in VC or OS is rebooted.

Also – if uplink cables are disconnected at any time after the servers are up and running, active connections will move to good X ports.  So if X3 and X4 were disconnected, all active logins would move to X1 and X2 and remain there until re-distributed, or the OS is rebooted.

And from Dan:

If you think it’s just a Distribution problem, this is easily solved by going to SAN Fabrics in VC on the left menu, then on the right side, click to highlight the Fabric you are working on, then after its highlighted, RIGHT click and choose “Redistribute” from the menu.

Note that Distribution only happens during FC Init/login which is usually at boot when the driver loads.

               Distribution will always take the FC uplink with the least amount of other servers on it, so things load up evenly.

REdistribution is how you move an existing connection to an alternate uplink either after it was added or fixed.

2 questions come to mind:

1)      Are you sure they wired all 4 cables to the same Fabric?  You cannot wire 2 to BrocadeA and 2 to BrocadeB like you would do for Ethernet.

2)      What kind of Fabric are we dealing with?  Some Fabrics enable NPIV per port while others are per Switch.

1. Check the Appendix sections at the end of the FC Cookbook for help with MDS and Brocade configs:
   http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c01702940-11.pdf

*******

Comments or questions?