VC Flex 10

[aali]

Hello,

I have the following configuration

[Hongjun Ma]

the physically cabling may look like a loop but loop will not occur. The blocking mechnism happens on VC side. The first thing VC needs to do is to make sure no looping toward network side.

The reason 6513 is not blocking is that you should treat VC as a host ports, enable BPDU guard and STP edge for edge security.

The No.1 thing you need to remember about VC is that it doesn't talk STP with any switch at all. It uses its own local internal decision to block redundant path.

you can check my blog for some VC tech overview slides and other cookbooks.

[aali]

Thanks for your quick reply. 

Regards,

[aali]

Thanks

[Hongjun Ma]

can you describe more about "networking" flapping? do you see ports between 6513 and flex-10 going up/down? or you have intermittent vm connectivity issue on a particular vlan?

what's your flex-10 firmware version? did you open a HP support case so they can help you on VC side?

in regard to your idea about connecting two chassis together like the way you described? my answer is don't do this way.  by removing flex-10 module 2, in reality you are losing half of bandwidth for all blades. all blades 2nd port(1g or 10g) are internally mapped to flex-10 module 2. Also there are more issues you'll encounter like converting non-stacking design to stacking design. your life will be miserable. your current setup should ok perfectly fine and it's very common for c7000 setup so stay this way and resolve this "flap" issue first.

[aali]

Thank You so much!  The VC firmware is 3.30.

[Hongjun Ma]

it's good that you send out error message on c7000 side. From the c7000 log message, you actually have OA mgmt link flapping. This should have nothing to do with 6513 and VC. Usually you'll have a pair of mgmt switch to connect c7000 OA ports, this is pure for mgmt access of enclosures.

OA and VC are doing different things in c7000. OA being mgmt and VC being I/O control for user traffic.

so you need to check where OA link is connected to, are they going to any TOR switch, the problem is with the GE link between that switch and OA and then you need to check cables/transceivers.

In regard to your 6513 config, just enable STP portfast trunk and BPDU guard under port channels. Remember treat VC ports as "big" host trunk ports.

[aali]

I was incorrect about firmware version.  Actually right now the version is 3.18 and we will be upgrading to 3.51.

We actually do not see any network ports flapping at all. The 6513 ports do not show any errors at all. No intermittent connectivity on a single VLAN.

The Virtual connect is at version 3.18 - we will schedule an upgrade, presumably for Sunday the 25th to upgrade to version 3.51

There are just times when we lose connectivity to the Blade Enclosure based on the fact that we receive ICE alerts indicating VMS are unreachable. When we receive these messages, it has been noticed that some of the Physical servers - connected via copper to an entirely different blade on the 6513, will get connectivity errors. I can not correlate an exact time, but I believe that if the Switch CPU is getting to 90%, then the packets destined for those physical servers may be dropped.

Thank you so much.

[Hongjun Ma]

i'm a little confused. Do you have OA link flapping? I saw it  in the log message.

if 6513 is running 90%, "show proc cpu" should tell you which process is taking cpu cycles. even you run high on cpu, if you don't have any routing/mac table flapping, 6513 should still switch traffic without interruption because those are in H/W asic and not depend on CPU if there is no reprogramming from CPU.

3.18 is not very old. 3.51 will give you more features and is our current latest release.

[aali]

Sorry for the late reply as I was trying to get more details.  The problem is totally beyond my imaginations.  What we have learnt that this problem only happens every 5-6 day and almost around the same time.  According to OA Logs, it shows that gateway is unreacheable.  The wierd thing is that the ESXi hosts, Guest VMs don't go down at all it just stops working.  Looks like that the Cisco Switches are unable to send traffic between VLANs and even in the same VLAN.  As I said that this usually happens every 5-6 day around same time and then everything geos normal after 15-20 minutes.  I was suspecting that may be some time of jobs or Application running at that time is causing this behavior.  Since OA logs doesn't show any issues other than gateway is down or unreachable, HP VC Technical Support is suspecting that Cisco 6513 switches are causing this abnormal behavior.

I am running out of ideas except may be running network analyzer and capture the traffic when this event occurs.  We can also enable traffic shapping or rate-limiting on those VLANs so the traffic will not put too much borden on Cisco 6513 switches.  But again, I beleive Cisco Switche with Sup 720 are good enough to handle large amount of traffic.

Any help will be appreciated.

.

[Hongjun Ma]

I doubt it's large amount of traffic issue. if the traffic is not hitting CPU, all should be forwarded in H/W asic. You need to watch out routing updates, broadcast/multicast store during that time.

at least you know what's the approximate time this problem will come back? I'd do the following on 6513.

1) clear log/counter ahead of time

2) write down some CLIs you feel in related to this issue and use cut/paste from notepad to do it multiple times during outage, usually  I put show clock at the first one, like

!!!!!!!!

show clock

show int summ (or show int x/y)

show proc cpu

show mem summ

show mac-address dynamic <address>

show ip route <ip>

show ip eigrp/ospf neigh

!!!!!!!

make sure your terminal length buffer is set to high so you can  scroll back and capture all output or just log the screen. test the cli ahead of time

also, grab "show logging" afterwards. Turn on logging even for link status  default so you can see link status change on terminal. I recall that I saw OA link flapping. If OA is connected to this 6513, definitely you should see the link status change on 6513 as well.

also, it's worth while to set up sniffer and run trace during that time, you only have 15-20mins every 1 week so you'd better to grab any many info as you can.

try to see if you or your colleague can run another session like ping from switch to host, host to host, vm to vm, in this way you can see which vlan is affected or all vlans are affected.

[aali]

Thanks for your feedback.  I have already advised to run network sniffer during the time of outage so will see what happens.  will keep you posted.

In the mean time, we may enable rate-limit/traffic shapping on a VLAN where all the server are connected to see if it prevents 6513 from dropping the packets.  My feeling is that the certain application or job running at that particular time causing this outage.

Regards,

[aali]

I beleive that I was able to find the problem.  The reason that they are having network connectivity loss due to Catalyst access modules 6148 and 6548.  These modules are not meant to be used for Server Farm connectivity due to 8:1 oversubscription.  I am going to recommend to replace these modules with either 6748 blades or having a dedicated server farm switches with line-rate ports and larger buffer per port.  I believe that HP A5800 Series switches with IRF capablitly will be better than Cisco server farm switches or 6748 modules.

This is a healtchare environment where a little downtime can have a major impact with life theratening.  My only debate is whether to recommend 6748 blades eliminating extra hop or having a dedicated server farm switches for C7000 enclosure with VC-Flex 10 Modules uplinking via (2) 10Gb LACP to each 6513 distribution switches.

Any thoughts! 

[Hongjun Ma]

you might want to take a look at new HPN5900. it has even lower latency and much more buffer than 5800. It has has 40G QSFP ports.

http://h18004.www1.hp.com/products/quickspecs/14252_na/14252_na.HTML

I wrote an IRF+VC white paper if you haven't seen it. you can find the doc off my blog.

6748 is a 1G line card while 5900 is really targeted for DC 10G above market(the ports can do 1G as well from spec)

You can also consider HPN IMC network management software to manage Cisco/HPN and Virtual Connect together. download and try it free for 60 days.

http://h17007.www1.hp.com/us/en/products/network-management/index.aspx

[aali]

Thanks! 

We are already in process of recommending HP IMC, but thanks for your suggestions.

I am totally with you regarding HPN 5900 series switches for server farm requirments.  The problem is that they only need 10Gb ports for their (3) C7000 Enclsousres with each having a dual VC-Flex 10 Modules.  They still have a lots of 1Gb requirments for Server Connectivity and that is why it is a little challange.  I got a check to make sure if IRF is possible between 5900s and 5800s.  The A5830AF-96 can do the job with up to (96) 1Gb and (10) 10gb ports, but with dual 5930s it migh be an overkill and a lots of 1Gb will go unused.

One possible solution would be to go with (2) 5800-48 port switches and  each switch with (4) port 10gb module can support up to (8) 10Gb ports.  So with 2 switches they can have up to (96) 1Gb ports and (16) 10Gb ports.  The only cavaet is that A5800 is designed for small-to-medium core deployment, and may not be ideal for server farm switching.

Let me know what is your thoughs and thanks.

[Hongjun Ma]

I see your point. you should also contact your local HP account team or partner to have them help you with more product info. sometimes there is some promotion going on, last time it's like you get a pair of 5820 for free if you order c7000 and some blades. you never know.