HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

c7000 and GbE2c setup

 
Robert Marek
Advisor

c7000 and GbE2c setup

Hello, I’m looking for help with setting up c7000 and switches. My knowledge on this is limited.

This is what I have:
C7000 enclosure
4 blades with two mezzanine cards – one with dual NIC and on with quad NIC – total 8 NIC’s
8 GbE2c L2/L3 switches
Left Hand P4500 (3 nodes)

I have created VLAN on Switch 1 for 4 blades and 3nodes. All switches are interconnected internally with cross link ports, and uplink ports to connect switches vertically. This way all switches are connected together. The problem is if I lose Switch1 then I lose everything. It makes sense since VLAN is set on Switch1.
I would like to have a setup where no matter what switch fails my connections are up. I wonder if this is possible without creating VLAN on my core switch 5406zl. Right now my core switch is only passing traffic between my main network and VLAN on GbE2c switches.

Thanks in advance for any suggestions.
18 REPLIES
Mike Schreiner
Occasional Contributor

Re: c7000 and GbE2c setup

have you gotten anywhere on this yet?
Robert Marek
Advisor

Re: c7000 and GbE2c setup

Only what I posted. I'm wondering what would be the best setup for what I have from networking perspective. SAN is accessible and my ESXi host s are fine but this need to be bullet proof setup.
Matt Palmer_2
Respected Contributor

Re: c7000 and GbE2c setup

Hi,

I used spanning tree groups and LACP along with MSTP to make my setup fully redundant. do you have spanning tree in place on these switches or any higher up in your core network??

regards

Matt
Robert Marek
Advisor

Re: c7000 and GbE2c setup

I do have spanning tree but I have no VLAN up on higher switch, in this case 5406zl. If I try contain VLAN only to switches in chasis it will never be fully redundant because if I loose first switch where VLAN is created I loose everything.

There only way to do this (in my opinion) is to setup VLAN higher up - on Procurve 5406zl and then make redundant setup on blade switches and if uplinked correctly I could loose 2 switches at the same and still have everything accessible... if I'm correct....Again, I have no prior experience with this so I may be totaly wrong.
Matt Palmer_2
Respected Contributor

Re: c7000 and GbE2c setup

Hi,

how I approached this was to break the interlinks on ports 17-18, and treat each switch as a separate entity.

then I created an aggregate on port 21-22 for 1 network 23-24 for other network, then used priority higher up in the network to manage failover if 1 or more cable was disconnected.

each switch has the same VLAN configuration on it, so within the OS (linux) I use active-backup bonding to cover myself if I loose an entire switch. I also used 802.1q in the os to tag the packets for each VLAN.

I can post up my switch config if you like, but you will have to tailor it a bit to your requirements, as I know nothing of your core network,etc.

regards

Matt
Robert Marek
Advisor

Re: c7000 and GbE2c setup

Matt, I tried to keep my core network out of this setup and only pass trafic between blade switches (one VLAN) and core network.
If I understand you correctly this is not possible.

Ports 17-18 are disabled by default. I had to enable them to have access from switch A tp switch B.
Matt Palmer_2
Respected Contributor

Re: c7000 and GbE2c setup

Hi,

ok, not sure how to proceed with that one then. One other thing not sure if youve looked at is the bladeconnect forum. Its a pretty good resource for config and setup issues. you could try there - http://www.bladenetwork.net/

regards

Matt
Antonio Milanese
Trusted Contributor

Re: c7000 and GbE2c setup

Hello Robert,

as Matt said a better solution is to configure MSTP and create a mesh topology within the encloser and use multiple(say 2)uplinks to your core switch.

>Matt, I tried to keep my core network out of
why ? insufficent modules/ports?
u need 6 port:
2x LACP (2x1Gbe x module) connected to two vertically/horizontally differents GbE2c (bay1/bay8 f.e.)

1x (1x1Gbe x module) connected to P4500 storage gateway with ALB loadbalancing

GbE2c are nothing more than regular L2/L3 switchs so MSTP is a must have with multiple redundant links

>blade switches (one VLAN) and core network
it's advisable to have a different dedicated VLAN for storage for at least 3 different reasons:

1) broadcast isolation
2) security
3) loadbalancing using MSTP instances

my suggestion is to create
2 MSTP instances, a storage VLAN mapped on all pNICs used as uplinks for ESX iSCSI portgrop/vswitch:

this way you can create 2 different paths within the encloser (i.e. balanced on left,righ bays) and on uplink pipes reserving effectively more bandwidth for your storage traffic.

Regards,

Antonio







Robert Marek
Advisor

Re: c7000 and GbE2c setup

Thanks Antonio,

I have one VLAN (on switch 1) for storage and ESX hosts to seperate traffic from main network. Switch 2 is connected on ports 17-18. Both of them are uplinked with core switch on default VLAN 1. If I power down Switch 1 VLAN is down and nothing is accessible from my network. This make sense sincethis Switch is setup with this VLAN IP.

I was hoping to be able to setup everything within the chasis without pushing any traffic through core switch.

I guess I'll have to setup VLAN's on my core switch...

Right now I have NIC#1 from each Lefthand node connected to switch 1 and NIC#2 connected to switch 2.

Robert Marek
Advisor

Re: c7000 and GbE2c setup

Matt, could you post your configuration?

Thanks,
Robert
Matt Palmer_2
Respected Contributor

Re: c7000 and GbE2c setup

Hi Robert,

please find config attached. please note that the exact config is also on my other switches, so you may need to move stuff about a bit.

please awards points if it helps.

regards

Matt
Robert Marek
Advisor

Re: c7000 and GbE2c setup

Matt and Antonio,

I'm attaching simple drawing of my switches and how I think they could be connected. Please let me know what you think.

If I uplink to my core switch and have VLAN's IP gateways on it will traffic goes through core switch or will be going between blade switches since they using ports 17 and 18.
The Brit
Honored Contributor

Re: c7000 and GbE2c setup

Hi Robert,
I am not a network person, therefore most of the discussion above is beyond me, however I did note that your primary issue is that if you lose the module in IC bay1, then you lose everything.

I wondered if you have considered using Virtual Connect Manager. The Software runs Active/Standby on IC Modules in Bays 1 & 2, and will fail over if the active module fails. Also, all VC modules can be "stacked" to form a single VC domain, which allows you to create multiple redundant uplinks to the Network.

As you will have noticed, the NIC ports on your blades (embedded and Mezz ports) map alternately to odd and even IC Bays which is intended to allow the use of NIC "teaming" at the OS level to maintain connectivity.

Communication between blades can be kept inside the enclosure by defining a VC "vLan" which has no uplink ports assigned.

I appreciate you may have already considered this, or that it may not solve your problem (since I dont fully understand that part).

Dave.
Robert Marek
Advisor

Re: c7000 and GbE2c setup

Thanks Dave, I wasn't aware of this...can you explain where it is and how to use it?
Robert Marek
Advisor

Re: c7000 and GbE2c setup

Dave, this will not work. I have GbE2c switches only. What you proposing would require Virtual Connect Ethernet modules.
Antonio Milanese
Trusted Contributor

Re: c7000 and GbE2c setup

Hello Robert,

looking at the your drawing now i have a better idea of what your problem:

a) you have created a full mesh topology between switches (cross connection+uplinks)
b) STP is running but with default on config: no MSTP and legacy IST (common spanning tree instance/domain)
b) cross links are active (p17,p18): that means you have a 2 ports LACP channel per bay horizontally
c) vertical uplinks are connected using a single port physical link

the problem lays in the fact that LACP groups have a better STP path cost (20000 vs 200000) so when STP builds the topology most of your "desired" vertical uplinks are in blocking state.

I suggest you:
a) use MSTP to build a partial mesh: how to link bays together depends on how you'll use downlink port (i.e. what kind of traffic they transport):
b) If you continue to use single port uplink you should lower STP path cost of those uplinks and lower xcross STP priority to build your desired topology
c) each uplink must be a trunk port i.e. carries each VLAN


an example:

suppose you use LOM:1a + MEZZ2:2b as active pNICs for the 2 ESX's vkernel iSCSI ports and
each P4500 cluster is connected with 1 bay left|right with ALB teaming (switch independent teaming);
the active P4500 cluster gateway nics are connected on bay1:bay2 VLAN 10 untagged ports
and LOM:1b, and other MEZZ1,MEZZ2 ports are carrying normal,non-storage traffic f.e. LAN,VMotion,ecc
and you have designed this partial mesh :

bay1->xcross->bay2
....
bayn->xcross->bayn+1

bay1->uplink->bay3
bay2->uplink->bay4
...
bayn->uplink->bayn+2

bay1->uplink->bay8
bay7->uplink->bay2


you can create 2 MSTP instances (STG in gb2ec gergon) mapping VLAN 10 on instance 1 and other VLAN on instance 0 (VLAN1 is locked into IST) and use path cost and priorities to build 2 independent specular paths: one for storage traffic and one for other traffic!

obviously only 1 uplink is active to your core switch the other is blocked/standby

I think this is what you want to achieve isn't it?

Regards,

Antonio
Robert Marek
Advisor

Re: c7000 and GbE2c setup

Thanks Antonio for your help so far. Any chance we could talk offline? My email address is robson13@gmail.com

I'm attaching another drawing. This is how switches are connected right now.
I'm doing very simple setup. There is only one VLAN on on switch1. Every other switch have IP interface for this VLAN and all ports that connect switches toghether including ports 17 and 18 belongs to this VLAN.

All switches are setup the same way but Switch3 is causing problems...
If this switch is powered up I can't ping switches2, 6 and 8. If I power off switch3 I'm pinging switches 2, 6 and 8 again. I don't understand this...
Adrian Clint
Honored Contributor

Re: c7000 and GbE2c setup