topic Re: Connecting Multiple Enclosures in BladeSystem - General

Connecting Multiple Enclosures

pz7rx8 — Fri, 07 May 2010 17:09:38 GMT

I have two c7000 enclosures next to each other in different racks. In each enclosure I have two BL870 servers, two flex-10 enet modules and two VC FC modules. The servers will be running Oracle 11g in a RAC configuration. I know I can create an internal connection within the enclosure to handle the database heartbeat connections but what about a heartbeat connection to the servers in the other enclosure ? Is there a way to connect the enclosures i.e using Ports X7 or X8 on the flex-10 modules ?

Re: Connecting Multiple Enclosures

HEM_2 — Fri, 07 May 2010 18:55:11 GMT

X7 and X8 by default are your internal stacking links between Bay 1 and Bay 2 of an enclosure. Therefore you might not want to use those for inter-enclosure linking.

Here are a couple options:

1) use multienclosure domain config, use stacking links between enclosures, define an internal network and configure your server profile network connections to use that internal network.

2) each enclosure maintains its own VC Domain, On each VC domain, create a network with an uplink on bay 1 and bay 2, set the connection mode to Failover and set one of the uplinks as primary. Connect the uplinks from each enclosure directly to each other and make sure the primary uplink from enc1 connects to the primary uplink on enc2.

Re: Connecting Multiple Enclosures

pz7rx8 — Fri, 07 May 2010 23:01:08 GMT

Thanks HEM. When you say that X7 and X8 by default are your internal stacking links between Bay 1 and Bay 2 of an enclosure how does that work ? I thought bays 1 and 2 were already connected internally through the midplane and did not need any cable connections.

Re: Connecting Multiple Enclosures

The Brit — Fri, 07 May 2010 23:49:47 GMT

Yes I had the same question. It seems like the X7 & X8 ports DO connect across the backplane, (i.e. they dont need external cables). If you look under stacking links, you will see them listed.

To connect the enclosures you can use CX4 cables to connect the X1 port on Enc1, Bay1 to Enc2 Bay1, and Enc1 Bay2 to Enc2 Bay2.

You can then create a single VC domain if you wish. Be sure you understand the implications of doing this however. (basically, one of you enclosures becomes the primary domain and the second enclosure has to be recreated from scratch).

Alternatively, you need not import the second enclosure, just keep them as separate VC domains. In this case the CX4 cables can be used as a high speed internal uplink, i.e. traffic does not go to the external network, it is direct, enclosure to enclosure.

Disclaimer: I haven't tried these solutions yet.

Dave.

Re: Connecting Multiple Enclosures

pz7rx8 — Mon, 10 May 2010 22:02:54 GMT

The issue I'm having, since we are using 1 Gb uplinks, is that I started with what I was told 8 ports available. Now X7 and X8 are not available leaving 6 ports. Now if I use a CX4 cable to connect the enclosures port X1 is no longer available leaving only 5 uplink ports available. I need 7 uplinks in a LACP configuration. Will the fact that the modules are stacked allow me to use uplinks from multiple eNet modules. For example can I run 5 cables from the VC enet module in enclosure 1 bay 1 and 2 from the enclosure 2 bay 1 to the same switch as an LACP configuration ?

Re: Connecting Multiple Enclosures

The Brit — Mon, 10 May 2010 23:49:20 GMT

Why is the number of uplinks so important?

Create an uplinkset with the 5 available ports, and assign your VC VLans to use the uplink set.

Dave.

Re: Connecting Multiple Enclosures

Adrian Clint — Wed, 12 May 2010 13:34:05 GMT

Well if we go back to the original request it was for just a heartbeat connection to an adjacent enclosure.
So we dont need a nice stacked 10Gb ring. Or redundant connections between adjacent Flex-10s.

The simplest way would be to just make sure that all your heartbeat connections are on NICs that connect to one Flex-10 module in each enclosure.
Then make a single link between these two Flex-10 modules with a 10Gb copper cable or two 1Gb SFPs. You will only then give up 1 uplink on each enclosure for bandwidth.