BladeSystem - General
1748073 Members
5024 Online
108758 Solutions
New Discussion юеВ

Connecting Multiple Enclosures

 
pz7rx8
Advisor

Connecting Multiple Enclosures

I have two c7000 enclosures next to each other in different racks. In each enclosure I have two BL870 servers, two flex-10 enet modules and two VC FC modules. The servers will be running Oracle 11g in a RAC configuration. I know I can create an internal connection within the enclosure to handle the database heartbeat connections but what about a heartbeat connection to the servers in the other enclosure ? Is there a way to connect the enclosures i.e using Ports X7 or X8 on the flex-10 modules ?
6 REPLIES 6
HEM_2
Honored Contributor

Re: Connecting Multiple Enclosures

X7 and X8 by default are your internal stacking links between Bay 1 and Bay 2 of an enclosure. Therefore you might not want to use those for inter-enclosure linking.

Here are a couple options:

1) use multienclosure domain config, use stacking links between enclosures, define an internal network and configure your server profile network connections to use that internal network.

2) each enclosure maintains its own VC Domain, On each VC domain, create a network with an uplink on bay 1 and bay 2, set the connection mode to Failover and set one of the uplinks as primary. Connect the uplinks from each enclosure directly to each other and make sure the primary uplink from enc1 connects to the primary uplink on enc2.
pz7rx8
Advisor

Re: Connecting Multiple Enclosures

Thanks HEM. When you say that X7 and X8 by default are your internal stacking links between Bay 1 and Bay 2 of an enclosure how does that work ? I thought bays 1 and 2 were already connected internally through the midplane and did not need any cable connections.
The Brit
Honored Contributor

Re: Connecting Multiple Enclosures

Yes I had the same question. It seems like the X7 & X8 ports DO connect across the backplane, (i.e. they dont need external cables). If you look under stacking links, you will see them listed.

To connect the enclosures you can use CX4 cables to connect the X1 port on Enc1, Bay1 to Enc2 Bay1, and Enc1 Bay2 to Enc2 Bay2.

You can then create a single VC domain if you wish. Be sure you understand the implications of doing this however. (basically, one of you enclosures becomes the primary domain and the second enclosure has to be recreated from scratch).

Alternatively, you need not import the second enclosure, just keep them as separate VC domains. In this case the CX4 cables can be used as a high speed internal uplink, i.e. traffic does not go to the external network, it is direct, enclosure to enclosure.

Disclaimer: I haven't tried these solutions yet.

Dave.
pz7rx8
Advisor

Re: Connecting Multiple Enclosures

The issue I'm having, since we are using 1 Gb uplinks, is that I started with what I was told 8 ports available. Now X7 and X8 are not available leaving 6 ports. Now if I use a CX4 cable to connect the enclosures port X1 is no longer available leaving only 5 uplink ports available. I need 7 uplinks in a LACP configuration. Will the fact that the modules are stacked allow me to use uplinks from multiple eNet modules. For example can I run 5 cables from the VC enet module in enclosure 1 bay 1 and 2 from the enclosure 2 bay 1 to the same switch as an LACP configuration ?
The Brit
Honored Contributor

Re: Connecting Multiple Enclosures

Why is the number of uplinks so important?

Create an uplinkset with the 5 available ports, and assign your VC VLans to use the uplink set.

Dave.
Adrian Clint
Honored Contributor

Re: Connecting Multiple Enclosures

Well if we go back to the original request it was for just a heartbeat connection to an adjacent enclosure.
So we dont need a nice stacked 10Gb ring. Or redundant connections between adjacent Flex-10s.

The simplest way would be to just make sure that all your heartbeat connections are on NICs that connect to one Flex-10 module in each enclosure.
Then make a single link between these two Flex-10 modules with a 10Gb copper cable or two 1Gb SFPs. You will only then give up 1 uplink on each enclosure for bandwidth.