topic Re: is Virtual Connect stacking required? in BladeSystem - General

is Virtual Connect stacking required?

Bob_Vance — Wed, 09 Nov 2011 22:13:07 GMT

is Virtual Connect stacking required?

<SHORT>

Is there any requirement other than possible
data flow that forces the need for stacking?
(e.g., required for firmware or config updates).

For example, say we have
VC1,2 for strictly production/client network,
with uplinks going to production backbone,
and
VC3,4 are using their own uplinks for isolated iSCSI traffic.

In this scenario, there will be no data packet flow
from VC3,4 out thru VC1,2.

So why would a vertical stacking be necessary?

Further,
Is horizontal stacking used for anything other than
possible standby to active data flow?

In an active/active setup,
with separate uplink sets for "left" and "right" traffic flow,
it seems that horizontal stacking would not be necessary.

In a VC-Flex10, this would free up X7 and X8,
allowing the use of all 8 uplinks for data packets.
((VC 1/10G enet are always
horizontally stacked via port X0
))

</SHORT>

<LONGER>

Current setup

VC1 1/10G Enet ......... VC2 1/10G Enet
used for production network into
Vmware ESX hosts.

VC3 1/10G Enet ......... VC4 1/10G Enet
used for iSCSI for Vmware ESX hosts

VC1 - VC3 - vertically stacked via CX4
VC2 - VC4 - vertically stacked via CX4

VC1 & VC2 use their other CX4 uplink
to connect to the Cisco backbone.

******
We are replacing the VC1&VC2 with new
VC-Flex10 modules.
******

The Flex10 VCs only have ONE CX4 connection,
which would be needed to stack vertically,
if stacking is required.
I was hoping NOT to stack vertically, thereby
freeing up the CX4 for additional 10G uplink.
((The backbone switches have an SFP slot open, to which
we will also be uplinking via "SFP cable"s.
))

</LONGER>

tks
bv

Re: is Virtual Connect stacking required?

Psychonaut — Thu, 10 Nov 2011 16:17:32 GMT

I see no reason for vertical stacking in your configuration.

I would leave 7 & 8 connected - or at least 8. The horizontal link will keep traffic internal to the chassis for any server to server traffic - rather than going up to the switch and back. By leaving both you gain a 20Gb pipe for side to side traffic.

Re: is Virtual Connect stacking required?

Bob_Vance — Thu, 10 Nov 2011 18:41:07 GMT

Thanks, Psyco
(that is your nickname, right ?>)

*********
1)
*********
>I see no reason for vertical stacking in your configuration.

I don't either, but my question was kinda,
"Is that absolutely true?"

I.e., is the vertical stack absolutely only for
vertical module-to-module vNet data traffic,
where uplinks from VC1,2 are being shared by
VC3,4 to get to the external network.
((E.g., a blade might be dual-homed, with, say,
        NICs 1,2 teamed in VLANx
    & NICs 3,4 teamed in VLANy.
))

I hear that you are saying "yes",
but do you know for certain ;>)

*********
2)
*********
>The horizontal link will keep traffic internal ...

Ahhh, yes!!
I was imagining traffic always within one VC
from BladeX/NIC1
to      BladeY/NIC1
.

!!!!
And
!!!!
I just realized that I have misconfigured
a vMotion network.

I configured two separate vNets
     vMotion-left
& vMotion-right
basically just to be consistent with all
the other vNets defined in left/right pairs
in an Active/Active design.

I was forgetting, that traffic might need to be
from BladeX/NIC1
to      BladeY/NIC2
.
E.g., in vMotion from BladeX to BladeY,
if
BladeX wants to send Vmotion out NIC1
but
BladeY/NIC1 is down
(unlikely but I suppose possible)
then the transfer cannot happen.

I was wondering why HP showed just ONE vMotion
vNet and put both NICs in it.

So, horizontal link will stay.
((And a redesign of the vMotion vNet!
))

tks

Re: is Virtual Connect stacking required?

Psychonaut — Thu, 10 Nov 2011 22:09:44 GMT

Just don't call me Francis :)

For that vertical link, with your config I don't see a need to have it. If you were sharing uplinks between those interconnects or if you want the two networks to talk to each other then yes. But since those are two separate networks with their own uplinks it isn't needed.

Actually let me amend that the only thing that comes to mind would be if that iSCSI network is entirely isolated that link could be used to possibly tunnel through for management or monitoring.

Re: is Virtual Connect stacking required?

The Brit — Fri, 11 Nov 2011 11:59:17 GMT

One thing which you may be overlooking.

VC Manager only runs on MOdule 1 or Module 2 (not 3 or 4).

Without stacking links, VCM knows nothing about 3 or 4, or anything connected to, or passing through 3 or 4.

You will not be able to configure the ports in modules 3 and 4, through VMC, and I also suspect that you will have problems with your profiles if VCM cannot talk to Modules 3 & 4.

The only way for VCM to talk to modules other than 1 and 2, is through stacking links.

If the stacking links are absent...

????

Dave.

Re: is Virtual Connect stacking required?

Psychonaut — Fri, 11 Nov 2011 17:07:13 GMT

"With VCM 3.10 and higher, the primary modules can be in bays other than 1 and 2."

With that in mind, unless those 4 interconnects are in the same domain you should be able to break that connection as long are you are current on your firmware.

Re: is Virtual Connect stacking required?

The Brit — Fri, 11 Nov 2011 17:47:52 GMT

You may be right about VCM being able to run on other modules than 1, or 2, however I still contend that without vertical stacking links, VCM will not be able to communicate or configure ports on modules which are not vertically connected (stacked).

Could be wrong (which is why we invested in a Test environment).

I would be interested in the results of working with unstacked modules.

Dave.

Re: is Virtual Connect stacking required?

The Brit — Fri, 11 Nov 2011 17:51:28 GMT

Sorry, just read your second sentence. This implies that you need to run two separate VC domains within the same enclosure, i.e. to instances of VCM. (I dont see any immediate objection to this, although I am a little concerned about server profiles spanning to different domains).

Dave.

Re: is Virtual Connect stacking required?

Bob_Vance — Fri, 11 Nov 2011 18:37:29 GMT

Wow,
I have to start typing faster.

I was in the middle of a long-winded reply addressing what you guys just discussed in the last hour!

Actually, I'm MAJORLY concerned about the server profile point in the two-domain scenario.

But, hey Dave, *you're* the one with the test chassis !>).

VC 3.15 Setup and Install Guide says
"All Virtual Connect Ethernet modules within the
Virtual Connect domain must be interconnected.
"
But next sentence says:
"Any combination of 1-Gb and 10-Gb cables can be
used to interconnect the modules.
"
So, how much Meta traffic could there really be?

In the particular config I discussed, where I wanted to free up the CX4 on Flex-10 VC1,2, I guess I could just use a 1Gb link to VC3,4 ....
assuming that stacking *is* required.

Let us know what you find out with your test system, Dave ;>)

tks

Re: is Virtual Connect stacking required?

The Brit — Fri, 11 Nov 2011 18:55:48 GMT

To be honest Bob, unless you can wait a couple of months, my workload doesn't give me time to do any of the testing you are hoping for.

I have to say though, the more I think about it the more objections rise. For example, how would two VC domains interact with your OA. Specifically, when you first insert a VC module into the enclosure, the OA detects it and creates a link on the left-hand navigation bar. If you create two VC domains, then the OA would have to provide two links, or else how would it know which VC domain you want to connect to.

Having said that, it is still the profile issue that stops me. I suppose it is feasible to split your servers so that they only use either the onboard nics via IC Bays 1 and 2, or they only use Mezzanine card in Mezz 1 to communicate with Bays 3 & 4. You would have to disable the onboard nics on these servers to stop them trying to link up to bays 1 & 2.

the servers can only have one profile, and the profile could only exist in one VC domain. Each domain would have to have its own set of WWD's and MAC's. (I dont think you mentioned whether your storage is internal or external, FC or SCSI), but you would have the same issues with FC VC modules (if you are using them.)

Bottom line is: I dont really know if this is doable, however it is sufficiently scary to make me stay away. I guess there would have to be an over-powering reason to put yourself out there. You need to do a "cost-benefit analysis", just to be sure that what you gain on the "swings" you don't lose on the "roundabout".

Dave.

Re: is Virtual Connect stacking required?

Bob_Vance — Fri, 11 Nov 2011 18:56:59 GMT

BTW,
I was able to imagine a design wherein the VCM on VC1 *could*, *theoretically*, know about the other VCs via the management network and the OA, which could communicate the VC arrangement to all the VCs.

So, that's how I came up with "no stacking required."
But, absent a "Theory of Operation" document ...

Pyscho's new scenario seemed to imply setting up the domain with stack-links and then breaking the stacking.

I'm wondering about just having two, mutually unaware domains from the get go.

Primary VCM in VC1 and another Primary VCM in VC3.

The OA mgt page probably would only have a link for the VC1 VCM, but we could connect directly to VC3 which should be running VCM, since it doesn't know about VC1,2

But still worried about the profiles.
I'm not even sure how the profiles get applied.
Thru the ILO?

bv

Re: is Virtual Connect stacking required?

Bart_Heungens — Fri, 11 Nov 2011 21:38:03 GMT

Mmm not really sure about what is mentioned above...

If there are no stacking links vertically the VCM running on modules 1 and 2 cannot communicate with the others and cannot configure them? Sorry but this is not true...

Simple example: how do U explain that in slot 3 and 4 are FC VC modules? There is no stacking possible between FC and Ethernet and yes U can configure them.

I have 27 enclosures at my customer site with 6 VC modules: 4 ethernet in 1, 2, 5 and 6 and FC modules in 3 and 4.

Without the stacking cables between 1 and 5 and another one between 2 and 6, I can configure all 6 modules... For VCM this is 1 VC domain...

Only: without the stacking cables U will get an error stating that this is not an ideal situation... Because with VC I can perfectly say that the onboard NIC 1 (physically connected to module 1) can get out with its data thru the uplinks on module 2, 5 or 6 (if I want), via the stacking cables...

Plenty of examples are mentioned in the VC Ethernet cookbook... Please check that one...

Kr,

Bart

Re: is Virtual Connect stacking required?

Bob_Vance — Fri, 11 Nov 2011 23:05:32 GMT

>> VC-FC
Good catch, Bart!!
Of course.

So, the Domain mgt Meta traffic must be occurring on the OA mgt network.
The "two-domain" scenario is unnecessary and out.
Although, I'm not sure about MAC tables.
Again, there's just not enough Theory of Op info.

>> Without the stacking cables between 1 and 5
>> and another one between 2 and 6,
The last phrase is a little ambiguous.
So, just to be clear, you did mean by that:

"No stacking cables between 1,5
and
no stacking cables between 2,6.
"
I.e, no vertical stacking at all between the VCs within a single enclosure.
(I assume, "Yes", since that's what the thread is about ;>)

So, what caused you not to use vertical stacking in your environment?
Or did you only try it out for curiosity.

>> ... VC Ethernet cookbook
That is a fine document, but it does address the issue of vertical stacking within an enclosure at all.
All the examples have just two VCs in each enclosure.

tks
bv

Re: is Virtual Connect stacking required?

Bart_Heungens — Sun, 13 Nov 2011 20:49:31 GMT

Hi,

All modules can communicate with each other through the signal midplane... However there is always communication via the OA...

2 VC domains in 1 enclosure is indeed not possible...

In my case Yes I have stacking cables between modules 1 and 5 and another one between 2 and 6... What I meant to explain is that, even without those cables, I can manage all modules... Via the OA and signal pidplane...

I know 1 scenario where the customers has all client and management trafic on modules 1 and 2, and only iSCSI trafic on modules 3 and 4 via the additional mezzanine card... Since it is a complete seperate type of trafic, stacking is not needed...

Kr,

Bart

Re: is Virtual Connect stacking required?

Bob_Vance — Mon, 14 Nov 2011 13:19:23 GMT

Thanks, Bart.

The separate iSCSI which you mentioned is the exact scenario that I will be dealing with, so I'm going to try it.

We will be doing a Flex-10 upgrade for VC 1/10G Enet modules in bays1,2 and will have a downtime window to be able to play with it --
basically just disconnecting the current vertical CX4 stacking and see what happens.

>> All modules can communicate with each other through the signal midplane..

My understanding is this:

VCs only communicate *horizontally* via the midplane and lacking vertical stack cables, cannot communicate vertically.

The VC 1/10G Enet has hidden X0 *always* connected horizontally across midplane.
VC-Flex10s use X7 & X8 horizontally, but if an SFP is installed in either or both ports, then the port(s) are disabled horizontally.

Do you disagree?

bv

Re: is Virtual Connect stacking required?

Bart_Heungens — Mon, 14 Nov 2011 13:32:54 GMT

Hi,

Correct. In the older modules the X0 was hardwired through the midplane and could not be changed.

In the Flex modules the customer can choose for himself where he wants to make the interconnection. Inside or outside the enclosure.

Best practice is however using the internal connections so that the external switches are not (ab)used for trafic that can perfectly be handled inside...

There is indeed a difference in data paths and mgmt data paths ... VC itself for its config goes thru the OA, 'real' data goes directly via the signmal midplane to the servers...

Kr,

Bart

Re: is Virtual Connect stacking required?

Bob_Vance — Mon, 14 Nov 2011 13:47:17 GMT

>> There is indeed a difference in data paths and mgmt data paths ... VC itself for its config goes thru the OA

This is the crucial idea and was the nub of my original query.

I'm going to leave this open until I have done my upgrade and will post results thereafter.

Re: is Virtual Connect stacking required?

Psychonaut — Mon, 14 Nov 2011 19:23:28 GMT

Bart, that is some great info - I was looking to white board this out a little today and take back some of what I had suggested earlier. But it looks like you guys have hashed it all out.

Re: is Virtual Connect stacking required?

Bart_Heungens — Mon, 14 Nov 2011 20:05:01 GMT

That's the reason why this forum exist... To share this kind of information.

Do not forget to assign kudo's, it is also appreciated... ;-)

Re: is Virtual Connect stacking required?

Brunot38 — Mon, 07 Jan 2013 17:21:17 GMT

Hello,

how did the tests go without vertical stacking ?

I have a similar setup ( that I am explaining in a new post " HP Virtual Connect - Design with no vertical Stacking" )

I would really like to know if a setup withoug vertical stacking is working. It would be much appreciated as I am stuck on this..

Thanks

Brunot38