Internet VC 1/10 and VC Flex-10?

[JonathanT]

Can I interconnect two VC 1/10GB with two VC Flex-10? I want the Flex-10 to have all the 10GB networking within the chassis, but would like the other VC module to do the uplink to the core switches with LACP trunks to reduce cost. Is it OK to upload the two types of units with CX-4 cables?

[Lmm_1]

That should work, what you can do is use the Multienclosure Stacking feature, that should be enoughg for your requirements, check this out:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=3794423&prodTypeId=3709945&objectID=c01724621

[JonathanT]

Sure looks like it will work. Plus the CX4 .5m cables are only like $100. I can also LACP 6GB/sec off the VC 1/10GB modules to the the core switches which is way more bandwidth and redundancy outside the chassis than we would need. Does anyone have any comments on this configuration?

VC Flex 10 <-> VC Flex 10
|CX4| |CX4|
VC 1/10GB <-> VC 1/10GB
|6x1GB| |6x1GB|
[Core 1 <-> Core 2] Stacked
|1GB| |1GB|
[COLOCATION NETWORK SWITCHES]


So to summarize I would 8 FlexNics per Blade and 2 1GB Nics per blade. If I team the Nics properly at the OS level I would have 5 teamed Nics running in TLB. In theory that would be 22GB TX and 11GB RX out of each blade. The C7000 chassies would upload to the core switch with 6GB to each side. Core 1 and Core 2 are PowerConnect 6248 switches with a stacking cable so they appear as a single switch. The load balancers, firewalls, etc. would all be connected to the core switches with dual GB links in LACP (one link to each side) for redundancy. What does everyone think of this kind of layout?

[JonathanT]

So much for ASCII art.

[JonathanT]

So far I'm off to a good start with my network configuration. I've got the Dell PowerConnect 6248 switches in a stacking configuration. I've configured 4 x 1GB uplinks from each VC 1/10 module to the power connect. I just setup each of the ports on the powerconnects to LAG w/ LACP and turned on the fastlink option (important). The VC module sees 4 links active and 4 standby. I even split the links with two links going to each switch. So if I completely power down one of the powerconnect switches I stay at 2GB active and 4GB standby. I'm debating if that is a good idea or not. Maybe all 4 should go to the same switch so if one fails it completely rolls to the other VC module's uplinks for maximum sustained bandwidth. Thoughts?

I'm waiting on the Flex-10 modules to come in and then I've got to get some CX4 cables to connect them with. CX4 cables are a standard so I should be able to use some quality aftermarket ones without any problem right?

Everyone please feel free to comment on my configuration. I'd love to hear feedback from experts who know far more about this stuff than I do and at the same time have something that is archived to help others on this forum in the future.

[JonathanT]

OK, so I got the Flex-10 modules in today. I moved my VC 1/10 from bays 1 + 2 to bays 3+4 and installed the Flex-10 modules. The VC manager got all kind of ticked off. It say bays 3+4 as flex-10 modules with incompatible hardware. So I called HP support for the first time. The lady that answered was totally knowledgeable. We wiped the VC configuration and let the wizard start over. After that it is seeing the proper hardware in each bay.

She also told me about the new Virtual Connect 2.12 firmware so I'm upgrading to that from 2.10. This new firmware allows the Flex-10 module to work properly even when the bottom of the enclosure is not fully populated. Apparently in v2.10 you had to first populate the bottom of the enclosure before populating the top.

On a side note I ordered my .5 meter CX4 cables to interconnect Bay 1/3 and 2/4 from pureformancecables.com. I'd say they were about 1/2 the cost of what CDW would have sold them for. Work like a charm as one would imagine. It's kind of like the monoprice of high end networking cables from I can tell.

I'll post another update on my networking quest once I get back down to the OS level and start passing traffic over the FlexNics through the VC 1/10 up to the core switches. In the mean time if anyone has any questions or comments I'd love to hear them.

[JonathanT]

So I defined two ethernet networks on the VC for iSCSI (9gb) and application (1gb). So that means each server has 4 FlexNICs on it. I went to team the two application nics and noticed the NCU kept showing one of the nics failsing on RX Path. I called HP support and they wanted be to use Firmware CD 8.60 to update some firmwares which I did, but that did not fix the problem. I then did some Googling and came across this advisory from Dec 2008 which says you have to turn off RX Path Validation in a Teamed FlexNIC scenario:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=1842750&prodTypeId=3709945&prodSeriesId=1842750&objectID=c01554957

I did what it said and the team is no longer going up and down.

I've gone ahead and turned on jumbo frames on the FlexNICs that I've defined for my iSCSI network. I'd like to find a good utility to test the bandwidth between the two servers. I'd like to see what the Windows 2008 TCP/IP stack is cable of in this configuration. I'm planning to attach some MDS600 storage off two of the blades and run Datacore SanMelody off them for the iSCSI SAN. I'm wondering what kind of performance I'm going to be able to drive in this configuration and how much of the 10GB I should carve up for the iSCSI FlexNICs. I purposely didn't team the FlexNICs for the iSCSI because I should be able to use MPIO so that both NICs are run wide open.

As always, questions and comments are welcomed.

[rick jones]

For network performance testing I generally suggest http://www.netperf.org/ but I must say I have a built-in bias in that regard :)

[JonathanT]

Couldn't really figure out how to stress the connection with netperf. What would be the best set of command line arguments to stress a 9Gb/sec FlexNIC with jumbo frames?

I did however copy a 2.6GB ISO from one server to another. It put the connection at about 35-40% which is probably about all the SSD drives in the server can handle.

[rick jones · Accepted solution]

I would use:

netperf -H -i 30,3 -c -C -t TCP_STREAM -- -m 64K -s 1M -S 1M

which will:

run a unidirectional transfer test using TCP to the system (where netserver should be running)

it will include local (-c) and remote (-C) CPU utilization, and calculate an efficiency (service demand) based on the assumption that all the CPU utilization comes from the test.

it will repeat that between three and thirty times (-i 30,3) until it is 99% certain/confident that the averages it has for local/remote CPU util and throughput are within +/- 2.5% of the "true" averages (if one were to have infinite samples).

it will call send() with 64KB of data at a time (-m 64K) and request the local socket buffer (-s) and remote socket buffer (-S) be 1MB in size.

[JonathanT]

Just to provide an update. I've got the PowerConnect 6448 stack working with the VC 1/10 nicely. I had some issues with a firmware reload on one of the switches, but I've been able to work through that. It does a fairly graceful failover if one of the core 6448 goes down.

I'm still having some issues with the TLB vs NFT on some of the FlexNIC teams with a Cisco RVS4000 device, but I'm not sure I'm going to be able to get that solved. Probably going to have to stick with NFT for now.

[Revantha Udugampola]

Hi Jonathan,

I've seen from your posts that you are an expert in the The Virtual connect arena for the HP blade solutions.

I am putting together a Blade config based on the c7000 chassis, BL460C blades and a LeftHand iSCSI SAN solution. The clinets is primarily into the media business (he is a major news paper publisher) and the key agenada is for Server Consolidation...most of the Servers are on Windows 2003 and I am planning for around 12 BL460'c.

Also we are planning to to migrate the Blade and the storage network into 10GigE with Brocade.. I would like to hear check from you wether you would be in a position to share your expereience with us and also to take a look on the c7000 config as well which we are hopeing to propose...


Many Thanks,

Revantha Udugampola
E-W Information Systems
Sri Lanka

skype - revanthau

[Adrian Clint]

Revantha

How are linking Lefthand with 10G and Brocade?

Lefthand is 1Gb RJ45 Networking, apparently if you put the Copper 10Gb connections in there it becomes so expensive you might as well buy a 4Gb Fibre Channel SAN.
Which is Brocade technology. So much in your statement is non compatible/no cost effective.
Start a new thread

[JonathanT]

Hi Revantha,

I don't mind giving you my two cents of advise, but I'm definitely not a guru when it comes to Flex-10. I'm new to it and learning like a sponge. I'm trying to document what I'm learning here so that others may gain knowledge.

I will tell you that I'm planning to use Datacore's Sanmelody on two of the BL490C G6 blades for my iSCSI SAN. With the 3.0 version you can do some obscenely large mega caches. As for storage to drive the 10Gb ethernet I'm planning to have the blades access a hybrid setup:

* SSDs in a SB40C storage blade.
* SAS drives in a MSA or MDS SAS storage enclosure.
* Later on option (Fusion-IO/HP IO Accelerator Mez Card)

The SanMelody turns the normal storage that will be attached to these blades into a high performance, high availability, iSCSI SAN. Maybe you should consider a similar thing instead of the Lefthand equipment. I agree with Adrian that you probably can't cost effectively drive the 10GB ethernet with an external iSCSI SAN.

[Revantha Udugampola]

Hi Adrian,

I am actually looking at the New Brocade CX- 624S

http://www.brocade.com/products-solutions/products/ethernet-switches-routers/enterprise-mobility/product-details/fastiron-cx-series/specifications.page


Your point is well taken.. I will shortly come back to you...

revantha

[Uy U.]

We are also considering the same exact solution:

* DataCore SANMelody 3.x
* MDS600
* FusionIO
* C7000 with Blade TBD

Did you get this solution to work? We are also looking at creating 2 SANMelody SANs for redundancy. Any pitfalls? Any gotchas?

[JonathanT]

At the time that config could not do redundant mirror networks between the SDS servers over iSCSI. With the nature of the network modules in a C7000 this left a single point of failure that would definitely be tripped if the module reset for anything like a firmware upgrade, etc. I don't know if they have fixed that issue since then.

We opted instead for an MSA2324 with an additional MSA disk shelf attached. We put SAS RAID cards in all our servers and used the shared SAS of the MSA. It works AWESOME and is super simple to setup. The best part is the performance with the maximum redundant cable configuration is awesome. It absolutely blows the pants off the previous EMC CX3 equipment that was in use. If you're in a VMWare environment this all HP solution is an incredibly cost effective way to have a good basic SAN.