Internet VC 1/10 and VC Flex-10?

[JonathanT]

Can I interconnect two VC 1/10GB with two VC Flex-10? I want the Flex-10 to have all the 10GB networking within the chassis, but would like the other VC module to do the uplink to the core switches with LACP trunks to reduce cost. Is it OK to upload the two types of units with CX-4 cables?

[Lmm_1]

That should work, what you can do is use the Multienclosure Stacking feature, that should be enoughg for your requirements, check this out:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=3794423&prodTypeId=3709945&objectID=c01724621

[JonathanT]

Sure looks like it will work. Plus the CX4 .5m cables are only like $100. I can also LACP 6GB/sec off the VC 1/10GB modules to the the core switches which is way more bandwidth and redundancy outside the chassis than we would need. Does anyone have any comments on this configuration?

VC Flex 10 <-> VC Flex 10
|CX4| |CX4|
VC 1/10GB <-> VC 1/10GB
|6x1GB| |6x1GB|
[Core 1 <-> Core 2] Stacked
|1GB| |1GB|
[COLOCATION NETWORK SWITCHES]


So to summarize I would 8 FlexNics per Blade and 2 1GB Nics per blade. If I team the Nics properly at the OS level I would have 5 teamed Nics running in TLB. In theory that would be 22GB TX and 11GB RX out of each blade. The C7000 chassies would upload to the core switch with 6GB to each side. Core 1 and Core 2 are PowerConnect 6248 switches with a stacking cable so they appear as a single switch. The load balancers, firewalls, etc. would all be connected to the core switches with dual GB links in LACP (one link to each side) for redundancy. What does everyone think of this kind of layout?

[JonathanT]

So much for ASCII art.

[JonathanT]

So far I'm off to a good start with my network configuration. I've got the Dell PowerConnect 6248 switches in a stacking configuration. I've configured 4 x 1GB uplinks from each VC 1/10 module to the power connect. I just setup each of the ports on the powerconnects to LAG w/ LACP and turned on the fastlink option (important). The VC module sees 4 links active and 4 standby. I even split the links with two links going to each switch. So if I completely power down one of the powerconnect switches I stay at 2GB active and 4GB standby. I'm debating if that is a good idea or not. Maybe all 4 should go to the same switch so if one fails it completely rolls to the other VC module's uplinks for maximum sustained bandwidth. Thoughts?

I'm waiting on the Flex-10 modules to come in and then I've got to get some CX4 cables to connect them with. CX4 cables are a standard so I should be able to use some quality aftermarket ones without any problem right?

Everyone please feel free to comment on my configuration. I'd love to hear feedback from experts who know far more about this stuff than I do and at the same time have something that is archived to help others on this forum in the future.

[JonathanT]

OK, so I got the Flex-10 modules in today. I moved my VC 1/10 from bays 1 + 2 to bays 3+4 and installed the Flex-10 modules. The VC manager got all kind of ticked off. It say bays 3+4 as flex-10 modules with incompatible hardware. So I called HP support for the first time. The lady that answered was totally knowledgeable. We wiped the VC configuration and let the wizard start over. After that it is seeing the proper hardware in each bay.

She also told me about the new Virtual Connect 2.12 firmware so I'm upgrading to that from 2.10. This new firmware allows the Flex-10 module to work properly even when the bottom of the enclosure is not fully populated. Apparently in v2.10 you had to first populate the bottom of the enclosure before populating the top.

On a side note I ordered my .5 meter CX4 cables to interconnect Bay 1/3 and 2/4 from pureformancecables.com. I'd say they were about 1/2 the cost of what CDW would have sold them for. Work like a charm as one would imagine. It's kind of like the monoprice of high end networking cables from I can tell.

I'll post another update on my networking quest once I get back down to the OS level and start passing traffic over the FlexNics through the VC 1/10 up to the core switches. In the mean time if anyone has any questions or comments I'd love to hear them.

[JonathanT]

So I defined two ethernet networks on the VC for iSCSI (9gb) and application (1gb). So that means each server has 4 FlexNICs on it. I went to team the two application nics and noticed the NCU kept showing one of the nics failsing on RX Path. I called HP support and they wanted be to use Firmware CD 8.60 to update some firmwares which I did, but that did not fix the problem. I then did some Googling and came across this advisory from Dec 2008 which says you have to turn off RX Path Validation in a Teamed FlexNIC scenario:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=1842750&prodTypeId=3709945&prodSeriesId=1842750&objectID=c01554957

I did what it said and the team is no longer going up and down.

I've gone ahead and turned on jumbo frames on the FlexNICs that I've defined for my iSCSI network. I'd like to find a good utility to test the bandwidth between the two servers. I'd like to see what the Windows 2008 TCP/IP stack is cable of in this configuration. I'm planning to attach some MDS600 storage off two of the blades and run Datacore SanMelody off them for the iSCSI SAN. I'm wondering what kind of performance I'm going to be able to drive in this configuration and how much of the 10GB I should carve up for the iSCSI FlexNICs. I purposely didn't team the FlexNICs for the iSCSI because I should be able to use MPIO so that both NICs are run wide open.

As always, questions and comments are welcomed.

[rick jones]

For network performance testing I generally suggest http://www.netperf.org/ but I must say I have a built-in bias in that regard :)

[JonathanT]

Couldn't really figure out how to stress the connection with netperf. What would be the best set of command line arguments to stress a 9Gb/sec FlexNIC with jumbo frames?

I did however copy a 2.6GB ISO from one server to another. It put the connection at about 35-40% which is probably about all the SSD drives in the server can handle.

[rick jones · Accepted solution]

I would use:

netperf -H -i 30,3 -c -C -t TCP_STREAM -- -m 64K -s 1M -S 1M

which will:

run a unidirectional transfer test using TCP to the system (where netserver should be running)

it will include local (-c) and remote (-C) CPU utilization, and calculate an efficiency (service demand) based on the assumption that all the CPU utilization comes from the test.

it will repeat that between three and thirty times (-i 30,3) until it is 99% certain/confident that the averages it has for local/remote CPU util and throughput are within +/- 2.5% of the "true" averages (if one were to have infinite samples).

it will call send() with 64KB of data at a time (-m 64K) and request the local socket buffer (-s) and remote socket buffer (-S) be 1MB in size.