4x P4300G2 SAS performance worries.

[kghammond]

Just a quick follow up.

My reason for posting our results is that HP engineers continuously preached that the overhead of running hardware RAID 5 versus hardware RAID 10 was at most about 10% hit in performance (IOPS).

When we ran into a real workload that was very write intensive, hardware RAID 5 did not hold up well.  HP insisted to us that we didn't have enough spindles and we needed more spindles.  Although we may need more spindles, we did this test to validate if RAID 10 would give us more bang for our buck versus more spindles.

Our eniornment:

• ALB on all nodes, we used to be LACP when we were on a single switch.
• All nodes dual connected into separate HP 5400 series ProCurve switches
• Dedicated HP 5400 series switches for iSCSI
• No gateway for iSCSI, layer 2 traffic only.
• We have a management server in our iSCSI that proxies any management traffic, such as email, dns, ntp, etc.
• flow control is enabled for the HP LeftHand ports.
• No jumbo frames

I think that covers the majority of the infrastructure questions.

I am not surprised by the results, but it does conflict with HP engineers.  My only surprise is that I expected the hit on IOPS to be negligible at a 70% read ratio.  I was not expecting the overhead of RAID 5 to be relatively linear regardless of the read/write ratio.

Thank You,

Kevin

[Gediminas Vilutis]

Just a few my cents...

3750 catalysts have almost no packet buffer at all - I would not recommend this switch for iscsi (not for heavy iscsi, at least)...

Gediminas

[David_Tocker]

The Packet buffer thing always confuses me.

HP recommend in all their bundles for P4000: Procurve 2910al

Dual ARM1156T2S @ 515 MHz, 4 MB flash, 1 GB compact flash, 512 MB SDRAM; packet buffer size: 6 MB

I am assuming this is a shared packet buffer and not per port, otherwise it is just huge.

We get great and consistent performance (heavily loaded) using 2810g switches at half the price:

MIPS @ 264 MHz, 16 MB flash, 64 MB SDRAM; packet buffer size: 0.75 MB (Per port? Shared?)

Now my understanding is that the 3750 in its various guises has at minimum a 0.75mb input packet buffer per 4 ports, along with (upto) a  2MB output buffer per every four ports (This is a shared buffer system with minimum values and the ability for each port to 'borrow' from a common pool of buffer memory)

This tells me that at a minimum 4.5MB of recieve buffer is *always* available over the switch which is not too far off what the 'recommended' switch (2910al, 6MB) has.

Add to this the dynamic output buffer and dont we get a very close (more likely larger) amount of frame buffer on the 3750 over the 2910al?

Or am I completely off the tracks?

If you really want your head to explode, consider the port buffers in a stackwise configuration... Latency higher or lower?

[Gediminas Vilutis]

Regarding 3750. Unfortunately I was unable to find official Cisco spec stating buffer size for this and all lower switches (I did research ~1 year ago, maybe now something new is released). So what I can tell is only from my own experience. 

I never used this switch for iscsi, but we had these in our WAN environment. Local internet peering exchange was connected to port on the switch. Packet drops started to appear when outbound average traffic reached ~600 Mbps, all due to lack of free buffers.... For bursty iscsi traffic the situation could be even worse. I haven't chance to play with cisco qos (as far as I remember packet buffer by default is divided to 4 separate subbuffers with no sharing between them), since we upgraded link to 10G. 

BTW, cats 3750 in stack do not do cross switch LACP trunks - LACP trunk can be done only within one switch. 3750 stack is mainly just for ease of management... 

BR,

Gediminas

[David_Tocker]

Hi There.

Apparently LACP can be configured cross-stack now. Previous IOS revisions could not:

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00806cb982.shtml#lacp

Link Aggregation Control Protocol (LACP) and Port Aggregation Protocol (PAgP)

EtherChannels have automatic configuration with either Port Aggregation Protocol (PAgP) or Link Aggregation Control Protocol (LACP). PAgP is a Cisco-proprietary protocol that you can only run on Cisco switches and on those switches that licensed vendors license to support PAgP. IEEE 802.3ad defines LACP. LACP allows Cisco switches to manage Ethernet channels between switches that conform to the 802.3ad protocol.

PAgP cannot be enabled on cross-stack EtherChannels while LACP is supported on cross-stack EtherChannels from Cisco IOS Software Release 12.2(25)SEC and later. Switch interfaces exchange LACP packets only with partner interfaces with the active or passive mode configuration. You can configure up to 16 ports to form a channel. Eight of the ports are in active mode, and the other eight are in standby mode. When any one of the active ports fails, a standby port becomes active. Interfaces with the on mode configuration do not exchange PAgP or LACP packets.

[Prakash Singh_1]

Hi,

You can also refer to the HP guided troubleshooting tree for P4000 series performance issues and some good solutions.

Click on link : 

http://h20584.www2.hp.com/hpgt/guides/select?lang=en&cc=us&prodTypeId=12169&prodSeriesId=4118659

click on : HP P4000 Performance Troubleshooting and white papers