slow read speeds

[ryan_1212]

Been battling this for over a year.. Read speeds are about 25% of write performance on any given day. I can write files to NSM volumes at ~100MB/S, but only read at about 15-30MB/S.

Environment:

4xP4300G2 cluster

Procurve 3400CL switches

VSA experiment:

I created a VSA on a vmware host with 2x10K SAS drives in its own cluster and the same network as the production, and was watching the total cluster throughput metric in the CMC.  I was reading data from a test volume on the VSA at 90-100MB/S. During this time the total cluster throughput metric matched the read performance I was seeing on the volume.

I took the same files and copied them to a volume on my P4300G2 cluster, and read the files. Read rate was approximately 20MB/S, and the CMC cluster throughput metric never peaked above 50MB/S.

Crystal Disk Mark and I\O meter show very similar results to this test as well.

Testing in both cases was done from a physical standalone server, so it was not a case that the VSA was on the same internal vmware switch as the system reading the volume.

Flow Control\Bonding experiment:

This led me to believe the issue was related to networking transmit from the P4300G2  NSMs since a VSA on the same network performed better than the entire cluster. I tried disabling flow control on each NSM, which made no difference, then disabled ALB and used a single NIC on each P4300G2 which also made no difference.

I graph all of the iSCSI Switches with Cacti, and the network is no where near saturated.  iperf tests between the NSMs (hidden feature of the CLIQ interface) report 800-900mbps throughput between NSMs in both directions.

The big things I cannot understand:

1) why 2x10K drives can outperform an entire P4300G2 cluster for Read operations when running through a VSA

2) Why writes are 2-4 times as fast as reads, especially considering these devices use RAID5.

Any ideas?

[Gediminas Vilutis]

How about errors/drops on network interfaces? Cacti can make graphs for these too. 

Gediminas