Fast reads but very poor writes - P4500 G2 x2 in Network Raid 10

[BenGadsden]

Hi All,

We're having a performance issue at the moment relating to writes.

To first give some background information, we have 2 x P4500 G5 each with 12 x 600GB 15K SAS drives, setup in Hardward R5 with Network R10 between the two. This is a single site setup.

About 6 months ago we installed the 10Gbe upgrade kit into both SANs with the plan to upgrade the hosts later. (We have 5 x Hyper-V 2012 in failover cluster) The hosts were previously setup with 1Gbe NIC's in MPIO config.

Throughput was pretty good and IOMeter matched all other results I could find throughout the web - ~120MB/s max reads and ~12Mb/s "real" test. (running 1Gbe on hosts and 10Gbe on SAN)

Max reads is 32K, 100% sequential, 100% reads.
Real is 8K, 60% random, 65% reads.

We moved rack over the weekend and also upgraded our hosts and switching fabric to 40GB using Mellanox switches and NIC's.

Jumbo frames has been enabled, along with flow control.

We are able to get ~500MB/s max reads with IOMeter from within a VM on the cluster which is very good but our writes have substantially dropped. We are now seeing 2MB/s- to 4MB/s on the "Real" test, with around 150ms-200ms I/O latency

I've tried shutting down one of the SANs and testing writes, disabling the 2nd NIC on each SAN, reducing the port speed on the Host ports on the switch to 1GB, all of which end up with the same poor write results.

I have also shutdown and cold boot one of the SANs and rebooted the other (I don't have iLO into one of the SANs). Still no change.

The CMC is showing no issues with IO latency (Reads @ avg 5ms, writes @ avg 0.5ms) or queue depth being extreme (avg 1-2 with spikes up to 12).

I'm actually trying some things as a type this and it appears, since rebooting one of the SAN's last night (the one I don't have iLO access to) the CMC Performance manage isn't getting any information from it and is even giving errors such as "The CMC performance monitor is having difficulty communicating the corresponding system via the performance monitor connection, the connection may have been lost or it times out because the system is busy.

I'm getting stats for the 2nd SAN, but none for the 1st.

Any suggestions would be great.

Also, we're running Lefthand OS 12.5.

I hope I've supplied enough information to get started.

Cheers,
Ben

[BenGadsden]

The issue with the performance statistics was resolved with a reboot of one of the P4500 G2 nodes. It was stuck using 100% cpu, unsure what caused it but hasn't happened since.

We are still seeing very poor write speeds though.