Slow Disk Read Performance in 6 Node Cluster

Peter J West — Wed, 22 Feb 2012 10:15:07 GMT

Hi,

We've been running a P4000 setup for a few months now and during this time we've generally had a good experience.

We've now come to the point where we need to implement a new backup solution and we're looking at using Data Protector in conjunction with a HP Tape Library to back up a variety of Servers, but particularly the data that's held on the SAN.

The problem i'm facing is that the backup throughput simply isn't quick enough. With the current data load and the speeds i'm seeing the backup is going to take in excess of 72 hours to complete. I've done some rough calculations and if we can achieve around 70% of the maximum throughput of the backup drive then in fact we should be able to complete the backup in <8 hours. But somewhere along the line we were seeing delays.

Initially it was thought the network might be the issue, but doing some tests using JPerf proved that not to be the case.

Following from that I did some IO tests on the SAN disks from a number of vmWare Servers. When doing this I was seeing read rates ranging between 18.6 megabytes/s and 120 megabytes/s. I also used vmWare to monitor the 3 vmNICs we are using to communicate with the SAN and could see that all three NICs were being utilised, with each achieving a peak transfer rate of around 30,000kbps.

I've also checked the NIC's on each of the nodes in the cluster and the majority of the interfaces are only showing 10-20% utilisation. When each disk read test is performed I see a spike up to around 50% on just one node, and i'm guessing that node is the gateway for the specific volume i'm testing.

I'm not sure if these figures are reasonable for a 6 node P4500 cluster or not - so i'd appreciate some feedback on this. To give a bit of background we operate 3 x vmWare ESX 4 Servers which are connected to a 6 node P4500 cluster. The cluster is configured as multi-site so data from 3 of the nodes is mirrored to the other nodes in the cluster.

Unfortunately until we can achieve a backup in a reasonable time-frame i'm not able to migrate any more systems and/or data into the environment.

Thanks in advance for any help.

Pete

Re: Slow Disk Read Performance in 6 Node Cluster

M.Braak — Thu, 23 Feb 2012 21:50:58 GMT

Did you already tried splitting the backup job in two and running them simultainously? Saniq limits (capping) single threaded read jobs.

topic Re: Slow Disk Read Performance in 6 Node Cluster in StoreVirtual Storage

Slow Disk Read Performance in 6 Node Cluster

Re: Slow Disk Read Performance in 6 Node Cluster