EVA 8000 Server 2008 R2 Performance issue

[Jimmy Flaptrap]

Hi,

First time post to the forum..

We've recently migrated a SAN attached physical host to Server 2008 R2 from Server 2003 R2 (32bit)

We reattached the LUN once the OS migration was complete. We're now seeing odd behaviour with regards to SAN disk performance for this host.

As a baseline we're running an 8K sequential read benchmark in IOmeter on a single worker.

We're seeing 1.5MB/s max on the Server 2008 R2 machine compared to up to 30MB/s on a SAN attached 2003 R2 host with identical hardware.

IOPS never go over 150 on 2008 R2 and latency seems high at up to 30ms.

Interestingly if we run 8 IOmeter workers on the 2008 R2 host it screams along @ 110MB/s

We're running an EVA 8000.
Disk group contains 48 FC disks and the Vdisks are all RAID5. Hosts are all running latest HP MPIO/HBA drivers for their respective OS.

Can anyone think of a reason for the drop in performance since migrating to 2008R2?

The VDISK was formatted by 2003 then presented to 2008 R2 but I can't imagine why this would cause such a drop in performance.

thanks!

Jim

[Prakash Singh_1]

Hi,

Try updating the EVA8000 firmware to the latest thats 6220.

and also confirm all s/w, drivers for the latest version.

[Jan Soska]

hello,
do no forget use latest multipath MPIO.
What softvare do you use for testing?

generally if your server is only client on such EVA, you should have performance limit close to your SAN bandwith, especially in multithreded test like iomoter od sqlio with more workers/threads. For example in 4Gb SAN and 4Gb EVA4400 we see up to 300MB/s or more write or read...
In your case (if I am not wrong Eva8000 has 2Gb backend) it could be less especially in writes, but still up to 150MB/s or more

Jan

[Zygmunt Krawczyk]

You need to install HP MPIO Full Featured DSM for EVA4x00/6x00/8x00 families of Disk Arrays

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=18964&prodSeriesId=421492&prodNameId=421494&swEnvOID=4064&swLang=13&mode=2&taskId=135&swItem=co-84060-1

and

HP MPIO DSM (Device Specific Module) Manager for Full Featured DSMs

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=18964&prodSeriesId=421492&prodNameId=421494&swEnvOID=4064&swLang=13&mode=2&taskId=135&swItem=co-79866-1

[Jimmy Flaptrap]

Hi All,

Prakash, we're running the most recent s/w and drivers although we are only running 6110 on our HSV210-A's. We're planning to move to 6220 shortly so I'll report back if it makes a difference.

Jan/Zygmunt we are running the latest HP MPIO DSM (including the manager) on the Windows 2003/2008 servers.

We have multiple clients attached to the EVA but there is certainly a discrepancy in performance for this particular host before/after the migration.

Other hosts using VDisks within the same disk group see much better performance as described in the initial post.

thanks

[Joseph Algieri · Accepted solution]

Jim,

From the information in web posting I see a couple of things going on here. First off, it doesnâ t appear to be an EVA overload issue (they have a small disk group containing on 48 HDDs). What would you like me to do to follow up on this?

We're seeing 1.5MB/s max on the Server 2008 R2 machine compared to up to 30MB/s on a SAN attached 2003 R2 host with identical hardware.

IOPS never go over 150 on 2008 R2 and latency seems high at up to 30ms.

Where are you measuring this latency? At the server or at the array port. Could be you have a very small queue depth setting on the server HBA and they are seeing high latencies at the server as a result.

Interestingly if we run 8 IOmeter workers on the 2008 R2 host it screams along @ 110MB/s

This is a good indication that the array is not the problem. If you increase the number of threads (i.e. workload sent to the array) the throughput scales nicely.

Prakash, we're running the most recent s/w and drivers although we are only running 6110 on our HSV210-A's. We're planning to move to 6220 shortly so I'll report back if it makes a difference.

You responded as to any changes after upgrading to VCS V6.220

We have multiple clients attached to the EVA but there is certainly a discrepancy in performance for this particular host before/after the migration.

Only this one server has an issue. Makes me believe it is a server centric problem not an array problem

Other hosts using VDisks within the same disk group see much better performance as described in the initial post.

Even other servers in the same disk group arenâ t seeing an issue. More indication that itâ s a server configuration problem. Maybe proxy reads or too small of an HBA queue depth setting on that Windows 2008 R2 server.


Joe

[Jimmy Flaptrap]

Hi Joe,

Thanks for your reply.

We're still waiting to move to 6.220 but we have investigated further and I agree with you that this is now looking like a server issue.

We have run identical IOMeter tests on the local drives (3 x 10K SAS 146GB RAID5) and we're seeing identical performance issues.

Specifically 121 IOPS/0.94MBps (single worker sequential read with 8K blocks in IOMeter )

We have checked server software/firmware versions with HP's PSP and everything looks up to date.

We're now trying to figure out what could negatively impact disk performance on both the servers local drives (running off the P400 smart array) and a SAN attached disk?

thanks

James

[Sivaramakrishna_1]

Hi,

please check the Command View EVA Host type should set to "Windows 2008".

EVA with Windows 2008 there is a Disk Performance issue observed. This is because of large IO size used by improved version of storport driver of Win2K8.

Install the lasted HBA driver and tune the IO Size parameter on the HBAs so that IO sizes will be tuned at HBA level before sending to EVA. Suggest you to tune the setting for 64KB or 128KB for best results.

http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=3664583&prodTypeId=12169&objectID=c02518189


We did the same at one customer place and found better results.

[AK_DT]

Hi Jim,

I seem to be experiencing the same I/O response issues after moving to 2008 R2. I'm connecting some BL465c G5 blades and a DL380 G5 server. All connecting via Brocade SAN to an EVA4400. Responses on the same fabric seem fine on 2003, however I/O on 2008 is much less than that of local HDD's. Did you have any luck with resolving this issue on your end?