Database Vdisk Writes causing very high CPU usage (EVA 8400)

[mattwaddy]

We have a couple of database servers that will send Writes at 100-150 MB/S to the EVA when doing heavy work with TempDB or when doing a recovery of a DB for example. This sends the EVA processor from an average of 30-40% up to 80-95% with some occasional 100% flatlines, these operations can last for 10-15 mins upwards, and when they happen we see the write latency on the EVA increase for all vdisks on that controller as a result. This has sometimes caused cluster disks to timeout and failover and is causing us other general storage issues. Now we can't understand why the writes are having such a dramatic effect and would like to try and rectify the issue if possible. I believe that if the writes are going to a VRAID 5 vdisk then the parity calculations wouldn't help matters, however the disks in questions are mainly VRAID 1.

Secondly I have been reading best practice guides and have noticed that the IO Size sent to the EVA from both ESX and Windows Server 2008 can be up to 32MB in size when the EVA is configured to deal with 128k IO size so has to split these down into smaller frames before commiting, I have started to configure hosts for 128K but have only done 2 out of 60+ so wonder if that might help if configured on everything.

If anyone has any thoughts that would be appreciated. We can directly tie the Write MB/s to the CPU increase through EVA Perf graphing and stats, heavy reads don't seem to have anywhere near the same effect

[giladzzz]

Hi

you must do the following:

Use of VMware ESX , which allows the maximum IO-size as issued by VMware to be restricted to 128Kbytes.

The default is 3Mbytes and changing this value requires a reboot of the ESX server.

Using Vcenter client, From ESX Server Configuration tab, on the left sidebar,

under Software, select Advanced Settings, select Disk and scroll down to find [Disk.DiskMaxIOSize] option and change the value in the [Disk.DiskMaxIOSize] field to 128.

Click OK and reboot the ESX server.

For Microsoft windows 2008 Emulex and QLogic Driver use CUSTOMER ADVISORY

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02518189&jumpid=reg_R1002_USEN

this is to prevent the EVA from splitting the IO into smaller size and cuasing the EVA write lettancy

Regards

[mattwaddy]

I'm aware of how to make the change and have already done this to one esx host and a windows host using HBAnywhere. However on both of the boxes when we produce large writes 100Mb/s+ we see the CPU go from 30-40% up to 100% until the work is complete. Now we were hoping that the 128K IO size change would resolve this issue but sadly it's still there. We will continue to do the change on all of the other hosts both ESX and windows. 

However there must be something else a miss, as when we are doing these writes it affects all vidks on the affected controller, we are wondering if it could be a cache related issue as we have the 8400 with only 14GB cache, 800MB write cache I believe. Can anyone advise of any metrics I can look at to try and work out why these writes cause the erratic behaviour on the EVA? We have logged it with HP Support as well, but any support would be appreciated

[giladzzz]

Hi

You should also on windows hosts install MPIO manager and enable ALB for all hosts

Also you should know that if you have not made the change to 128K IO on all hosts one such host can cause problem for all other hosts.

Regards

[mattwaddy]

I guess that could be the case. But as mentioned this issue is present for luns presented to the windows host where the change was made and it still happens. The issue can be tied back to specific luns when it happens and is always when doing large writes. Still doesn't make sense but will continue with the 128k changes anyway. Anything else it could be? Or anything else to help us identify What the cause could be?

[giladzzz]

Hi

As I said before it's also very important on windows hosts to enable ALB from MPIO manager

Regards

[mattwaddy]

We have esx using alua and windows hosts are static until we can get alb working correctly. We have no proxy mirror port activity where luns are accessed via the none owning controller.

[Jan Soska]

What is your disk count? Aren't you simply limited by overwhelming disks with to much I/O requests?

Jan

[mattwaddy]

Ok something isn't working as I have been setting the LimTransferSize for emulex using HBAnywhere, however the change doesn't seem to be reflected on our EVA as I can see that the IO Request size is still over 512K. It must be the host not retaining the settings for the driver, as even if I set this at both host and port level it keeps reverting after a reboot? Anyone had a similar experience with HBAnywhere? I have the latest drivers and firmware also

[giladzzz]

Hi

You are using OneCommand not HBAnywhere and you can check the parameter after the reboot. I have not seen any problems once it's been set only after driver upgrades.

Regards